Page automation operation method, device, equipment, storage medium and program product

Abstract

This application provides a method, apparatus, device, storage medium, and program product for automated page operation. It relates to automated processing technology. The method includes: receiving operation content input by a user; generating a sequence of operation steps for the page to be operated based on the operation content; wherein the operation step sequence includes multiple sequentially arranged operation steps and the expected page state change corresponding to each operation step; for each operation step in the operation step sequence, executing the current operation step and obtaining the actual page state change after executing the current operation step; if the actual page state change is consistent with the expected page state change, continuing to execute the next operation step until all operation steps corresponding to the operation content are completed. This method aims to improve the execution efficiency and stability of automated operations while maintaining high adaptability to dynamic pages.

Description

Technical Field

[0001] This application relates to automated processing technology, and more particularly to a method, apparatus, device, storage medium, and program product for automated page operation. Background Technology

[0002] With the deepening of digital transformation, browser-based business systems are widely used in finance, e-commerce, and government sectors. These systems contain a large number of repetitive manual tasks, such as form filling, data querying, and workflow approval. These tasks are not only inefficient but also highly dependent on the operator's familiarity with specific page layouts.

[0003] Currently, the mainstream technologies for automating such tasks mainly fall into two categories: one is robotic process automation (RPA) tools based on recording and static planning, which have high execution efficiency but are sensitive to changes in page structure and have high maintenance costs; the other is general-purpose browser agents based on large language models, which have strong adaptability but require calling large models for decision-making at each step, resulting in large response delays and low execution efficiency.

[0004] Therefore, the relevant technologies struggle to maintain both high flexibility and execution efficiency, necessitating an automated operation method that can adapt to dynamic page changes and is highly efficient and reliable. Summary of the Invention

[0005] This application provides methods, apparatus, devices, storage media, and program products for automated page operation, which improve the execution efficiency and stability of automated operation while maintaining high adaptability to dynamic pages.

[0006] In a first aspect, embodiments of this application provide a method for automating page operations, including:

[0007] Receive user input of operation content; wherein, the operation content is used to perform automated operations on the page to be operated;

[0008] Based on the operation content, an operation step sequence is generated for the page to be operated; wherein, the operation step sequence includes multiple sequentially arranged operation steps, and the expected page state change corresponding to each operation step;

[0009] For each operation step in the operation step sequence, execute the current operation step and obtain the actual state change of the page to be operated after executing the current operation step;

[0010] If the actual change in the page state matches the expected change in the page state, continue to execute the next operation step until all operation steps corresponding to the operation content are completed.

[0011] In the above technical solution, by receiving user input and generating a sequence of operation steps containing expected state changes, the actual page state is compared with the expected state in real time during execution, and subsequent steps are only advanced when the states match. This method achieves a structured transformation from user intent to automated execution, and through progressive state verification, reduces single-step and multi-step cumulative errors caused by network latency, dynamic page changes, or asynchronous interactions, avoiding complex rollback and correction operations, thereby improving the adaptability, robustness, and execution success rate of the automated process.

[0012] In one possible implementation, receiving user input includes:

[0013] The system receives text descriptions input by the user and uses these text descriptions as the user's input operation content; wherein, the text descriptions are natural language text describing the goals and intentions of the automated operation.

[0014] In the above approach, when receiving user input, users are allowed to describe their automation needs in natural language that conforms to everyday expression habits, without needing to understand the details of the page structure or master professional automation script syntax beforehand. This lowers the barrier to entry and enhances the intuitiveness and flexibility of human-computer interaction. Simultaneously, by combining the contextual understanding and reasoning capabilities of natural language processing technology, it is possible to reasonably infer and complete ambiguous or implicit user intentions, further enhancing the accuracy and robustness of the operation content parsing.

[0015] In one possible implementation, receiving user input includes:

[0016] Receive text description content input by the user;

[0017] After receiving the text description, the user is prompted to provide a user operation example; wherein, the user operation example is a record of the process demonstrated by the user through actual interactive operation to achieve the task corresponding to the text description.

[0018] If the user confirms the additional input, then the user operation example input by the user is received;

[0019] Based on the user operation example, a corresponding operation description is generated; wherein, the operation description is a standardized instruction sequence that defines the specific execution steps and logic of the automated operation;

[0020] The text description and the operation description are used as the operation content input by the user.

[0021] In the above process, by combining user intent descriptions with specific operation examples, richer contextual information can be obtained during planning. This allows us to understand the user's operation goals, grasp their preferred execution paths and specific interaction details, thereby improving the accuracy, adaptability, and execution efficiency of operation step sequence generation.

[0022] In one possible implementation, an example of the user operation of receiving user input includes:

[0023] Before the user inputs the user operation example, the page data collection tool is injected into the page to be operated;

[0024] The user operation examples are collected based on the page data collection tool.

[0025] The aforementioned data collection tools enable the continuous, non-intrusive collection of structured interaction data while the user performs actions, ensuring the integrity and accuracy of the action examples.

[0026] In one possible implementation, the page data acquisition tool includes a JavaScript script; the JavaScript script is used to monitor and collect user interaction behaviors and changes in the page's DOM structure.

[0027] In one possible implementation, based on the operation content, a sequence of operation steps for the page to be operated is generated, including:

[0028] Obtain a pre-built large language model and a first prompt word template; wherein, the first prompt word template includes a preset slot for receiving the operation content, so as to guide the generation of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step; the slot for receiving the operation description content in the preset slot is an optional slot;

[0029] Fill the operation content into the first prompt word template to generate an operation planning prompt word;

[0030] Based on the large language model and the operation planning prompts, a sequence of operation steps for the page to be operated is generated.

[0031] In the above approach, by using a large language model for adaptive planning, multi-step operation sequences can be flexibly generated based on user input, improving the adaptability to different pages to be operated on. At the same time, by completing multi-step planning at once, the efficiency of overall operation planning and execution is improved.

[0032] In one possible implementation, performing the current operation step includes:

[0033] Invoke the preset browser automation testing tool and send the execution command corresponding to the current operation step to the browser automation testing tool;

[0034] Based on the browser's automated testing tool, the current operation steps are executed on the page to be operated.

[0035] In the above method, by calling a preset browser automation testing tool, the intelligent agent can transform the planned sequence of operation steps into executable instructions, realize the automated operation of the page to be operated, and thus improve the accuracy, efficiency and repeatability of the page automation operation.

[0036] In one possible implementation, before sending the execution instruction corresponding to the current operation step to the browser automated testing tool, the method further includes:

[0037] All operation data for each step is sent to the browser automated testing tool.

[0038] In this way, during execution, the agent only needs to send control command signals such as "continue execution", "pause" or "stop" to the browser's automated testing tool, without having to repeatedly send specific execution commands after each step, thereby reducing communication overhead and improving the coherence and overall efficiency of the execution process.

[0039] In one possible implementation, before performing the current operation step based on the browser automated testing tool, the method further includes:

[0040] Obtain the preset page data collection tool;

[0041] The page data collection tool is injected into the page to be operated.

[0042] In one possible implementation, injecting the page data acquisition tool into the page to be operated includes:

[0043] If the user does not confirm and supplement the user operation example when inputting operation content, the page data collection tool will be injected into the page to be operated before the current operation step is executed based on the browser automated testing tool.

[0044] Through the above implementation methods, even in the absence of user operation examples, the intelligent agent can still achieve comprehensive and real-time data collection of the operation execution process by actively injecting monitoring tools, providing a reliable data foundation for closed-loop control and status verification of automated processes.

[0045] In one possible implementation, obtaining the actual state change of the page to be operated on after performing the current operation step includes:

[0046] The start time is the moment when the current planned operation step is executed, and the end time is the moment when the page to be operated stops changing after the current planned operation step is executed;

[0047] Obtain all single-point events between the start time and the end time; wherein, the single-point events include user interaction events and page DOM structure change events;

[0048] Based on each of the single-point events, determine the actual state change of the page to be operated on after the current operation step is executed.

[0049] In the above method, by precisely defining the time window from the start of the operation to the page stabilization, and capturing all user interaction events and DOM structure change events within this window, the actual state evolution process of the page after the operation can be accurately and completely reconstructed from the underlying event data, thereby improving the reliability of the verification of the results of subsequent steps.

[0050] In one possible implementation, determining the actual state change of the page to be operated on after executing the current operation step, based on each of the single-point events, includes:

[0051] Based on a preset window threshold, all single-point events are aggregated using a sliding window to generate multiple aggregated events;

[0052] Invoke a preset first description content generation model; wherein, the first description generation model is a model trained based on natural language processing technology, used to convert structured event sequences into natural language descriptions;

[0053] Based on the first description content generation model, page change analysis is performed according to each of the aggregated events to generate the actual page state change after executing the current operation step.

[0054] In the above approach, fine-grained single-point events are integrated into semantically richer aggregate events through sliding window aggregation. Then, a dedicated model trained based on natural language processing technology is used to transform these aggregate events into natural language descriptions of the actual changes in the page state. The agent can clearly and structurally present the actual results after the operation is executed, thereby comparing and verifying them with the expected state. This provides a reliable and interpretable basis for determining the successful execution of automated steps, improving the accuracy and reliability of the automated process.

[0055] In one possible implementation, determining the actual state change of the page to be operated on after executing the current operation step, based on each of the single-point events, includes:

[0056] Based on a preset window threshold, all single-point events are aggregated using a sliding window to generate multiple aggregated events;

[0057] Obtain the starting page image corresponding to the starting time and the ending page image corresponding to the ending time;

[0058] The preset second description content generation model is invoked; wherein, the second description generation model is a model trained jointly based on natural language processing technology and image processing technology, used to convert structured event sequences into natural language descriptions;

[0059] Based on the second description content generation model, page change analysis is performed according to each of the aggregated events, the starting page image, and the ending page image to generate the actual page state change after executing the current operation step.

[0060] The above implementation method enhances the robustness and accuracy of state change recognition by introducing page visual information and performing multimodal fusion analysis with event sequences. In particular, it can capture visual details that pure event monitoring may overlook, thereby providing a more comprehensive and reliable description of the actual state for subsequent comparison and verification with the expected state.

[0061] In one possible implementation, the actual change in the page state is consistent with the expected change in the page state, including:

[0062] Obtain a preset state change comparison model; wherein, the state change comparison model is a model based on semantic similarity calculation or rule matching, used to determine whether the state changes of two natural language descriptions are consistent;

[0063] Based on the state change comparison model, a similarity comparison is performed between the actual state change of the page and the expected state change of the page to obtain the comparison result;

[0064] If the similarity of the comparison results meets the preset threshold, then it is confirmed that the actual change in the page state is consistent with the expected change in the page state.

[0065] In the above implementation, semantic comparison enables efficient and objective verification of operation results, reduces reliance on manual judgment, and improves the reliability and scalability of automated testing or execution processes.

[0066] In one possible implementation, the method further includes:

[0067] If the similarity of the comparison result does not meet the preset threshold, it is confirmed that the actual change in the page state is inconsistent with the expected change in the page state.

[0068] Send a stop command to the browser automation testing tool to stop executing subsequent planned operation steps.

[0069] The above implementation method improves the robustness of the agent when facing unexpected states by interrupting the execution process in a timely manner and coordinating with exception handling, avoids the spread of errors, and provides effective support for quickly locating and repairing problems.

[0070] In one possible implementation, the method further includes:

[0071] Obtain the actual and expected changes in the page state corresponding to the current operation step and the previously executed historical operation steps, and use them as operation process data;

[0072] The operation process data and the operation content are filled into a pre-built second prompt word template to generate a re-planning prompt word; wherein, the second prompt word template includes a preset slot for receiving the operation process data and the operation content, so as to guide the regeneration of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step;

[0073] Based on the preset large language model and the re-planned prompt words, a new sequence of operation steps for the page to be operated is generated.

[0074] In the above implementation, by utilizing the reasoning and planning capabilities of a large language model, the automated process is intelligently and dynamically adjusted when encountering unexpected states, thereby improving the agent's adaptability to complex and ever-changing page environments and its robustness in task completion.

[0075] Secondly, embodiments of this application provide a page automation operation device, including:

[0076] An operation content receiving module is used to receive operation content input by the user; wherein, the operation content is used to perform automated operations on the page to be operated;

[0077] An operation step sequence generation module is used to generate an operation step sequence for the page to be operated based on the operation content; wherein, the operation step sequence includes multiple sequentially arranged operation steps, and the expected page state change corresponding to each operation step;

[0078] The first execution module is used to execute the current operation step for each operation step in the operation step sequence, and to obtain the actual state change of the page to be operated after the current operation step is executed.

[0079] The second execution module is used to continue executing the next operation step when the actual change in the page state is consistent with the expected change in the page state, until all operation steps are completed.

[0080] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0081] The memory stores computer-executed instructions;

[0082] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0083] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0084] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0085] The page automation operation method, apparatus, device, storage medium, and program product provided in this application receive user operation content and generate an operation step sequence containing expected state changes. During execution, the actual page state is compared with the expected state in real time, and subsequent steps are only advanced when the states match. This method achieves a structured transformation from user intent to automated execution, and through progressive state verification, reduces single-step and multi-step cumulative errors caused by network latency, dynamic page changes, or asynchronous interactions, avoiding complex rollback and correction operations, thereby improving the adaptability, robustness, and execution success rate of the automated process. Attached Figure Description

[0086] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0087] Figure 1 An application scenario diagram of the page automation operation method provided in this application;

[0088] Figure 2 Flowchart of the page automation operation method provided in this application Figure 1 ;

[0089] Figure 3 A schematic diagram of the page automation operation device provided in this application;

[0090] Figure 4 A schematic diagram of the structure of the electronic device provided in this application.

[0091] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0092] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0093] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0094] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of the relevant data shall comply with relevant laws, regulations and standards, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0095] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0096] With the deepening of digital transformation, business systems in finance, e-commerce, and government sectors generally adopt browser-based management interfaces. These systems involve a large number of repetitive manual tasks, such as form filling, data querying, workflow approval, and payment processing. These tasks are not only inefficient but also highly dependent on the operator's familiarity with specific page layouts and processes. Furthermore, the page structure of these business systems often changes dynamically due to version iterations or external service integration, further increasing the complexity and error risk of manual operations.

[0097] Currently, the mainstream technologies for automating such tasks mainly fall into two categories: one is Robotic Process Automation (RPA) tools based on recording and static planning, which execute operations through predefined scripts, resulting in high efficiency, but are extremely sensitive to changes in page structure, have high maintenance costs, and are difficult to handle unstructured interaction logic; the other is general-purpose browser agents (such as Browser-Use) based on large language models, which implement operations through dynamic planning and single-step interaction, making them highly adaptable, but each step of the operation requires calling a large model for decision-making and state verification, resulting in response delays, low overall execution efficiency, and difficulty in meeting business scenarios with high real-time requirements.

[0098] Therefore, the relevant technologies struggle to maintain both high flexibility and execution efficiency, necessitating a more efficient and reliable method for automating page operations that can adapt to dynamic page changes.

[0099] This application provides a method for automating page operations, aiming to solve the aforementioned technical problems. Specifically, by receiving user input and generating a sequence of operation steps containing expected state changes, the method compares the actual page state with the expected state in real time during execution, advancing subsequent steps only when the states match. This method achieves a structured transformation from user intent to automated execution, and through progressive state verification, reduces single-step and multi-step cumulative errors caused by network latency, dynamic page changes, or asynchronous interactions, avoiding complex rollback and correction operations, thereby improving the adaptability, robustness, and execution success rate of the automated process.

[0100] The page automation method provided in this application is applicable to various browser-based business systems or web application interfaces that require repetitive and procedural operations.

[0101] For example, in business management scenarios such as financial transaction approval, e-commerce order processing, and online government application, the system typically includes numerous standardized operational steps such as form filling, data querying, status verification, and process navigation. Furthermore, the page structure may change due to adjustments in business rules or system upgrades. In addition, the above methods can also be applied to fields requiring automated interaction with web pages, such as software testing, data collection, and cross-system data synchronization.

[0102] In summary, any scenario involving the execution of structured operations on dynamically changing web pages, while balancing execution efficiency and system adaptability, falls under the application scenarios of the page automation operation technology solution in this application.

[0103] The following is combined Figure 1 The application scenarios applicable to the embodiments of this application are described below. Figure 1This is an application scenario diagram of the page automation operation method provided in this application.

[0104] See Figure 1 Taking the data query scenario as an example, this scenario mainly involves electronic devices that interact with users, and these electronic devices are equipped with intelligent agents and large language models.

[0105] Specifically, the agent is used to receive the user's input description of the operation task and send it to the large language model.

[0106] The large language model generates a sequence of operation steps for the page to be operated on based on the task description. This sequence of operation steps includes multiple sequentially arranged operation steps and the expected change in the page state corresponding to each operation step. The generated sequence of operation steps is fed back to the agent.

[0107] The agent executes each operation step sequentially according to the operation step sequence. After each operation step is completed, the agent obtains the actual state change of the page after execution and compares it with the expected state change of the page corresponding to that step. If the comparison matches, the agent continues to execute the next operation step; if the comparison does not match, the agent resends the actual and expected state change information corresponding to the current step and the previously executed historical steps to the large language model.

[0108] The large language model regenerates the sequence of operation steps based on this information, and the agent continues to execute it based on the new plan. This process is repeated until all operation steps corresponding to the user-input operation task description are completed.

[0109] In some implementations, the large language model can also be deployed on other electronic devices that communicate with the electronic device where the agent is located; however, this application does not specifically limit this.

[0110] Through the above steps, this application employs a large language model to adaptively generate a sequence of operation steps for the page to be operated based on user input, demonstrating high adaptability to dynamic pages. Simultaneously, by planning multiple operations at once and executing them sequentially, planning and execution efficiency are improved. Furthermore, during execution, a local agent performs real-time status verification for each step, reducing the frequency of interaction between the agent and the large language model. This ensures the accuracy of operation execution while also improving the efficiency and stability of the process.

[0111] Figure 2 Flowchart of the page automation operation method provided in this application Figure 1This method can be executed by a page automation device, which can be a server or an electronic device; the following explanation uses an electronic device as an example. The method in this embodiment can be implemented through software, hardware, or a combination of both. Figure 2 As shown, the method includes:

[0112] S201, Receive user input for operation content.

[0113] In this embodiment, the operation content is text information input by the user through an interactive interface to the intelligent agent responsible for automated operations, describing the task to be performed, such as "complete order review and export report" or "fill out loan application form and submit". In other words, the operation content is used to instruct a series of automated operations to be performed on a specified page to be operated.

[0114] The page to be operated on is typically a browser-based business interface, such as a specific function page in a financial approval system, e-commerce backend, or government service platform. The user's input may include the operation objective, the object of the operation, and relevant constraints.

[0115] Specifically, the intelligent agent receives user input on the interactive interface, obtains the input content, parses the input to understand the user's intent, and initiates subsequent automated operation processes based on the parsing results. Thus, the intelligent agent can adaptively generate a sequence of operation steps for the page to be operated on based on the user input and automate its execution, thereby possessing a high degree of adaptability and execution flexibility to dynamic page changes.

[0116] S202. Based on the operation content, generate a sequence of operation steps for the page to be operated.

[0117] Specifically, the intelligent agent can perform semantic understanding and task decomposition on the received operation content, transform it into a series of executable specific operation instructions, and arrange these operation instructions into an operation in a logical order.

[0118] In some scenarios, the intelligent agent can also call a pre-set large language model to perform semantic parsing and operation step planning on the received operation content in order to obtain the operation step sequence.

[0119] In this embodiment, the operation step sequence includes multiple sequentially arranged operation steps, and each operation step may include an action to be performed, such as clicking, inputting, selecting, etc.

[0120] In addition, the sequence of operation steps also synchronously defines the expected page state changes after each operation step is executed, such as preset controls becoming operable, the page jumping to a specified URL, or a certain data area displaying the expected content.

[0121] In this way, by generating the above sequence of operation steps, user intent can be transformed into structured specific operation instructions, thereby providing a clear and executable planning basis for the effective execution of subsequent automated operations, and improving the efficiency and reliability of automated execution.

[0122] S203. For each operation step in the operation step sequence, execute the current operation step and obtain the actual state change of the page to be operated after the current operation step is executed.

[0123] In this embodiment, the intelligent agent can execute each operation step in the order of the operation step sequence.

[0124] During execution, corresponding operation instructions can be sent to the page to be operated through browser drivers or page automation interfaces, such as simulating clicks, text input, or selection of options.

[0125] After completing the current operation step, the system also uses a preset page status monitoring mechanism to obtain the actual changes in the page status of the page to be operated.

[0126] For example, the preset page state monitoring mechanism includes, but is not limited to, monitoring changes in the page DOM structure, capturing network request responses, obtaining the attributes or content of specified page elements, or analyzing page screenshots through visual recognition technology.

[0127] Here, the actual changes in the page state obtained can be compared with the expected state in subsequent steps to verify whether the current operation step has been successfully executed, that is, whether to continue to execute the next operation step.

[0128] S204. If the actual change in the page state is consistent with the expected change in the page state, continue to execute the next operation step until all operation steps corresponding to the operation content are completed.

[0129] In this embodiment, the agent compares the actual page state changes obtained earlier with the expected page state changes defined during planning. If they match, the current operation step is determined to have been executed successfully. At this point, the agent can continue to execute the next operation step in the sequence and repeat the state acquisition and comparison process.

[0130] The above process is repeated until all steps in the sequence of all operation steps corresponding to the operation content have been successfully executed, indicating that all automated operations corresponding to the operation content have been completed.

[0131] By using state-based progressive execution, subsequent operations can be carried out only after confirming that the page response meets expectations at each step. This can reduce single-step execution errors caused by factors such as network latency, dynamic page changes, or asynchronous interactions, and can also avoid multi-step consecutive execution failures caused by the accumulation of errors.

[0132] In the above technical solution, by receiving user input and generating a sequence of operation steps containing expected state changes, the actual page state is compared with the expected state in real time during execution, and subsequent steps are only advanced when the states match. This method achieves a structured transformation from user intent to automated execution, and through progressive state verification, reduces single-step and multi-step cumulative errors caused by network latency, dynamic page changes, or asynchronous interactions, avoiding complex rollback and correction operations, thereby improving the adaptability, robustness, and execution success rate of the automated process.

[0133] The following section will further detail the execution process of the page automation technology solution. It should be noted that the following description is merely an exemplary implementation of the technical solution in this application and does not constitute a limitation on the technical solution of this application.

[0134] In this embodiment, when a user inputs operation content, they can provide only a text description. Therefore, an optional implementation of receiving user-input operation content may include: receiving the text description content input by the user and using the text description content as the user-input operation content.

[0135] In this embodiment, the text description content is natural language text describing the goals and intentions of the automated operation. For example, the user can enter a similar statement in the text input box of the interactive interface, such as "After logging into the system, filter out the orders with the status of 'Pending Shipment' on the order management page and export them as an Excel file".

[0136] Specifically, the intelligent agent performs semantic parsing and structured extraction of text descriptions through an integrated natural language understanding module or by invoking a pre-defined large language model. For example, entity extraction is used to identify and extract key semantic elements such as the operation object (e.g., "order management page" in the example above), operation actions (e.g., "filter" and "export"), operation conditions (e.g., "status is 'awaiting shipment'"), and expected output (e.g., "Excel file"). The extracted results are then integrated to obtain structured operation content. Furthermore, based on the parsing results, the operation step sequence is processed to execute an automated process.

[0137] In the above approach, when receiving user input, users are allowed to describe their automation needs in natural language that conforms to everyday expression habits, without needing to understand the details of the page structure or master professional automation script syntax beforehand. This lowers the barrier to entry and enhances the intuitiveness and flexibility of human-computer interaction. Simultaneously, by combining the contextual understanding and reasoning capabilities of natural language processing technology, it is possible to reasonably infer and complete ambiguous or implicit user intentions, further enhancing the accuracy and robustness of the operation content parsing.

[0138] To further enhance the planning effectiveness of automated operations, users can also provide text descriptions and corresponding operation examples. By combining text and example information, more comprehensive contextual information can be obtained when planning automated operations, thereby more accurately generating action sequences and expected state changes, effectively improving the efficiency and accuracy of the planning process.

[0139] Based on this, another optional implementation of receiving user input operation content may include: receiving text description content input by the user; after receiving the text description content, prompting the user whether to add a user operation example; wherein, the user operation example is a process record demonstrated by the user through actual interactive operation to achieve the task corresponding to the text description content; if the user confirms to add the input, then receiving the user operation example input by the user; generating corresponding operation description content based on the user operation example; wherein, the operation description content is a standardized instruction sequence defining the specific execution steps and logic of the automated operation; and using the text description content and the operation description content as the operation content input by the user.

[0140] In this embodiment, after the intelligent agent receives the text description, it can proactively prompt the user to provide a user operation example. Here, the user operation example refers to a record of the process demonstrated by the user through actual interactive operations to achieve the task corresponding to the text description. Specifically, this process record is interactive behavior data generated when the user manually performs relevant steps on the page to be operated, such as a browser or application interface, and may include the user's operation trajectory, click sequence, input content, page jump path, and interface state changes, etc.

[0141] If the user confirms the additional input, the intelligent agent activates the operation recording function to capture and record the user's interaction process on the page to be operated in real time, generating operation example data. For example, this data can be collected through a browser extension, desktop client, or embedded recording module, or it can be achieved in other ways, which are not limited in this embodiment.

[0142] The agent parses the operation logic and execution sequence of the obtained user operation examples to obtain the corresponding operation description. Here, the operation description is a standardized sequence of instructions that defines the specific execution steps and logic of the automated operation, such as operation instructions based on DOM paths, API call sequences, or executable script fragments.

[0143] Based on this, the intelligent agent associates and integrates the text description initially entered by the user with the operation description generated based on the operation example, and uses them together as the operation content entered by the user, and performs subsequent operation step sequence processing to execute the automated process.

[0144] In the above process, by combining user intent descriptions with specific operation examples, richer contextual information can be obtained during planning. This allows us to understand the user's operation goals, grasp their preferred execution paths and specific interaction details, thereby improving the accuracy, adaptability, and execution efficiency of operation step sequence generation.

[0145] In the above process, if the user confirms and supplements the user operation example, one possible implementation of obtaining the user operation example may include: injecting a page data collection tool into the page to be operated before the user inputs the user operation example; and collecting the user operation example based on the page data collection tool.

[0146] To achieve complete collection of user action examples, a page data collection tool can be dynamically injected into the page to be operated on before the user begins to input the example. As the user inputs the example, the page data collection tool collects user behavior data on the page in real time and transmits it to the intelligent agent to obtain a complete user action example.

[0147] In this embodiment, the tool can be any of the following: a lightweight JavaScript script, a browser extension module, or an embedded data collection agent. It is securely loaded into the page runtime environment through the browser's developer interface or the application's plugin mechanism to monitor and collect user interaction behavior and page DOM structure changes on the page to be operated.

[0148] For example, the data collection process may include, but is not limited to: real-time monitoring and recording of various user interaction events on the page, such as mouse clicks, keyboard input, page scrolling, focus switching, and form submissions. Simultaneously, the data collection tool also monitors changes in the page's DOM structure. This includes changes in page element identifiers related to the operation, including but not limited to element XPath paths, CSS selectors, or unique IDs. Furthermore, the page URL redirection process and changes in browser history are also fully recorded by the data collection tool.

[0149] In situations where a more intuitive representation of the operation scenario is needed, page snapshots can be automatically captured at key steps, or screen recording can be activated to generate video recordings of the operation process, serving as an auxiliary reference for subsequent analysis and instruction generation.

[0150] The aforementioned data collection tools enable the continuous, non-intrusive collection of structured interaction data while the user performs actions, ensuring the integrity and accuracy of the action examples.

[0151] Based on the user's operation examples and the generated operation content, an automated operation plan is executed to generate a sequence of operation steps for the page to be operated.

[0152] Optionally, one possible implementation of generating the operation step sequence may include: obtaining a pre-built large language model and a first prompt word template; wherein, the first prompt word template includes preset slots for receiving operation content to guide the generation of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step; the slots in the preset slots for receiving operation description content are optional slots; filling the operation content into the first prompt word template to generate operation planning prompt words; and generating an operation step sequence for the page to be operated based on the large language model and the operation planning prompt words.

[0153] In this embodiment, the agent can call a general large language model or use a large language model pre-trained based on a corpus of a specific domain, and simultaneously acquire or dynamically construct a first prompt word template for reasoning using the large language model.

[0154] Specifically, the first prompt word template is a structured text framework used to guide the generation of a complete sequence of operation steps to be executed sequentially, and the expected changes in the page state corresponding to each step. Here, the first prompt word template includes preset slots for receiving user input. Among these preset slots, the slot for receiving text descriptions is mandatory, while the slot for receiving operation descriptions—that is, standardized instructions generated based on user operation examples—is optional, accommodating input modes that provide only text descriptions or both text and examples.

[0155] Based on this, the agent fills the corresponding slots in the first prompt word template with the user-input operation content, i.e., the text description content, or both the text description content and the operation description content generated based on the user's operation example, thereby generating operation planning prompt words. Furthermore, this operation planning prompt word is used as input to the large language model, which, based on its semantic understanding and logical reasoning capabilities, generates a sequence of operation steps for the page to be operated, with a clear execution order and expected state.

[0156] For example, the sequence is typically output as structured data such as JSON or a specific instruction format, which can be directly used for subsequent automated execution.

[0157] In the above approach, by using a large language model for adaptive planning, multi-step operation sequences can be flexibly generated based on user input, improving the adaptability to different pages to be operated on. At the same time, by completing multi-step planning at once, the efficiency of overall operation planning and execution is improved.

[0158] Based on the sequence of operation steps, the agent can execute each operation step in the order of the sequence.

[0159] During execution, taking any operation step as an example, one possible implementation of executing the current operation step may include: calling a preset browser automation testing tool and sending the execution instruction corresponding to the current operation step to the browser automation testing tool; and executing the current operation step on the page to be operated based on the browser automation testing tool.

[0160] Specifically, the agent invokes pre-defined browser automation testing tools, such as open-source tools like Selenium, Puppeteer, and Playwright, and sends the execution instructions corresponding to the current operation step to the tool.

[0161] Taking Playwright, a browser automation testing tool, as an example, Playwright locates the corresponding elements on the page and simulates user interaction behaviors based on the received execution instructions, such as entering text in a specified input box, clicking a button or link, selecting a drop-down option, and submitting a form.

[0162] In the above method, by calling a preset browser automation testing tool, the intelligent agent can transform the planned sequence of operation steps into executable instructions, realize the automated operation of the page to be operated, and thus improve the accuracy, efficiency and repeatability of the page automation operation.

[0163] To ensure seamless integration between operational steps during execution and to guarantee the rapid and continuous execution of the next step after the previous one has been successfully completed, the agent can send a complete sequence of operational steps, including operational data, to the browser-based automated testing tool all at once before executing all operational steps.

[0164] Specifically, to facilitate efficient execution of each operation step by browser automation testing tools, the operation data sent by the agent can include the operation type, such as click, input, scroll, etc., target element positioning information, such as XPath, CSS selectors, and necessary operation parameters, such as input text content.

[0165] In this way, during execution, the agent only needs to send control command signals such as "continue execution", "pause" or "stop" to the browser's automated testing tool, without having to repeatedly send specific execution commands after each step, thereby reducing communication overhead and improving the coherence and overall efficiency of the execution process.

[0166] In some alternative implementations, since dynamic adjustments may occur during the execution of operation steps, such as conditional branches, error rollbacks, or user interventions, in order to avoid wasting resources by not executing some of the pre-sent operation step data, the intelligent agent can adopt an on-demand sending method, that is, simultaneously sending the specific operation data required for the current operation step while sending the current operation step execution instruction.

[0167] Of course, a preloading optimization strategy can also be adopted. For example, when comparing or verifying the status of the current operation step, the operation data for the next operation step can be sent in advance to reduce the waiting delay between steps. This embodiment does not strictly limit the timing of sending operation data and can be flexibly selected according to the actual execution efficiency and system load requirements.

[0168] In order to obtain the actual state changes of the page to be operated after the current operation step is executed, it is necessary to ensure that the page data collection tool is ready and in working condition before execution.

[0169] Specifically, if the user provides an example of their operation when inputting the operation content, the data collection tool that records the example can directly collect data for the current operation step.

[0170] If the user does not provide an operation example when inputting operation content, in order to achieve data collection during the execution of the current operation step, this embodiment also includes the following steps: before executing the current operation step based on the browser automated testing tool, obtain a preset page data collection tool and inject it into the running environment of the page to be operated, so as to ensure that page state changes can be monitored and collected in real time.

[0171] Specifically, before executing the current operation step, the intelligent agent obtains a preset page data collection tool from a local or remote configuration.

[0172] For example, the obtained page data collection tool can be a JavaScript script. Based on this, the collection script can be loaded into the page context by executing preset script injection methods such as Selenium's `execute_script` or Puppeteer's `evaluate`, making it part of the page to be operated and starting to run.

[0173] Through the above implementation methods, even in the absence of user operation examples, the intelligent agent can still achieve comprehensive and real-time data collection of the operation execution process by actively injecting monitoring tools, providing a reliable data foundation for closed-loop control and status verification of automated processes.

[0174] Once the injection is complete, the page data collection tool enters a ready state and begins to continuously monitor and record various state changes of the page during the operation process.

[0175] The agent then drives the browser's automated testing tool to execute the current operation step. During or after execution, it calls the interface provided by the data collection tool or reads the log data it outputs and analyzes the log data to obtain information on the actual changes in the page state caused by the current step.

[0176] In one optional implementation, obtaining the actual state change of the page to be operated on after executing the current operation step may include: taking the moment when the current planned operation step is executed as the start time, and the moment when the page to be operated on stops changing after executing the current planned operation step as the end time; obtaining all single-point events between the start time and the end time; wherein, single-point events include user interaction events and page DOM structure change events; and determining the actual state change of the page to be operated on after executing the current operation step based on each single-point event.

[0177] To achieve accurate capture of page state changes, the data collection time window corresponding to the current operation step can be predetermined before starting data collection.

[0178] Specifically, the start time is the moment when the intelligent agent drives the browser automated testing tool to start executing the currently planned operation steps, and the end time is the moment when the page to be operated stops undergoing any detectable changes after execution.

[0179] It should be understood that the cessation of any detectable changes can be determined comprehensively by monitoring factors such as DOM structure stability, idle network requests, and a certain time threshold of no user interaction events on the page, without making specific limitations on this.

[0180] Within the time window from the start time to the end time, all single-point events that occur during this time period are obtained using the injected page data collection tool.

[0181] These events mainly fall into two categories: one is user interaction events, which are simulated user interaction behaviors directly triggered by automated tools, such as mouse clicks, keyboard input, focus switching, scrolling, etc.; the other is page DOM structure change events, which are changes to the document object model (DOM) structure caused by the aforementioned operations, such as adding or deleting page elements, modifying attributes, updating styles, and changing text content.

[0182] For example, a sample of a partial single-point event may include data in the form of: [{time:20:18:00,type:keyboard,content:''}, {time:20:19:00,type:DomChange,content:'input value changes from empty to 1234, the label of the input is the merchant number'}, {time:20:20:20,type:click,content:query}].

[0183] Furthermore, the type, target element, timestamp, and related data of each single event can be recorded in chronological order to form a complete event sequence log.

[0184] Based on this, event aggregation and state deduction can be performed on the acquired individual events to determine the actual state changes of the page after the current operation step is executed.

[0185] In the above method, by precisely defining the time window from the start of the operation to the page stabilization, and capturing all user interaction events and DOM structure change events within this window, the actual state evolution process of the page after the operation can be accurately and completely reconstructed from the underlying event data, thereby improving the reliability of the verification of the results of subsequent steps.

[0186] In the implementation of the above scheme, an optional implementation method for determining the actual state change of the page after executing the current operation step based on each single event may include: performing sliding window aggregation on all single events based on a preset window threshold to generate multiple aggregated events; calling a preset first description content generation model; wherein, the first description generation model is a model trained based on natural language processing technology for converting structured event sequences into natural language descriptions; and performing page change analysis based on the first description content generation model according to each aggregated event to generate the actual state change of the page after executing the current operation step.

[0187] Specifically, by using a fixed-step sliding time window, multiple single-point events falling within the same window are merged into a single aggregated event. It should be understood that each aggregated event may include information such as statistics on the event types occurring within the window, the set of target elements involved, a summary of key attribute changes, and the window's start and end timestamps.

[0188] This allows a large number of fine-grained events to be integrated into event units at a higher semantic level, reducing data redundancy and highlighting event clusters with temporal correlation.

[0189] During this process, a preset first description content generation model is synchronously invoked. In this embodiment, the model can be a dedicated model trained based on natural language processing technology. Its training data typically contains a large number of paired samples from "structured event sequences" to natural language descriptions, enabling it to convert event sequences into coherent and semantically accurate text descriptions.

[0190] Based on the model and the aggregated events, the sequence of aggregated events arranged in chronological order is input into the model. The model can understand the semantics and structure of the event sequence through the encoder, and then generate the corresponding natural language description through the decoder. The generated description is the actual change in the page state caused by the current step.

[0191] In the above approach, fine-grained single-point events are integrated into semantically richer aggregate events through sliding window aggregation. Then, a dedicated model trained based on natural language processing technology is used to transform these aggregate events into natural language descriptions of the actual changes in the page state. The agent can clearly and structurally present the actual results after the operation is executed, thereby comparing and verifying them with the expected state. This provides a reliable and interpretable basis for determining the successful execution of automated steps, improving the accuracy and reliability of the automated process.

[0192] To improve the accuracy of the analysis of actual page state changes and thus enhance the reliability of subsequent comparison and verification, another optional implementation method for determining the actual page state changes after executing the current operation step, based on each single-point event, may include: performing sliding window aggregation on all single-point events based on a preset window threshold to generate multiple aggregated events; acquiring the starting page image corresponding to the start time and the ending page image corresponding to the end time; calling a preset second description content generation model; wherein the second description generation model is a model jointly trained based on natural language processing and image processing technologies, used to convert structured event sequences into natural language descriptions; and based on the second description content generation model, performing page change analysis according to each aggregated event, the starting page image, and the ending page image to generate the actual page state changes after executing the current operation step.

[0193] Specifically, during the process of obtaining the aggregation result based on the aforementioned implementation method, the starting page image corresponding to the start time and the ending page image corresponding to the end time can also be acquired simultaneously. In other words, a screenshot of the page before the operation is performed and a screenshot of the page after the page stops changing are acquired. These two images visually preserve the overall appearance of the page before and after execution, and can supplement any visual changes that may be missed in the event sequence, such as layout offsets, color changes, and non-DOM-driven animation effects.

[0194] The second description content generation model is invoked. Here, this model can be a multimodal model jointly trained based on natural language processing and image processing techniques. Its training data includes paired samples of event sequences, previous and subsequent page images, and corresponding natural language descriptions. In addition, the model typically adopts a fusion encoder structure, which can simultaneously understand the semantic logic of the event sequence and the visual features in the image, and establish the relationship between the two.

[0195] Based on this, the aggregated event sequence and the images of the preceding and following pages are input into the second descriptive content generation model. The model uses its internal multimodal fusion mechanism to comprehensively identify the behavioral logic triggered by the event and the differences in the visual state reflected by the image, and outputs a natural language description, which is the actual change in the page state caused by the current step.

[0196] The above implementation method enhances the robustness and accuracy of state change recognition by introducing page visual information and performing multimodal fusion analysis with event sequences. In particular, it can capture visual details that pure event monitoring may overlook, thereby providing a more comprehensive and reliable description of the actual state for subsequent comparison and verification with the expected state.

[0197] After obtaining the actual state change of the page to be operated on after executing the current operation step, the technical solution of this embodiment further includes: comparing it with the expected state change of the page corresponding to the current operation step, and confirming that the current operation step was successfully executed when the two are consistent.

[0198] Based on this, one possible implementation for confirming the successful execution of the current operation step may include: obtaining a preset state change comparison model; wherein, the state change comparison model is a model based on semantic similarity calculation or rule matching, used to determine whether the state changes described by two natural language are consistent; according to the state change comparison model, performing similarity comparison processing on the actual state change of the page and the expected state change of the page to obtain the comparison result; if the similarity of the comparison result meets the preset threshold, then confirming that the actual state change of the page is consistent with the expected state change of the page.

[0199] Specifically, a pre-trained state change comparison model is obtained. In this embodiment, the model can be a dedicated model built based on semantic similarity calculation or rule matching mechanism, used to determine whether the state changes described by two natural language are semantically consistent.

[0200] For example, in a semantic similarity-based implementation, the model can use a pre-trained natural language understanding model, such as BERT or Sentence BERT, to extract semantic vectors of texts describing the changes between two states, and quantify semantic consistency by calculating the cosine similarity or Euclidean distance between the vectors.

[0201] For example, in a rule-based matching implementation, the model may rely on predefined keywords, phrase patterns, or syntactic structure rules for logical matching.

[0202] Of course, in some scenarios, the state comparison model can also be based on other algorithms and models for comparison processing, and there are no limitations on this.

[0203] For example, an NLI (Natural Language Inference) dataset of actions and page results is constructed, and the BERT model is fine-tuned to infer the causal relationship between actions and page results, resulting in a trained state comparison model. Furthermore, this state model is used to infer changes in two page states to determine whether they are consistent.

[0204] After the state comparison model processes the data to obtain the comparison result, the comparison result is compared with a preset similarity threshold. If the similarity of the comparison result meets or exceeds the threshold, it is confirmed that the actual state change of the page is consistent with the expected state change of the page, and the current operation step can be determined to have been executed successfully. If the similarity is lower than the threshold, it is determined to be inconsistent, and the current step may have been executed abnormally or failed to achieve the expected effect. The agent can trigger the corresponding abnormal handling process, such as logging the error, attempting to retry, or notifying the user to intervene.

[0205] In the above implementation, semantic comparison enables efficient and objective verification of operation results, reduces reliance on manual judgment, and improves the reliability and scalability of automated testing or execution processes.

[0206] In the above implementation process, if the similarity of the comparison results does not meet the preset threshold and it is confirmed that the actual change of the page state is inconsistent with the expected change of the page state, the technical solution of this embodiment further includes: sending a stop execution command to the browser automated testing tool to stop the execution of subsequent planned operation steps.

[0207] Specifically, the agent can send a stop command to the browser's automated testing tool. This command is used to forcibly interrupt the currently executing automated operation or operation verification and stop the execution of all subsequent planned operation steps.

[0208] This prevents the accumulation of errors or further disruption of the system state caused by continuing to perform operations on an erroneous basis, thereby ensuring the safety and controllability of automated processes.

[0209] In some scenarios, while or after the agent outputs the instruction to stop execution, it can automatically record the anomaly, trigger alarms, or attempt to resume operations to improve problem-solving efficiency and system robustness.

[0210] The above implementation method improves the robustness of the agent when facing unexpected states by interrupting the execution process in a timely manner and coordinating with exception handling, avoids the spread of errors, and provides effective support for quickly locating and repairing problems.

[0211] During the execution of the above implementation method, the technical solution provided in this embodiment further includes: obtaining the actual page state changes and expected page state changes corresponding to the current operation step and the previously executed historical operation steps, and using them as operation process data; filling the operation process data and operation content into a pre-built second prompt word template to generate re-planned prompt words; wherein, the second prompt word template includes preset slots for receiving operation process data and operation content to guide the regeneration of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step; and regenerating the operation step sequence for the page to be operated based on the preset large language model and the re-planned prompt words.

[0212] Specifically, the actual and expected changes in the page state corresponding to the current operation step and previously executed historical operation steps are obtained, and this information is collectively referred to as operation process data. In this embodiment, this process data together constitutes a complete operation trajectory and state comparison record from the start of the automated operation to the current execution anomaly.

[0213] The agent invokes a general-purpose large language model, or employs a large language model pre-trained on a domain-specific corpus, and simultaneously acquires or dynamically constructs a second cue word template for inference using this large language model. Here, the second cue word template guides the large language model to understand the actual situation of the current process, where deviations have occurred, and the expected target state, thereby providing sufficient context for its replanning.

[0214] Specifically, the second prompt template also includes multiple preset slots for receiving information such as the operation content of each executed step, the operation object, the description of the actual page state changes, the description of the expected page state changes, and the comparison results. Furthermore, among these preset slots, the slot for receiving text descriptions is mandatory, while the slot for receiving operation descriptions—that is, standardized instructions generated based on user operation examples—is optional, accommodating both text descriptions and examples as input modes.

[0215] Based on this, the agent fills the second prompt word template with the operation process data and the operation content entered by the user, and generates the re-planned prompt words.

[0216] Using the operation planning prompt as input to the large language model, the model will understand the historical execution context and the current anomaly, infer feasible subsequent operation paths, and output a new sequence of operation steps containing multiple new operation steps and the expected changes in the page state corresponding to each step.

[0217] The newly generated sequence of operation steps can be submitted to the browser's automated testing tool immediately or after confirmation to replace the original interrupted process and continue execution until all operation steps in all operation step sequences corresponding to the operation content have been successfully executed, indicating that all automated operations corresponding to the operation content have been completed.

[0218] In the above implementation, by utilizing the reasoning and planning capabilities of a large language model, the automated process is intelligently and dynamically adjusted when encountering unexpected states, thereby improving the agent's adaptability to complex and ever-changing page environments and its robustness in task completion.

[0219] Figure 3 A schematic diagram of the structure of the page automation operation device provided in this application is shown below. Figure 4 As shown, the page automation operation device 30 provided in this embodiment includes:

[0220] The operation content receiving module 301 is used to receive operation content input by the user; wherein, the operation content is used to perform automated operations on the page to be operated.

[0221] The operation step sequence generation module 302 is used to generate an operation step sequence for the page to be operated based on the operation content; wherein, the operation step sequence includes multiple sequentially arranged operation steps, and the expected page state change corresponding to each operation step;

[0222] The first execution module 303 is used to execute the current operation step for each operation step in the operation step sequence, and to obtain the actual state change of the page to be operated after the current operation step is executed.

[0223] The second execution module 304 is used to continue executing the next operation step when the actual change in the page state matches the expected change in the page state, until all operation steps are completed. In one possible implementation,

[0224] In one optional embodiment, the operation content receiving module 301 is specifically used for:

[0225] It receives text descriptions input by the user and uses these text descriptions as the user's input for operations; the text descriptions are natural language texts that describe the goals and intentions of the automated operations.

[0226] In one optional embodiment, the operation content receiving module 301 is specifically used for:

[0227] Receive text description content input by the user;

[0228] After receiving the text description, the system prompts the user to provide a user action example. The user action example is a record of the process demonstrated by the user through actual interaction to achieve the task corresponding to the text description.

[0229] If the user confirms the additional input, then the user operation example is received.

[0230] Based on user operation examples, generate corresponding operation descriptions; the operation descriptions are standardized instruction sequences that define the specific execution steps and logic of automated operations.

[0231] Use the text description and operation description as the operation content for user input.

[0232] In one optional embodiment, the operation content receiving module 301 is specifically used for:

[0233] Before the user inputs an example of their action, the page data collection tool is injected into the page to be operated.

[0234] Based on a page data collection tool, collect user operation examples.

[0235] In one alternative implementation, the page data collection tool includes a JavaScript script; the JavaScript script is used to monitor and collect user interaction behaviors and changes in the page's DOM structure.

[0236] In one optional embodiment, the operation step sequence generation module 302 is specifically used for:

[0237] Obtain a pre-built large language model and a first prompt word template; wherein, the first prompt word template contains preset slots for receiving operation content, so as to guide the generation of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step; the slots in the preset slots for receiving operation description content are optional slots;

[0238] Fill the operation content into the first prompt word template to generate operation planning prompt words;

[0239] Based on the large language model and operation planning prompts, a sequence of operation steps for the page to be operated is generated.

[0240] In one alternative implementation, the first execution module 303 is specifically used for:

[0241] Invoke the preset browser automation testing tool and send the execution command corresponding to the current operation step to the browser automation testing tool;

[0242] Based on browser-based automated testing tools, perform the current operation steps on the page to be operated.

[0243] In one alternative implementation, the first execution module 303 is further configured to:

[0244] Before sending the execution instructions corresponding to the current operation step to the browser automation testing tool, send the operation data of all operation steps to the browser automation testing tool.

[0245] In one alternative implementation, the first execution module 303 is further configured to:

[0246] Before executing the current operation step using browser-based automated testing tools, obtain preset page data collection tools;

[0247] Inject the page data collection tool into the page to be operated.

[0248] In one alternative implementation, the first execution module 303 is specifically used for:

[0249] If the user does not confirm and supplement the user operation example when entering the operation content, the page data collection tool will be injected into the page to be operated before the browser-based automated testing tool executes the current operation step.

[0250] In one alternative implementation, the first execution module 303 is specifically used for:

[0251] The start time is the moment when the current planned operation step is executed, and the end time is the moment when the pending operation page stops changing after the current planned operation step is executed;

[0252] Retrieve all single-point events between the start and end times; these single-point events include user interaction events and page DOM structure change events.

[0253] Based on each single event, determine the actual state change of the page to be operated on after the current operation step is executed.

[0254] In one alternative implementation, the first execution module 303 is specifically used for:

[0255] Based on a preset window threshold, all single-point events are aggregated using a sliding window to generate multiple aggregated events;

[0256] Invoke the preset first description content generation model; wherein, the first description generation model is a model trained based on natural language processing technology, used to convert structured event sequences into natural language descriptions;

[0257] Based on the first description content generation model, page change analysis is performed according to each aggregated event to generate the actual page state changes after executing the current operation step.

[0258] In one alternative implementation, the first execution module 304 is specifically used for:

[0259] Based on a preset window threshold, all single-point events are aggregated using a sliding window to generate multiple aggregated events;

[0260] Get the starting page image corresponding to the start time and the ending page image corresponding to the end time;

[0261] The preset second description content generation model is invoked; wherein, the second description generation model is a model trained jointly based on natural language processing technology and image processing technology, used to convert structured event sequences into natural language descriptions;

[0262] Based on the second description content generation model, page change analysis is performed according to each aggregated event, the starting page image, and the ending page image to generate the actual page state changes after executing the current operation step.

[0263] In one alternative implementation, the first execution module 304 is specifically used for:

[0264] Obtain a preset state change comparison model; wherein, the state change comparison model is a model based on semantic similarity calculation or rule matching, used to determine whether the state changes of two natural language descriptions are consistent;

[0265] Based on the state change comparison model, the actual state change of the page and the expected state change of the page are compared to obtain the comparison results.

[0266] If the similarity of the comparison results meets the preset threshold, it is confirmed that the actual change in the page state is consistent with the expected change in the page state.

[0267] In one alternative implementation, the second execution module 304 is further configured to:

[0268] If the similarity of the comparison results does not meet the preset threshold, it is confirmed that the actual change in the page state is inconsistent with the expected change in the page state.

[0269] Send a stop command to the browser's automated testing tool to halt the execution of subsequent planned steps.

[0270] In one alternative implementation, the second execution module 304 is further configured to:

[0271] Obtain the actual and expected changes in the page state corresponding to the current operation step and the previously executed historical operation steps, and use them as operation process data;

[0272] The operation process data and operation content are filled into the pre-built second prompt word template to generate re-planned prompt words; wherein, the second prompt word template contains preset slots for receiving operation process data and operation content, so as to guide the regeneration of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step;

[0273] Based on the preset large language model, and by re-planning the prompt words, a new sequence of operation steps for the page to be operated is generated.

[0274] The page automation operation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0275] Figure 4 This is a block diagram of an electronic device provided in an embodiment of this application. The device may be a computer, a digital broadcasting terminal, etc. See also... Figure 4 The device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output interface 412, sensor component 414, and communication component 416.

[0276] Processing component 402 typically controls the overall operation of device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0277] Memory 404 is configured to store various types of data to support the operation of device 400. Examples of this data include instructions for any application or method operating on device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0278] Power supply component 406 provides power to various components of device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 400.

[0279] Multimedia component 408 includes a screen that provides an output interface between device 400 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera and / or a rear-facing camera. When device 400 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0280] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0281] Input / output interface 412 provides an interface between processing component 402 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0282] Sensor assembly 414 includes one or more sensors for providing state assessments of various aspects of device 400. For example, sensor assembly 414 may detect the on / off state of device 400, the relative positioning of components such as the display and keypad of device 400, changes in the position of device 400 or a component of device 400, the presence or absence of user contact with device 400, the orientation or acceleration / deceleration of device 400, and temperature changes of device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0283] Communication component 416 is configured to facilitate wired or wireless communication between device 400 and other devices. Device 400 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0284] In an exemplary embodiment, device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0285] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of the device 400 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0286] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a server's processor, enable the server to perform the methods described above.

[0287] This application also provides a chip for executing instructions, which is used to execute the technical solutions in the above embodiments.

[0288] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed on a computer, cause the computer to perform the above-described technical solution.

[0289] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the above-described technical solution can be implemented when the at least one processor executes the computer program.

[0290] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0291] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0292] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0293] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for automating page operations, characterized in that, The method includes: Receive user input of operation content; wherein, the operation content is used to perform automated operations on the page to be operated; Based on the operation content, an operation step sequence is generated for the page to be operated; wherein, the operation step sequence includes multiple sequentially arranged operation steps, and the expected page state change corresponding to each operation step; For each operation step in the operation step sequence, execute the current operation step and obtain the actual state change of the page to be operated after executing the current operation step; If the actual change in the page state matches the expected change in the page state, continue to execute the next operation step until all operation steps corresponding to the operation content are completed.

2. The method according to claim 1, characterized in that, Receive user input, including: The system receives text descriptions input by the user and uses these text descriptions as the user's input operation content; wherein, the text descriptions are natural language text describing the goals and intentions of the automated operation.

3. The method according to claim 1, characterized in that, Receive user input, including: Receive text description content input by the user; After receiving the text description, the user is prompted to provide a user operation example; wherein, the user operation example is a record of the process demonstrated by the user through actual interactive operation to achieve the task corresponding to the text description. If the user confirms the additional input, then the user operation example input by the user is received; Based on the user operation example, a corresponding operation description is generated; wherein, the operation description is a standardized instruction sequence that defines the specific execution steps and logic of the automated operation; The text description and the operation description are used as the operation content input by the user.

4. The method according to claim 3, characterized in that, Examples of user operations that receive user input include: Before the user inputs the user operation example, the page data collection tool is injected into the page to be operated; The user operation examples are collected based on the page data collection tool.

5. The method according to claim 4, characterized in that, The page data collection tool includes a JavaScript script; the JavaScript script is used to monitor and collect user interaction behavior and page DOM structure changes on the page to be operated.

6. The method according to claim 1, characterized in that, Based on the operation content, a sequence of operation steps for the page to be operated is generated, including: Obtain a pre-built large language model and a first prompt word template; wherein, the first prompt word template includes a preset slot for receiving the operation content, so as to guide the generation of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step; the slot for receiving the operation description content in the preset slot is an optional slot; Fill the operation content into the first prompt word template to generate an operation planning prompt word; Based on the large language model and the operation planning prompts, a sequence of operation steps for the page to be operated is generated.

7. The method according to claim 1, characterized in that, Perform the current operation steps, including: Invoke the preset browser automation testing tool and send the execution command corresponding to the current operation step to the browser automation testing tool; Based on the browser's automated testing tool, the current operation steps are executed on the page to be operated.

8. The method according to claim 7, characterized in that, Before sending the execution instruction corresponding to the current operation step to the browser automated testing tool, the method further includes: All operation data for each step is sent to the browser automated testing tool.

9. The method according to claim 7, characterized in that, Before executing the current operation step based on the browser automated testing tool, the method further includes: Obtain the preset page data collection tool; The page data collection tool is injected into the page to be operated.

10. The method according to claim 9, characterized in that, Injecting the page data collection tool into the page to be operated includes: If the user does not confirm and supplement the user operation example when inputting operation content, the page data collection tool will be injected into the page to be operated before the current operation step is executed based on the browser automated testing tool.

11. The method according to claim 1, characterized in that, Obtaining the actual state changes of the page to be operated on after executing the current operation step, including: The start time is the moment when the current planned operation step is executed, and the end time is the moment when the page to be operated stops changing after the current planned operation step is executed; Obtain all single-point events between the start time and the end time; wherein, the single-point events include user interaction events and page DOM structure change events; Based on each of the single-point events, determine the actual state change of the page to be operated on after the current operation step is executed.

12. The method according to claim 11, characterized in that, Based on each of the aforementioned single-point events, determine the actual state change of the page to be operated on after executing the current operation step, including: Based on a preset window threshold, all single-point events are aggregated using a sliding window to generate multiple aggregated events; Invoke a preset first description content generation model; wherein, the first description generation model is a model trained based on natural language processing technology, used to convert structured event sequences into natural language descriptions; Based on the first description content generation model, page change analysis is performed according to each of the aggregated events to generate the actual page state change after executing the current operation step.

13. The method according to claim 11, characterized in that, Based on each of the aforementioned single-point events, determine the actual state change of the page to be operated on after executing the current operation step, including: Based on a preset window threshold, all single-point events are aggregated using a sliding window to generate multiple aggregated events; Obtain the starting page image corresponding to the starting time and the ending page image corresponding to the ending time; The preset second description content generation model is invoked; wherein, the second description generation model is a model trained jointly based on natural language processing technology and image processing technology, used to convert structured event sequences into natural language descriptions; Based on the second description content generation model, page change analysis is performed according to each of the aggregated events, the starting page image, and the ending page image to generate the actual page state change after executing the current operation step.

14. The method according to claim 1, characterized in that, The actual changes in the page state are consistent with the expected changes in the page state, including: Obtain a preset state change comparison model; wherein, the state change comparison model is a model based on semantic similarity calculation or rule matching, used to determine whether the state changes of two natural language descriptions are consistent; Based on the state change comparison model, a similarity comparison is performed between the actual state change of the page and the expected state change of the page to obtain the comparison result; If the similarity of the comparison results meets the preset threshold, then it is confirmed that the actual change in the page state is consistent with the expected change in the page state.

15. The method according to claim 14, characterized in that, The method further includes: If the similarity of the comparison result does not meet the preset threshold, it is confirmed that the actual change in the page state is inconsistent with the expected change in the page state. Send a stop command to the browser's automated testing tool to halt the execution of subsequent planned steps.

16. The method according to claim 15, characterized in that, The method further includes: Obtain the actual and expected changes in the page state corresponding to the current operation step and the previously executed historical operation steps, and use them as operation process data; The operation process data and the operation content are filled into a pre-built second prompt word template to generate a re-planning prompt word; wherein, the second prompt word template includes a preset slot for receiving the operation process data and the operation content, so as to guide the regeneration of an operation step sequence containing multiple operation steps and the expected page state changes corresponding to each operation step; Based on the preset large language model and the re-planned prompt words, a new sequence of operation steps for the page to be operated is generated.

17. A page automation operation device, characterized in that, include: An operation content receiving module is used to receive operation content input by the user; wherein, the operation content is used to perform automated operations on the page to be operated; An operation step sequence generation module is used to generate an operation step sequence for the page to be operated based on the operation content; wherein, the operation step sequence includes multiple sequentially arranged operation steps, and the expected page state change corresponding to each operation step; The first execution module is used to execute the current operation step for each operation step in the operation step sequence, and to obtain the actual state change of the page to be operated after the current operation step is executed. The second execution module is used to continue executing the next operation step when the actual change in the page state is consistent with the expected change in the page state, until all operation steps are completed.

18. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-16.

20. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-16.

Images