Large model driven web task automatic execution method and system

Abstract

The application discloses a large model driven Web task automatic execution method and system, and relates to the field of artificial intelligence.The application perceives and analyzes Web page content through a large model, constructs and maintains a state-action graph, stores and manages a task sequence library, performs semantic matching and task automatic execution.The application solves the problems of high manual intervention demand and low task processing efficiency in traditional Web task automation, and achieves the technical effects of significantly improving automation level, enhancing intelligent decision-making capability, perfecting knowledge memory mechanism and efficient execution.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a large model-driven Web task automation execution method and system. Background Art

[0002] Web task automation is an automation technology that aims to automate the execution of some web tasks based on software systems. By simulating human actions to perform actual operations on web pages or applications, it can greatly improve work efficiency and effectively reduce human labor in repetitive or highly regular tasks.

[0003] A large model refers to a type of neural network model that has been pre-trained on large amounts of data and possesses a massive number of parameters. Because they have far more parameters than traditional AI models, often measured in billions, they are collectively referred to as large models. Through pre-training with large amounts of data, large models internalize a vast amount of human knowledge and accumulate a wealth of human semantics, making them the most human-like models to date. Based on large model technology, intelligent agents are further constructed, capable of interacting with their environments, becoming true "proxies" for humans, capable of completing real-world tasks on their behalf.

[0004] Traditional web task automation systems are mainly implemented in engineering-based and artificial intelligence-based methods. Among them, engineering implementation methods mainly include demonstration programming technology and execution flow design technology. Demonstration programming technology is based on the record-and-replay concept. By manually demonstrating the web task process and recording it, the task script is generated. When needed, the script can be executed by simply replaying it. Execution flow technology relies on users to build the execution flow diagram of the web task in a low-code drag-and-drop format, and clearly define each step of the action such as clicks and inputs, so as to build an executable execution flow and execute it. It can be seen that the engineering method still requires a lot of manpower and is not generalizable.

[0005] Traditional AI-based approaches primarily utilize AI models to understand and execute sequences. For example, reinforcement learning-based techniques, through training with large amounts of expert demonstration data and reward design, enable models to learn relevant knowledge about web task execution processes, thereby applying the models to task execution in real-world scenarios. However, the difficulties of reinforcement learning reward design and its weak generalization have led to a catastrophic decline in its effectiveness for complex and diverse web site systems. Action learning-based techniques, using traditional semantic models, attempt to capture and learn semantic information from action execution, thereby mastering the semantic knowledge of task execution. However, this still requires significant manual effort to generate and label interaction traces. This shows that traditional AI methods also require significant data labeling costs, and their intelligence level still needs to be improved.

[0006] With the development of big model technology, in order to further solve the manpower overhead in web task automation and improve the intelligence and generalization of the system, people have begun to build web intelligent agents based on big models. Based on the decision-making ability of big models, by adding perception, action, memory and other capabilities to the big models, they can directly simulate the entire process of perception, planning, decision-making and action when performing web tasks, thus achieving true full-process automation, further improving the degree of automation of web tasks, and receiving widespread attention.

[0007] However, the existing related technologies have the following limitations:

[0008] First, there is a lack of effective methods for perceiving and executing decisions on web pages. Compared to other applications, web pages have a much larger number of elements and information. The HTML of a web page contains a large number of elements and redundant information representation, making it difficult for web agents to understand web content. Furthermore, since the output of large models is natural language descriptions, there is a challenge in parsing natural language decisions to correctly match target elements and execute the correct actions.

[0009] Secondly, there is a lack of effective methods for representing and importing web task knowledge. The correct decision-making of intelligent agents relies on the incorporation of sufficient domain knowledge. For example, a large medical model requires sufficient medical knowledge. The domain knowledge of web tasks is the operational knowledge of the website. However, the current lack of effective knowledge representation and import methods makes it difficult to ensure the correctness of web agents' decision-making.

[0010] Furthermore, there's a lack of effective task memory management mechanisms. Existing web agents are hampered by the time it takes to understand and generate large models, resulting in long execution times for single-step web operations. Without effective task memory management and call mechanisms, even repeated tasks require restarting, making them unbearably time-consuming in practice.

[0011] Therefore, it is very important to design a large-model driven Web task automation method and system that can effectively perceive the content of Web pages, improve the correctness of Web agent planning through effective knowledge introduction methods, and further greatly reduce the system time consumption through the management and call of task memory, so as to further improve the automation, intelligence level and efficiency of Web task automation. Summary of the Invention

[0012] In order to solve the above problems, the present invention provides a large model-driven Web task automation execution method and system, which eliminates manual operations in the process of Web task automation execution and improves the overall intelligence level and execution efficiency.

[0013] The technical solutions adopted by the present invention to solve the above problems are as follows:

[0014] A large model-driven Web task automation execution method includes the following steps:

[0015] 1) Based on the web task description proposed by the user, semantic matching is performed in the task sequence library, the most relevant task sequence is selected, and whether it meets the task requirements is determined;

[0016] 2) If the matching task sequence meets the requirements, the task sequence is executed; otherwise, the web task description is semantically matched with the state description in the constructed state-action graph, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements;

[0017] 3) If the new task sequence meets the requirements, execute the task sequence; otherwise, obtain the status information of the current web page, build an accessibility tree, and obtain a screenshot of the current web page; then, align the visual information and text information of the current web page screenshot and the accessibility tree to generate a description of the current web page;

[0018] 4) Input the web task description and the description information of the current web page into the large model for perception and generate single-step decision information; parse the single-step decision information to obtain single-step web actions, execute these actions step by step, and store the user's tasks after the tasks are completed.

[0019] Furthermore, the step of performing semantic matching in step 1) includes:

[0020] Use semantic vectorization model to convert the web task description proposed by the user into vector representation;

[0021] Use the semantic vectorization model to vectorize all task sequences in the task sequence library to obtain vectorized representations;

[0022] Using the semantic similarity calculation method, the vectorized representation of the Web task description is compared with the vectorized representation of the task sequence in the task sequence library, and the task sequence that is most similar to the user's task description is selected from the task sequence library.

[0023] Furthermore, in step 1), the task sequence library stores and maintains the task sequence, and the steps include:

[0024] Generate a full-process description of the task by splicing the semantic descriptions of the task sequence's states and actions;

[0025] Input the full process description of the task into the multimodal large model to generate an overall description of the task;

[0026] Using the overall description of the task as a semantic index, combined with the full process description of the task and its state-action sequence, the task sequence is vectorized.

[0027] The generated task sequence information, including the overall description, the whole process description and the state-action sequence, is stored in the task sequence library;

[0028] Manage and update the entries in the task sequence library, including deleting task sequences that have not been used for a long time based on storage time and frequency of use.

[0029] Furthermore, the steps of constructing the state-action diagram in step 2) include:

[0030] Based on the different states of the website and the corresponding action execution relationship, an original state-action graph is constructed. The state nodes in the graph represent the different states of the web page, and the one-way edges represent the actions executed between the states.

[0031] Based on the original web page information stored in the state nodes, the similarity between the state nodes is calculated and similar state nodes are merged; at the same time, based on the similarity between the action API and the target element, similar actions in the graph are merged;

[0032] The merged state-action graph is represented by state semantics and action semantics to obtain an abstract state-action graph.

[0033] Furthermore, the step of semantically matching the Web task description with the state description in the constructed state-action diagram in step 2) includes:

[0034] Use the semantic vectorization model to convert the web task description into a vector representation, and also vectorize all state nodes in the abstract state-action graph;

[0035] Based on the above two vectorized representations, the semantic similarity between the web task description and the state nodes is calculated, and the top k state nodes with the highest semantic similarity are selected as candidate intermediate nodes;

[0036] Retrieve all possible paths from the initial state or current state to the candidate intermediate nodes from the state-action graph, combine the semantic descriptions of the states and actions of these paths, and generate description information of all paths;

[0037] Determine whether there is a state node that meets the description of the web task among the candidate intermediate nodes. If there is an intermediate state node that meets the conditions, select an optimal path based on the path description information and automatically execute the operation process corresponding to the path using the original state-action graph;

[0038] If there is no suitable intermediate state, the large model will gradually explore from the initial state or current state to combine a feasible path until the expected goal is reached.

[0039] Furthermore, the step of constructing the accessibility tree in step 3) includes:

[0040] Get a list of all accessible elements on the current web page and iterate through the elements in the list;

[0041] Clean the elements according to their attribute information to remove irrelevant or invalid element information;

[0042] After the cleaning is completed, an accessibility tree is constructed using a depth-first traversal method based on the hierarchical relationship between elements. Each element node in the tree contains name, role, attribute and unique identifier information.

[0043] Furthermore, the step of aligning the visual information and textual information of the current web page screenshot and the accessibility tree in step 3) includes:

[0044] For the constructed accessibility tree, filter out inoperable elements based on the role and attribute information of the elements, and generate an operable element tree for the web page;

[0045] Based on the coordinates and unique number of each element in the operable element tree, the operable element area at the corresponding position is marked on the web page screenshot, and a number is added to each area.

[0046] Furthermore, the single-step decision information generated by the large model in step 4) includes:

[0047] Observation field: requires the large model to describe the main content of the current web page in order to output the main information and structure of the current page;

[0048] Thinking field: requires the large model to analyze the execution purpose and logic of the current step and output the analysis and reasoning of the current task step;

[0049] Target element: requires the large model to select the target element from the accessibility tree and output the unique number of the target element;

[0050] Action API: requires the big model to select an action API from the Web action API library to execute the current step;

[0051] Action API parameters: require the big model to output the parameter values ​​required by the selected action API so that the action API can be executed correctly;

[0052] Task completion judgment: The large model is required to judge the execution status of the current task and confirm whether the task has been completed;

[0053] Planning of subsequent steps: The large model is required to plan the subsequent steps and generate a sequence of operations or a task plan to be performed next.

[0054] A large model-driven web task automation execution system, comprising:

[0055] The multimodal web agent module is used to understand user task requirements and web page content, generate single-step decision information required to execute tasks through multimodal processing, and execute web actions;

[0056] The state-action graph module is used to build and maintain the state-action graph of Web tasks, supporting task sequence generation and agent knowledge learning based on states and actions;

[0057] The task sequence library module is used to store and manage semantic task sequences, supporting the abstraction, semantic matching, and execution of task sequences;

[0058] Scheduling module, used to select the corresponding module according to the scheduling strategy;

[0059] The executor module is used to execute the generated web actions;

[0060] Among them, the task sequence library module performs semantic matching in the task sequence library based on the Web task description proposed by the user, selects the most relevant task sequence, and determines whether it meets the task requirements;

[0061] If the matching task sequence meets the requirements, the task sequence is executed through the executor module; otherwise, the web task description is semantically matched with the state description in the constructed state-action graph through the state-action graph module, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements;

[0062] If the new task sequence meets the requirements, it is executed by the executor module. Otherwise, the multimodal web agent module obtains the status information of the current web page, builds an accessibility tree, and obtains a screenshot of the current web page. Then, by aligning the visual information and text information of the current web page screenshot and the accessibility tree, a description of the current web page is generated.

[0063] The Web task description and the description information of the current Web page are input into the large model for perception, and single-step decision information is generated. Based on the single-step decision information, the executor module parses the single-step Web actions, executes these actions step by step, and stores the user's tasks through the task sequence library module after the task is completed.

[0064] Furthermore, the steps performed by the multimodal web agent module include:

[0065] Obtain a list of accessible elements for the current page based on the browser's CDP interface, clean the element attribute information in the list, and remove irrelevant or invalid elements. Then, construct a simplified accessibility tree based on the hierarchical relationships between elements, and uniquely number each element for identification in the tree.

[0066] Analyze the role and attribute information of elements in the accessibility tree, match them using rules, build the operational element tree of the web page, filter out inoperable elements, and ensure that only interactive elements are retained;

[0067] Get a screenshot of the current web page as the visual modal representation of the page; based on the coordinates and numbers of the elements in the operable element tree, mark the corresponding operable element areas on the screenshot and add a number to each area;

[0068] The task execution information and the current web page description are input into the multimodal large model. The large model generates single-step decision information by understanding the page content and task requirements.

[0069] Parse the single-step decisions generated by the large model, extract the target elements and target actions, and combine this information into executable API calls to perform related operations;

[0070] Determine whether the current task has been completed. If the task is completed, end the execution; otherwise, repeat the above steps until the task is completed.

[0071] Furthermore, the steps performed by the state-action diagram module include:

[0072] Based on automated technology, the original state-action graph of the website is quickly explored and recorded. The state nodes in the graph represent different states of the web page and store related web page information. The unidirectional edges represent the action execution relationship between states and record the original information of the action.

[0073] Merge similar states in the original state-action graph by calculating the multi-dimensional weighted similarity between state nodes and merging similar state nodes. At the same time, based on the similarity between the API and the target element, merge similar actions in the graph and integrate similar actions.

[0074] The large model is used to perceive page information and generate a semantic description of the page status and a function list; at the same time, by analyzing the original execution information, a semantic representation of the action is generated to form an abstract state-action diagram.

[0075] Furthermore, the steps performed by the state-action diagram module include:

[0076] Use the semantic vectorization model to convert the user's natural language task description into a vectorized representation for subsequent semantic similarity calculation;

[0077] The merged and semantically abstracted state-action graph is processed to generate a vectorized representation of the state nodes. The semantic similarity between the task description vector and the state node vector is calculated, and the top k state nodes with the highest similarity are selected as candidate intermediate nodes. All paths from the initial state or current state to these candidate nodes are retrieved to generate path description information.

[0078] Based on the generated path description information, the large model or the user determines the intermediate state nodes that meet the task requirements; if there are intermediate state nodes that meet the conditions, a suitable path is selected for execution; if there are no matching intermediate state nodes, the path is gradually reached from the original state or the current state;

[0079] Starting from the current state node, the large model combines the user's task description and the semantic description of the current state to plan the task phase and select the action that matches the task from the subsequent actions as the next step. Based on the large model's decision, the original state-action graph is used to execute the action and reach the specified next state node.

[0080] Determine whether the task is completed. If the large model confirms that the final state of the task has been reached or determines that the subsequent actions of the current state do not meet the task requirements, the task is ended; otherwise, repeat the previous step and proceed step by step until the task completion requirements are met.

[0081] Furthermore, the steps performed by the task sequence library module include:

[0082] For the executed task sequence, the semantic descriptions of states and actions are arranged in sequence and the whole process description of the task is generated using connectives.

[0083] Input the full process description of the task into the multimodal large model to generate an overall description of the task sequence;

[0084] Use the overall description as a semantic index to build a task sequence storage in the task sequence library; input each task sequence into the vectorization model to generate and store its vectorized representation;

[0085] The web task description proposed by the user is converted into a semantic vector, and the similarity is calculated with the vectorized representation in the task sequence library. The top k most similar task sequences are selected as candidate sequences. The large model or the user judges the candidate sequences and selects the sequence that meets the requirements for execution.

[0086] Furthermore, the steps performed by the scheduling module include:

[0087] Based on the Web task description proposed by the user, the scheduling module calls the task sequence library module to match the task sequence; if there is a matching task sequence in the task sequence library, it will be directly executed after confirmation by the user or the big model, and the task sequence will end after the task is completed;

[0088] If no matching task sequence is found in the task sequence library, the scheduling module calls the state-action diagram module for exploration. The large model drives the exploration of the state-action diagram. If a complete task sequence is found, the sequence is executed and added to the task sequence library.

[0089] If no matching task sequence is found in the state-action graph, the scheduling module will organize the explored information and call the multimodal Web agent module to make a single-step decision; the agent of the multimodal Web agent module will step by step execute these actions until the task is completed, and update the new state and action to the state-action graph; after the task is completed, the new task sequence will be added to the task sequence library.

[0090] Compared with the prior art, the present invention has achieved the following technical effects:

[0091] 1. This invention significantly improves the automation level of web tasks through a large model-driven web agent. Compared to traditional web task automation technologies, this invention reduces the need for human intervention in task execution and achieves a higher degree of automation.

[0092] 2. This invention provides Web agents with deep perception capabilities of web page content and intelligent decision-making analysis through large models, enabling Web agents to better understand and handle complex task requirements. Compared with traditional methods, it has stronger intelligence and better generalization capabilities.

[0093] 3. The present invention establishes a comprehensive knowledge memory mechanism through the state-action diagram module and the task sequence library module. This mechanism effectively reduces the reasoning cost of processing repetitive tasks and improves the execution efficiency of the system by quickly matching and reusing existing task sequences.

[0094] 4. This invention uses a processed accessibility tree and screenshots to represent web page content, and uniquely numbers the elements in the accessibility tree. By filtering the actionable elements and marking them on the screenshots, effective alignment of the visual and textual modalities is achieved.

[0095] 5. The present invention utilizes the encapsulated Web action API and the numbering during page representation to convert the Web element decision of the large model into a number selection that can be directly parsed by the program, determines the unique target element, and generates an executable Web action through the API selection output by the large model.

[0096] 6. The present invention provides a complete system, including an intelligent agent module, a graph knowledge base module and a task sequence library module, as well as its scheduling strategy, which can achieve efficient execution and intelligent decision-making, quickly respond to user needs, reduce task execution time, and improve overall usage effect.

[0097] 7. This paper demonstrates the practical application of big model technology in Web task scenarios, proves how big models can effectively solve practical problems, and lays the foundation for its further application in the field of Web task automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 This is an overall architecture diagram of a large model-driven Web task automation execution system according to an embodiment of the present invention.

[0099] Figure 2 A flowchart of the multimodal Web agent module executing tasks according to an embodiment of the present invention.

[0100] Figure 3 A flowchart of constructing a state-action diagram for a state-action diagram module according to an embodiment of the present invention.

[0101] Figure 4 This is a flowchart of performing abstract graph exploration for the state-action graph module according to an embodiment of the present invention.

[0102] Figure 5 This is a flowchart of task sequence abstraction and matching performed by the task sequence library module according to an embodiment of the present invention.

[0103] Figure 6 This is a flowchart of a scheduling module executing a scheduling strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0104] In order to make the various technical features and advantages or technical effects of the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.

[0105] The embodiment of the present invention discloses a large model-driven Web task automation execution method, the main processing steps of which include:

[0106] 1) For the web task description proposed by the user, a vectorization model is used to generate a web task description vector. The vector is then compared with the task sequences in the task sequence library for semantic similarity calculation. The most relevant task sequences are then selected and handed over to the user or the large model to determine whether they meet the task requirements.

[0107] 2) If the matching task sequence meets the task requirements, the task sequence is directly executed and the task is marked as completed;

[0108] 3) If the matched task sequence does not meet the task requirements, the semantic similarity between the web task description vector and the state description in the state-action diagram is calculated, and the most relevant state node is selected for fast arrival;

[0109] 4) After quickly reaching the target node, the large model is used to gradually explore the state-action diagram to generate a task execution process. If the process meets the task requirements, the task sequence is executed and added to the task sequence library, marking the task as completed.

[0110] 5) If the task execution process obtained by exploration still does not meet the task requirements, the state information of the current web page is obtained, and an accessibility tree is constructed based on the hierarchical relationship of the accessibility elements of the web page;

[0111] 6) Take a screenshot of the current web page and mark the location and number of elements on the screenshot according to the accessibility tree to align visual information with text information;

[0112] 7) Input the current task information and web page information into the large model for perception and decision-making, and generate a single-step web action;

[0113] 8) Analyze the decisions generated by the large model and execute the corresponding single-step Web actions;

[0114] 9) Update the new page status to the state-action diagram;

[0115] 10) Repeat steps 6) to 9) until the task is completed;

[0116] 11) After the task is completed, the task sequence corresponding to the task is added to the task sequence library.

[0117] To implement the above method, the embodiment of the present invention also discloses a large model driven Web task automation execution system, such as Figure 1 As shown in the figure, the system divides the architecture into two main parts according to the capabilities of the Web agent: the decision layer and the execution layer.

[0118] The decision-making layer is mainly responsible for understanding user tasks and planning and making decisions on task execution. The decision-making layer includes the following modules:

[0119] Multimodal Web Agent Module: Used to generate the actions required to complete each step of the task based on the user's task requirements and understanding of the Web page content.

[0120] State-action graph module: used to implement the graph representation of Web task knowledge and realize the learning process of intelligent agent knowledge by exploring on the graph.

[0121] Task sequence library module: provides semantic task sequence abstraction to achieve fast task sequence matching and execution.

[0122] Scheduling module: Make decisions based on scheduling strategies and select appropriate decision paths.

[0123] The execution layer includes the following modules:

[0124] Executor module: parses and actually executes the decision actions generated by the decision layer, such as clicking and inputting on the target elements of the web page, and feeds back the execution results to the decision layer.

[0125] 1. Multimodal Web Agent Module for Task Execution

[0126] The multimodal web agent module provides a web agent built on a multimodal large model. This agent can perform multiple rounds of perception and decision-making based on the web task requirements proposed by the user, thereby completing the entire web task. This agent is built based on a general multimodal large model, preferably using the GPT-4-turbo model, which optimizes its task execution capabilities through contextual learning. The task execution process of the multimodal web agent module is as follows: Figure 2 As shown, the specific execution process includes the following steps:

[0127] S1: Page accessibility tree construction

[0128] First, the browser CDP interface is used to obtain the list of accessibility elements of the current page and these elements are traversed. Then, the elements are cleaned according to their attributes and other information, including: whether the element is ignorable, whether it has a real size (that is, the length and width of the element are not zero), whether it is visible on the page, and whether it is valid. If the element is ignorable, has no real size, is invisible, or is a meaningless empty element, it is cleared. Then, based on the hierarchical relationship between elements and the depth-first traversal method, a simplified accessibility tree is constructed. Each element in the tree includes: name, role, attribute, and unique number. The name describes the semantic information of the element, such as "search" and "refresh"; the role describes the category of the element, such as "button" and "link"; the attribute describes the configuration properties of the element, such as "focusable"; the number is unique, and each element is uniquely numbered.

[0129] S2: Filtering of actionable elements on the page

[0130] After building a simplified accessibility tree, we further analyze and match rules based on the element's role and attribute information to filter out inoperable elements, thereby constructing the web page's operational element tree. Inoperable elements are those that cannot respond to user actions, such as static text. Operational elements are divided into clickable elements and editable elements. Clickable elements are elements that can respond to mouse clicks, such as buttons and links. Editable elements are elements that, after being focused by a mouse click, can be input through input media such as the keyboard, such as search boxes.

[0131] S3: Capture and annotate page screenshots

[0132] After constructing the actionable element tree, a screenshot of the current web page is obtained as a visual modality representation. To align the textual information of the elements in the tree with the visual information of the corresponding elements in the screenshot, the actionable elements at the corresponding locations in the screenshot are marked with boxes and numbers based on the coordinates and numbers of each element in the actionable element tree. In this embodiment, to maintain readability of the visual image, only the actionable elements are boxed and marked, thus achieving alignment between the visual information and the textual information.

[0133] S4: Multimodal Large Model Single-Step Perception Decision

[0134] First, task execution information and the current web page description are input into the multimodal macro model through command prompts. Task execution information includes: user task requirements, completed steps, previous plans, etc.; the current web page description includes: a simplified accessibility tree text of the current page and annotated page screenshot. Then, the multimodal macro model understands and perceives the current task status and web page information, generating single-step decision information. Single-step decision information includes:

[0135] a) Observation: This field requires the large model to describe its observations of the current page, for example: "This is the Baidu search homepage, and this is the search result page for XX."

[0136] b) Thought: This field requires the large model to analyze the current step. For example, "On the Baidu search homepage, to complete the search task XX, you need to fill in XX content in the input box."

[0137] c) Target element: The large model is required to select the target element from the provided simplified accessibility tree and output its unique number.

[0138] d) Action API: The big model is required to select an API name from the provided Web Action API library, such as "click" or "input." The Web Action API library encapsulates common Web operations at the API level and provides the API name, function description, parameters, and parameter descriptions in the rule instructions, helping the big model understand and call the API when appropriate. The big model outputs the decision content in JSON format for subsequent decision analysis.

[0139] e) Action API parameters: The large model is required to output the parameter content required by the selected API. For example, click parameters include left / right key, single / double click, etc.

[0140] f) Task completion judgment: The large model is required to judge whether the current stage of the task has completed the user's task goal. If it has been completed, the completion status is output; otherwise, the execution continues.

[0141] g) Planning of subsequent steps: The big model is required to make preliminary plans for the subsequent steps, which will serve as the planning information input to the big model in the next step for reference.

[0142] S5: Decision parsing and execution

[0143] Because the single-step decision information output by the large model is in natural language, it must be parsed to execute the actual action. By parsing the JSON-formatted decision output by the large model, the various decision components are analyzed. The target element is identified by its unique ID, and the action API instruction is synthesized from the API name and parameters. The actual action is then executed on the target element.

[0144] S6: Task completion judgment

[0145] Based on the judgment of the task stage by the large model, if the task has been completed, the execution ends; otherwise, steps S1 to S5 are repeated until the task is completed. After the task is completed, the corresponding task sequence is added to the task sequence library for subsequent use.

[0146] 2. State-action diagram module to build state-action diagram

[0147] In order to improve the planning ability of the Web agent, it is necessary to introduce Web task knowledge to the agent. To this end, the state-action diagram module provides the knowledge base representation and retrieval method required for the Web agent to understand, plan and make decisions about Web tasks. It also supports the memory function of the executed task process, making the state-action diagram both the knowledge base of the agent and the memory base of task execution. The state-action diagram construction process of the state-action diagram module is as follows: Figure 3 As shown, the specific construction process includes:

[0148] S1: Original state-action graph construction.

[0149] First, the original state-action graph is quickly recorded and generated through automation technology. The automation technology includes manually demonstrated task flow recording, Web intelligent agent task execution recording and automated detection technology to improve the efficiency of graph construction. In the examples of the present invention, automated detection technology is mainly used, such as random detection methods or detection methods based on reinforcement learning, to quickly explore the state and action execution relationship of the Web website and construct the original state-action graph. At the same time, combined with the manually demonstrated task flow recording technology and the Web intelligent agent task execution record, the information of the original state-action graph is further improved. In the constructed state-action graph, the node is called the state node, which represents the state of the web page and stores the original web page information such as HTML and accessibility tree; the edge is a unidirectional edge, which represents the executed action, connects the state before and after the action is executed, and stores the original action information, such as API calls and element locators. The locator is used to uniquely determine the target element. Commonly used locators include coordinates, XPath, CSS selectors, etc., which help the automated script find the target element during execution.

[0150] S2: Graph state and action merging

[0151] Since the original state-action graph is based on the original execution of direct records, it may contain a large amount of similar state and action information, which can easily lead to the explosion of the state space and increase storage and retrieval costs. Therefore, it is necessary to merge the states and actions of the original state-action graph. Specifically, similar states are first merged. The original state-action graph may contain a large number of similar states, such as search engine homepages with different content, shopping cart pages with different numbers of products, etc. It is necessary to calculate the text and image similarity based on the HTML node information, page URL, screenshot information, etc. stored in the state nodes, and calculate the similarity between state nodes by weighted summation. According to the set threshold, states with similarity above the threshold are merged. Furthermore, in the process from state A to state B, if there are multiple actions, the similarity is compared based on the action API and the target element, and highly similar actions are merged for representation. By merging similar states and similar actions, the representation space of the state-action graph can be significantly reduced.

[0152] S3: Semantic abstraction of graph states and actions

[0153] Since the state-action diagram provides knowledge representation for Web agents to perform Web tasks, and the original state HTML information and action API and locator information are not conducive to the understanding of Web agents, it is necessary to perform semantic abstraction on the merged state-action diagram, including state semantic representation and action semantic representation.

[0154] Specifically, for the semantic representation of state, the big model's perception of page information is used to generate a semantic description of the page state. By inputting screenshots of the state storage and accessibility tree information into the multimodal big model, a semantic description of the page and a list of executable functions for the page are generated. For example, for a search engine homepage, the generated semantic description is "XX search engine's homepage"; the list of executable functions for the page may include "content search, hot news list, images, news, weather navigation," etc. These descriptions construct a representation of the webpage state that is easy for the big model to understand.

[0155] For the semantic representation of actions, the original execution information is analyzed to generate a semantic description of the action. For example, for the original execution information click(element_id,left,single), its API function and parameter description are "click, left button, single click". The target element is matched by the element locator to obtain its description information (for example: <button>'Search') can further generate semantic descriptions, such as "left-click the search button." By semantically describing states and actions, the original state-action graph is converted into an abstract semantic description graph, namely an abstract state-action graph. This allows the large model to perform semantic understanding on the graph, thereby enabling the large model to learn and explore web task knowledge on the semantic graph. The aforementioned multimodal large model is obtained through contextual learning from a general multimodal large model, preferably a GPT-4-turbo model.

[0156] 3. State-Action Graph Module for Abstract Graph Exploration

[0157] In addition to the construction, merging, and semantic abstraction of state-action graphs, the state-action graph module also includes an exploration method for abstract state-action graphs. The process of the abstract graph exploration method of the state-action graph module is as follows: Figure 4 As shown, it includes two stages: fast matching and arrival (steps S1 to S3) and gradual arrival (steps S4 to S5).

[0158] S1: Vectorized representation of user tasks

[0159] First, for tasks described in natural language by users, a semantic vectorization model is used to convert the natural language description into a vectorized representation of the task for subsequent semantic similarity calculation. This semantic vectorization model is obtained through large-scale pre-training, and the preferred model is the text-embedding-3-small model.

[0160] S2: Matching task description and state semantic similarity

[0161] For the merged and semantically abstracted state-action graph, the semantic state descriptions of all state nodes are used to generate vectorized representations of the state through a semantic vectorization model. Furthermore, the cosine similarity between the task description vector and the vectors of each state node is calculated to represent the semantic similarity between the two. Through this semantic similarity calculation, the top k state nodes with the highest semantic similarity to the task description are selected as candidate intermediate nodes. Then, all paths from the initial state or current state to these candidate intermediate nodes are retrieved from the graph. The state and action semantic descriptions in these paths are combined to generate path description information, such as: "From the search engine homepage, left-click the search button to reach page XX, left-click the XX button to reach page XX..."

[0162] S3: Large model or user selection matching state arrives

[0163] Based on all the matched path description information, the large model or the user determines whether there is an intermediate state node that meets the task requirements among the candidate intermediate nodes. For example, for the task "Query Beijing Weather", if the matched intermediate state node is "Weather Network Homepage", after the large model or the user confirms that the intermediate state node is correct, a suitable path is selected according to the path description information. The system will use the original state-action graph to automatically execute the process corresponding to the path (i.e., the task sequence), so as to quickly reach the confirmed intermediate state node and complete the fast matching and arrival stage. This stage can effectively accelerate the task matching and execution process. If the intermediate state node that meets the requirements cannot be matched, it will directly enter the gradual arrival stage (i.e., gradually execute each path). The large model is preferably a general multimodal large model obtained through context learning. As a preferred option, the GPT-4-turbo model can be used.

[0164] S4: Large model task state understanding and state-next-action selection

[0165] Starting from the current state node, the large model combines the user's task description with the semantic description of the current state node to form an understanding of the current task stage and plan the subsequent steps. After planning is completed, the large model makes a decision based on the subsequent actions available at the current state node, selecting the action that matches the task execution as the next step to be executed.

[0166] S5: Decision execution reaches the next state

[0167] According to the decision of the large model in the previous step, the corresponding action is executed using the original state-action graph to reach the next state node pointed to by the action.

[0168] S6: Task completion judgment

[0169] If the large model determines that the current state node has reached the task's final state, meaning the task is complete, or if it determines that the subsequent actions and the state pointed to by the state do not meet the requirements of the next step in the task, the process ends. Otherwise, steps S4 to S5 are repeated, gradually reaching the final state through multiple steps, until the end requirements are met. During the exploration process in the graph, the large model continuously records its exploration process and corresponding sequence and executes them synchronously. Therefore, when the complete task sequence is explored, the execution of the related state actions is also completed synchronously, thereby improving system execution efficiency. If the complete sequence cannot be explored, the previously recorded execution sequence will provide knowledge reference for subsequent decision-making of the web agent.

[0170] 4. Task sequence library module performs task sequence abstraction and matching

[0171] In order to improve the task execution efficiency of Web agents and realize the reuse of task memory, the task sequence library module further abstracts the executed tasks and summarizes them into semantic forms so that they can be quickly matched and executed when needed. The abstraction and matching process of task sequences is as follows: Figure 5 The specific steps are as follows:

[0172] S1: Generate semantic description of the entire task sequence

[0173] For a completed task sequence, the semantic descriptions of the states and actions are first combined to form a semantic description of the entire task process. By arranging the semantic descriptions of the states and actions in sequence and connecting them with conjunctions, a full description of the task process can be generated. For example: "From page XX, click on element XX, reach page XX, enter XX, reach page XX, ..., reach page XX, task completed."

[0174] S2: Generate overall description of task sequence

[0175] The generated full-process description of the task is fed into the multimodal model. The model will understand the task sequence and output a summary of its objectives, thus forming an overall description of the task. For example, "Use the XX search engine to search for and visit the homepage of the XX website." This multimodal model is derived from the general model through contextual learning. The preferred general model is the GPT-4-turbo model.

[0176] S3: Semantic, vectorized representation and storage of task sequences

[0177] The generated overall task description is used as the semantic index of the task to construct the task entries in the task sequence library, including: the overall description index of the task, the full process description of the task, the execution sequence corresponding to the task (the state and action sequence on the state-action diagram), etc. In order to achieve fast matching of task sequences, each task sequence in the task sequence library is input into the semantic vectorization model, and the natural language description is converted into its vectorized representation. In this way, fast matching can be achieved in the matching stage by calculating the similarity of semantic vectors. The semantic vectorization model is obtained by large-scale pre-training, and the preferred semantic vectorization model is the text-embedding-3-small model.

[0178] S4: Task sequence matching

[0179] When a user submits a task requirement, the semantic vectorization model converts the user's natural language requirement into a semantic vector. Cosine similarity is calculated with the vectorized representations in the task sequence library, and the top k most similar task sequences are selected as candidate sequences. The overall description, full process description, and corresponding state-action sequence of these candidate sequences are extracted. The large model or the user then makes a judgment and selects the sequence that meets the requirement. This allows the system to quickly execute the selected state-action sequence, significantly improving system efficiency.

[0180] S5: Task sequence library management

[0181] Taking into account the storage and retrieval costs of the task sequence library, in an embodiment of the present invention, by performing a weighted evaluation on factors such as the storage time and the last use time of the task sequence, some task sequences that have not been used for a long time are deleted to optimize storage resources and retrieval efficiency.

[0182] 5. The scheduling module executes the scheduling strategy

[0183] The scheduling module integrates the multimodal Web agent module, the state-action diagram module, and the task sequence library module through scheduling strategies to maximize the execution efficiency of the system. The scheduling strategy can be implemented based on a program or using a large model, among which the preferred large model is the GPT-4-turbo model. The scheduling strategy process of the scheduling module is as follows: Figure 6 As shown, the specific steps include:

[0184] S1: Task sequence library matching

[0185] For task requirements proposed by users, the scheduling module first calls the task sequence library to match task sequences. If a matching task sequence exists in the task sequence library, the system can directly execute the task sequence after confirmation by the user or the large model. The task is considered completed after completion.

[0186] S2: State-Action Graph Exploration

[0187] When the task sequence library fails to match a relevant sequence, the scheduling module calls the state-action graph module, which is driven by the large model to explore the state-action graph. If a complete task sequence is found in the graph, the system executes the task sequence and adds it to the task sequence library for subsequent rapid matching.

[0188] S3: Single-step decision making for multimodal web agents

[0189] If a complete task sequence cannot be found in the state-action graph, the scheduling module will consolidate the explored information and invoke the multimodal web agent module to make single-step decisions until the task is completed. During task execution, new states and actions are updated in the state-action graph. Once a task is completed, the task sequence is added to the task sequence library to enhance its coverage.

[0190] By recording and updating new states and actions, the state-action graph and task sequence library will be gradually improved. This improvement enables the rapid matching and execution of repetitive tasks, significantly reducing the call overhead of the large model, thereby improving the execution efficiency of the system. At the same time, as the state-action graph is improved, the large model can explore more task sequences within the graph, reducing its reliance on the web agent's decision-making. Improvements to the state-action graph also help reduce decision-making errors of the web agent, such as hallucinations. Even when multimodal web agent decisions are required, as the state-action graph is gradually improved, the agent will gain more comprehensive task knowledge, thereby improving the accuracy of its decisions.

[0191] 6. Actuator module

[0192] The Executor module, located in the Execution Layer, is responsible for executing the operational instructions generated by the Decision Layer within a real browser environment. This module interprets the decisions made by the Decision Layer and executes the corresponding actions to achieve the task objectives. The browser environment and operational functions are preferably implemented using the Playwright automation tool, which provides efficient browser automation capabilities.

[0193] Specifically, the functions of the actuator module include:

[0194] S1: Operation parsing: Parsing the operation instructions output by the decision layer, including click, input, navigation, etc. The instructions contain the location information of the target element and the operation parameters.

[0195] S2: Operation execution: Execute the parsed operation instructions in a real browser environment. For example, click on elements, input text, navigate pages, etc. through the Playwright API.

[0196] S3: Result Feedback: Feedback the results of the operation to the decision-making layer, including information on the success or failure of the operation and the change in the page status. This feedback information is used by the decision-making layer for further decision-making and adjustments.

[0197] The implementation of the actuator module ensures efficient conversion from the decision-making level to actual operations, and guarantees the operating efficiency of the system and the accurate completion of tasks.

[0198] Although the present invention has been disclosed as above by way of embodiments, they are not intended to limit the present invention. Any appropriate modification or equivalent substitution of the technical solution of the present invention by a person skilled in the art should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on that defined in the claims.< / button>

Claims

1. A large model driven Web task automation execution method, characterized in that: The following steps are involved: 1) Based on the web task description proposed by the user, semantic matching is performed in the task sequence library, the most relevant task sequence is selected, and whether it meets the task requirements is determined; 2) If the matching task sequence meets the requirements, the task sequence is executed; otherwise, the web task description is semantically matched with the state description in the constructed state-action graph, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements; 3) If the new task sequence meets the requirements, execute the task sequence; otherwise, obtain the status information of the current web page, build an accessibility tree, and obtain a screenshot of the current web page; then, align the visual information and text information of the current web page screenshot and the accessibility tree to generate a description of the current web page; 4) Input the web task description and the description information of the current web page into the large model for perception and generate single-step decision information; parse the single-step decision information to obtain single-step web actions, execute these actions step by step, and store the user's tasks after the tasks are completed.

2. The method according to claim 1, wherein The steps for semantic matching in step 1) include: Use semantic vectorization model to convert the web task description proposed by the user into vector representation; Use the semantic vectorization model to vectorize all task sequences in the task sequence library to obtain vectorized representations; Using the semantic similarity calculation method, the vectorized representation of the Web task description is compared with the vectorized representation of the task sequence in the task sequence library, and the task sequence that is most similar to the user's task description is selected from the task sequence library.

3. The method according to claim 1, wherein In step 1), the task sequence library stores and maintains the task sequence, and the steps include: Generate a full-process description of the task by splicing the semantic descriptions of the task sequence's states and actions; Input the full process description of the task into the multimodal large model to generate an overall description of the task; Using the overall description of the task as a semantic index, combined with the full process description of the task and its state-action sequence, the task sequence is vectorized. The generated task sequence information, including the overall description, the whole process description and the state-action sequence, is stored in the task sequence library; Manage and update the entries in the task sequence library, including deleting task sequences that have not been used for a long time based on storage time and frequency of use.

4. The method according to claim 1, wherein The steps for constructing the state-action graph in step 2) include: Based on the different states of the website and the corresponding action execution relationship, an original state-action graph is constructed. The state nodes in the graph represent the different states of the web page, and the one-way edges represent the actions executed between the states. Based on the original web page information stored in the state nodes, the similarity between the state nodes is calculated and similar state nodes are merged; at the same time, based on the similarity between the action API and the target element, similar actions in the graph are merged; The merged state-action graph is represented by state semantics and action semantics to obtain an abstract state-action graph.

5. The method according to claim 1, wherein The steps of semantically matching the web task description with the state description in the constructed state-action diagram in step 2) include: Use the semantic vectorization model to convert the web task description into a vector representation, and also vectorize all state nodes in the abstract state-action graph; Based on the above two vectorized representations, the semantic similarity between the web task description and the state nodes is calculated, and the top k state nodes with the highest semantic similarity are selected as candidate intermediate nodes; Retrieve all possible paths from the initial state or current state to the candidate intermediate nodes from the state-action graph, combine the semantic descriptions of the states and actions of these paths, and generate description information of all paths; Determine whether there is a state node that meets the description of the web task among the candidate intermediate nodes. If there is an intermediate state node that meets the conditions, select an optimal path based on the path description information and automatically execute the operation process corresponding to the path using the original state-action graph; If there is no suitable intermediate state, the large model will gradually explore from the initial state or current state to combine a feasible path until the expected goal is reached.

6. The method according to claim 1, wherein The steps of building the accessibility tree in step 3) include: Get a list of all accessible elements on the current web page and iterate through the elements in the list; Clean the elements according to their attribute information to remove irrelevant or invalid element information; After the cleaning is completed, an accessibility tree is constructed using a depth-first traversal method based on the hierarchical relationship between elements. Each element node in the tree contains name, role, attribute and unique identifier information.

7. The method according to claim 1, wherein The steps of aligning the visual information and textual information of the current web page screenshot and the accessibility tree in step 3) include: For the constructed accessibility tree, filter out inoperable elements based on the role and attribute information of the elements, and generate an operable element tree for the web page; Based on the coordinates and unique number of each element in the operable element tree, the operable element area at the corresponding position is marked on the web page screenshot, and a number is added to each area.

8. The method according to claim 1, wherein The single-step decision information generated by the large model in step 4) includes: Observation field: requires the large model to describe the main content of the current web page in order to output the main information and structure of the current page; Thinking field: requires the large model to analyze the execution purpose and logic of the current step and output the analysis and reasoning of the current task step; Target element: requires the large model to select the target element from the accessibility tree and output the unique number of the target element; Action API: requires the big model to select an action API from the Web action API library to execute the current step; Action API parameters: require the big model to output the parameter values ​​required by the selected action API so that the action API can be executed correctly; Task completion judgment: The large model is required to judge the execution status of the current task and confirm whether the task has been completed; Planning of subsequent steps: The large model is required to plan the subsequent steps and generate a sequence of operations or a task plan to be performed next.

9. A large model driven Web task automation execution system, characterized in that: include: The multimodal web agent module is used to understand user task requirements and web page content, generate single-step decision information required to execute tasks through multimodal processing, and execute web actions; The state-action graph module is used to build and maintain the state-action graph of Web tasks, supporting task sequence generation and agent knowledge learning based on states and actions; The task sequence library module is used to store and manage semantic task sequences, supporting the abstraction, semantic matching, and execution of task sequences; Scheduling module, used to select the corresponding module according to the scheduling strategy; The executor module is used to execute the generated web actions; Among them, the task sequence library module performs semantic matching in the task sequence library based on the Web task description proposed by the user, selects the most relevant task sequence, and determines whether it meets the task requirements; If the matching task sequence meets the requirements, the task sequence is executed through the executor module; otherwise, the web task description is semantically matched with the state description in the constructed state-action graph through the state-action graph module, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements; If the new task sequence meets the requirements, it is executed by the executor module. Otherwise, the multimodal web agent module obtains the status information of the current web page, builds an accessibility tree, and obtains a screenshot of the current web page. Then, by aligning the visual information and text information of the current web page screenshot and the accessibility tree, a description of the current web page is generated. The Web task description and the description information of the current Web page are input into the large model for perception, and single-step decision information is generated. Based on the single-step decision information, the executor module parses the single-step Web actions, executes these actions step by step, and stores the user's tasks through the task sequence library module after the task is completed.

10. The system according to claim 9, wherein: The steps performed by the multimodal web agent module include: Obtain a list of accessible elements for the current page based on the browser's CDP interface, clean the element attribute information in the list, and remove irrelevant or invalid elements. Then, construct a simplified accessibility tree based on the hierarchical relationships between elements, and uniquely number each element for identification in the tree. Analyze the role and attribute information of elements in the accessibility tree, match them using rules, build the operational element tree of the web page, filter out inoperable elements, and ensure that only interactive elements are retained; Get a screenshot of the current web page as the visual modal representation of the page; based on the coordinates and numbers of the elements in the operable element tree, mark the corresponding operable element areas on the screenshot and add a number to each area; The task execution information and the current web page description are input into the multimodal large model. The large model generates single-step decision information by understanding the page content and task requirements. Parse the single-step decisions generated by the large model, extract the target elements and target actions, and combine this information into executable API calls to perform related operations; Determine whether the current task has been completed. If the task is completed, end the execution; otherwise, repeat the above steps until the task is completed. The steps performed by the state-action diagram module include: Based on automated technology, the original state-action graph of the website is quickly explored and recorded. The state nodes in the graph represent different states of the web page and store related web page information. The unidirectional edges represent the action execution relationship between states and record the original information of the action. Merge similar states in the original state-action graph by calculating the multi-dimensional weighted similarity between state nodes and merging similar state nodes. At the same time, based on the similarity between the API and the target element, merge similar actions in the graph and integrate similar actions. The large model is used to perceive page information and generate a semantic description of the page status and a function list. At the same time, by analyzing the original execution information, a semantic representation of the action is generated to form an abstract state-action graph. The steps performed by the state-action diagram module include: Use the semantic vectorization model to convert the user's natural language task description into a vectorized representation for subsequent semantic similarity calculation; The merged and semantically abstracted state-action graph is processed to generate a vectorized representation of the state nodes. The semantic similarity between the task description vector and the state node vector is calculated, and the top k state nodes with the highest similarity are selected as candidate intermediate nodes. All paths from the initial state or current state to these candidate nodes are retrieved to generate path description information. Based on the generated path description information, the large model or the user determines the intermediate state nodes that meet the task requirements; if there are intermediate state nodes that meet the conditions, a suitable path is selected for execution; if there are no matching intermediate state nodes, the path is gradually reached from the original state or the current state; Starting from the current state node, the large model combines the user's task description and the semantic description of the current state to plan the task phase and select the action that matches the task from the subsequent actions as the next step. Based on the large model's decision, the original state-action graph is used to execute the action and reach the specified next state node. Determine whether the task is completed. If the large model confirms that the task has reached the final state or determines that the subsequent actions of the current state do not meet the task requirements, then end the task; otherwise, repeat the previous step and gradually reach the next state until the task completion requirements are met; The steps performed by the Task Sequence Library module include: For the executed task sequence, the semantic descriptions of states and actions are arranged in sequence and the whole process description of the task is generated using connectives. Input the full process description of the task into the multimodal large model to generate an overall description of the task sequence; Use the overall description as a semantic index to build a task sequence storage in the task sequence library; input each task sequence into the vectorization model to generate and store its vectorized representation; The web task description proposed by the user is converted into a semantic vector, and the similarity is calculated with the vectorized representation in the task sequence library. The top k most similar task sequences are selected as candidate sequences. The large model or the user judges the candidate sequences and selects the sequence that meets the requirements for execution. The steps performed by the scheduling module include: Based on the Web task description proposed by the user, the scheduling module calls the task sequence library module to match the task sequence; if there is a matching task sequence in the task sequence library, it will be directly executed after confirmation by the user or the big model, and the task sequence will end after the task is completed; If no matching task sequence is found in the task sequence library, the scheduling module calls the state-action diagram module for exploration. The large model drives the exploration of the state-action diagram. If a complete task sequence is found, the sequence is executed and added to the task sequence library. If no matching task sequence is found in the state-action graph, the scheduling module will organize the explored information and call the multimodal Web agent module to make a single-step decision; the agent of the multimodal Web agent module will step by step execute these actions until the task is completed, and update the new state and action to the state-action graph; after the task is completed, the new task sequence will be added to the task sequence library.

Images