Large model driven Web task automatic execution method and system

Abstract

The invention discloses a large-model-driven Web task automatic execution method and system, and relates to the field of artificial intelligence. According to the method, Web page content is perceived and analyzed through a large model, a state-action graph is constructed and maintained, a task sequence library is stored and managed, and semantic matching and automatic task execution are carried out. According to the method, the problems of high manual intervention requirement and low task processing efficiency in traditional Web task automation are solved, and the technical effects of remarkably improving the automation level, enhancing the intelligent decision-making capability, perfecting a knowledge memory mechanism and efficiently executing are achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a large model-driven Web task automation execution method and system. Background Art

[0002] Web task automation is an automation technology that aims to automate the execution of some Web tasks based on software systems. It simulates human actions to achieve actual operations on Web pages or applications, which can greatly improve work efficiency and effectively reduce human labor in repetitive or highly regular tasks.

[0003] A big model refers to a type of neural network model that has been pre-trained with large amounts of data and has a huge number of parameters. Because it has a much larger number of parameters than traditional artificial intelligence models, usually calculated in billions, it is also collectively referred to as a big model. Through pre-training with large amounts of data, the big model has internalized a large amount of human world knowledge and accumulated a large amount of human semantics, becoming the most human-like model to date. Based on the big model technology, people further construct intelligent agents to enable them to interact with the environment, thus becoming real human "agents" that can complete real-world tasks on behalf of humans.

[0004] Traditional Web task automation systems are mainly implemented in the form of engineering and artificial intelligence methods. Among them, the engineering implementation method mainly includes demonstration programming technology and execution flow design technology. Demonstration programming technology is based on the idea of ​​recording and replaying. It generates task scripts by manually demonstrating the Web task process and recording it. When needed, you only need to replay the script to execute it. The execution flow technology relies on users to build the execution flow diagram of Web tasks in a low-code form of dragging and dropping, and clarify each step of action such as clicking, inputting, etc., so as to build an executable execution flow and execute it. It can be seen that the engineering method still requires a lot of manpower overhead and is not generalizable.

[0005] The methods based on traditional artificial intelligence mainly use artificial intelligence models to understand and execute sequences. For example, reinforcement learning-based technology uses a large amount of expert demonstration data and reward design training to enable the model to learn relevant knowledge of the Web task execution process, thereby applying the model to task execution in actual scenarios. However, the difficulty of reinforcement learning reward design and its weak generalization characteristics have led to a catastrophic decline in its effectiveness in the face of complex and diverse Web site systems. The technology based on action learning attempts to capture and learn semantic information in action execution through traditional semantic models, thereby mastering the semantic knowledge of task execution. However, it still requires a lot of manual costs for the generation and annotation of interaction traces. It can be seen that traditional artificial intelligence methods also require a lot of data annotation costs, and the level of intelligence still needs to be improved.

[0006] With the development of big model technology, in order to further solve the manpower overhead in Web task automation and improve the intelligence and generalization of the system, people began to build Web agents based on big models. Based on the decision-making ability of big models, by adding perception, action, memory and other capabilities to big models, they can directly simulate the entire process of perception, planning, decision-making and action when people perform Web tasks, thereby realizing true full-process automation, further improving the degree of automation of Web tasks, and receiving widespread attention.

[0007] However, the existing related technologies have the following limitations:

[0008] First, there is a lack of effective perception and decision execution methods for web pages. Compared with other applications, web pages have a larger number of elements and information. The HTML of a web page has a large number of elements and redundant information representation, which makes it difficult for web agents to understand web content. In addition, since the output of large models is a natural language description, how to parse natural language decisions to correctly match target elements and execute correct actions is a problem that needs to be faced.

[0009] Secondly, there is a lack of effective methods for representing and introducing Web task knowledge. The correct decision of the intelligent agent depends on the introduction of sufficient domain knowledge. For example, a large medical model requires sufficient medical knowledge. The domain knowledge of Web tasks is the operational knowledge of Web sites. However, there is currently a lack of effective knowledge representation and introduction methods, making it difficult to ensure the correctness of Web intelligent agents in task decision-making.

[0010] In addition, there is a lack of effective task memory management mechanism. Existing Web agents are affected by the time-consuming understanding and generation of large models, which makes the execution time of single-step Web operations longer. In the absence of effective task memory management and calling mechanism, even repeated tasks still need to be restarted, making the actual use time-consuming unbearable.

[0011] Therefore, it is very important to design a large model-driven Web task automation method and system that can effectively perceive the content of Web pages, improve the correctness of Web agent planning through effective knowledge introduction methods, and further greatly reduce the system time consumption through the management and call of task memory, so as to further improve the automation, intelligence level and efficiency of Web task automation. Summary of the invention

[0012] In order to solve the above problems, the present invention provides a large model-driven Web task automation execution method and system, which eliminates the manual operation in the process of Web task automation execution and improves the overall intelligence level and execution efficiency.

[0013] The technical solution adopted by the present invention to solve the above problems is as follows:

[0014] A large model-driven Web task automation execution method comprises the following steps:

[0015] 1) According to the Web task description proposed by the user, semantic matching is performed in the task sequence library, the most relevant task sequence is selected, and it is judged whether it meets the task requirements;

[0016] 2) If the matched task sequence meets the requirements, the task sequence is executed; otherwise, the Web task description is semantically matched with the state description in the constructed state-action graph, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements;

[0017] 3) If the new task sequence meets the requirements, execute the task sequence; otherwise, obtain the status information of the current Web page, build an accessibility tree, and obtain a screenshot of the current Web page; then, align the visual information and text information of the screenshot of the current Web page and the accessibility tree to generate the description information of the current Web page;

[0018] 4) Input the Web task description and the description information of the current Web page into the large model for perception and generate single-step decision information; parse the single-step decision information to obtain single-step Web actions, execute these actions step by step, and store the user's tasks after the task is completed.

[0019] Furthermore, the step of performing semantic matching in step 1) includes:

[0020] Use semantic vectorization model to convert the Web task description proposed by the user into vector representation;

[0021] Use the semantic vectorization model to vectorize all task sequences in the task sequence library to obtain vectorized representation;

[0022] Using the semantic similarity calculation method, the vectorized representation of the Web task description is compared with the vectorized representation of the task sequence in the task sequence library, and the task sequence that is most similar to the user's task description is selected from the task sequence library.

[0023] Furthermore, in step 1), the task sequence library stores and maintains the task sequence, and the steps include:

[0024] Generate a full-process description of the task by splicing the semantic descriptions of the task sequence's states and actions;

[0025] Input the full process description of the task into the multimodal large model to generate an overall description of the task;

[0026] Using the overall description of the task as a semantic index, combined with the full process description of the task and its state-action sequence, the task sequence is vectorized.

[0027] The generated task sequence information, including the overall description, the whole process description and the state-action sequence, is stored in the task sequence library;

[0028] Manage and update the entries in the task sequence library, including deleting task sequences that have not been used for a long time based on storage time and frequency of use.

[0029] Furthermore, the steps of constructing the state-action graph in step 2) include:

[0030] Based on the different states of the Web site and the corresponding action execution relationship, an original state-action graph is constructed. The state nodes in the graph represent the different states of the Web page, and the one-way edges represent the actions executed between the states.

[0031] Based on the original web page information stored in the state nodes, the similarity between the state nodes is calculated and similar state nodes are merged; at the same time, based on the similarity between the action API and the target element, similar actions in the graph are merged;

[0032] The merged state-action graph is represented by state semantics and action semantics to obtain an abstract state-action graph.

[0033] Furthermore, the step of semantically matching the Web task description with the state description in the constructed state-action diagram in step 2) includes:

[0034] Use the semantic vectorization model to convert the Web task description into a vector representation, and also vectorize all state nodes in the abstract state-action graph;

[0035] Based on the above two vectorized representations, the semantic similarity between the Web task description and the state node is calculated, and the top k state nodes with the highest semantic similarity are selected as candidate intermediate nodes;

[0036] Retrieve all possible paths from the initial state or current state to the candidate intermediate nodes from the state-action graph, combine the semantic descriptions of the states and actions of these paths, and generate description information of all paths;

[0037] Determine whether there is a state node that meets the description of the Web task among the candidate intermediate nodes. If there is an intermediate state node that meets the conditions, select an optimal path according to the path description information, and use the original state-action graph to automatically execute the operation process corresponding to the path;

[0038] If there is no suitable intermediate state, the large model will gradually explore from the initial state or current state to combine a feasible path until the expected goal is reached.

[0039] Furthermore, the step of constructing the accessibility tree in step 3) includes:

[0040] Get a list of all accessible elements on the current web page and iterate through the elements in the list;

[0041] Clean the elements according to their attribute information and remove irrelevant or invalid element information;

[0042] After the cleaning is completed, the accessibility tree is constructed using the depth-first traversal method based on the hierarchical relationship between elements. Each element node in the tree contains name, role, attribute and unique identifier information.

[0043] Furthermore, the step of aligning the visual information and text information of the current web page screenshot and the accessibility tree in step 3) includes:

[0044] For the constructed accessibility tree, filter out the inoperable elements according to the role and attribute information of the elements, and generate the operable element tree of the Web page;

[0045] Based on the coordinates and unique number of each element in the operable element tree, the operable element area at the corresponding position is marked on the web page screenshot, and a number mark is added to each area.

[0046] Furthermore, the single-step decision information generated by the large model in step 4) includes:

[0047] Observation field: requires the large model to describe the main content of the current web page to output the main information and structure of the current page;

[0048] Thinking field: requires the big model to analyze the execution purpose and logic of the current step, and output the analysis and reasoning of the current task steps;

[0049] Target element: requires the large model to select a target element from the accessibility tree and output the unique number of the target element;

[0050] Action API: requires the big model to select an action API from the Web action API library to execute the current step;

[0051] Action API parameters: require the big model to output the parameter values ​​required by the selected action API so that the action API can be executed correctly;

[0052] Task completion judgment: The large model is required to judge the execution status of the current task and confirm whether the task has been completed;

[0053] Planning of subsequent steps: The large model is required to plan the subsequent steps and generate a sequence of operations or a task plan to be performed next.

[0054] A large model-driven Web task automation execution system, comprising:

[0055] The multimodal Web agent module is used to understand the user's task requirements and Web page content, generate the single-step decision information required to perform the task through multimodal processing, and execute Web actions;

[0056] The state-action graph module is used to build and maintain the state-action graph of Web tasks, supporting the generation of task sequences based on states and actions and the learning of agent knowledge;

[0057] The task sequence library module is used to store and manage semantic task sequences, and supports the abstraction, semantic matching, and execution of task sequences.

[0058] The scheduling module is used to select the corresponding module according to the scheduling strategy;

[0059] The executor module is used to execute the generated web actions;

[0060] Among them, the task sequence library module performs semantic matching in the task sequence library according to the Web task description proposed by the user, selects the most relevant task sequence, and determines whether it meets the task requirements;

[0061] If the matched task sequence meets the requirements, the task sequence is executed through the executor module; otherwise, the Web task description is semantically matched with the state description in the constructed state-action graph through the state-action graph module, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements;

[0062] If the new task sequence meets the requirements, the task sequence is executed through the executor module; otherwise, the state information of the current web page is obtained through the multimodal web agent module, the accessibility tree is constructed, and the screenshot of the current web page is obtained; then, the description information of the current web page is generated by aligning the visual information and text information of the screenshot of the current web page and the accessibility tree;

[0063] The Web task description and the description information of the current Web page are input into the large model for perception, and single-step decision information is generated; based on the single-step decision information, the single-step Web action is parsed through the executor module, and these actions are executed step by step. After the task is completed, the user's task is stored through the task sequence library module.

[0064] Furthermore, the steps performed by the multimodal Web agent module include:

[0065] Obtain the list of accessible elements of the current page based on the browser CDP interface, clean the attribute information of the elements in the list, and remove irrelevant or invalid elements; then, build a simplified accessibility tree based on the hierarchical relationship between the elements, and uniquely number each element for identification in the tree;

[0066] Analyze the role and attribute information of elements in the accessibility tree, match them using rules, build the operable element tree of the web page, filter out inoperable elements, and ensure that only interactive elements are retained;

[0067] Get a screenshot of the current web page as the visual modal representation of the page; based on the element coordinates and numbers in the operable element tree, mark the corresponding operable element areas on the screenshot and add a number mark to each area;

[0068] The task execution information and the current web page description are input into the multimodal large model. The large model generates single-step decision information by understanding the page content and task requirements.

[0069] Parse the single-step decisions generated by the large model, extract the target elements and target actions, and combine this information into executable API calls to perform related operations;

[0070] Determine whether the current task has been completed. If the task is completed, end the execution; otherwise, repeat the above steps until the task is completed.

[0071] Furthermore, the steps performed by the state-action diagram module include:

[0072] Based on automation technology, the original state-action graph of the website is quickly explored and recorded. The state nodes in the graph represent different states of the web page and store related web page information. The one-way edges represent the action execution relationship between states and record the original information of the action.

[0073] Merge similar states in the original state-action graph by calculating the multi-dimensional weighted similarity between state nodes and merging similar state nodes. At the same time, based on the similarity between the API and the target element, merge similar actions in the graph and integrate similar actions.

[0074] The large model is used to perceive page information and generate a semantic description of the page status and a function list. At the same time, by analyzing the original execution information, a semantic representation of the action is generated to form an abstract state-action diagram.

[0075] Furthermore, the steps performed by the state-action diagram module include:

[0076] Use the semantic vectorization model to convert the user's natural language task description into a vectorized representation for subsequent semantic similarity calculation;

[0077] The merged and semantically abstracted state-action graph is processed to generate a vectorized representation of the state nodes. The semantic similarity between the task description vector and the state node vector is calculated, and the top k state nodes with the highest similarity are selected as candidate intermediate nodes. All paths from the initial state or current state to these candidate nodes are retrieved to generate path description information.

[0078] Based on the generated path description information, the large model or the user determines the intermediate state nodes that meet the task requirements; if there are intermediate state nodes that meet the conditions, a suitable path is selected for execution; if there are no matching intermediate state nodes, the nodes are gradually reached from the original state or the current state;

[0079] Starting from the current state node, the big model combines the user's task description and the semantic description of the current state to plan the task phase and select the action that matches the task from the subsequent actions as the next step; according to the decision of the big model, the original state-action graph is used to execute the action and reach the specified next state node;

[0080] Determine whether the task is completed. If the large model confirms that the final state of the task has been reached or determines that the subsequent actions of the current state do not meet the task requirements, the task is ended; otherwise, repeat the previous step and proceed step by step until the task completion requirements are met.

[0081] Furthermore, the steps performed by the task sequence library module include:

[0082] For the executed task sequence, the semantic descriptions of states and actions are arranged in sequence and the whole process description of the task is generated using connectives;

[0083] Input the full process description of the task into the multimodal large model to generate an overall description of the task sequence;

[0084] Use the overall description as a semantic index to build a task sequence storage in the task sequence library; input each task sequence into the vectorization model to generate and store its vectorized representation;

[0085] The Web task description proposed by the user is converted into a semantic vector, and the similarity is calculated with the vectorized representation in the task sequence library. The top k most similar task sequences are selected as candidate sequences. The candidate sequences are judged by the large model or the user, and the sequences that meet the requirements are selected for execution.

[0086] Furthermore, the steps performed by the scheduling module include:

[0087] According to the Web task description proposed by the user, the scheduling module calls the task sequence library module to match the task sequence; if there is a matching task sequence in the task sequence library, after confirmation by the user or the big model, the task sequence is directly executed and ends after the task is completed;

[0088] If no matching task sequence is found in the task sequence library, the scheduling module calls the state-action graph module for exploration; the state-action graph is explored by the large model, and if a complete task sequence is found, the sequence is executed and added to the task sequence library;

[0089] If no matching task sequence is found in the state-action graph, the scheduling module will organize the explored information and call the multimodal Web agent module to make a single-step decision; the agent of the multimodal Web agent module will step by step execute these actions until the task is completed, and update the new state and action to the state-action graph; after the task is completed, the new task sequence will be added to the task sequence library.

[0090] Compared with the prior art, the present invention has achieved the following technical effects:

[0091] 1. The present invention significantly improves the automation level of Web tasks through Web agents driven by large models. Compared with traditional Web task automation technology, the present invention reduces the need for manual intervention in task execution and achieves a higher degree of automation.

[0092] 2. The present invention provides Web agents with deep perception capabilities of web page content and intelligent decision-making analysis through a large model, enabling Web agents to better understand and handle complex task requirements. Compared with traditional methods, the present invention has stronger intelligence and better generalization capabilities.

[0093] 3. The present invention establishes a comprehensive knowledge memory mechanism through the state-action diagram module and the task sequence library module, which effectively reduces the reasoning cost of processing repetitive tasks and improves the execution efficiency of the system by quickly matching and reusing existing task sequences.

[0094] 4. The present invention uses the processed accessibility tree and screenshots to jointly represent the content of the Web page, and uniquely numbers the elements of the accessibility tree. By filtering the operable elements and marking these elements on the screenshots, effective alignment of the visual modality and the textual modality is achieved.

[0095] 5. The present invention utilizes the encapsulated Web action API and the numbering during page representation to convert the Web element decision of the large model into a numbering selection that can be directly parsed by the program, determines the unique target element, and generates an executable Web action through the API selection output by the large model.

[0096] 6. The present invention provides a complete system, including an intelligent agent module, a graph knowledge base module and a task sequence library module, and its scheduling strategy, which can achieve efficient execution and intelligent decision-making, quickly respond to user needs, reduce task execution time, and improve overall usage effect.

[0097] 7. This paper demonstrates the practical application of big model technology in Web task scenarios, proves how big models can effectively solve practical problems, and lays the foundation for its further application in the field of Web task automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 The figure is an overall architecture diagram of a large model-driven Web task automation execution system according to an embodiment of the present invention.

[0099] Figure 2 A flowchart of the multimodal Web agent module executing tasks according to an embodiment of the present invention.

[0100] Figure 3 A flow chart of constructing a state-action diagram for a state-action diagram module of an embodiment of the present invention.

[0101] Figure 4 A flowchart of performing abstract graph exploration on a state-action graph module according to an embodiment of the present invention.

[0102] Figure 5 The present invention is a flowchart of a task sequence library module that performs task sequence abstraction and matching.

[0103] Figure 6 This is a flow chart of a scheduling module executing a scheduling strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0104] In order to make the various technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.

[0105] The embodiment of the present invention discloses a large model driven Web task automation execution method, the main processing steps of which include:

[0106] 1) For the Web task description proposed by the user, the vectorization model is used to generate the Web task description vector, and the semantic similarity is calculated with the task sequences in the task sequence library, and the most relevant task sequences are selected from them. The user or the big model determines whether they meet the task requirements;

[0107] 2) If the matching task sequence meets the task requirements, the task sequence is directly executed and the task is marked as completed;

[0108] 3) If the matched task sequence does not meet the task requirements, the semantic similarity between the Web task description vector and the state description in the state-action graph is calculated, and the most relevant state node is selected for quick arrival;

[0109] 4) After quickly reaching the target node, the large model is used to gradually explore the state-action diagram to generate a task execution process; if the process meets the task requirements, the task sequence is executed and added to the task sequence library, marking the task as completed;

[0110] 5) If the explored task execution process still does not meet the task requirements, the status information of the current Web page is obtained, and an accessibility tree is constructed based on the hierarchical relationship of the accessibility elements of the Web page;

[0111] 6) Get a screenshot of the current web page and mark the location and number of the elements on the screenshot according to the accessibility tree to align the visual information with the text information;

[0112] 7) Input the current task information and web page information into the large model for perception and decision-making, and generate a single-step web action;

[0113] 8) Parse the decisions generated by the large model and execute the corresponding single-step Web actions;

[0114] 9) Update the new page status to the status-action diagram;

[0115] 10) Repeat steps 6) to 9) until the task is completed;

[0116] 11) After the task is completed, the task sequence corresponding to the task is added to the task sequence library.

[0117] To implement the above method, the embodiment of the present invention also discloses a large model driven Web task automation execution system, such as Figure 1 As shown, the system divides the architecture into two main parts according to the capabilities of the Web agent: the decision-making layer and the execution layer.

[0118] The decision-making layer is mainly responsible for understanding user tasks and planning and making decisions on task execution. The decision-making layer includes the following modules:

[0119] Multimodal Web Agent Module: It is used to generate the actions required to complete each step of the task based on the user's task requirements and the understanding of the Web page content.

[0120] State-action graph module: It is used to implement the graph representation of Web task knowledge and realize the learning process of intelligent agent knowledge by exploring on the graph.

[0121] Task sequence library module: provides semantic task sequence abstraction to achieve fast task sequence matching and execution.

[0122] Scheduling module: Make decisions based on scheduling strategies and select appropriate decision paths.

[0123] The execution layer includes the following modules:

[0124] Executor module: parses and actually executes the decision actions generated by the decision layer, such as clicking and inputting operations on the target elements of the Web page, and feeds back the execution results to the decision layer.

[0125] 1. Multimodal Web Agent Module for Task Execution

[0126] The multimodal Web agent module provides a Web agent built on a multimodal large model. The agent can perform multiple rounds of perception and decision-making based on the Web task requirements proposed by the user, thereby completing the entire Web task. The agent is built based on a general multimodal large model, preferably using the GPT-4-turbo model, which optimizes its task execution capability through contextual learning. The task execution process of the multimodal Web agent module is as follows: Figure 2 As shown, the specific execution process includes the following steps:

[0127] S1: Page accessibility tree construction

[0128] First, the browser CDP interface is used to obtain the list of accessible elements of the current page and traverse these elements. Then, the elements are cleaned according to their attributes, including whether the element is ignorable, whether it has a real size (i.e., the length and width of the element are not zero), whether it is visible on the page, and whether it is valid. If the element is ignorable, has no real size, is invisible, or is a meaningless empty element, it is cleared. Next, based on the hierarchical relationship between elements and the depth-first traversal method, a simplified accessibility tree is constructed. Each element on the tree includes: name, role, attribute, and unique number. The name describes the semantic information of the element, such as "search" and "refresh"; the role describes the category of the element, such as "button" and "link"; the attribute describes the configuration attribute of the element, such as "focusable"; the number is unique, and each element is uniquely numbered.

[0129] S2: Filtering of actionable elements on the page

[0130] After building a simplified accessibility tree, we further analyze and match rules based on the role and attribute information of the elements, filter out the inoperable elements, and thus build an operable element tree for the web page. Inoperable elements refer to those elements that cannot respond to user actions, such as static text. Operable elements are divided into clickable elements and editable elements. Clickable elements refer to elements that can respond by clicking with the mouse, such as buttons and links; editable elements refer to elements that can be focused by clicking with the mouse and can be input through input media such as the keyboard, such as search boxes.

[0131] S3: Get screenshots and mark them on the screen

[0132] After constructing the operable element tree, a screenshot of the current web page is obtained as a representation of the visual modality. In order to align the text information of the elements on the tree with the visual information of the corresponding elements in the screenshot, the operable elements at the corresponding positions are marked with boxes and numbers on the screenshot according to the coordinates and numbers of each element on the operable element tree. In this embodiment, in order to maintain the readability of the visual image, only the operable elements are boxed and marked, thereby achieving alignment of the visual information with the text information.

[0133] S4: Multimodal large model single-step perception decision

[0134] First, through the instruction prompt, the task execution information and the current web page description are input into the multimodal large model. The task execution information includes: user task requirements, completed steps, previous plans, etc.; the current web page description information includes: the simplified accessibility tree text of the current page and the annotated page screenshot. Then, the multimodal large model understands and perceives the current task status and web page information, and generates single-step decision information. Single-step decision information includes:

[0135] a) Observation field: requires the large model to describe the observation of the current page, for example: "This is the homepage of Baidu search, and this is the result page of searching XX."

[0136] b) Thought field: requires the big model to analyze the current step, for example: "On the Baidu search homepage, in order to complete the task of searching for XX, you need to fill in XX content in the input box."

[0137] c) Target element: The large model is required to select the target element from the provided simplified accessibility tree and output its unique number.

[0138] d) Action API: The big model is required to select an API name from the provided Web action API library, such as click, input, etc. The Web action API library encapsulates common Web operations at the API level and provides the API name, function description, parameters, and parameter description in the rule instructions to help the big model understand and call the API when appropriate. The big model outputs the decision content in JSON format for subsequent decision parsing.

[0139] e) Action API parameters: The large model is required to output the parameter content required by the selected API. For example, click parameters include left / right key, single / double click, etc.

[0140] f) Task completion judgment: The large model is required to judge whether the current stage of the task has completed the user's task goal. If it has been completed, the completion status is output; otherwise, the execution continues.

[0141] g) Planning of subsequent steps: The big model is required to make preliminary plans for the subsequent steps, which will be used as planning information input by the big model in the next step for reference.

[0142] S5: Decision parsing and execution

[0143] Since the single-step decision information output by the big model is in natural language, it is necessary to parse these outputs to implement the actual action execution. By parsing the JSON format decision output by the big model, the various parts of the decision are analyzed. The target element is confirmed by the unique number of the target element, the action API instruction is synthesized by the API name and parameters, and the actual action is executed on the target element.

[0144] S6: Task completion judgment

[0145] Based on the judgment of the task stage by the large model, if the task has been completed, the execution is terminated; otherwise, steps S1 to S5 are repeated until the task is completed. After the task is completed, the corresponding task sequence is added to the task sequence library for subsequent use.

[0146] 2. State-action diagram module to build state-action diagram

[0147] In order to improve the planning ability of Web agents, it is necessary to introduce Web task knowledge to the agents. To this end, the state-action graph module provides the knowledge base representation and retrieval method required for Web agents to understand, plan and make decisions on Web tasks. It also supports the memory function of the executed task process, making the state-action graph both a knowledge base for the agent and a memory for task execution. The state-action graph construction process of the state-action graph module is as follows: Figure 3 As shown in the figure, the specific construction process includes:

[0148] S1: Original state-action graph construction.

[0149] First, the original state-action graph is quickly recorded and generated through automation technology. The automation technology includes manually demonstrated task flow recording, Web intelligent agent task execution recording and automated detection technology to improve the efficiency of graph construction. In the example of the present invention, automated detection technology, such as random detection method or reinforcement learning-based detection method, is mainly used to quickly explore the state and action execution relationship of the Web site and construct the original state-action graph. At the same time, combined with the manually demonstrated task flow recording technology and the task execution record of the Web intelligent agent, the information of the original state-action graph is further improved. In the constructed state-action graph, the node is called the state node, which represents the state of the web page and stores the original web page information such as HTML and accessibility tree; the edge is a unidirectional edge, which represents the executed action, connects the state before and after the action is executed, and stores the original action information, such as API calls and element locators. The locator is used to uniquely determine the target element. Commonly used locators include coordinates, XPath, CSS selectors, etc., which help the automated script find the target element when executing.

[0150] S2: Graph state and action merging

[0151] Since the original state-action graph is based on the original execution of direct recording, it may contain a large amount of similar state and action information, which can easily lead to the explosion of the state space and increase the storage and retrieval costs. Therefore, it is necessary to merge the states and actions of the original state-action graph. Specifically, similar states are first merged. The original state-action graph may contain a large number of similar states, such as search engine homepages with different contents, shopping cart pages with different numbers of goods, etc. It is necessary to calculate the text and image similarity based on the HTML node information, page URL, screenshot information, etc. stored in the state node, and calculate the similarity between the state nodes by weighted summation. According to the set threshold, the states with similarity higher than the threshold are merged. Furthermore, in the process from state A to state B, if there are multiple actions, the similarity comparison is performed based on the action API and the target element, and the highly similar actions are merged and represented. By merging similar states and similar actions, the representation space of the state-action graph can be significantly reduced.

[0152] S3: Semantic abstraction of graph states and actions

[0153] Since the state-action graph provides knowledge representation for Web agents to perform Web tasks, and the original state HTML information and action API and locator information are not conducive to the understanding of Web agents, it is necessary to perform semantic abstraction on the merged state-action graph, including state semantic representation and action semantic representation.

[0154] Specifically, for the semantic representation of the state, the big model's perception of the page information is used to generate a semantic description of the page state. By inputting the screenshots of the state storage and the accessibility tree information into the multimodal big model, a semantic description of the page and a list of executable functions of the page are generated. For example, for a search engine homepage, the generated semantic description is "the homepage of XX search engine"; the list of executable functions of the page may include "content search, hot news list, pictures, news, weather navigation", etc. These descriptions construct a representation of the web page state that is easy for the big model to understand.

[0155] For the semantic representation of actions, the original execution information is analyzed to generate a semantic description of the action. For example, for the original execution information click(element_id,left,single), its API function and parameter description are "click, left key, single click". The target element is matched by the element locator to obtain its description information (for example: <button>'Search'"), a semantic description can be further generated, such as "left-click the search button". By semantically describing states and actions, the original state-action graph is converted into an abstract semantic description graph, i.e., an abstract state-action graph, so that the big model can perform semantic understanding on the graph, thereby realizing the Web task knowledge learning and exploration process of the big model on the semantic graph. The above-mentioned multimodal big model is obtained by context learning from a general multimodal big model, preferably a GPT-4-turbo model.

[0156] 3. State-Action Graph Module for Abstract Graph Exploration

[0157] In addition to the construction, merging, and semantic abstraction of state-action graphs, the state-action graph module also includes an exploration method for abstract state-action graphs. The process of the abstract graph exploration method of the state-action graph module is as follows: Figure 4 As shown, it includes two stages: fast matching and arrival (steps S1 to S3) and gradual arrival (steps S4 to S5).

[0158] S1: Vectorized representation of user tasks

[0159] First, for the natural language description task proposed by the user, the natural language description is converted into a vectorized representation of the task through a semantic vectorization model for subsequent semantic similarity calculation. The semantic vectorization model is obtained through large-scale pre-training, and the preferred model is the text-embedding-3-small model.

[0160] S2: Task description and state semantic similarity matching

[0161] For the merged and semantically abstracted state-action graph, for all state nodes, the semantic state description is used to generate a vectorized representation of the state through the semantic vectorization model. Furthermore, the cosine similarity between the task description vector and the vector of each state node is calculated to represent the semantic similarity between the two. Through the semantic similarity calculation, the first k state nodes with the highest semantic similarity to the task description are selected as candidate intermediate nodes. Then, all paths from the initial state or current state to these candidate intermediate nodes are retrieved from the graph, and the state and action semantic descriptions in these paths are combined to generate path description information, such as: "From the search engine homepage, left-click the search button to reach the XX page, left-click the XX button to reach the XX page..."

[0162] S3: Large model or user selection matching state reaches

[0163] According to all the matched path description information, the large model or the user determines whether there is an intermediate state node that meets the task requirements in the candidate intermediate nodes. For example, for the task "Query Beijing Weather", if the matched intermediate state node is "Weather Network Home Page", after the large model or the user confirms that the intermediate state node is correct, a suitable path is selected according to the path description information. The system will use the original state-action graph to automatically execute the process (i.e., the task sequence) corresponding to the path, so as to quickly reach the confirmed intermediate state node and complete the fast matching and arrival stage. This stage can effectively accelerate the task matching and execution process. If the intermediate state node that meets the requirements cannot be matched, it directly enters the step-by-step arrival stage (i.e., step-by-step execution of each path). The large model is preferably a general multimodal large model, which is obtained through context learning. As a preferred option, the GPT-4-turbo model can be used.

[0164] S4: Large model task state understanding and state next action selection

[0165] Starting from the current state node, the big model combines the user's task description and the semantic description of the current state node to form an understanding of the current task stage and plan the subsequent steps. After the planning is completed, the big model makes a decision from the subsequent actions available to the current state node and selects the action that matches the task execution as the action to be executed in the next step.

[0166] S5: Decision execution reaches the next state

[0167] According to the decision of the large model in the previous step, the corresponding action is executed using the original state-action graph to reach the next state node pointed to by the action.

[0168] S6: Task completion judgment

[0169] If the large model determines that the current state node has reached the final state of the task, that is, the task has been completed, or believes that the subsequent actions and the pointed state of the state do not meet the requirements of the next step of the task, then the end is reached; otherwise, steps S4 to S5 are repeated, and gradually arrive through multiple steps until the end requirements are met. During the exploration process in the graph, the large model will continue to record its exploration process and corresponding sequence, and execute them synchronously. Therefore, when the exploration of the complete task sequence is completed, the execution of related state actions will also be completed synchronously, thereby improving the execution efficiency of the system. If the complete sequence cannot be explored, the execution sequence recorded in the early stage will provide knowledge reference for the subsequent decision-making of the Web agent.

[0170] 4. Task sequence library module performs task sequence abstraction and matching

[0171] In order to improve the task execution efficiency of Web agents and realize the reuse of task memory, the task sequence library module further abstracts the executed tasks and summarizes them into semantic forms so that they can be quickly matched and executed when needed. Figure 5 As shown, the specific steps are as follows:

[0172] S1: Generate semantic description of the entire task sequence

[0173] For a completed task sequence, first use the semantic descriptions of states and actions to form a semantic description of the entire task process. By arranging the semantic descriptions of states and actions in order and connecting them with conjunctions, a full description of the task process can be generated, for example: "From XX page, click XX element, arrive at XX page, enter XX, arrive at XX page, ..., arrive at XX page, task completed."

[0174] S2: Generate overall description of task sequence

[0175] The generated task process description is input into the multimodal model, which will understand the task sequence and output a summary of the task sequence goals, thereby forming an overall description of the task. For example: "Use the XX search engine to query and visit the homepage of the XX website." The multimodal model is obtained by context learning from the general model, and the preferred general model is the GPT-4-turbo model.

[0176] S3: Semantic, vectorized representation and storage of task sequences

[0177] The generated overall task description is used as the semantic index of the task to construct the task entries in the task sequence library, including: the overall description index of the task, the full process description of the task, the execution sequence corresponding to the task (the state and action sequence on the state-action diagram), etc. In order to achieve fast matching of task sequences, each task sequence in the task sequence library is input into the semantic vectorization model, and the natural language description is converted into its vectorized representation. In this way, fast matching can be achieved in the matching stage by calculating the similarity of semantic vectors. The semantic vectorization model is obtained by large-scale pre-training, and the preferred semantic vectorization model is the text-embedding-3-small model.

[0178] S4: Task sequence matching

[0179] When the user proposes a task requirement, the semantic vectorization model is used to convert the user's natural language requirements into semantic vectors. By calculating the cosine similarity with each vectorized representation in the task sequence library, the top k most similar task sequences are selected as candidate sequences. The overall description, full process description and corresponding state-action sequence of these candidate sequences are extracted, and the large model or user makes a judgment and selects the sequence that meets the requirements. In this way, the system can use the selected state-action sequence to achieve fast execution, thereby significantly improving the system's operating efficiency.

[0180] S5: Task sequence library management

[0181] Taking into account the storage and retrieval costs of the task sequence library, in an embodiment of the present invention, by performing a weighted evaluation on factors such as the storage time of the task sequence and the last use time, some task sequences that have not been used for a long time are deleted to optimize storage resources and retrieval efficiency.

[0182] 5. The scheduling module executes the scheduling strategy

[0183] The scheduling module integrates the multimodal Web agent module, the state-action graph module, and the task sequence library module through scheduling strategies to maximize the execution efficiency of the system. The scheduling strategy can be implemented based on the program or using a large model, where the preferred large model is the GPT-4-turbo model. The scheduling strategy flow of the scheduling module is as follows: Figure 6 As shown, the specific steps include:

[0184] S1: Task sequence library matching

[0185] For the task requirements proposed by the user, the scheduling module first calls the task sequence library to match the task sequence. If there is a matching task sequence in the task sequence library, after confirmation by the user or the big model, the system can directly execute the task sequence, and the task is considered completed after completion.

[0186] S2: State-Action Graph Exploration

[0187] When the task sequence library cannot match the relevant sequence, the scheduling module calls the state-action graph module, and the large model drives the exploration on the state-action graph. If a complete task sequence is explored from the graph, the system will execute the task sequence and add it to the task sequence library for subsequent quick matching.

[0188] S3: Single-step decision making for multimodal web agents

[0189] If the complete task sequence cannot be found in the state-action graph, the scheduling module will sort out the explored information and call the multimodal Web agent module to make single-step decisions until the task is completed. During the task execution, the new state and action are updated to the state-action graph. After the task is completed, the task sequence is added to the task sequence library to enhance the coverage of the task sequence library.

[0190] By recording and updating new states and actions, the state-action graph and task sequence library will be gradually improved. This improvement can enable repeated tasks to be quickly matched and executed, significantly reducing the call overhead of the large model, thereby improving the execution efficiency of the system. At the same time, as the state-action graph is improved, the large model can explore more task sequences in the graph and reduce dependence on Web agent decision-making. Improvements to the state-action graph can also help reduce decision-making errors of Web agents, such as hallucinations. Even if a multimodal Web agent's decision is required, as the state-action graph is gradually improved, the agent will obtain more comprehensive task knowledge, thereby improving the accuracy of the decision.

[0191] 6. Actuator module

[0192] The executor module is located in the execution layer and is responsible for executing the operation instructions generated from the decision layer in the real browser environment. This module analyzes the decisions of the decision layer and performs corresponding operations to achieve the task objectives. The implementation of the browser environment and operation functions preferably uses the Playwright automation tool, which provides efficient browser automation capabilities.

[0193] Specifically, the functions of the actuator module include:

[0194] S1: Operation analysis: Analyze the operation instructions output by the decision layer, including click, input, navigation and other operations. The instructions contain the location information and operation parameters of the target element.

[0195] S2: Operation execution: Execute the parsed operation instructions in the real browser environment. For example, click on elements, input text, navigate pages, etc. through Playwright's API.

[0196] S3: Result feedback: Feedback the results of the operation to the decision-making layer, including information on the success or failure of the operation and changes in the page status. This feedback information is used for further decision-making and adjustments by the decision-making layer.

[0197] The implementation of the actuator module ensures efficient conversion from the decision-making level to actual operation, and guarantees the operating efficiency of the system and the accurate completion of tasks.

[0198] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.< / button>

Claims

1. A large model driven Web task automation execution method, characterized in that: The following steps are involved: 1) According to the Web task description proposed by the user, semantic matching is performed in the task sequence library, the most relevant task sequence is selected, and it is judged whether it meets the task requirements; 2) If the matched task sequence meets the requirements, the task sequence is executed; otherwise, the Web task description is semantically matched with the state description in the constructed state-action graph, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements; 3) If the new task sequence meets the requirements, execute the task sequence; otherwise, obtain the status information of the current Web page, build an accessibility tree, and obtain a screenshot of the current Web page; then, align the visual information and text information of the screenshot of the current Web page and the accessibility tree to generate the description information of the current Web page; 4) Input the Web task description and the description information of the current Web page into the large model for perception and generate single-step decision information; parse the single-step decision information to obtain single-step Web actions, execute these actions step by step, and store the user's tasks after the task is completed.

2. The method according to claim 1, characterized in that The steps of semantic matching in step 1) include: Use semantic vectorization model to convert the Web task description proposed by the user into vector representation; Use the semantic vectorization model to vectorize all task sequences in the task sequence library to obtain vectorized representation; Using the semantic similarity calculation method, the vectorized representation of the Web task description is compared with the vectorized representation of the task sequence in the task sequence library, and the task sequence that is most similar to the user's task description is selected from the task sequence library.

3. The method according to claim 1, characterized in that In step 1), the task sequence library stores and maintains the task sequence, and the steps include: Generate a full-process description of the task by splicing the semantic descriptions of the task sequence's states and actions; Input the full process description of the task into the multimodal large model to generate an overall description of the task; Using the overall description of the task as a semantic index, combined with the full process description of the task and its state-action sequence, the task sequence is vectorized. The generated task sequence information, including the overall description, the whole process description and the state-action sequence, is stored in the task sequence library; Manage and update the entries in the task sequence library, including deleting task sequences that have not been used for a long time based on storage time and frequency of use.

4. The method according to claim 1, characterized in that The steps of constructing the state-action graph in step 2) include: Based on the different states of the Web site and the corresponding action execution relationship, an original state-action graph is constructed. The state nodes in the graph represent the different states of the Web page, and the one-way edges represent the actions executed between the states. Based on the original web page information stored in the state nodes, the similarity between the state nodes is calculated and similar state nodes are merged; at the same time, based on the similarity between the action API and the target element, similar actions in the graph are merged; The merged state-action graph is represented by state semantics and action semantics to obtain an abstract state-action graph.

5. The method according to claim 1, characterized in that The step of semantically matching the Web task description with the state description in the constructed state-action diagram in step 2) includes: Use the semantic vectorization model to convert the Web task description into a vector representation, and also vectorize all state nodes in the abstract state-action graph; Based on the above two vectorized representations, the semantic similarity between the Web task description and the state node is calculated, and the top k state nodes with the highest semantic similarity are selected as candidate intermediate nodes; Retrieve all possible paths from the initial state or current state to the candidate intermediate nodes from the state-action graph, combine the semantic descriptions of the states and actions of these paths, and generate description information of all paths; Determine whether there is a state node that meets the description of the Web task among the candidate intermediate nodes. If there is an intermediate state node that meets the conditions, select an optimal path according to the path description information, and use the original state-action graph to automatically execute the operation process corresponding to the path; If there is no suitable intermediate state, the large model will gradually explore from the initial state or current state to combine a feasible path until the expected goal is reached.

6. The method according to claim 1, characterized in that The steps of building the accessibility tree in step 3) include: Get a list of all accessible elements on the current web page and iterate through the elements in the list; Clean the elements according to their attribute information and remove irrelevant or invalid element information; After the cleaning is completed, the accessibility tree is constructed using the depth-first traversal method based on the hierarchical relationship between elements. Each element node in the tree contains name, role, attribute and unique identifier information.

7. The method according to claim 1, characterized in that The step of aligning the visual information and text information of the current web page screenshot and the accessibility tree in step 3) includes: For the constructed accessibility tree, filter out the inoperable elements according to the role and attribute information of the elements, and generate the operable element tree of the Web page; Based on the coordinates and unique number of each element in the operable element tree, the operable element area at the corresponding position is marked on the web page screenshot, and a number mark is added to each area.

8. The method according to claim 1, characterized in that The single-step decision information generated by the large model in step 4) includes: Observation field: requires the large model to describe the main content of the current web page to output the main information and structure of the current page; Thinking field: requires the big model to analyze the execution purpose and logic of the current step, and output the analysis and reasoning of the current task steps; Target element: requires the large model to select a target element from the accessibility tree and output the unique number of the target element; Action API: requires the big model to select an action API from the Web action API library to execute the current step; Action API parameters: require the big model to output the parameter values ​​required by the selected action API so that the action API can be executed correctly; Task completion judgment: The large model is required to judge the execution status of the current task and confirm whether the task has been completed; Planning of subsequent steps: The large model is required to plan the subsequent steps and generate a sequence of operations or a task plan to be performed next.

9. A large model-driven Web task automation execution system, characterized in that: include: The multimodal Web agent module is used to understand the user's task requirements and Web page content, generate the single-step decision information required to perform the task through multimodal processing, and execute Web actions; The state-action graph module is used to build and maintain the state-action graph of Web tasks, supporting the generation of task sequences based on states and actions and the learning of agent knowledge; The task sequence library module is used to store and manage semantic task sequences, and supports the abstraction, semantic matching, and execution of task sequences. The scheduling module is used to select the corresponding module according to the scheduling strategy; The executor module is used to execute the generated web actions; Among them, the task sequence library module performs semantic matching in the task sequence library according to the Web task description proposed by the user, selects the most relevant task sequence, and determines whether it meets the task requirements; If the matched task sequence meets the requirements, the task sequence is executed through the executor module; otherwise, the Web task description is semantically matched with the state description in the constructed state-action graph through the state-action graph module, the most relevant state nodes are selected from the graph to form a path, and a new task sequence is generated based on the path description information to determine whether it meets the task requirements; If the new task sequence meets the requirements, the task sequence is executed through the executor module; otherwise, the state information of the current web page is obtained through the multimodal web agent module, the accessibility tree is constructed, and the screenshot of the current web page is obtained; then, the description information of the current web page is generated by aligning the visual information and text information of the screenshot of the current web page and the accessibility tree; The Web task description and the description information of the current Web page are input into the large model for perception, and single-step decision information is generated; based on the single-step decision information, the single-step Web action is parsed through the executor module, and these actions are executed step by step. After the task is completed, the user's task is stored through the task sequence library module.

10. The system according to claim 9, characterized in that The steps performed by the multimodal web agent module include: Obtain the list of accessible elements of the current page based on the browser CDP interface, clean the attribute information of the elements in the list, and remove irrelevant or invalid elements; then, build a simplified accessibility tree based on the hierarchical relationship between the elements, and uniquely number each element for identification in the tree; Analyze the role and attribute information of elements in the accessibility tree, match them using rules, build the operable element tree of the web page, filter out inoperable elements, and ensure that only interactive elements are retained; Get a screenshot of the current web page as the visual modal representation of the page; based on the element coordinates and numbers in the operable element tree, mark the corresponding operable element areas on the screenshot and add a number mark to each area; The task execution information and the current web page description are input into the multimodal large model. The large model generates single-step decision information by understanding the page content and task requirements. Parse the single-step decisions generated by the large model, extract the target elements and target actions, and combine this information into executable API calls to perform related operations; Determine whether the current task has been completed. If the task is completed, end the execution; otherwise, repeat the above steps until the task is completed; The steps performed by the state-action diagram module include: Based on automation technology, the original state-action graph of the website is quickly explored and recorded. The state nodes in the graph represent different states of the web page and store related web page information. The one-way edges represent the action execution relationship between states and record the original information of the action. Merge similar states in the original state-action graph by calculating the multi-dimensional weighted similarity between state nodes and merging similar state nodes. At the same time, based on the similarity between the API and the target element, merge similar actions in the graph and integrate similar actions. The large model is used to perceive page information and generate a semantic description of the page status and a function list. At the same time, the original execution information is analyzed to generate a semantic representation of the action and form an abstract state-action graph. The steps performed by the state-action diagram module include: Use the semantic vectorization model to convert the user's natural language task description into a vectorized representation for subsequent semantic similarity calculation; The merged and semantically abstracted state-action graph is processed to generate a vectorized representation of the state nodes. The semantic similarity between the task description vector and the state node vector is calculated, and the top k state nodes with the highest similarity are selected as candidate intermediate nodes. All paths from the initial state or current state to these candidate nodes are retrieved to generate path description information. Based on the generated path description information, the large model or the user determines the intermediate state nodes that meet the task requirements; if there are intermediate state nodes that meet the conditions, a suitable path is selected for execution; if there are no matching intermediate state nodes, the nodes are gradually reached from the original state or the current state; Starting from the current state node, the big model combines the user's task description and the semantic description of the current state to plan the task phase and select the action that matches the task from the subsequent actions as the next step; according to the decision of the big model, the original state-action graph is used to execute the action and reach the specified next state node; Determine whether the task is completed. If the large model confirms that the final state of the task has been reached or determines that the subsequent actions of the current state do not meet the task requirements, the task is terminated; otherwise, repeat the previous step and proceed step by step until the task completion requirements are met; The steps performed by the Task Sequence Library module include: For the executed task sequence, the semantic descriptions of states and actions are arranged in sequence and the whole process description of the task is generated using connectives; Input the full process description of the task into the multimodal large model to generate an overall description of the task sequence; Use the overall description as a semantic index to build a task sequence storage in the task sequence library; input each task sequence into the vectorization model to generate and store its vectorized representation; The web task description proposed by the user is converted into a semantic vector, and the similarity is calculated with the vectorized representation in the task sequence library. The top k most similar task sequences are selected as candidate sequences. The candidate sequences are judged by the big model or the user, and the sequences that meet the requirements are selected for execution. The steps performed by the scheduling module include: According to the Web task description proposed by the user, the scheduling module calls the task sequence library module to match the task sequence; if there is a matching task sequence in the task sequence library, after confirmation by the user or the big model, the task sequence is directly executed and ends after the task is completed; If no matching task sequence is found in the task sequence library, the scheduling module calls the state-action graph module for exploration; the state-action graph is explored by the large model, and if a complete task sequence is found, the sequence is executed and added to the task sequence library; If no matching task sequence is found in the state-action graph, the scheduling module will organize the explored information and call the multimodal Web agent module to make a single-step decision; the agent of the multimodal Web agent module will step by step execute these actions until the task is completed, and update the new state and action to the state-action graph; after the task is completed, the new task sequence will be added to the task sequence library.

Images