An agent task processing method and apparatus

Abstract

The application discloses an intelligent agent task processing method and device, and relates to the technical field of computers.A specific embodiment of the method comprises: receiving current page image information and intelligent agent task information uploaded by a client, determining state type information corresponding to a current page based on the current page image information; determining a state conversion path according to the state type information corresponding to the current page and the intelligent agent task information; generating intelligent agent execution instructions based on the state conversion path, and issuing the intelligent agent execution instructions to the client to enable the client to execute the intelligent agent execution instructions to process the intelligent agent task.This embodiment enables the intelligent agent to plan a task based on explicit page state information, generates and issues intelligent agent execution instructions according to the state conversion path, realizes state constraint and path control on a task execution process, improves the controllability and accuracy of the intelligent agent in executing a complex multi-step task, and improves the execution efficiency of task processing.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for intelligent agent task processing. Background Technology

[0002] Mobile GUI (Graphical User Interface) agents represent a cutting-edge research direction integrating artificial intelligence and human-computer interaction technologies. Their core objective is to enable agents to understand and operate graphical user interfaces on mobile devices like humans, achieving automated task execution. Existing mobile GUI agent technologies are primarily based on a large language model-based planning-execution architecture. They acquire images of mobile screens, utilize multimodal understanding technology to identify page elements and semantic information, and generate operation instructions based on task objectives to automate task processing. However, these technologies directly make operation decisions after extracting page feature information, relying solely on implicit page state understanding. This makes it difficult to accurately determine the current stage of the task flow or clearly define the transition relationships between pages, resulting in insufficient controllability and low execution efficiency in agent task processing. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an intelligent agent task processing method and apparatus. By determining the state type information corresponding to the current page based on page image information and combining it with intelligent agent task information to determine the state transition path, the intelligent agent can perform task planning based on explicit page state information and generate and issue intelligent agent execution instructions according to the state transition path. This realizes state constraints and path control of the task execution process, improves the controllability and accuracy of the intelligent agent in executing complex multi-step tasks, and enhances the execution efficiency of task processing.

[0004] To achieve the above objectives, according to one aspect of the present invention, an agent task processing method is provided, comprising: Receive the current page image information and agent task information uploaded by the client, and determine the corresponding state type information of the current page based on the current page image information; Determine the state transition path based on the current page's state type information and the agent's task information; The agent execution instructions are generated based on the state transition path and sent to the client so that the client can execute the agent execution instructions to process the agent task.

[0005] Optionally, based on the state type information and agent task information corresponding to the current page, a state transition path is determined, including: The task information of the intelligent agent is parsed and processed to obtain the task target information; Based on the pre-configured state knowledge base retrieved from the task target information, the set of page states and the set of transition rules corresponding to the task target information are obtained. A state transition function is constructed based on the set of page states and the set of transition rules, and the state transition path is determined based on the state transition function.

[0006] Optionally, based on the state type information and agent task information corresponding to the current page, a state transition path is determined, including: Determine the current state node based on the state type information corresponding to the current page; Determine the target status nodes based on task objective information; The state transition path between the current state node and the target state node is determined based on the state transition function.

[0007] Optionally, the state type information corresponding to the current page is determined based on the current page image information, including: Feature extraction is performed on the image information of the current page to obtain page feature information; The matching result is obtained by matching the state knowledge base based on page feature information; The current page's status type information is determined based on the matching results.

[0008] Optionally, after generating agent execution instructions based on the state transition path and sending the agent execution instructions to the client so that the client executes the agent execution instructions to process the agent task, the method further includes: Receive status feedback messages sent by the client and determine the actual page status based on the status feedback messages; The actual page state is compared with the expected page state corresponding to the state transition path to obtain the comparison result; In response to the inconsistency in the comparison results, a state transition exception message is generated, and an exception recovery operation is performed according to the exception handling strategy corresponding to the state transition exception message.

[0009] Optionally, the method further includes: If the comparison result is consistent, determine whether the actual page state corresponds to the target state node; If the judgment result is yes, the processing result of the agent's task is determined to be task completion; If the judgment result is negative, the state transition path is determined again based on the actual page state. Based on the newly determined state transition path, the agent execution instruction is generated and sent to the client.

[0010] According to a second aspect of the present invention, an intelligent agent task processing apparatus is provided, comprising: The receiving module is used to receive the current page image information and agent task information uploaded by the client, and determine the corresponding state type information of the current page based on the current page image information; The determination module is used to determine the state transition path based on the state type information corresponding to the current page and the agent's task information; The generation module is used to generate agent execution instructions based on the state transition path and send the agent execution instructions to the client so that the client can execute the agent execution instructions to process the agent task.

[0011] According to a third aspect of the present invention, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement the methods of any of the above embodiments.

[0012] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method of any of the above embodiments.

[0013] According to a fifth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0014] One embodiment of the above invention has the following advantages or beneficial effects: by receiving the current page image information and agent task information uploaded by the client, the state type information corresponding to the current page is determined based on the current page image information; a state transition path is determined based on the current page state type information and agent task information; an agent execution instruction is generated based on the state transition path, and the agent execution instruction is sent to the client so that the client executes the agent execution instruction to process the agent task; this embodiment determines the current page state type information based on the page image information and determines the state transition path in combination with the agent task information, enabling the agent to perform task planning based on explicit page state information, and generate and send agent execution instructions based on the state transition path, thereby realizing state constraints and path control of the task execution process, improving the controllability and accuracy of the agent's execution of complex multi-step tasks, and improving the execution efficiency of task processing.

[0015] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0016] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main flow of the intelligent agent task processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of an intelligent agent task processing system according to a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the main modules of the intelligent agent task processing device according to an embodiment of the present invention; Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] It should be noted that the acquisition, storage, and application of personal information involved in the embodiments of the present invention comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] Mobile GUI agents represent a cutting-edge research direction integrating artificial intelligence and human-computer interaction technology. Their core objective is to enable agents to understand and operate graphical user interfaces on mobile devices like humans, achieving automated task execution. Existing mobile GUI agent technologies are primarily based on a large language model-based planning-execution architecture. They acquire images of mobile screens, utilize multimodal understanding technology to identify page elements and semantic information, and generate operation instructions based on task objectives to automate task processing. However, these technologies directly make operation decisions after extracting page feature information, relying solely on implicit page state understanding. This makes it difficult to accurately determine the current stage of the task flow or clearly define the transition relationships between pages, resulting in insufficient controllability and low execution efficiency in agent task processing.

[0020] In view of this, according to one aspect of the present invention, an intelligent agent task processing method is provided.

[0021] Figure 1 This is a schematic diagram of the main flow of the intelligent agent task processing method according to an embodiment of the present invention. Figure 1 As shown, the intelligent agent task processing method according to an embodiment of the present invention includes the following steps S101 to S103.

[0022] Step S101: Receive the current page image information and agent task information uploaded by the client, and determine the state type information corresponding to the current page based on the current page image information.

[0023] In this embodiment, the current page image information refers to the image data corresponding to the currently displayed page of the mobile device, which is collected and uploaded by the client. The current page image information may include a full-screen screenshot of the mobile device, or it may include partial page image information captured based on a preset area, such as image information containing key interactive areas like buttons, input boxes, and prompt text. The intelligent agent task information refers to the task description information input by the user to the client, used to instruct the intelligent agent to perform a target business operation, such as text information, voice information, or structured task request information corresponding to tasks like handling business, querying information, or filling out forms. The status type information is used to characterize the business status type of the current page in the intelligent agent task execution process, such as entry status, form status, confirmation status, processing status, result status, or abnormal status, etc.

[0024] Specifically, after receiving the agent task information input by the user, the client collects the page image data of the currently displayed page on the mobile device and uploads the current page image information and the agent task information to the server through a preset communication connection. This preset communication connection can use a Hypertext Transfer Protocol (HTTP) connection, a long-lived connection, or other communication methods that enable data interaction between the client and server. After receiving the current page image information and the agent task information, the server performs page state recognition processing on the current page image information.

[0025] For example, the server can analyze the current page image information using a page element detection model to identify page elements such as page controls, input areas, buttons, and prompts. It can also extract text content from the current page image using a text recognition model, combining the page elements and text content to determine page feature information. Furthermore, the page feature information is matched with pre-stored state template information, and the corresponding state type information for the current page is determined based on the matching result. Alternatively, the server can input the current page image information into a pre-trained page state classification model, which directly outputs the corresponding state type information based on the page image features. This page state classification model can be trained based on page image samples and corresponding state type labels from historical task execution processes. Through these methods, the visual information corresponding to the mobile page can be converted into explicit state type information, providing a state basis for subsequent agent task planning based on state transition relationships.

[0026] Step S102: Determine the state transition path based on the state type information and agent task information corresponding to the current page.

[0027] In this embodiment, a state transition path refers to the path information used to describe the path an agent traverses from the current page's state type information to the task target's state type information during the execution of a target task, as well as the corresponding transition relationships between these page states. A state transition path may include multiple state nodes and state transition relationships connecting these nodes. State nodes represent page states in a mobile application, and state transition relationships represent the operations, transition conditions, and target states required to transition from one page state to another. For example, when processing a business task, the state transition path may include path information transitioning from an entry state to a form state, from a form state to a confirmation state, and from a confirmation state to a result state.

[0028] Specifically, the server performs state path planning based on the state type information corresponding to the current page and the agent's task information. As one implementation method, the server first performs semantic parsing on the agent's task information to determine the corresponding task target information, and then determines the final target state based on the task target information. Further, based on a pre-built state transition model, it queries the state transition relationships associated with the state type information corresponding to the current page, and determines the state transition path from the current state node to the final target state node based on these relationships. The state transition model can be stored in a state knowledge base, and it includes multiple page states and the corresponding transition conditions and operations between different page states.

[0029] In other implementations, the server can also determine the state transition path using a path planning model based on the current page's state type information, the agent's task information, and historical task execution information. Specifically, the current page's state type information and the agent's task information are input into a pre-trained path planning model. The model then predicts the state transition path from the current page state to the task target state based on the page state change patterns and operation success rates during historical task execution. This approach dynamically determines the appropriate state transition path based on the current page state and the task target, enabling the agent to execute tasks according to defined state transition relationships and improving controllability and accuracy in complex task processing.

[0030] Step S103: Generate agent execution instructions based on state transition paths and send the agent execution instructions to the client so that the client can execute the agent execution instructions to process agent tasks.

[0031] In this embodiment, the agent execution instruction refers to the operation instruction information used to control the client to perform page interaction operations, thereby driving the current page state to transition according to the state transition path. The agent execution instruction may include target operation type, operation object information, and operation parameter information, etc.; wherein, the target operation type indicates the interaction operation to be performed, such as a click operation, a swipe operation, a text input operation, a selection operation, etc.; the operation object information indicates the page element to be operated, such as a button, an input box, a menu option, etc.; the operation parameter information indicates the specific parameters required to perform the target operation, such as the click position coordinates, input content, swipe direction, and swipe distance, etc. Through the agent execution instruction, the state transition relationship in the state transition path can be transformed into a specific operation behavior that the client can execute.

[0032] Specifically, after determining the state transition path, the server determines the target operation information corresponding to each state transition relationship based on the state transition relationships between adjacent state nodes in the path, and generates corresponding agent execution instructions based on the page element information in the current page. For example, when the state transition path includes a transition from a form state to a confirmation state, the server can determine that a form submission operation needs to be performed based on the state transition relationships, and generate agent execution instructions containing a submit button identifier and click parameters.

[0033] In one implementation, the server can generate a set of corresponding agent execution instructions sequentially according to multiple state nodes in the state transition path, and send the set of agent execution instructions to the client through the communication connection between the client and the server. The client then executes the agent execution instructions sequentially. In another implementation, the server can also determine the next state node corresponding to the current stage based on the state transition path, and generate only the agent execution instructions required for the current state node to transition to the next state node. After the client executes the agent execution instructions, it feeds back the execution result to the server. The server then re-determines the subsequent state transition path based on the feedback result and generates the agent execution instructions corresponding to the next stage. Through the above methods, the execution instructions can be dynamically adjusted according to the actual page state during task execution, enabling the agent to complete task processing step by step according to the state transition path, thereby improving the accuracy and reliability of the task execution process.

[0034] This embodiment receives the current page image information and agent task information uploaded by the client, determines the state type information corresponding to the current page based on the current page image information, determines the state transition path based on the current page state type information and agent task information, generates agent execution instructions based on the state transition path, and sends the agent execution instructions to the client so that the client can execute the agent execution instructions to process the agent task. This embodiment determines the current page state type information based on page image information and determines the state transition path in combination with agent task information, enabling the agent to perform task planning based on explicit page state information, and generate and send agent execution instructions based on the state transition path. This realizes state constraints and path control of the task execution process, improves the controllability and accuracy of the agent's execution of complex multi-step tasks, and improves the execution efficiency of task processing.

[0035] Optionally, the state transition path is determined based on the state type information and agent task information corresponding to the current page, including: parsing and processing the agent task information to obtain task target information; retrieving a pre-configured state knowledge base based on the task target information to obtain the page state set and transition rule set corresponding to the task target information; and constructing a state transition function based on the page state set and transition rule set to determine the state transition path based on the state transition function.

[0036] In this embodiment, task target information is used to characterize the target business content that the intelligent agent needs to complete, such as the target operation result corresponding to handling business, querying information, and submitting applications. After receiving the intelligent agent task information, semantic parsing processing is performed on the intelligent agent task information to extract information such as the task object, task operation, and task result, and generate the corresponding task target information. Further, a pre-configured state knowledge base is queried based on the task target information. The state knowledge base stores page state information corresponding to different task scenarios and the transformation relationship between page states. The corresponding page state set and transition rule set are obtained by matching according to the task target information. The page state set includes page states such as entry state, form state, confirmation state, processing state, result state, and abnormal state. The transition rule set includes operation information and transformation conditions corresponding to state transformation. The server determines the state nodes according to the page state set and establishes the transformation relationship between each state node according to the transition rule set, constructing a state transition function. The state transition function is used to represent the target state that can be transformed to after performing the corresponding operation in the current state. Based on the state transition function, combined with the state type information corresponding to the current page and the task target information, the state transformation path from the current state to the target state is searched. By using the above methods, the page change relationships during the execution of intelligent agents can be explicitly modeled, enabling intelligent agents to plan tasks based on clear state transition relationships, thereby improving the controllability and accuracy of complex task execution processes.

[0037] Optionally, the state transition path is determined based on the state type information corresponding to the current page and the agent's task information, including: determining the current state node based on the state type information corresponding to the current page; determining the target state node based on the task target information; and determining the state transition path between the current state node and the target state node based on the state transition function.

[0038] In this embodiment, when determining the state transition path based on the state type information of the current page and the agent's task information, the current state node is first determined based on the state type information of the current page, and the corresponding target state node is determined based on the task target information. The target state node represents the page state that needs to be reached after completing the agent's task. Then, based on a pre-built state transition function, the state transition relationship between the current state node and the target state node is queried. The state transition path is determined based on the queried reachable state nodes, state transition conditions, and state transition operations. Through this method, the agent's task execution process can be transformed into a path planning process based on state nodes, enabling the agent to clearly understand the current task stage and subsequent execution direction, thus improving the controllability of the task processing process.

[0039] Alternatively, a path planning request can be generated based on the current page's state type information and the agent's task information. This request is then input into a pre-trained path planning model, which predicts the state transition path from the current state node to the target state node based on historical task execution data, page state change patterns, and task objective information. This approach enables dynamic generation of state transition paths based on different task scenarios, improving the adaptability of state transition path determination and the agent's task processing efficiency.

[0040] Optionally, determining the state type information corresponding to the current page based on the current page image information includes: performing feature extraction processing on the current page image information to obtain page feature information; matching the page feature information against a state knowledge base to obtain a matching result; and determining the state type information corresponding to the current page based on the matching result.

[0041] In this embodiment, a pre-trained image feature extraction model can be used to analyze and process the current page image information, extracting page feature information such as page layout features, page element features, and text content features. Page element features can include page control information such as buttons, input boxes, lists, and tooltips, while text content features can include semantic information such as page titles, tooltips, and input content. Further, the page feature information is compared with state template information in a state knowledge base to obtain corresponding matching results. Based on the matching results, the state type information corresponding to the current page image information is determined, such as entry state, form state, confirmation state, processing state, result state, or abnormal state. Through this method, page image information can be converted into state type information with business semantics, enabling explicit recognition of mobile page states, avoiding implicit judgments based solely on visual information, and improving the accuracy of state recognition.

[0042] In other embodiments, the current page's state type information can be determined based on a pre-trained page state recognition model. Specifically, the current page image information is input into the page state recognition model, which then performs joint analysis of visual and textual features in the page image and directly outputs the corresponding state type information. This method eliminates the need for pre-configured state templates and enables adaptive recognition based on historical page state samples, improving the flexibility of page state recognition in different application scenarios.

[0043] Optionally, after generating agent execution instructions based on the state transition path and sending the agent execution instructions to the client so that the client can execute the agent execution instructions to process the agent task, the method further includes: receiving a state feedback message sent by the client and determining the actual page state based on the state feedback message; comparing the actual page state with the expected page state corresponding to the state transition path to obtain a comparison result; and generating state transition exception information in response to the inconsistency of the comparison result, and performing an exception recovery operation according to the exception handling strategy corresponding to the state transition exception information.

[0044] In this embodiment, after generating the agent execution instruction based on the state transition path and sending it to the client to execute the agent task, the state of the agent execution process is further verified. Specifically, after the client executes the agent execution instruction, it collects the current page change information and generates a state feedback message containing the operation execution result, page image information, and current page state information, and sends the state feedback message to the server. After receiving the state feedback message, the server parses and processes the state feedback message to determine the actual page state after the client executes the agent execution instruction. Further, the server determines the expected page state corresponding to the current operation based on the state transition path, and compares the actual page state with the expected page state to determine whether the current page state has completed the transition according to the preset state transition relationship. When the actual page state is inconsistent with the expected page state, a state transition anomaly is determined to have occurred, state transition anomaly information is generated, and the corresponding anomaly handling strategy is matched according to the state transition anomaly information to perform anomaly recovery operations such as reoperation, state rollback, or replanning of the state transition path. By using the above methods, page state changes can be monitored in real time during the execution of the agent, avoiding task flow deviations due to operation failures or abnormal page jumps, and improving the reliability and fault tolerance of agent task processing.

[0045] In other embodiments, an anomaly prediction model can be built based on historical task execution data. Status feedback messages are input into the anomaly prediction model, which predicts whether the current page state is abnormal and outputs a corresponding anomaly handling strategy. The server then executes the corresponding recovery operation according to the anomaly handling strategy. This approach combines historical execution experience to identify potential anomalies in advance, improving the proactiveness of anomaly handling and the efficiency of agent task execution.

[0046] Optionally, the method further includes: in response to a matching result, determining whether the actual page state corresponds to the target state node; in response to a yes result, determining that the processing result of the agent task is task completion; in response to a no result, re-determining the state transition path based on the actual page state, generating an agent execution instruction based on the re-determined state transition path, and sending the agent execution instruction to the client.

[0047] In this embodiment, after confirming a match, the server further determines whether the actual page state of the agent's current execution stage has reached the corresponding target state node. Specifically, the server determines whether the target state node is the final target state node corresponding to the agent's task; in response to the target state node being the final target state node, the server determines that the processing result of the agent's task is task completion and generates corresponding task completion information. When the actual page state has not reached the final target state node, it indicates that the agent's task is still in execution. The server re-queries the state transition function based on the current actual page state, determines the subsequent state transition path, and generates a new agent execution instruction based on the subsequent state transition path. The agent execution instruction is then sent to the client to continue executing the agent's task. Through this method, the execution flow can be dynamically adjusted according to the actual page state during task execution, enabling the agent to complete complex tasks state by state, thereby improving the continuity, accuracy, and reliability of the task execution process.

[0048] In addition, a task execution status evaluation result can be generated based on the actual page state and remaining task information. The next state node can then be determined based on this evaluation result, and a new state transition path can be dynamically generated using a path planning algorithm. This approach enables adaptive updates to the execution path when the page state changes or the task flow is adjusted, improving the agent's adaptability to complex business scenarios.

[0049] Figure 2 This is a schematic diagram of the architecture of an intelligent agent task processing system according to a preferred embodiment of the present invention. The system of the present invention adopts a client-server separation architecture, comprising three main parts: a client layer, a server layer, and a state knowledge base layer. The layers are connected via WebSocket or HTTP communication protocols to achieve real-time data interaction and command issuance.

[0050] like Figure 2 As shown, the client layer is deployed on mobile devices and comprises four main components: a page screenshot module, an operation execution module, a state awareness module, and a communication module. The page screenshot module is responsible for capturing screenshot data of the current screen of the mobile device, supporting both full-screen and partial screenshot modes. The operation execution module receives operation instructions from the server and converts them into atomic operations executable by the mobile device, including clicks, swipes, and text input. The state awareness module monitors page changes after operation execution, collects state change information, and feeds it back to the server. The communication module maintains the communication connection between the client and server, enabling bidirectional communication for data uploading and instruction reception.

[0051] The server-side layer, deployed on a cloud server or local server, is the core processing unit of the system. It includes core components such as a task parsing module, a page state machine engine, and a state knowledge base. The task parsing module receives user-submitted task requests and parses the task objectives and constraints. The page state machine engine is the core of the system, comprising five sub-modules: a state recognition module, a state transition graph construction module, a task planning module, an operation execution control module, and a state verification module. The state recognition module identifies the current state type based on a page screenshot. The state transition graph construction module constructs a state transition path graph based on the task objectives. The task planning module generates an operation planning sequence based on the state transition graph. The operation execution control module converts the operation plan into an executable instruction sequence. The state verification module verifies state changes after operation execution to determine if the expected state has been achieved.

[0052] The state knowledge base layer stores prior knowledge and historical experience of page states, including a state type definition library, a state transition rule library, and an operation-state mapping library. The state knowledge base provides knowledge support for the page state machine engine, enabling the system to quickly and accurately identify states and plan transition paths.

[0053] Specifically, the page state machine engine is used to formally model the state of mobile GUI pages. The page state machine proposed in this embodiment is a formal model defined by a six-tuple, including a set of states, a set of input symbols, a state transition function, an initial state, a set of accepting states, and a state attribute mapping function.

[0054] The state set S is defined as the set of all possible states a page can be in. This invention classifies page states in government service scenarios into six types: Entry, Form, Confirm, Processing, Result, and Error. Entry state represents the starting page of a business process, such as the homepage or function entry page of government services. Form state represents a page requiring information entry, which may contain multiple form fields. Confirm state represents a page requiring user confirmation or approval. Processing state represents an intermediate state page where the system is processing the data. Result state represents a page displaying the result of task completion. Error state represents a page where an error has occurred or requires special handling.

[0055] The input symbol set I is defined as the set of triggerable operation types, including atomic operation types such as Click, Swipe, Input, and LongPress. Each operation type can carry additional information such as target element identifier and operation parameters.

[0056] The state transition function δ is defined as a mapping relationship S×I→S, representing the new state transitioned to after performing a specific operation in the current state. State transition functions are stored in the form of a state transition table or a state transition graph, supporting efficient querying and reasoning. Each state transition relationship is also associated with a transition condition Cond and a transition operation Op, indicating that a state transition is triggered only when a specific condition is met and a specific operation is performed.

[0057] The initial state S0 is defined as the page state at the start of task execution, typically the entry state. The set of accepted states F is defined as the set of states that should be reached when the task is successfully completed, typically including the result state and the success state. The state attribute mapping function Attr is defined as the mapping from S to P, mapping each state to its attribute set. Attributes include state name, state description, a list of key elements, a list of required fields, etc.

[0058] This embodiment also proposes a GUI agent task planning mechanism based on state transition graphs. By constructing state transition graphs to explicitly represent task execution paths, the mechanism guides the agent in operation planning. The task planning mechanism includes four stages: state recognition, state transition graph construction, path search, and operation sequence generation.

[0059] State Recognition Phase. The state recognition module receives a screenshot of the page uploaded by the client, extracts page features using multimodal understanding technology, including visual features and text semantic features, and then matches these page features with state templates in a state knowledge base to identify the current state type of the page. The specific method for state recognition is as follows: First, a target detection model is used to identify key elements on the page, such as buttons, input boxes, and tooltips. Then, OCR technology is used to extract the page text content and perform semantic analysis. Finally, the extracted element features and text semantics are compared with the state templates to calculate similarity, and the state type with the highest similarity is selected as the recognition result.

[0060] The state transition graph construction phase involves building a state transition graph from the initial state to the target state based on task objective information and a state knowledge base. The state transition graph is a directed graph where nodes represent page states, edges represent state transition relationships, and transition conditions and operations are labeled on the edges. The construction method is as follows: First, the set of states and transition rules related to the current task are retrieved from the state knowledge base. Then, starting from the initial state and ending at the target state, a reachable path is constructed using a breadth-first search or depth-first search algorithm. Finally, the constructed path is optimized, redundant state transitions are removed, and the shortest or optimal path is selected.

[0061] Pathfinding Phase. The task planning module searches for the optimal path from the current state to the next target state based on the state transition graph. Pathfinding employs a heuristic search algorithm, such as A... The algorithm uses the estimated cost of state transitions as a heuristic function to search for the minimum transition path. The heuristic function is designed considering factors such as the operational complexity of state transitions, the historical probability of successful transitions, and user preferences. The search result is a sequence of state transition paths, where each transition step includes the target state, transition conditions, and transition operations.

[0062] Operation sequence generation phase. The task planning module converts state transition paths into specific operation sequences. For each state transition step, based on the transition operation and the current page element identification result, specific atomic operation instructions are generated, including operation type, target element coordinates, operation parameters, etc. The operation sequence is arranged in the order of the transition path to form a complete task execution plan.

[0063] This embodiment also proposes a state-driven operation execution and verification process. Through a state verification mechanism, it ensures that each operation correctly triggers the expected state transition, improving the reliability of task execution. The operation execution and verification process includes four stages: operation execution, state awareness, state verification, and exception handling.

[0064] Operation Execution Phase. The operation execution control module sequentially sends the sequence of operation instructions to the client for execution. After receiving the instructions, the client's operation execution module executes the corresponding atomic operations through accessibility services or UI automation frameworks, such as clicking a specified coordinate or entering specified text. After the operation is executed, the client waits for the page to stabilize and then captures a new page screenshot.

[0065] State Awareness Phase. The state awareness module monitors page changes after an operation is executed and collects state change information. Specifically, it captures a new page screenshot, extracts page features, identifies the current new state, and encapsulates the state change information into a state feedback message, which is then sent to the server. The state feedback message includes information such as the state before the operation, the operation type, the state after the operation, and a description of the page changes.

[0066] State Verification Phase. The state verification module receives state feedback from the client, compares the expected state with the actual state, and determines whether the state transition meets expectations. The specific method for state verification is as follows: First, obtain the expected target state corresponding to the current operation from the task plan. Then, compare the actual state identified by the state awareness module with the expected state. If they match, the state transition is considered successful, and the next operation continues. If they do not match, the state transition is considered abnormal, triggering the exception handling process.

[0067] Exception Handling Phase. When the state check detects an abnormal state transition, the system enters the exception handling process. Exception handling strategies include the following: First, a retry strategy: when a temporary error is detected (such as network latency or incomplete page loading), the current operation is re-executed. Second, a rollback strategy: when an unrecoverable error is detected, the system rolls back to the previous stable state and replans the execution path. Third, a manual intervention strategy: when the exception cannot be handled automatically, task execution is paused, and manual intervention is notified. The selection of exception handling strategies is based on the exception type and historical experience in the state knowledge base.

[0068] In this embodiment, the state knowledge base serves as the knowledge foundation of the page state machine engine, storing prior knowledge and historical experience of page states. The state knowledge base comprises three main parts: a state type definition library, a state transition rule library, and an operation-state mapping library.

[0069] The state type definition library stores formal definitions for various state types, including state names, state descriptions, key element templates, and text keywords. This library provides matching templates for the state recognition module, enabling the system to quickly and accurately identify page states.

[0070] The state transition rule base stores the definitions of transition relationships between states, including source state, target state, transition conditions, and transition operations. The state transition rule base provides these transition relationships for the state transition graph construction module, supporting the rapid construction of task execution paths.

[0071] The operation-state mapping library stores the correspondence between operations and state changes, including operation type, operation target, and expected state change. This library provides the state verification module with an expected state reference, supporting accurate state change verification.

[0072] The state knowledge base is constructed using two methods: knowledge extraction and knowledge fusion. Knowledge extraction extracts state definitions and transition rules from official documents and operation manuals of government applications. Knowledge fusion learns state transition patterns from historical task execution logs, continuously enriching and optimizing the knowledge base content.

[0073] According to a second aspect of the present invention, an intelligent agent task processing apparatus is provided.

[0074] Figure 3 This is a schematic diagram of the main modules of an intelligent agent task processing device according to an embodiment of the present invention. Figure 3 As shown, the intelligent agent task processing device 300 includes: The receiving module 301 is used to receive the current page image information and intelligent agent task information uploaded by the client, and determine the state type information corresponding to the current page based on the current page image information; The determination module 302 is used to determine the state transition path based on the state type information and agent task information corresponding to the current page; The generation module 303 is used to generate agent execution instructions based on the state transition path and send the agent execution instructions to the client so that the client can execute the agent execution instructions to process the agent task.

[0075] Optionally, the determining module 302 is also used for: The task information of the intelligent agent is parsed and processed to obtain the task target information; Based on the pre-configured state knowledge base retrieved from the task target information, the set of page states and the set of transition rules corresponding to the task target information are obtained. A state transition function is constructed based on the set of page states and the set of transition rules, and the state transition path is determined based on the state transition function.

[0076] Optionally, the determining module 302 is also used for: Determine the current state node based on the state type information corresponding to the current page; Determine the target status nodes based on task objective information; The state transition path between the current state node and the target state node is determined based on the state transition function.

[0077] Optionally, the receiving module 301 is also used for: Feature extraction is performed on the image information of the current page to obtain page feature information; The matching result is obtained by matching the state knowledge base based on page feature information; The current page's status type information is determined based on the matching results.

[0078] Optionally, the intelligent agent task processing device 300 further includes a comparison module, which is also used for: Receive status feedback messages sent by the client and determine the actual page status based on the status feedback messages; The actual page state is compared with the expected page state corresponding to the state transition path to obtain the comparison result; In response to the inconsistency in the comparison results, a state transition exception message is generated, and an exception recovery operation is performed according to the exception handling strategy corresponding to the state transition exception message.

[0079] Optionally, the comparison module is also used for: If the comparison result is consistent, determine whether the actual page state corresponds to the target state node; If the judgment result is yes, the processing result of the agent's task is determined to be task completion; If the judgment result is negative, the state transition path is determined again based on the actual page state. Based on the newly determined state transition path, the agent execution instruction is generated and sent to the client.

[0080] It should be noted that the specific implementation details of the intelligent agent task processing device of the present invention have been described in detail in the above intelligent agent task processing method, so the content will not be repeated here.

[0081] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0082] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0083] According to a fifth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0084] Figure 4 An exemplary system architecture 400 is shown that can be applied to the agent task processing method or agent task processing apparatus of the present invention.

[0085] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0086] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0087] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0088] Server 405 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 401, 402, and 403 (for example only). The backend management server can analyze and process data such as received agent task processing requests, and feed back the processing results (such as agent task processing results - for example only) to the terminal devices.

[0089] It should be noted that the intelligent agent task processing method provided in the embodiments of the present invention is generally run by server 405, and correspondingly, the intelligent agent task processing device is generally set in server 405.

[0090] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0091] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing a terminal device of the present invention. Figure 5 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0092] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0093] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. Drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0094] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is run by the central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0095] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more operable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually operate substantially in parallel, and they may sometimes operate in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a receiving module, a determining module, and a generating module. The names of these modules do not necessarily limit the module itself; for example, the generating module can also be described as "a module for generating agent execution instructions based on state transition paths and sending the agent execution instructions to the client."

[0098] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: receiving current page image information and agent task information uploaded by a client; determining the state type information corresponding to the current page based on the current page image information; determining a state transition path based on the state type information and agent task information corresponding to the current page; generating agent execution instructions based on the state transition path; and sending the agent execution instructions to the client so that the client executes the agent execution instructions to process the agent task.

[0099] The computer program product provided in this embodiment of the invention includes a computer program that, when executed by a processor, implements the intelligent agent task processing method in this embodiment of the invention.

[0100] According to the technical solution of the present invention, the following advantages or beneficial effects are achieved: by receiving the current page image information and agent task information uploaded by the client, the state type information corresponding to the current page is determined based on the current page image information; a state transition path is determined according to the state type information and agent task information corresponding to the current page; an agent execution instruction is generated based on the state transition path, and the agent execution instruction is sent to the client so that the client executes the agent execution instruction to process the agent task; this embodiment determines the state type information corresponding to the current page based on the page image information and determines the state transition path in combination with the agent task information, enabling the agent to perform task planning based on explicit page state information, and generate and send agent execution instructions according to the state transition path, thereby realizing state constraints and path control of the task execution process, improving the controllability and accuracy of the agent's execution of complex multi-step tasks, and improving the execution efficiency of task processing.

[0101] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0102] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for intelligent agent task processing, characterized in that, include: Receive the current page image information and agent task information uploaded by the client, and determine the current page status type information based on the current page image information; The state transition path is determined based on the state type information corresponding to the current page and the agent task information; Based on the state transition path, an agent execution instruction is generated and sent to the client so that the client can execute the agent execution instruction to process the agent task.

2. The method according to claim 1, characterized in that, Based on the current page's state type information and the agent's task information, determine the state transition path, including: The task information of the intelligent agent is parsed and processed to obtain the task target information; Based on the task target information, a pre-configured state knowledge base is retrieved to obtain the page state set and transition rule set corresponding to the task target information; A state transition function is constructed based on the set of page states and the set of transition rules, and a state transition path is determined based on the state transition function.

3. The method according to claim 2, characterized in that, Based on the current page's state type information and the agent's task information, determine the state transition path, including: The current state node is determined based on the state type information corresponding to the current page; Determine the target status node based on the aforementioned task objective information; The state transition path between the current state node and the target state node is determined based on the state transition function.

4. The method according to claim 2, characterized in that, Based on the current page image information, the current page's corresponding state type information is determined, including: The current page image information is subjected to feature extraction processing to obtain page feature information; Based on the page feature information, the state knowledge base is matched to obtain the matching result; The status type information corresponding to the current page is determined based on the matching result.

5. The method according to claim 1, characterized in that, After generating agent execution instructions based on the state transition path and sending the agent execution instructions to the client so that the client executes the agent execution instructions to process the agent task, the process further includes: Receive status feedback messages sent by the client, and determine the actual page status based on the status feedback messages; The actual page state is compared with the expected page state corresponding to the state transition path to obtain the comparison result; In response to the inconsistency of the comparison result, a state transition exception information is generated, and an exception recovery operation is performed according to the exception handling strategy corresponding to the state transition exception information.

6. The method according to claim 5, characterized in that, The method further includes: If the comparison result is consistent, determine whether the actual page state corresponds to the target state node; If the judgment result is yes, the processing result of the agent's task is determined to be task completion; If the judgment result is negative, the state transition path is determined again based on the actual page state. An agent execution instruction is generated based on the re-determined state transition path, and the agent execution instruction is sent to the client.

7. A smart agent task processing device, characterized in that, include: The receiving module is used to receive the current page image information and the intelligent agent task information uploaded by the client, and determine the state type information corresponding to the current page based on the current page image information; The determination module is used to determine the state transition path based on the state type information corresponding to the current page and the agent task information; The generation module is used to generate agent execution instructions based on the state transition path, and send the agent execution instructions to the client so that the client can execute the agent execution instructions to process the agent task.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images