Task execution method and device based on trajectory construction, equipment and medium

Abstract

The application discloses a task execution method and device based on trajectory construction, equipment and medium, relates to the field of artificial intelligence technology, and the method comprises the following steps: performing semantic analysis on a task request of a user end to obtain N coarse-grained subtasks; in a page information database, according to semantic information of the nth coarse-grained subtask, a plurality of related page blocks are acquired from a page information data set of an application associated with the nth coarse-grained subtask; the page information database comprises a plurality of page information data sets, one page information data set corresponds to one application, and each page information data set is used for storing a plurality of page blocks related to the corresponding application; according to the plurality of related page blocks and the coarse-grained subtask, an nth fine-grained operation sequence corresponding to the nth coarse-grained subtask is generated; the nth fine-grained operation sequence comprises a plurality of action description information for completing the nth coarse-grained subtask; and the task request is executed based on the N fine-grained operation sequences.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a task execution method, apparatus, device, and medium based on trajectory construction. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) and Multimodal Language Models (MLLMs), artificial intelligence technology has made groundbreaking progress in fields such as natural language understanding, visual perception, logical reasoning, and autonomous decision-making. Agents based on these models possess human-like task understanding and cross-domain operational capabilities, and have been widely applied in various fields such as software development, information retrieval, autonomous driving, and human-computer interaction.

[0003] In recent years, researchers have attempted to apply this technology to GUI (Graphical User Interface) operations on mobile devices, automating tasks such as opening apps, filling out forms, and changing system settings. Among these, AppAgent, based on multimodal perception, understands the app interface by capturing screenshots and XML structure information, inputting these into a multimodal model to generate operational decisions, thus enabling mobile task interaction based on vision and language.

[0004] However, these methods still have the following problems that limit their application in real-world complex scenarios: First, LLM cannot understand the semantic structure and interaction logic of many real apps. Relying solely on pre-trained data knowledge will lead to operational distortion.

[0005] Secondly, the system or framework for task execution lacks long-term memory and cannot retain historical task execution experience in multiple task executions. The repeated trial and error of tasks and the lack of experience transfer result in execution efficiency far lower than that of human users. Furthermore, LLM typically generates a complete sequence of task executions directly from user input (such as a series of clicks or inputs). However, mobile tasks often have multi-level and multi-contextual dependency characteristics. Summary of the Invention

[0006] The purpose of this application is to provide a task execution method, apparatus, device, and medium based on trajectory construction, which can solve the above-mentioned problems.

[0007] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a task execution method based on trajectory construction, the method comprising: Semantic parsing is performed on the user's task request to obtain N coarse-grained subtasks, each of which is a logical operation; N is an integer not less than 1. In the page information database, based on the semantic information of the nth coarse-grained subtask, multiple related page blocks are obtained from the page information data set of the application associated with the nth coarse-grained subtask; the page information database contains multiple page information data sets, one page information data set corresponds to one application, and each page information data set is used to store multiple page blocks related to the corresponding application; Based on the multiple related page blocks and the coarse-grained subtasks, generate the nth fine-grained operation sequence corresponding to the nth coarse-grained subtask; the nth fine-grained operation sequence contains multiple action description information for completing the nth coarse-grained subtask; The task request is executed based on N fine-grained operation sequences.

[0008] Optionally, the state information of the page block includes at least the following fields: page description, key UI elements, path information, and page tags. The execution of the task request based on N fine-grained operation sequences includes: The nth operation instruction is generated based on the nth fine-grained operation sequence, the observation information on the first page, and the operation records of the previous n-1 operation instructions; Execute the nth operation instruction, and evaluate the execution result of the nth operation instruction to obtain the operation evaluation result of the nth operation instruction; After the nth coarse-grained subtask is completed, the page observation information, actions, inference text and operation evaluation results generated during the execution of the nth coarse-grained subtask are integrated to obtain the task trajectory information of the nth coarse-grained subtask. The state information of each page contained in the task trajectory information is semantically compressed to obtain a new page block, and the new page block is stored in the corresponding page information data set for reuse in new task requests.

[0009] Optionally, the step of obtaining multiple related page blocks from the page information data set of the application associated with the nth coarse-grained subtask based on the semantic information of the nth coarse-grained subtask includes: Based on the semantic information of the nth coarse-grained subtask and the application associated with the nth coarse-grained subtask, the task context description of the nth coarse-grained subtask is obtained; Calculate the similarity between the task context description of the nth coarse-grained subtask and each page block in the page information data set of the application associated with the nth coarse-grained subtask; The multiple page blocks in the page information data set are sorted in descending order of similarity to obtain a similarity ranking result. The top K page blocks in the similarity ranking result are selected as multiple related page blocks for the nth coarse-grained subtask.

[0010] Optionally, the step of generating the nth operation instruction based on the nth fine-grained operation sequence, the first page observation information, and the operation records of the previous n-1 operation instructions includes: The decision generator generates inference text based on the nth fine-grained operation sequence, the observation information of the first page, and the operation records of the previous n-1 operation instructions. Then, it generates the nth operation instruction according to the inference text. The step of evaluating the execution result of the nth operation instruction to obtain the operation evaluation result of the nth operation instruction includes: The task reviewer evaluates the page state changes between the observation information on the first page and the observation information on the second page to obtain the operation evaluation result of the nth operation instruction; the operation evaluation result is used to indicate whether the nth operation instruction was successfully executed, the current task progress, and the next recommended action; The first page observation information is the observation information of the current page before the execution of the nth operation instruction; the second page observation information is the observation information of the current page after the execution of the nth operation instruction.

[0011] Optionally, after obtaining the operation evaluation result of the nth operation instruction, the method further includes: The task reviewer feeds back the operation evaluation result of the nth operation instruction to the decision generator. In the case where the operation evaluation result indicates that the nth coarse-grained subtask has failed, the decision generator modifies the nth operation instruction based on the operation evaluation result to obtain the updated nth inference text and the updated nth operation instruction. The task executor executes the updated nth operation instruction according to the updated nth reasoning text; After the task executor executes the updated nth operation instruction, the task reviewer evaluates the page state changes between the third page observation information and the fourth page observation information to obtain the operation evaluation result of the updated nth operation instruction. The third page observation information refers to the observation information of the current page before the execution of the updated nth operation instruction; the fourth page observation information refers to the observation information of the current page after the execution of the updated nth operation instruction.

[0012] Optionally, the status information of the page block further includes a timestamp, and the method further includes: Calculate the similarity between any two page blocks in the multiple page blocks contained in each page information data set to obtain the similarity result between any two page blocks; If the similarity between two page blocks exceeds a preset similarity threshold, the page block with the earlier timestamp will be deleted based on the timestamps of the two page blocks.

[0013] Optionally, the integration of page observation information, actions, inference text, and operation evaluation results generated during the execution of the nth coarse-grained subtask to obtain the task trajectory information of the nth coarse-grained subtask includes: If the nth coarse-grained subtask is successfully executed, the page observation information, actions, inference text corresponding to the successfully executed nth operation instruction, and operation evaluation results corresponding to the successfully executed nth operation instruction generated during the execution process are integrated to obtain the task trajectory information of the nth coarse-grained subtask.

[0014] Secondly, embodiments of this application provide a task execution device based on trajectory construction, the device comprising: The coarse-grained subtask partitioning module is used to perform semantic parsing on the user's task requests to obtain N coarse-grained subtasks, each of which is a logical operation; N is an integer not less than 1. The related page block acquisition module is used to acquire multiple related page blocks from the page information data set of the application associated with the nth coarse-grained subtask in the page information database, based on the semantic information of the nth coarse-grained subtask; the page information database contains multiple page information data sets, one page information data set corresponds to one application, and each page information data set is used to store multiple page blocks related to the corresponding application. The fine-grained operation sequence generation module is used to generate the nth fine-grained operation sequence corresponding to the nth coarse-grained subtask based on the multiple related page blocks and the coarse-grained subtask; the nth fine-grained operation sequence includes multiple action description information for completing the nth coarse-grained subtask; The task execution module is used to execute the task request based on N fine-grained operation sequences.

[0015] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the trajectory-based task execution method described in the first aspect.

[0017] The technical solution described above maintains a page information database containing multiple application-related page blocks, each page block recording detailed page information of the application. When a new task request is received, it is first divided into multiple coarse-grained subtasks based on semantic understanding. Then, for the semantic information of each coarse-grained subtask, the application associated with that subtask is determined. Next, the page information data set associated with that application is found in the page information database, and the multiple page blocks most relevant to the current task are retrieved from this data set to generate a fine-grained operation sequence for the coarse-grained subtask. The task request is then executed sequentially according to this fine-grained operation sequence. This allows the agent to directly perceive the application's interface hierarchy and interaction logic during task execution, significantly improving its understanding of real-world applications and reducing operational distortion. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of a task execution method based on trajectory construction provided in an embodiment of this application; Figure 2 This is a schematic diagram of a page information database provided in one embodiment of this application; Figure 3 This is a schematic diagram of the framework of a task execution device based on trajectory construction according to an embodiment of this application; Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The following description, in conjunction with the accompanying drawings, details a task execution method based on trajectory construction provided by the embodiments of this application through specific examples and application scenarios.

[0022] Mobile task automation agents in related technologies suffer from the following significant problems in complex real-world scenarios: The first problem: lack of understanding of real-world application knowledge.

[0023] While current LLMs possess strong language understanding and reasoning capabilities, in mobile device operation scenarios, their knowledge primarily comes from general text corpora, lacking a true understanding of the interface hierarchy, interaction logic, and operation paths of real apps. These issues lead to a disconnect between the "understanding" and "execution" capabilities of LLMs in mobile task automation, becoming a core obstacle limiting their practical application.

[0024] The second problem is the lack of a long-term, reusable task memory mechanism, which leads to repetitive learning and inefficient decision-making.

[0025] Existing LLM agents typically employ a "short-term context" strategy, which records step history within a limited context window (prompt) during a single task session. This results in the agent being unable to form reusable experience for successfully executed tasks, repeatedly going through an "exploration-trial and error" process. Unlike human users, they cannot build "page-level mental models" when operating the same app, and lack a structured memory storage and retrieval mechanism for completed tasks, leading to inefficiency.

[0026] Figure 1 This is a flowchart illustrating a task execution method based on trajectory construction according to an embodiment of this application. Figure 2 This is a schematic diagram of a page information database provided in one embodiment of this application.

[0027] refer to Figure 1 To address the aforementioned problems, this application provides a task execution method based on trajectory construction, the method comprising steps S11 to S14: Step S11: Semantic parsing is performed on the task request from the user end to obtain N coarse-grained subtasks. Each coarse-grained subtask is a logical operation, and N is an integer not less than 1.

[0028] This method is applied to a task execution system, which includes a coarse-grained planning submodule (LLM Coarse-Grained Planner), a fine-grained planning submodule (LLM Fine-Grained Planner), a decision generator (LLM Decision-Maker), a task executor, and a task judge (LLM Judge).

[0029] Specifically, in this embodiment, a task request submitted by the user in natural language is first received. Then, the received task request is semantically parsed to understand its true intent, and the task request is decomposed into multiple specific logical operations. A corresponding coarse-grained subtask is generated based on each logical operation.

[0030] The system invokes the coarse-grained planning submodule (LLM Coarse-Grained Planner) in response to the task request. As input, the semantic reasoning capabilities of the large language model are used to generate a set of several coarse-grained sub-tasks. Each coarse-grained subtask corresponds to an independent logical operation, such as "open the Settings app," "enter the display page," or "switch to dark mode." Based on the information of applications installed in the system, the system determines the specific application to which each coarse-grained subtask belongs. The system then groups each coarse-grained subtask in the task set according to its application, forming an "application-coarse-grained subtask mapping table." And, in the order of execution, they are cached in short-term memory units.

[0031] For example, suppose the task request is: "Change the color display mode of application A from dark mode to follow system mode".

[0032] After semantic parsing the task request through the coarse-grained planning submodule, it was determined that the task request contained the following logical operations: Coarse-grained subtask 1: Open application A.

[0033] Coarse-grained subtask 2. Enter the settings page.

[0034] Coarse-grained subtask 3: Locate the color display method option.

[0035] Coarse-grained subtask 4. Set dark mode to follow system mode.

[0036] Step S12: In the page information database, based on the semantic information of the nth coarse-grained subtask, obtain multiple related page blocks from the page information data set of the application associated with the nth coarse-grained subtask; the page information database contains multiple page information data sets, one page information data set corresponds to one application, and each page information data set is used to store multiple page blocks related to the corresponding application.

[0037] In this embodiment, after dividing the task into multiple coarse-grained subtasks, for each coarse-grained subtask, the relevant page block is retrieved from the page information database. (See reference...) Figure 2 The page information database is a database that stores a large number of application page chunks. In the page information database, each application-related page chunk is stored in a corresponding page information data set. Based on the semantic information of each coarse-grained subtask, the application associated with that coarse-grained subtask is determined, and multiple page chunks associated with that coarse-grained subtask are retrieved from the page information data set corresponding to that application as relevant page chunks.

[0038] Step S13: Based on the multiple related page blocks and the coarse-grained subtasks, generate the nth fine-grained operation sequence corresponding to the nth coarse-grained subtask; the nth fine-grained operation sequence contains multiple action description information for completing the nth coarse-grained subtask.

[0039] In this embodiment, the Fine-Grained Planning (LLM) submodule generates a fine-grained operation sequence for each coarse-grained subtask, combining the multiple relevant page blocks obtained in step S12. The fine-grained operation sequence contains action descriptions of multiple specific actions required to complete the coarse-grained subtask. These action descriptions are generated based on detailed information from the page blocks. Examples include clicking a button or entering text. By refining coarse-grained subtasks into specific fine-grained operation sequences, it is easier to execute task requests more accurately subsequently.

[0040] For example, for coarse-grained subtask 2 (entering the settings page), based on the obtained relevant page blocks "main interface page block" and "settings page block", generate the fine-grained operation sequence for coarse-grained subtask 2: click the button to return to the main interface, find "settings" and click the "settings" button.

[0041] Step S14: Execute the task request based on N fine-grained operation sequences.

[0042] In this embodiment, the task request is executed based on N generated fine-grained operation sequences. Executable operation instructions are generated using the specific action description information in each fine-grained operation sequence. During execution, the operation results are monitored in real time to ensure that each operation instruction is completed correctly. If an operation instruction fails, it is retried or the task is terminated after a preset number of retries until the task request is completed.

[0043] The technical solution described above maintains a page information database containing multiple application-related page blocks, each page block recording detailed page information of the application. When a new task request is received, it is first divided into multiple coarse-grained subtasks based on semantic understanding. Then, for the semantic information of each coarse-grained subtask, the application associated with that subtask is determined. Next, the page information data set associated with that application is found in the page information database, and the multiple page blocks most relevant to the current task are retrieved from this data set to generate a fine-grained operation sequence for the coarse-grained subtask. The task request is then executed sequentially according to this fine-grained operation sequence. This allows the agent to directly perceive the application's interface hierarchy and interaction logic during task execution, significantly improving the agent's understanding of real applications, reducing operational distortion, and significantly increasing the success rate of complex tasks (especially cross-application tasks).

[0044] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another task execution method based on trajectory construction. In this method, step S12, "obtaining multiple related page blocks from the page information data set of the application associated with the nth coarse-grained subtask based on the semantic information of the nth coarse-grained subtask," specifically includes steps S12-1 to S12-3: Step S12-1: Based on the semantic information of the nth coarse-grained subtask and the application associated with the nth coarse-grained subtask, obtain the task context description of the nth coarse-grained subtask.

[0045] In this embodiment, based on the semantic information of the nth coarse-grained subtask, the application associated with the coarse-grained subtask is determined from the application list. Combining the semantic information of the nth coarse-grained subtask and the relevant information of the application associated with the coarse-grained subtask, a task context description of the nth coarse-grained subtask is generated. The task context description not only includes the operation intention of the nth coarse-grained subtask, but also incorporates the relevant information of the application associated with the coarse-grained subtask, thereby more accurately describing the meaning of the coarse-grained subtask in the associated application environment.

[0046] For example, assuming semantic information for coarse-grained subtasks: Go to the settings page.

[0047] Related application: Application A.

[0048] Task context description: In application A, the operation of entering the settings page.

[0049] Step S12-2, calculate the similarity between the task context description of the nth coarse-grained subtask and each page block in the page information data set associated with the application of the nth coarse-grained subtask.

[0050] In this embodiment, in the page information database according to the application associated with the nth coarse-grained subtask, find the page information data set corresponding to the application, and calculate the similarity between the task context description of the nth coarse-grained subtask and each page block included in the page information data set respectively. The status information of each page block is stored in a vectorized form.

[0051] For the calculation of similarity, the cosine similarity can be calculated. Convert the task context description of the nth coarse-grained subtask into a vectorized form and calculate the cosine similarity with the vectorized status information of each page block respectively, to obtain the similarity between the task context description of the nth coarse-grained subtask and each page block. The range of the cosine similarity is between 0 and 1. The higher the similarity, the more relevant the page block is to the nth coarse-grained subtask.

[0052] Step S12-3, sort the multiple page blocks in the page information data set in descending order of similarity to obtain a similarity sorting result, and select the first K page blocks from the similarity sorting result as the multiple relevant page blocks of the nth coarse-grained subtask.

[0053] In this embodiment, after obtaining the similarity between the task context description of the nth coarse-grained subtask and each page block, sort the similarity results in descending order to obtain a similarity sorting result, ensuring that the most relevant page blocks are ranked in the front. And select the first K page blocks from the similarity sorting result as the relevant page blocks. K is a positive integer and not greater than the number of page blocks in the page information data set. K is a preset parameter, indicating the number of the most relevant page blocks that are desired to be considered. Specifically, the process of selecting the first K page blocks as relevant page blocks can be formalized as: Pappi = vj ∈ Di | rank(vj, Emb(Sappi)) < k, where Di represents the page information data set corresponding to application i, vj ∈ Di represents the jth vectorized page block in the page information data set, Emb() is a vectorization model function, and rank represents the ranking of vector vj based on cosine similarity under the query Sappi.

[0054] For example, assuming K=2, when the task context description of the nth coarse-grained subtask is entering the settings page of application A, the top two page blocks in the similarity ranking results are selected as the relevant page blocks: 1. Page Block 3: Similarity 0.9 (Describes the settings page of application A); 2. Page Block 1: Similarity 0.7 (Describes the main interface of application A); 3. Page block 2: Similarity 0.1 (Describes the functional interface of application A).

[0055] Therefore, page block 3 and page block 1 are selected as the relevant page blocks for the nth coarse-grained subtask.

[0056] It should be noted that, in order to avoid introducing irrelevant page blocks when the number of page blocks in the page information data set is small, after selecting K page blocks, it is necessary to determine whether the similarity of the selected K page blocks is greater than a preset similarity threshold (e.g., 0.3). If there are page blocks among the selected K page blocks whose similarity is not greater than the preset similarity threshold, the page blocks whose similarity is not greater than the preset similarity threshold are removed and are not considered as relevant page blocks. That is, the number of relevant page blocks selected in the end is no greater than K and the similarity of each relevant page block is greater than the preset similarity threshold.

[0057] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another task execution method based on trajectory construction. In this method, the state information of the page block includes at least the following fields: page description, key UI elements, path information, and page tags. The step S14, "execute the task request based on N fine-grained operation sequences," specifically includes steps S14-1 to S14-4: Step S14-1: Generate the nth operation instruction based on the nth fine-grained operation sequence, the first page observation information, and the operation records of the previous n-1 operation instructions.

[0058] In this embodiment, since the fine-grained operation sequence is text-based action description information generated based on task logic, but in actual execution, it is necessary to consider the specific state of the current page and the previously executed operation records, and convert them into executable operation instructions. For example, the "Settings" button may differ in different screen positions or states. Based on this, for the nth fine-grained operation sequence, it is necessary to combine the n fine-grained operation sequences, the first page observation information, and the first n-1 operation instructions to obtain the nth operation instruction corresponding to the nth coarse-grained subtask.

[0059] For example, for the fine-grained operation sequence "click the settings button of application A", the current page observation information is "currently displaying the personal page of application A, with the 'Settings' button located at the bottom of the personal page", and the previous operation record is "successfully clicked the 'Me' button on the main interface to enter the personal page". By combining the specific state of the current page (the first page observation information) and the previously executed operation records (the first n-1 operation instructions), a specific and executable operation instruction that can realize "click the settings button of application A" is generated: "click the 'Settings' button at the bottom of the personal page".

[0060] Step S14-2: Execute the nth operation instruction and evaluate the execution result of the nth operation instruction to obtain the operation evaluation result of the nth operation instruction.

[0061] In this embodiment, the sequential execution of multiple coarse-grained subtasks is maintained through the task order of the short-term memory unit. After generating operation instructions, the generated operation instructions are executed, and the execution results are evaluated. The purpose of operation evaluation is to determine whether the operation was successfully completed and to record the operation evaluation results. The operation evaluation results include whether the operation was successful, the progress of the current task, and the recommended next operation, helping the decision generator to dynamically adjust operation instructions and ensure that task requests can be processed smoothly.

[0062] Step S14-3: After the nth coarse-grained subtask is completed, the page observation information, actions, inference text and operation evaluation results generated during the execution of the nth coarse-grained subtask are integrated to obtain the task trajectory information of the nth coarse-grained subtask.

[0063] In this embodiment, after the nth coarse-grained subtask is completed, the page observation information, actions, inference text, and operation evaluation results generated during the execution of the nth coarse-grained subtask are integrated to form the task trajectory information of the nth coarse-grained subtask. The task trajectory information contains all relevant information for completing the coarse-grained subtask and is used for subsequent analysis and reuse.

[0064] The task trajectory information of the nth coarse-grained subtask may contain multiple items, and each task trajectory information corresponds to the execution of an operation instruction. Specifically, if the operation instruction of the nth coarse-grained subtask fails to execute, a new operation instruction will be generated and re-executed until the coarse-grained subtask is successfully executed. The execution process of each operation corresponds to a task trajectory information.

[0065] Step S14-4: Semantic compression and vectorization encoding are performed on the state information of each page contained in the task trajectory information to obtain new page blocks, and the new page blocks are stored in the corresponding page information data set for reuse in new task requests.

[0066] In this embodiment, the state information of each page contained in the task trajectory information is semantically compressed to generate new page blocks, and these new page blocks are stored in the corresponding page information data set. Through semantic compression, the core content of the page can be extracted and redundant information can be removed, making the resulting page blocks more concise and efficient. Storing new page blocks can be reused for future task requests, enhancing the system's memory capacity and reuse efficiency, and realizing self-learning and memory accumulation.

[0067] Through the above embodiments, by dynamically generating coarse-grained subtasks into fine-grained operation sequences, executing operation instructions, and evaluating operation results in real time, the system can better adapt to complex mobile task scenarios, improving the system's robustness and adaptability.

[0068] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another task execution method based on trajectory construction. In this method, the step S14-1 of "generating the nth operation instruction based on the nth fine-grained operation sequence, the first page observation information, and the operation records of the previous n-1 operation instructions" specifically includes: Step S14-1-1: The decision generator generates inference text based on the nth fine-grained operation sequence, the first page observation information, and the operation records of the previous n-1 operation instructions, and generates the nth operation instruction according to the inference text.

[0069] In this embodiment, the operation execution is generated by a decision generator (LLM Decision-Maker), a module based on a Large Language Model (LLM) that utilizes natural language processing techniques to generate operation instructions. Specifically, it receives three main inputs: the current fine-grained operation sequence (the nth fine-grained operation sequence), the observation information of the current page (the first page observation information), and the operation records of previous and executed operation instructions (the operation records of the first n-1 operation instructions). These inputs provide the decision generator with the context and current state of the subtask. By understanding these inputs, the decision generator generates a reasoning text, which represents its thought process in analyzing the inputs. Then, based on this reasoning text, the decision generator generates specific, executable operation instructions. In this way, the decision generator can generate accurate operation instructions that are adapted to the current state.

[0070] For example, suppose the current coarse-grained subtask is "enter the settings page of application A", and the fine-grained operation sequence indicates that the "Settings" button needs to be clicked. The current page observation information shows that we are currently on the personal page of application A, and the operation record of the previously executed operation instructions shows that the "Me" button on the personal page has been successfully clicked.

[0071] Then, the decision generator combines this input information to generate inference text: "Because the current page is application A's personal page, the settings button is located at the bottom of the page, clicking the settings button will take you to the settings page." Based on this inference text, the decision generator generates the operation instruction: "Click the 'Settings' button at the top of the screen." This achieves the generation of inference text using input information and the generation of accurate operation instructions based on the inference text, forming a "think-action" structure, symbolically expressed as: (Θt,at)=DM(Tfg,ot), where Θt represents the inference text behind the action at in the operation instruction, Tfg represents the fine-grained operation sequence, and ot represents the first page observation information. Since the current operation instruction corresponds to the first coarse-grained subtask, DM() only contains the fine-grained operation sequence and the first page observation information.

[0072] The step S14-2, "evaluating the execution result of the nth operation instruction to obtain the operation evaluation result of the nth operation instruction," specifically includes: Step S14-2-1: The task reviewer evaluates the page state changes between the first page observation information and the second page observation information to obtain the operation evaluation result of the nth operation instruction; the operation evaluation result is used to indicate whether the nth operation instruction was successfully executed, the current task progress, and the next recommended action; The first page observation information is the observation information of the current page before the execution of the nth operation instruction; the second page observation information is the observation information of the current page after the execution of the nth operation instruction.

[0073] In this embodiment, the task reviewer is also a module generated based on a Large Language Model (LLM), responsible for evaluating the execution results of operation instructions. Specifically, the task reviewer receives two inputs: page observation information before executing the operation instruction (first page observation information) and page observation information after executing the operation instruction (second page observation information). By comparing these two page observations, the task reviewer can evaluate the changes in page state before and after the operation instruction is executed, thereby determining whether the operation instruction was successful. The operation evaluation result includes whether the operation was successful, the progress of the current task, and the recommended next action. The task reviewer uses natural language processing technology to understand and compare page observation information, thereby generating accurate evaluation results. In this way, the task reviewer can provide real-time feedback to help the decision generator dynamically adjust subsequent operation instructions.

[0074] For example, suppose the executed command is "tap the 'Settings' button at the bottom of the screen." The first page observation information shows that the user is currently on the profile page of application A, while the second page observation information shows that the user has entered the settings page of application A. The task reviewer compares these two page observation information and generates an operation evaluation result: "Operation successful, entered the settings page. Current task progress: The operation of entering the settings page has been completed. The next step is to find the 'Dark Mode' option in the settings page." Thus, by comparing the page observation information before and after the execution of the command, the task reviewer evaluates the execution result of the command and provides guidance for subsequent operations.

[0075] If the operation command fails to execute, the generated operation evaluation result is: "Operation failed, did not enter the settings page; Current task progress: Currently in the personal page of application A; Next step recommendation: Relocate the actual position of the "Settings" button in the settings page and click it." In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another task execution method based on trajectory construction. In this method, step S14-3, "integrating the page observation information, actions, inference text, and operation evaluation results generated during the execution of the nth coarse-grained subtask to obtain the task trajectory information of the nth coarse-grained subtask," includes: Step S14-3-1: If the nth coarse-grained subtask is successfully executed, the page observation information, actions, inference text corresponding to the successfully executed nth operation instruction, and operation evaluation results corresponding to the successfully executed nth operation instruction generated during the execution process are integrated to obtain the task trajectory information of the nth coarse-grained subtask.

[0076] In this embodiment, to ensure that the page blocks stored subsequently have reuse value, the system can generate and store only the page blocks corresponding to the successfully executed operation instructions. This makes it easier to directly refer to the page blocks corresponding to these successfully executed operation instructions when encountering similar task requests in the future, quickly generate operation instructions, improve the efficiency of task execution, and avoid storing a large amount of useless failure information, thus reducing data redundancy.

[0077] Information corresponding to failed operation instructions can be stored in a separate database and used for subsequent task anomaly analysis. For example, if an operation instruction of a coarse-grained subtask fails multiple times, the information corresponding to the failed operation instructions can be analyzed to determine whether the failure is due to network problems, application version incompatibility, or other reasons, which helps to optimize and repair the system.

[0078] The information corresponding to the operation instructions includes: page observation information, actions, inference text, and operation evaluation results. Page observation information represents the state description information of each page involved in the execution of the operation instructions, which is used to record the specific environment during the execution of the operation instructions. Actions represent the behavior of the actual executed operation instructions, such as clicking or swiping. Inference text is the inference text generated by the decision generator for the execution of the operation. Operation evaluation results are the evaluation of the execution results of the operation instructions by the task reviewer.

[0079] Through the above embodiments, by generating task trajectory information and storing page blocks in long-term memory, the system can acquire experience from successfully executed tasks, avoid repeated trial and error, significantly improve task execution efficiency, and reduce inefficient decision-making.

[0080] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another task execution method based on trajectory construction. In this method, after obtaining the operation evaluation result of the nth operation instruction in step S14-2, the method further includes steps S21 to S24: Step S21: The task reviewer feeds back the operation evaluation result of the nth operation instruction to the decision generator.

[0081] In this embodiment, after generating the operation evaluation result, the task reviewer will feed the operation evaluation result back to the decision generator so that the decision generator can understand the execution status of the operation instructions in real time based on the operation evaluation result and determine whether the operation instructions need to be adjusted.

[0082] Step S22: Using a decision generator, if the operation evaluation result indicates that the nth coarse-grained subtask has failed, the nth operation instruction is modified based on the operation evaluation result to obtain the updated nth inference text and the updated nth operation instruction.

[0083] In this embodiment, the decision executor can modify the operation instructions based on the operation evaluation results. If the operation evaluation results indicate that the coarse-grained subtask has failed, the decision executor will regenerate the updated inference text and updated operation instructions based on the operation evaluation results, thereby dynamically adjusting the operation instructions according to the current page observation information and the reason for failure to improve the success rate of task execution.

[0084] For example, suppose the operation evaluation result indicates that clicking the 'Settings' button at the bottom of the screen fails because the actual location of the settings button is not as expected. The decision executor regenerates the inference text and operation instruction based on this evaluation result. The generated updated inference text is: "Because the current page is application A's personal page, the settings button is located in the middle of the page; clicking the settings button will take you to the settings page." The updated operation instruction is: "Click the 'Settings' button in the middle of the page."

[0085] Step S23: Execute the updated nth operation instruction according to the updated nth reasoning text through the task executor.

[0086] In this embodiment, after generating the updated operation instructions, the operation instructions are sent to the task executor, which then calls the Android Debug Bridge (ADB) interface to execute the corresponding operation instructions (click, input, scroll, etc.).

[0087] Step S24: After the task executor executes the updated nth operation instruction, the task reviewer evaluates the page state change between the third page observation information and the fourth page observation information to obtain the operation evaluation result of the updated nth operation instruction. The third page observation information refers to the observation information of the current page before the execution of the updated nth operation instruction; the fourth page observation information refers to the observation information of the current page after the execution of the updated nth operation instruction.

[0088] In this embodiment, after the updated operation instruction is executed, the updated operation instruction also needs to be evaluated by the task reviewer to obtain the operation evaluation result of the updated nth operation instruction. This process is referred to step S14-2-1 and will not be repeated here.

[0089] Through the technical solutions described in the above embodiments, the operation evaluation results can be input again into the decision generator to correct subsequent operation instructions, thereby achieving dynamic closed-loop control. This dual LLM structure of the decision generator and task reviewer forms a three-stage intelligent execution loop of "operation instruction generation – operation result review – operation instruction correction," significantly improving the robustness and safety of execution.

[0090] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another task execution method based on trajectory construction. In this method, the state information of the page block further includes a timestamp, and the method further includes steps S31 to S32: Step S31: Calculate the similarity between any two page blocks in the multiple page blocks contained in each page information data set to obtain the similarity result between any two page blocks.

[0091] In this embodiment, during the execution of multiple task requests, a large number of page blocks are stored for each application. To avoid redundant storage of identical page blocks, the similarity between any two page blocks within each page information dataset is calculated. Each page block contains a page description, key UI elements, path information, page tags, and status information such as timestamps. By calculating similarity, it is possible to determine whether two page blocks are highly similar in content. Similarity can be measured using natural language processing techniques, such as cosine similarity and Jaccard similarity, to measure the degree of similarity between text, thereby identifying potentially duplicated or highly similar page blocks in the page information dataset.

[0092] Step S32: If the similarity between two page blocks is greater than the preset similarity threshold, delete the page block with the earlier timestamp based on the timestamps of the two page blocks.

[0093] In this embodiment, based on the calculated similarity results, if the similarity between two page blocks is found to be greater than a preset similarity threshold (e.g., 0.95), the two page blocks are considered highly similar in content. In this case, it is necessary to further delete the page block with the earlier timestamp based on the timestamps of when the two page blocks were stored. The timestamp records the generation time of the page block. When storing page blocks, the timestamp of the generated page block needs to be stored as state information as well. By deleting the page block with the earlier timestamp when removing redundant page blocks, it can be ensured that the database stores the latest page block information, so as to maintain the sparsity and timeliness of the database. This process forms a closed-loop memory accumulation mechanism, enabling the system to continuously strengthen its own knowledge base during long-term operation and achieve continuous evolution of "use once, remember once".

[0094] Furthermore, by storing page blocks, these experiences can be reused in future tasks, reducing redundant calculations and improving the scalability of the system.

[0095] Based on the various technical solutions shown in the above embodiments, in order to verify the effectiveness and robustness of this application when performing complex tasks in a real mobile device environment, the standardized public benchmarks SPA-Bench and CHOP datasets, which include both Chinese and English scenarios, were used for experimental evaluation.

[0096] The evaluation metric for this application is the success rate. The experimental system was deployed in a Windows 11 environment, with the host equipped with an i5-12600KF processor and an NVIDIA RTX 4070 Super GPU. Testing was conducted on a real smartphone device, specifically an Xiaomi 14. The system executed commands via the Android Debug Bridge (ADB) interface. Milvus was used for storing page information in the database, and the text-embedding-v3 embedding model was employed. This application uses the large language model GPT-4o as the unified inference core to ensure fairness and consistency in the experimental comparisons. Quantitative comparative tests were conducted on the aforementioned datasets, and the quantitative results are shown in Table 1. This application outperforms the existing state-of-the-art MobileAgentV2 method on both the SPA-Bench and CHOP datasets, demonstrating a higher success rate and superior robustness.

[0097] Table 1

[0098] In summary, this application proposes a trajectory-based memory-enhanced task execution method. By transforming the task trajectory information of historical task requests into reusable page blocks, it achieves long-term accumulation and semantic reuse of task knowledge. Combined with a memory-enhanced hierarchical task planning mechanism, task requests are sequentially divided into multiple coarse-grained subtasks and fine-grained operation sequences. Furthermore, a dual LLM collaborative execution architecture—a decision generator and a task reviewer—implements a three-stage intelligent execution loop for operation instructions. This enables high-precision task understanding and dynamic correction of generated operation instructions in complex mobile environments. The experimental results clearly demonstrate that this application significantly improves the success rate and execution stability of task requests across multiple languages ​​and application scenarios, overcoming the limitations of existing mobile agents in planning illusion and cross-task generalization. It has broad application value and promotion potential in various intelligent terminal application scenarios such as mobile intelligent assistants and human-computer collaboration.

[0099] In summary, the key point of this application lies in the deep integration of the semantic planning capability of the large language model with the long-term memory storage mechanism of page blocks in the historical task execution process based on trajectory. Through the hierarchical planning of sub-tasks, the task requests are divided in a coarse-to-fine process. Then, by combining dual LLM collaboration and storing the new page blocks, a closed-loop memory reconstruction is achieved, realizing the intelligent evolution in the field of mobile task automation.

[0100] First, this application generates page memory units, namely page blocks, by recording and structurally analyzing the task trajectory information during the execution of mobile tasks. These units consist of page descriptions, interface elements, operation paths, semantic tags, and timestamps. The page blocks are then stored in the corresponding page information data set in the page information database in vector form, thereby enabling the long-term accumulation and reuse of historical task experience.

[0101] Secondly, this application designs a two-stage hierarchical task planning method that is "from coarse to fine". First, the task scheduling module performs task intent recognition and task decomposition to obtain multiple coarse-grained subtasks. Then, fine-grained operation sequences are generated by retrieving page blocks and combining them with large language model reasoning. This enables the agent to make up for the lack of real perception of the interface hierarchy, interaction logic and operation path of the existing agent in real applications, and solves the problem of "planning illusion" that existing LLM is prone to in mobile tasks.

[0102] In addition, to improve the stability and self-correction capability of the agent in multi-step interactive tasks, this application designs a dual LLM execution architecture consisting of a decision generator and a task reviewer. The former is responsible for generating the steps of reasoning and operation instructions, while the latter performs state evaluation and correction feedback on the execution results of the task's operation instructions, which significantly reduces the error rate.

[0103] The method designed in this application, which enhances planning by generating task trajectory information and storing page blocks, improves the accuracy and success rate of task automation in dynamic and complex real mobile device environments, and has good potential for engineering applications.

[0104] Based on the above innovations, the content to be protected in this application includes, but is not limited to: a framework for constructing structured page blocks based on task trajectories, a coarse-to-fine subtask planning mechanism combined with memory enhancement, and a task execution and review architecture with dual LLM collaboration.

[0105] It should be noted that the task execution method based on trajectory construction provided in this application embodiment can be executed by a task execution device based on trajectory construction, or a control module in the task execution device for executing the task execution method based on trajectory construction. This application embodiment uses the execution of the task execution method based on trajectory construction by the task execution device as an example to illustrate the task execution method based on trajectory construction provided in this application embodiment.

[0106] Figure 3 This is a schematic diagram of the framework of a task execution device based on trajectory construction according to an embodiment of this application, with reference to... Figure 3 One embodiment of this application provides a task execution device based on trajectory construction, the device comprising: The coarse-grained subtask partitioning module 11 is used to perform semantic parsing on the user's task request to obtain N coarse-grained subtasks, each of which is a logical operation; N is an integer not less than 1. The related page block acquisition module 12 is used to acquire multiple related page blocks from the page information data set of the application associated with the nth coarse-grained subtask in the page information database, based on the semantic information of the nth coarse-grained subtask; the page information database contains multiple page information data sets, one page information data set corresponds to one application, and each page information data set is used to store multiple page blocks related to the corresponding application. The fine-grained operation sequence generation module 13 is used to generate the nth fine-grained operation sequence corresponding to the nth coarse-grained subtask based on the multiple related page blocks and the coarse-grained subtask; the nth fine-grained operation sequence includes multiple action description information for completing the nth coarse-grained subtask; The task execution module 14 is used to execute the task request based on N fine-grained operation sequences.

[0107] The task execution device based on trajectory construction in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0108] The task execution device based on trajectory construction in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0109] The trajectory-based task execution device provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented by the trajectory-based task execution device in the method embodiment will not be described again here to avoid repetition.

[0110] Optionally, Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. This application also provides an electronic device; it should be noted that the electronic device in this application includes the mobile electronic device and non-mobile electronic device described above.

[0111] The electronic device includes, but is not limited to, components such as: radio frequency unit, network module, audio output unit, input unit, sensor, display unit, user input unit, interface unit, memory, and processor.

[0112] Those skilled in the art will understand that electronic devices may also include power supplies (such as batteries) that supply power to various components. The power supply may be connected to the processor logic through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0113] As an example, such as Figure 4 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps in the trajectory-based task execution method disclosed in the above embodiments of this application.

[0114] As the apparatus is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0116] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects.

[0117] Furthermore, this application embodiment also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described trajectory-based task execution method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0118] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0119] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described trajectory-based task execution method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0120] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0121] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0123] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A task execution method based on trajectory construction, characterized in that, The method includes: Semantic parsing is performed on the user's task request to obtain N coarse-grained subtasks, each of which is a logical operation; N is an integer not less than 1. In the page information database, based on the semantic information of the nth coarse-grained subtask, multiple related page blocks are obtained from the page information data set of the application associated with the nth coarse-grained subtask; the page information database contains multiple page information data sets, one page information data set corresponds to one application, and each page information data set is used to store multiple page blocks related to the corresponding application; Based on the multiple related page blocks and the coarse-grained subtasks, generate the nth fine-grained operation sequence corresponding to the nth coarse-grained subtask; the nth fine-grained operation sequence contains multiple action description information for completing the nth coarse-grained subtask; The task request is executed based on N fine-grained operation sequences.

2. The task execution method based on trajectory construction according to claim 1, characterized in that, The state information of the page block includes at least the following fields: page description, key UI elements, path information, and page tags. The execution of the task request based on N fine-grained operation sequences includes: The nth operation instruction is generated based on the nth fine-grained operation sequence, the observation information on the first page, and the operation records of the previous n-1 operation instructions; Execute the nth operation instruction, and evaluate the execution result of the nth operation instruction to obtain the operation evaluation result of the nth operation instruction; After the nth coarse-grained subtask is completed, the page observation information, actions, inference text and operation evaluation results generated during the execution of the nth coarse-grained subtask are integrated to obtain the task trajectory information of the nth coarse-grained subtask. The state information of each page contained in the task trajectory information is semantically compressed to obtain a new page block, and the new page block is stored in the corresponding page information data set for reuse in new task requests.

3. The task execution method based on trajectory construction according to claim 1, characterized in that, The step involves obtaining multiple related page blocks from the page information data set of the application associated with the nth coarse-grained subtask based on the semantic information of the nth coarse-grained subtask, including: Based on the semantic information of the nth coarse-grained subtask and the application associated with the nth coarse-grained subtask, the task context description of the nth coarse-grained subtask is obtained; Calculate the similarity between the task context description of the nth coarse-grained subtask and each page block in the page information data set of the application associated with the nth coarse-grained subtask; The multiple page blocks in the page information data set are sorted in descending order of similarity to obtain a similarity ranking result. The top K page blocks in the similarity ranking result are selected as multiple related page blocks for the nth coarse-grained subtask.

4. The task execution method based on trajectory construction according to claim 2, characterized in that, The process of generating the nth operation instruction based on the nth fine-grained operation sequence, the first page observation information, and the operation records of the previous n-1 operation instructions includes: The decision generator generates inference text based on the nth fine-grained operation sequence, the observation information of the first page, and the operation records of the previous n-1 operation instructions. Then, it generates the nth operation instruction according to the inference text. The step of evaluating the execution result of the nth operation instruction to obtain the operation evaluation result of the nth operation instruction includes: The task reviewer evaluates the page state changes between the observation information on the first page and the observation information on the second page to obtain the operation evaluation result of the nth operation instruction; the operation evaluation result is used to indicate whether the nth operation instruction was successfully executed, the current task progress, and the next recommended action; The first page observation information is the observation information of the current page before the execution of the nth operation instruction; the second page observation information is the observation information of the current page after the execution of the nth operation instruction.

5. The task execution method based on trajectory construction according to claim 4, characterized in that, After obtaining the operation evaluation result of the nth operation instruction, the method further includes: The task reviewer feeds back the operation evaluation result of the nth operation instruction to the decision generator. In the case where the operation evaluation result indicates that the nth coarse-grained subtask has failed, the decision generator modifies the nth operation instruction based on the operation evaluation result to obtain the updated nth inference text and the updated nth operation instruction. The task executor executes the updated nth operation instruction according to the updated nth reasoning text; After the task executor executes the updated nth operation instruction, the task reviewer evaluates the page state changes between the third page observation information and the fourth page observation information to obtain the operation evaluation result of the updated nth operation instruction. The third page observation information refers to the observation information of the current page before the execution of the updated nth operation instruction; the fourth page observation information refers to the observation information of the current page after the execution of the updated nth operation instruction.

6. The task execution method based on trajectory construction according to claim 1, characterized in that, The status information of the page block also includes a timestamp, and the method further includes: Calculate the similarity between any two page blocks in the multiple page blocks contained in each page information data set to obtain the similarity result between any two page blocks; If the similarity between two page blocks exceeds a preset similarity threshold, the page block with the earlier timestamp will be deleted based on the timestamps of the two page blocks.

7. The task execution method based on trajectory construction according to claim 2, characterized in that, The process of integrating page observation information, actions, inference text, and operation evaluation results generated during the execution of the nth coarse-grained subtask to obtain the task trajectory information of the nth coarse-grained subtask includes: If the nth coarse-grained subtask is successfully executed, the page observation information, actions, inference text corresponding to the successfully executed nth operation instruction, and operation evaluation results corresponding to the successfully executed nth operation instruction generated during the execution process are integrated to obtain the task trajectory information of the nth coarse-grained subtask.

8. A task execution device based on trajectory construction, characterized in that, The device includes: The coarse-grained subtask partitioning module is used to perform semantic parsing on the user's task requests to obtain N coarse-grained subtasks, each of which is a logical operation; N is an integer not less than 1. The related page block acquisition module is used to acquire multiple related page blocks from the page information data set of the application associated with the nth coarse-grained subtask in the page information database, based on the semantic information of the nth coarse-grained subtask; the page information database contains multiple page information data sets, one page information data set corresponds to one application, and each page information data set is used to store multiple page blocks related to the corresponding application. The fine-grained operation sequence generation module is used to generate the nth fine-grained operation sequence corresponding to the nth coarse-grained subtask based on the multiple related page blocks and the coarse-grained subtask; the nth fine-grained operation sequence includes multiple action description information for completing the nth coarse-grained subtask; The task execution module is used to execute the task request based on N fine-grained operation sequences.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the trajectory-based task execution method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the trajectory-based task execution method as described in any one of claims 1-7.

Images