A task execution method and device

Abstract

Embodiments of the present application provide a task execution method and device, and relate to the technical field of artificial intelligence. The method comprises: obtaining a matched set of action block templates from an action block library based on a task execution instruction input by a user and environment state information; constructing a prompt text based on the task execution instruction, the environment state information and the set of action block templates; inputting the prompt text as a prompt instruction into a large language model to obtain task execution planning information, the task execution planning information comprising a sequence of action blocks to be executed; executing the sequence of action blocks in a task execution environment to obtain a task execution result. The entire task execution process only needs to call the large language model once, greatly compressing the decision space of the large language model and simplifying the planning problem. Furthermore, the task execution instruction is efficiently and reliably executed using the solidified action block library, and the entire process has high reliability, extremely low cost and fast execution speed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a task execution method and apparatus. Background Technology

[0002] As artificial intelligence (AI) continues to develop and permeate all aspects of people's lives, people expect AI agents to automatically complete complex, multi-step tasks (also known as long-chain tasks) through graphical user interfaces, much like skilled human employees. For example, completing the entire shopping process on an e-commerce platform, from logging in, searching, comparing prices, placing an order, to making a payment; or completing a series of office tasks in a corporate system, such as data querying, report generation, and email sending.

[0003] In related technologies, dynamic hierarchical planning is usually used to handle long-chain tasks. Specifically, the task is first dynamically divided into multiple sub-goals by a large language model (LLM), and then another (or the same) LLM dynamically plans the atomic steps of each sub-goal.

[0004] However, in the above process, completing a long task requires dozens of calls to the application programming interface (API) of an expensive LLM (such as GPT-4), resulting in high costs and significant latency, making it impossible to deploy at scale; moreover, it is difficult to cope with dynamic environmental changes such as UI updates, unexpected pop-ups, and network latency, resulting in poor robustness. Summary of the Invention

[0005] This application provides a task execution method and apparatus to improve the execution reliability and fault tolerance of long-chain tasks.

[0006] On one hand, embodiments of this application provide a task execution method, the method comprising: Receive task execution instructions from the user and obtain environmental status information of the task execution environment; Based on the task execution instructions and the environment state information, a matching action block template set is obtained from the action block library. Each action block template in the action block template set includes: multiple pre-encapsulated atomic operations. A prompt text is constructed based on the task execution instruction, the environmental status information, and the action block template set; Using the prompt text as a prompt instruction, inputting it into a large language model for reasoning, and obtaining task execution planning information, the task execution planning information includes a sequence of action blocks to be executed; The action block sequence is executed in the task execution environment to obtain the task execution result.

[0007] On one hand, embodiments of this application provide a task execution apparatus, the apparatus comprising: The receiving module is used to receive the task execution instructions input by the user and to obtain the environmental status information of the task execution environment; The acquisition module is used to acquire a matching action block template set from the action block library based on the task execution instruction and the environment state information. Each action block template in the action block template set includes: multiple pre-encapsulated atomic operations. The execution module is used to construct prompt text based on the task execution instruction, the environment state information, and the action block template set; use the prompt text as a prompt instruction, input it into a large language model for reasoning, and obtain task execution planning information, which includes a sequence of action blocks to be executed; execute the action block sequence in the task execution environment to obtain the task execution result.

[0008] Optionally, the acquisition module is specifically used for: Obtain multiple candidate operation sequences from log data; Feature extraction is performed on each candidate operation sequence to obtain the corresponding semantic vector; Based on the semantic vectors of the multiple candidate operation sequences, the multiple candidate operation sequences are clustered to obtain multiple operation sequence groups; For each operation sequence group, the common structure of each candidate operation sequence in the operation sequence group is extracted, and the parameters of the common structure are constructed to obtain the action block template; Save the action block template to the action block library.

[0009] Optionally, the acquisition module is specifically used for: Obtain multiple raw operation sequences from the log data; For each original operation sequence, each atomic operation in the original operation sequence is mapped to its corresponding operation type to obtain a mapped operation sequence; The mapping operation sequence that meets the preset screening conditions is taken as the candidate operation sequence. The preset screening conditions include: the product of the occurrence frequency and the success rate is greater than a preset threshold.

[0010] Optionally, the acquisition module is specifically used for: Determine the comprehensive semantic similarity between the semantic vectors of any two candidate operation sequences, wherein the semantic vectors include at least one of the following: operation intention features, contextual environment features, target state change features, and temporal relationship features; Based on the obtained comprehensive semantic similarity, the multiple candidate operation sequences are clustered to obtain multiple operation sequence groups.

[0011] Optionally, the acquisition module is specifically used for: Align each candidate operation sequence in the operation sequence group to obtain a corresponding alignment matrix. The alignment matrix includes multiple alignment positions, and the multiple alignment positions form the common structure. For each alignment position, the variable type of the alignment position is obtained based on the differences between the candidate operation sequences at that alignment position; Construct a parameterized template for each alignment position according to the variable type of each alignment position; The action block template is obtained based on the parameterized templates corresponding to the multiple alignment positions.

[0012] Optionally, the execution module is further configured to: When one action block in the action block sequence fails to execute, the execution information of each atomic operation in the action block is obtained; Based on the execution information of each atomic operation in the action block, the atomic operations that failed to execute in the action block are obtained, and the atomic operations are marked as breakpoints where the task is paused.

[0013] Optionally, the execution module is specifically used for: Use at least one of the following recovery strategies to recover failed action blocks: Adjust the remaining atomic operations in the failed action block and continue executing the adjusted remaining atomic operations from the breakpoint; The failed action block is replaced with a substitute block, and the remaining action blocks in the action block sequence are continued from the substitute block. Re-plan the remaining action blocks in the action block sequence that follow the failed action block, and re-execute the updated action block sequence; Revert to the previous stable checkpoint and continue execution of the action block sequence from the previous stable checkpoint, where the stable checkpoint is an atomic operation or an action block.

[0014] Optionally, the execution module is further configured to: When the frequency of consecutive execution of multiple action block templates exceeds a preset threshold and the execution success rate exceeds a preset success rate, the multiple action block templates are encapsulated into a super block template and the super block template is stored in the action block library. When the number of execution failures of an atomic operation in an action block module exceeds a preset number, the action block module is split into multiple sub-action block templates, and the multiple sub-action block templates are stored in the action block library.

[0015] Optionally, the execution module is further configured to: When the action block library does not contain an action block template that matches the task execution instruction and the environment state information, multiple candidate paths are generated based on the task execution instruction and the environment state information. Each candidate path includes at least one of the following: atomic operation and basic action block. Execute each candidate path in the test environment and obtain the corresponding execution results; Based on the execution results of each candidate path, a comprehensive score is obtained for each candidate path; The candidate path with the highest comprehensive score is generalized to obtain an action block template, and the action block template is saved to the action block library.

[0016] On one hand, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described task execution method.

[0017] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described task execution method.

[0018] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described task execution method.

[0019] In this embodiment, based on the user-input task execution command and environmental state information, a matching action block template set is obtained from the action block library. Then, a prompt text is constructed based on the task execution command, environmental state information, and action block template set. This prompt text serves as the prompt command, which is input into a large language model for reasoning to obtain task execution planning information. Only one call to the large language model is needed, and the decision space of the large language model is compressed from hundreds of atomic operations to dozens of action blocks with clear business semantics, greatly simplifying the planning problem. The action block sequence is executed in the task execution environment to obtain the task execution result. Utilizing a pre-defined action block library, the execution task is executed efficiently and reliably, with high reliability, extremely low cost, and fast execution speed throughout the entire process. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram of a system architecture provided in an embodiment of this application; Figure 2 A flowchart illustrating a task execution method provided in an embodiment of this application; Figure 3 A flowchart illustrating the generation of an action block template provided in an embodiment of this application; Figure 4 This is a schematic diagram of another task execution method provided in an embodiment of this application; Figure 5 A flowchart illustrating an offline mining and solidification process provided in an embodiment of this application; Figure 6 A flowchart illustrating an online planning and execution process provided in this application embodiment; Figure 7 A flowchart illustrating a continuous evolution process provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a task execution device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] The following is a brief introduction to the system architecture diagram applicable to the technical solutions of the embodiments of this application. It should be noted that the system architecture diagram described below is only used to illustrate the embodiments of this application and is not intended to limit the scope of the application.

[0024] refer to Figure 1 This is a system architecture diagram applicable to the embodiments of this application. The system architecture includes at least terminal device 101 and server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. This application does not specifically limit the number of terminal devices 101 and servers 102.

[0025] Terminal device 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart home appliance, smart voice interaction device, smart vehicle device, etc., but is not limited to these.

[0026] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminal device 101 and server 102 can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this.

[0027] It should be noted that the method in this embodiment can be executed by terminal device 101 or server 102 alone, or by terminal device 101 and server 102 interacting. When executed by terminal device 101 or server 102 alone, the following steps are performed: receiving a task execution instruction input by the user and obtaining the environmental state information of the task execution environment; based on the task execution instruction and the environmental state information, obtaining a matching action block template set from the action block library; constructing a prompt text based on the task execution instruction, environmental state information, and action block template set; using the prompt text as a prompt instruction, inputting it into a large language model for reasoning to obtain task execution planning information; executing the action block sequence in the task execution environment to obtain the task execution result.

[0028] When the server 102 and the terminal device 101 interact, the terminal device 101 displays an operation interface. After the user inputs a task execution command on the operation interface, the terminal device 101 sends the task execution command to the server 102. The server 102 executes the task execution method of this embodiment based on the task execution command, and the two interact to complete the long-chain task execution process. In practical applications, specific configurations can be made according to the situation, and this application does not make specific limitations here.

[0029] Both server 102 and terminal device 101 may include one or more processors, memory, and interactive I / O interfaces. Furthermore, server 102 may be configured with a database to store model parameters of large language models. The memory of server 102 and terminal device 101 may also store program instructions required for execution in the task execution method provided in this application embodiment. These program instructions, when executed by the processor, can be used to implement the task execution process provided in this application embodiment.

[0030] It should be noted that when the task execution method provided in this application embodiment is executed by either server 102 or terminal device 101 alone, the system architecture of this application may also include only a single device, either server 102 or terminal device 101. Alternatively, server 102 and terminal device 101 may be considered as the same device. Of course, in practical applications, when the task execution method provided in this application embodiment is executed jointly by server 102 and terminal device 101, server 102 and terminal device 101 may also be the same device. That is, server 102 and terminal device 101 may be different functional modules of the same device, or virtual devices virtualized by the same physical device.

[0031] In this embodiment, the terminal device 101 and the server 102 can communicate directly or indirectly through one or more networks. The network can be a wired network or a wireless network; for example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network, or other possible networks. This embodiment does not limit the types of networks used.

[0032] based on Figure 1 The system architecture diagram shown in this application illustrates the flow of a task execution method. Figure 2 As shown, the process of this method is executed by a computer device, which can be... Figure 1 The terminal device 101 and / or server 102 shown include the following steps: Step S201: Receive the task execution command input by the user and obtain the environmental status information of the task execution environment.

[0033] Specifically, in long-chain task execution, the task execution instructions entered by the user are usually a high-level, complex, vague, or multi-step goal description, rather than a detailed, directly executable command. For example, help me plan a weekend trip to Beijing, including booking flights, hotels, and recommended attractions, and generate a detailed itinerary document.

[0034] These types of instructions typically express intent rather than specific operations, and a single instruction may imply multiple steps. Such instructions require intelligent systems, such as Large Language Models (LLMs), to perform in-depth understanding, decomposition, and planning in order to be effectively executed.

[0035] After receiving the task execution command from the user, the system obtains the environmental status information of the task execution environment in real time through the multimodal perception system.

[0036] Among them, a multimodal perception system refers to an intelligent system that can process, integrate and understand information from multiple different sources or types (i.e., "modalities") simultaneously or alternately. Through information complementarity, it can obtain a more comprehensive, robust and accurate understanding of the environment or task than any single modality.

[0037] The task execution environment refers to the overall environment in which an intelligent agent performs a task, including physical or virtual space, which contains various objects, other intelligent agents, and any conditions and factors that may affect task execution. The task environment defines the external world that the intelligent agent can perceive and interact with.

[0038] Environmental state information refers to a snapshot of the specific state, attributes, and relationships of all relevant elements in a task environment at a specific point in time. It is a concrete instantiation of the environment at a certain moment, including visual state analysis, system context monitoring, and session state maintenance.

[0039] Among them, visual state analysis refers to using a lightweight screen understanding model to identify active applications, window layouts, and the states of key UI elements; system context monitoring refers to acquiring runtime information such as clipboard content, the current focused element, and network connection status; and session state maintenance refers to recording the progress of the current task execution and the intermediate data generated.

[0040] Step S202: Based on the task execution instructions and environment status information, obtain a matching action block template set from the action block library. Each action block template in the action block template set includes: multiple pre-encapsulated atomic operations. Specifically, based on the task execution instructions and environmental status information input by the user, the system dynamically retrieves and loads the appropriate and optimal action block template set from the action block library. Each action block template in each action block template set is accompanied by detailed specifications such as its function description, parameter requirements, applicable conditions, and historical success rate.

[0041] This application provides a process for obtaining an action block template, such as... Figure 3 As shown, it includes the following steps: Step S301: Obtain multiple candidate operation sequences from the log data.

[0042] In some embodiments, multiple raw operation sequences are obtained from log data; for each raw operation sequence, each atomic operation in the raw operation sequence is mapped to its corresponding operation type to obtain a mapped operation sequence; and the mapped operation sequences that meet the preset filtering conditions are selected as candidate operation sequences.

[0043] Specifically, the log data includes GUI operation logs, which are structured information collections recording various interactive behaviors, operation contexts, and system response results performed by the user on the graphical user interface. Atomic operations from historical demonstrations or successful historical tasks are extracted from the GUI operation logs.

[0044] The original operation sequence consists of multiple atomic operations. Semantic categorization of atomic operations (e.g., treating "click('login')" and "tap('login')" as the same operation in a given context) improves the generalization ability of the mined patterns. Data preprocessing and semantic normalization are performed on each atomic operation, unifying atomic operations with different expressions but the same function into their corresponding operation types, resulting in a mapped operation sequence. For example, 'username', 'username box', and 'email / username' are uniformly mapped to [field:username].

[0045] Next, the mapping operation sequence is filtered using preset filtering conditions, including: the product of the frequency of occurrence and the success rate is greater than a preset threshold.

[0046] Specifically, the algorithm not only calculates the frequency of occurrence of mapping operation sequences, but also considers the proportion of task success when the pattern occurs (i.e., the success rate), prioritizing the discovery of high-success-rate patterns. The frequency and success rate are weighted and summed to obtain weighted support, and then mapping operation sequences with weighted support greater than a preset threshold are selected.

[0047] In addition, the preset screening conditions include: time constraint embedding, which requires that consecutive atomic operations in the mapping operation sequence must occur within a reasonable time window (e.g., the interval between two operations does not exceed 5 seconds) in order to exclude accidental spurious associations.

[0048] An improved sequence pattern mining algorithm is applied to sequence mining. The sequence pattern mining algorithm recursively discovers subsequences and calculates their weighted support (weighted support = original support × success rate). The algorithm outputs subsequences that meet the preset screening conditions and uses them as candidate operation sequences.

[0049] Sequence pattern mining is a data mining technique that automatically discovers frequently occurring subsequences (i.e., "patterns") with a specific order from ordered event sequence data.

[0050] For example, taking the mining of the "user login" action block as an example, assuming the system collects 5 login operation logs, the first step is to perform data preprocessing and semantic normalization, and to uniformly map operations with different expressions but the same function to obtain the mapped operation sequence. For example, the atomic operations "username", "username box", and "email / username" are uniformly mapped to the operation type: [field: username]; the atomic operations: click ('login button'), click ('login'), and press (Enter key) after entering the password are uniformly mapped to the operation type: [action: submit login].

[0051] We set the minimum support to 2 (meaning the pattern appears in at least two sequences), and calculated the weighted support for each mapping operation sequence. Then, we applied an improved sequence pattern mining algorithm to mine each mapping operation sequence, obtaining the following candidate operation sequences: Candidate operation sequence A: [Click ([username]), Enter (text A), Click ([password]), Enter (text B), [Submit login]], support 3, success rate 100%.

[0052] Candidate operation sequence B: [Click ([username]), Enter (text A), Switch focus, Enter (text B), [Submit login]], support 1, success rate 100%.

[0053] Step S302: Extract features from each candidate operation sequence to obtain the corresponding semantic vector.

[0054] Specifically, multi-dimensional features are extracted for each candidate operation sequence to construct a computable semantic vector, wherein the semantic vector includes at least one of the following: operation intention features, contextual environment features, target state change features, temporal relationship features, etc.

[0055] Operation intent features specifically refer to analyzing the textual semantics of the operation target element and extracting keywords (such as "save" and "login"); contextual features specifically refer to recording contextual information such as the application type and interface state in which the operation occurs; target state change features specifically refer to analyzing the system state changes caused by the operation sequence and identifying input-output relationships; temporal relationship features specifically refer to recording the time intervals and dependency patterns between operations.

[0056] Step S303: Cluster the multiple candidate operation sequences according to their respective semantic vectors to obtain multiple operation sequence groups.

[0057] In some embodiments, a comprehensive semantic similarity between the semantic vectors of any two candidate operation sequences is determined, wherein the semantic vector includes at least one of the following: operation intention features, contextual environment features, target state change features, and temporal relationship features; and multiple candidate operation sequences are clustered according to the obtained comprehensive semantic similarity to obtain multiple operation sequence groups.

[0058] Specifically, a hybrid similarity calculation method is used to evaluate the comprehensive semantic similarity between candidate operation sequences. Specifically, based on the operation intent features between candidate operation sequences, the similarity of intent models is obtained. This similarity is then used to determine the sequence intent using a pre-trained small intent recognition model. Figure 1 To the point of being compatible; Based on the contextual features between candidate operation sequences, the similarity of the context is obtained. The similarity of the context is used to compare the similarity of the application environment and preconditions in which the operation occurs. Based on the target state change features between candidate operation sequences, the similarity of state changes is obtained. The similarity of state changes is used to compare whether the state changes before and after the operation sequence are consistent. Based on the temporal relationship features between candidate operation sequences, the similarity of keywords is obtained, which is used to compare the semantic overlap of the target text of the operation.

[0059] A comprehensive semantic similarity is obtained based on at least one of the following similarities: keyword similarity, intent model similarity, state change similarity, and context similarity. Then, according to the obtained comprehensive semantic similarity, agglomerative hierarchical clustering algorithm is used to cluster multiple candidate operation sequences to obtain multiple operation sequence groups.

[0060] The specific process of using agglomerative hierarchical clustering algorithm to cluster multiple candidate operation sequences is as follows: First, each candidate operation sequence is initialized as an independent cluster. Then, the comprehensive semantic similarity matrix among all clusters is calculated. Next, the two clusters with the highest comprehensive semantic similarity are iteratively merged until the similarity among all clusters is lower than a preset threshold (such as 0.6), resulting in multiple operation sequence groups. Finally, semantic labels are automatically generated for each operation sequence group.

[0061] For example, suppose the following three candidate operation sequences are mined from different applications: Candidate operation sequence A (Word save): Click the "File" menu → Click the "Save" option → Enter the file name → Click the "Save" button; Candidate operation sequence B (save in Photoshop): Press "Ctrl+S" shortcut key → enter file name → press "Enter" key; Candidate operation sequence C (save in VSCode): Right-click the tab → select "Save As" → enter the file name → click "OK".

[0062] Although the specific operation paths of the three candidate operation sequences are different, through the semantic clustering process described above, they will be identified as having the same semantic intent of "saving files" and thus be grouped into the same operation sequence group.

[0063] Step S304: For each operation sequence group, extract the common structure of each candidate operation sequence in the operation sequence group, and construct parameters for the common structure to obtain the action block template.

[0064] In some embodiments, each candidate operation sequence in the operation sequence group is aligned to obtain a corresponding alignment matrix. The alignment matrix includes multiple alignment positions, which form a common structure. For each alignment position, the variable type of the alignment position is obtained based on the differences between each candidate operation sequence at the alignment position. A parameterized template for the alignment position is constructed according to the variable type of each alignment position. An action block template is obtained based on the parameterized templates corresponding to each of the multiple alignment positions.

[0065] Specifically, an improved multi-sequence alignment method is used to align candidate operation sequences within an operation sequence group. The core of this method is "semantic similarity dynamic programming," which focuses on defining and calculating the semantic similarity between two atomic operations (e.g., SemSim(op_i, op_j)). This semantic similarity integrates the operation type, the textual semantics of the target element (calculated using a lightweight semantic embedding model), and the operation context, thus enabling the identification of functionally equivalent but superficially different operations (e.g., clicking ("Save button") and pressing ("Enter key") have high similarity). Based on this semantic similarity, a dynamic programming algorithm is used for double-sequence alignment, and a progressive strategy (determining the optimal merging order by constructing a guide tree) is used to complete multi-sequence alignment, ultimately resulting in an alignment matrix where each column represents an alignment position.

[0066] In the sequence matrix, the consistency of operations is analyzed column by column. If a certain alignment position has semantically equivalent operations in all sequences (ignoring gaps) (i.e., the semantic consistency score of the column exceeds the high threshold), then the alignment position is marked as a stable point, which represents the core operation steps necessary to achieve the semantic goal.

[0067] Next, the alignment positions where differences exist between candidate operation sequences in the alignment matrix are identified, i.e., variable points. Variable points are categorized into value variables, mode variables, and condition variables. Value variables refer to operations with the same type and semantics but different specific values ​​(e.g., entering different filenames in the same location); mode variables refer to different operation methods used to achieve the same sub-goal (e.g., triggering a save dialog box can be done by clicking the "File" menu, pressing "Ctrl+S", or using the right-click menu); condition variables refer to certain operations or steps that only occur under specific conditions (such as a specific application or interface state), and are represented as "gaps" in other sequences.

[0068] The specific steps for constructing a parameterized template based on the variable type at each alignment position to obtain the action block template are as follows: Value parameters are extracted for value variables, which means analyzing all specific samples that appear and inferring the name, data type and constraints of the parameters. For example, the file name parameter filename is a string type and must conform to the file naming convention.

[0069] For method variables and condition variables, analyze the context or state conditions that cause differences (such as application type, whether a specific interface element is visible), construct condition judgment branch logic, so that the action block can automatically select the most appropriate operation path according to the runtime environment.

[0070] Additionally, for optional operations, it is determined whether an operation appearing only in a portion of the sequence is necessary to complete the objective. If not, it is marked as a conditional execution step in the template.

[0071] Based on the above analysis, the input parameter list and output result format of the action block are formally defined for the input and output interfaces.

[0072] For example, taking the "save file" cluster mentioned above as an example, after multiple sequence alignment, the following alignment positions were found and the variable type was determined: Alignment Position 1 (Trigger dialog box): [Click ("File"), press ("Ctrl+S"), right-click...] → Method Variable.

[0073] Alignment position 2 (middle option): [Click (“Save”), select (“Save As”)] → Condition Variables.

[0074] Alignment position 3 (input name): [input (file name A), input (file name B), input (file name C)] → value variable.

[0075] Alignment position 4 (Confirm operation): [Click (“Save button”), press (“Enter”), click (“OK”)] → Method variable.

[0076] Based on this, the generated generic action block template Block_SaveFile(filename) will encapsulate the following logic: Conditional Trigger: Intelligently selects the best method to trigger the save dialog box based on the application and interface status; Conditional Execution: If the "Save" or "Save As" option appears, select the corresponding option; Parameterized Execution: Enter the value of the parameter filename in the input box; Conditional Confirmation: Click the OK button or press Enter to complete the save according to the actual situation in the dialog box.

[0077] Step S305: Save the action block template to the action block library.

[0078] Specifically, for each generated action block template, diverse test cases (including normal values, boundary values, and abnormal situations) are automatically built and robustness verification is performed in a sandbox environment. Only action blocks that pass rigorous testing (e.g., success rate > 95%, robustness score > threshold, etc.) are given descriptive names and metadata and are officially stored in the persistent action block library.

[0079] In this embodiment, through the complete semantic clustering and parameterized generalization process described above, multiple distinct operation sequences can be intelligently abstracted into a unified, conditionally judgment-capable, and highly reliable parameterized action block template. For example, in the "login" scenario, the final generated Block_Login(username, password) template can adaptively handle multiple positioning methods for the username / email input box and password box, as well as multiple submission methods such as clicking the login button or pressing the Enter key, thereby reliably executing the login task in various actual interfaces.

[0080] Step S203: Construct prompt text based on task execution instructions, environment status information, and action block template set; Specifically, the task execution instructions are the original instructions input by the user, the environmental status information describes the current rich environmental status, the action block template set contains detailed specifications for each action block template, and there are also planning constraints to indicate the business rules and best practices related to the task.

[0081] Step S204: Using the prompt text as a prompt instruction, input it into the large language model for reasoning to obtain task execution planning information, which includes the sequence of action blocks to be executed; Specifically, LLM reasones the aforementioned prompt text at a high-level semantic level to obtain task execution planning information and output an executable macro plan.

[0082] Step S205: Execute the action block sequence in the task execution environment to obtain the task execution result.

[0083] Specifically, based on the aforementioned executable macro plan, the execution engine sequentially calls the action block sequence and ensures task robustness through fine-grained monitoring and intelligent replanning, thereby obtaining the result of task execution success or failure.

[0084] Each action block in the action block sequence consists of a series of atomic operations (such as click, input, scroll, etc.). When executing an action block, the execution engine records and tracks the execution information of each atomic operation. The execution information of multiple atomic operations forms a fine-grained execution trajectory. The execution information of atomic operations includes: success or failure status, identification information of the target element (such as locator, text content, etc.), intermediate results generated by the operation (such as text in the input box, obtained attribute values, etc.), and screenshots of the screen state before and after the operation.

[0085] For example, suppose the user inputs the task instruction: "Buy a laptop priced under 5000 yuan from an e-commerce website." This application can complete the task by performing the following steps: First, the macro planner generates a plan based on the task execution instructions and the current browser environment, combined with the action block library: [Block_Login(myAccount), Block_SearchItem(“Laptop”), Block_FilterByPrice(0, 5000), Block_SelectTopResult, Block_AddToCart, Block_Checkout], and only one LLM call is needed here.

[0086] Secondly, the execution engine sequentially executes six highly reliable, pre-defined action blocks. Each block encapsulates all necessary atomic operations and error handling (such as login CAPTCHA recognition, price filter positioning, and shopping cart button clicks). Even with minor changes to the website UI, only the corresponding action block needs to be updated, and the planning logic remains unaffected. The entire process boasts extremely high reliability (success rate of each block >95%, overall success rate >90%), extremely low cost (requiring only one LLM call), and fast execution speed.

[0087] In this embodiment, based on the user-input task execution command and environmental state information, a matching action block template set is obtained from the action block library. Then, a prompt text is constructed based on the task execution command, environmental state information, and action block template set. This prompt text serves as the prompt command, which is input into a large language model for reasoning to obtain task execution planning information. Only one call to the large language model is needed, and the decision space of the large language model is compressed from hundreds of atomic operations to dozens of action blocks with clear business semantics, greatly simplifying the planning problem. The action block sequence is executed in the task execution environment to obtain the task execution result. Utilizing a pre-defined action block library, task execution commands are executed efficiently and reliably, resulting in high reliability, extremely low cost, and fast execution speed throughout the entire process.

[0088] In some embodiments, when an action block in the action block sequence fails to execute, the execution information of each atomic operation in the action block is obtained; based on the execution information of each atomic operation in the action block, the atomic operation that failed to execute in the action block is obtained, and the atomic operation is marked as a breakpoint where the task is paused.

[0089] Specifically, when an action block fails to execute, a multi-level diagnosis is performed on the failed action block based on its execution trajectory. This includes: identifying the precise location of the failure (e.g., which specific atomic operation failed and marking that atomic operation as a breakpoint for task pause), analyzing the cause of the failure (e.g., element not found, operation timeout, status verification failure, etc.), and recording the complete context of the failure. The complete context includes the sequence of successfully executed atomic operations and their outputs.

[0090] Then, the information of the action block is received and integrated, including: the name and parameters of the failed action block, the specific index and description of the failed atomic operation, the analysis results of the failure reason, the list of successfully executed atomic operations and their output, the current GUI status and environment snapshot, and the overall progress of task execution.

[0091] Finally, save the complete execution state at the moment of failure, including: all completed action blocks and their outputs, the sequence of atomic operations successfully executed in the current action block, the current GUI state and environment snapshot, and all intermediate variables and data.

[0092] In some embodiments, at least one of the following recovery strategies is used to recover failed action blocks: Strategy A: Adjust the remaining atomic operations in the failed action block and continue executing the adjusted remaining atomic operations from the breakpoint; Specifically, this strategy applies to situations where the failure occurs within an action block, but the action block can be adjusted and continued.

[0093] For example, if a user login action block (Block_Login) fails at the "click login button" step but has successfully entered their username and password, the login button settings can be adjusted to restart the login operation from the "click login button" step.

[0094] Strategy B: Replace the failed action block with a substitute block, and continue executing the remaining action blocks in the action block sequence from the substitute block; Specifically, this strategy applies to situations where the current action block completely fails, but there is a functionally equivalent alternative block. For example, if the action block Block_SearchItem, which displays search results, fails on an e-commerce website, but there is an action block Block_SearchByCategory that performs a search by category, then Block_SearchByCategory is used to replace the failed Block_SearchItem. Then, the search operation is performed starting from the replacement block, and the original plan continues.

[0095] Strategy C: Re-plan the remaining action blocks in the action block sequence that follow the failed action block, and re-execute the updated action block sequence; Specifically, this strategy is applicable when failure exposes fundamental errors in the original plan.

[0096] For example, if the purchase action block Block_AddToCart fails because the product is out of stock, the purchase strategy needs to be re-planned based on the current state, and then a completely new purchase plan needs to be executed.

[0097] Strategy D: Roll back to the previous stable checkpoint and continue executing the action block sequence from the previous stable checkpoint. A stable checkpoint is either an atomic operation or an action block.

[0098] Specifically, this strategy is applicable to situations where failure may be caused by transient states.

[0099] For example, if a network timeout causes a page to fail to load, it is recommended to go back a few steps to the previous stable check state, refresh the page, and then retry the original plan.

[0100] Based on the above replanning strategy, the large language model implements precise rather than general replanning based on fine-grained failure parameter information such as the name and parameters of the failed action block, the specific index and description of the failed atomic operation, the failure reason analysis, the list of successfully executed atomic operations and their outputs, the current GUI state and environment snapshot, and the overall progress of task execution. It uses at least one of the above recovery strategies to recover the failed action blocks and retains and reuses all successful operation outputs and states to avoid duplication of work.

[0101] Finally, update the knowledge base: store successful and unsuccessful execution cases (after anonymization) into the knowledge base; update the success rate statistics and applicable condition descriptions of action blocks; and provide the macro planner with more accurate tool selection criteria.

[0102] For example, suppose the user inputs the task execution command "Buy a wireless mouse under 5000 yuan on JD.com", and the executed action block is: [Block_Login, Block_SearchItem("wireless mouse"), Block_FilterByPrice(5000), Block_SelectTopResult, Block_AddToCart, Block_Checkout]. Among them, the action blocks that are successfully executed are: Block_Login (login) and Block_SearchItem (return to the product list); the execution fails in the third step of the price filtering action block Block_FilterByPrice (clicking the "price filter" button) because the button is not found.

[0103] A multi-level diagnosis was performed to determine the cause of the failure. First, the precise location of the failure was identified based on the atomic operation trajectory: ① Input the price limit of 5000 (success) → ② Click the "Price Range" dropdown (success) → ③ Click the "Under 5000 Yuan" option (failure). Then, the cause of the failure was analyzed as follows: the website UI was updated, and the Cascading Style Sheets (CSS) class name for the "Under 5000 Yuan" option had changed. Finally, the complete context of the failure was recorded: logged in, products found, price limit entered.

[0104] After receiving and integrating the information from the action block, the large language model selects to execute strategy A to restore the action block and generates an adjustment plan: in the current price filter area, find the option whose text content contains '5000' and click it, and adjust the remaining steps of Block_FilterByPrice.

[0105] Continue execution of the remaining atomic operations after the breakpoint: resume execution from the third step of Block_FilterByPrice (clicking the price filter button), use the new element positioning strategy to find and click the correct option, and continue to execute subsequent operations such as Block_SelectTopResult.

[0106] Finally, retain and reuse all successful operation outputs and states to avoid duplication of work: record the failure, update the UI element positioning logic of Block_FilterByPrice, and use this fix as an alternative execution path for this action block.

[0107] In some embodiments, when the frequency of continuous execution of multiple action block templates exceeds a preset threshold and the execution success rate exceeds a preset success rate, the multiple action block templates are encapsulated into a super block template and the super block template is stored in the action block library; when the number of execution failures of an atomic operation in an action block module exceeds a preset number, the action block module is split into multiple sub-action block templates and the multiple sub-action block templates are stored in the action block library.

[0108] Specifically, during the execution of tasks, by collecting execution data (such as success rate, time consumption, common failure modes, etc.) of each action block template, the dependencies and execution patterns between atomic operations are analyzed in order to dynamically adjust the granularity of the action block template.

[0109] Specific adjustment methods include: for action block templates that frequently fail in specific atomic operations, the system suggests splitting them, that is, splitting the action block template into multiple smaller, more focused sub-action block templates; For action block templates that are executed more frequently than a preset threshold and have a success rate greater than a preset success rate, the system suggests merging them, that is, encapsulating multiple action block templates into a "super block template" to improve planning efficiency.

[0110] In addition, the system will continuously optimize the internal implementation of action blocks based on actual execution data, such as updating UI element locators and adding fault tolerance mechanisms.

[0111] In this embodiment, atomic operation-level monitoring, intelligent failure diagnosis, precise replanning, and breakpoint continuation mechanisms are introduced. These mechanisms not only enable graceful recovery when action blocks fail, but also continuously optimize the action block library and planning strategies based on fine-grained execution data, forming a virtuous cycle of "execution-diagnosis-optimization". This significantly improves the overall reliability and execution efficiency of long-chain tasks in dynamic environments.

[0112] In some embodiments, when the action block library does not contain action block templates that match the task execution instructions and environment state information, multiple candidate paths are generated based on the task execution instructions and environment state information. Each candidate path includes at least one of the following: atomic operations and basic action blocks. Each candidate path is executed in a test environment to obtain the corresponding execution results. Based on the execution results of each candidate path, a comprehensive score for each candidate path is obtained. The candidate path with the highest comprehensive score is generalized to obtain an action block template, and the action block template is saved to the action block library.

[0113] Specifically, when an instruction is received or a high-frequency user target is discovered, but the existing action block library cannot directly complete it, the system will proactively attempt to "create" a new method to complete the target.

[0114] First, the task execution instruction (such as "set the selected text to red bold") and the current application context (such as "in a Word editing window, with text already selected") are submitted to a large language model (LLM) with code generation capabilities. The LLM's task is not to directly plan atomic operations, but to generate one or more candidate paths that could accomplish the goal, composed of existing atomic operations or basic action blocks. For example, the LLM might generate the following two candidate paths: Candidate Path 1: [Click the "Font Color" button, select red, and click the "Bold" button]; Candidate Path 2: [Open the "Font" dialog box, set the color to red, set the font to bold, and click OK].

[0115] Then, in a controlled sandbox environment, each candidate path is automatically executed sequentially. After each execution, an assertion verification mechanism is used to automatically check whether the goal has been achieved. Verification methods include: GUI state check: Detects whether specific UI elements appear or their properties change (such as whether the "bold" button in the toolbar is highlighted and pressed). Visual result verification: Use computer vision to compare the screen area to confirm whether the text has indeed turned red and bold; Application data query: Query whether the properties of a document object have changed through the accessibility interface or application API.

[0116] After verification, the system records the execution success rate, execution efficiency, and robustness (such as adaptability to changes in window size) of each path.

[0117] Finally, the system analyzes the verification results of all candidate paths and selects the candidate path with the highest comprehensive score (considering factors such as success rate, efficiency, and simplicity) as the winner. The winning path is then parameterized and generalized (for example, "red" is abstracted as the parameter color, and "bold" is abstracted as the parameter bold). After more comprehensive automated testing (boundary conditions, exception handling), the winning path is encapsulated and solidified into a new action block template (such as Block_FormatText(color, bold, ...)) and officially stored in the database. At the same time, other feasible candidate paths can be recorded in the metadata of the action block as backup solutions or conditional branches to increase its adaptability in complex environments.

[0118] One possible implementation is to initiate a single-session learning session when the system encounters a completely new UI or a task it has never handled before, and there are no directly available blocks in the action block library.

[0119] Users complete this task through a single graphical operation (or by providing a screen recording). The specific learning process is as follows: The computer vision module identifies the operation objects and UI state changes in the screen recording, and the sequence analysis module parses the logic and dependencies between operation steps to obtain a new, parameterized action block draft and infer its input parameters (such as file name, search keywords) and possible conditional branches.

[0120] Once the action block draft has been quickly validated in the sandbox environment (running several times to ensure basic functionality), it can be added to the library as a temporary or formal action block template, enabling "learn and use immediately".

[0121] For example, suppose a user frequently needs to copy data from a webpage table to Excel and adjust it to a specific format. The traditional method would require the user to record a macro or write a script.

[0122] In this embodiment, the user only needs to provide a few natural language commands (or the system observes the pattern) to trigger the active exploration process. The LLM may generate multiple solutions through "copy-paste-format painter" or "export CSV and then import". After the system automatically tries and verifies, it solidifies the optimal solution into an action block template for all subsequent users to call with one click.

[0123] In this embodiment, through three core mechanisms—demonstration learning, block combination, and proactive exploration and automated verification—the constructed intelligent agent system possesses powerful continuous evolution capabilities. It can gradually grow from an initial system with basic capabilities into an "expert-level" automated executor covering a wide range of tasks and possessing rich best practice skills. The system no longer waits for human demonstrations or historical data; it proactively explores unknown task spaces, transforming unrealized user intentions into executable capabilities. Through a closed loop of "generation-execution-verification," it ensures that newly discovered action blocks are truly effective in the real world, rather than merely theoretical possibilities. By exploring multiple implementation paths and selecting the best, the system can solidify optimal practices and may even discover more efficient operating methods not conceived by humans. The rapid capability expansion process can be automated or semi-automated, enabling the action block library to expand rapidly and at low cost as user needs grow, greatly enhancing the system's long-term adaptability and vitality.

[0124] To better explain the embodiments of this application, the following describes the complete task execution method provided by the embodiments of this application, from generating the action block template to completing the task execution and then to subsequent optimization and updates. See [link to relevant documentation]. Figure 4 This includes: offline pre-fixed mining process, online planning and execution process, and continuous evolution process.

[0125] The offline mining pre-solidification process includes: collecting atomic operation logs, using an improved mining algorithm to mine the original operation sequences in the atomic operation logs to obtain candidate operation sequences, performing semantic clustering and generalization on the candidate operation sequences to obtain multiple operation sequence groups, then aligning and parameterizing each candidate operation sequence in the operation sequence group to obtain action block templates, generating test cases for each generated action block template for automated verification, and storing the verified action block templates into the action block library.

[0126] The planning and execution process includes: receiving task execution instructions from the user; obtaining environmental state information of the task execution environment through emotion perception and tool loading steps; dynamically retrieving and loading applicable action block template sets from the action block library; using a large language model for reasoning and planning to generate an executable macro plan; parsing the macro plan, executing the action block sequence and recording the execution information of the action sequence until the task is completed.

[0127] The continuous evolution process includes: During task execution, the large language model can monitor and optimize the task execution process and update the action block library through the following three methods: Demonstration Learning: Initiate a single demonstration learning session to identify changes in the objects being operated on and the UI state in the screen recording of the user's demonstration operation, automatically synthesize an action block draft, and after quick verification, quickly add it to the action block library; Block Combination: Multiple action block templates that are executed consecutively with a frequency and success rate greater than a threshold are combined and encapsulated into a super block template, and the super block template is stored in the action block library; Active exploration and verification: When there is no matching action block template in the action block library, multiple candidate paths are automatically generated and executed in the test environment. The candidate path with the highest comprehensive score is selected as the action block template and stored in the action block library.

[0128] The following section details the offline mining pre-solidification process, such as... Figure 5 As shown, the specific steps include: Multiple raw operation sequences are obtained from the log data. Data preprocessing and semantic normalization are performed on the raw operation sequences to obtain multiple mapped operation sequences.

[0129] Pattern mining: The mapping operation sequences are filtered using preset filtering conditions. The minimum support is set and the weighted support of each mapping operation sequence is calculated to obtain candidate operation sequences.

[0130] Semantic clustering: Extract multi-dimensional features for each candidate operation sequence to construct a computable semantic vector, calculate the comprehensive semantic similarity between the semantic vectors of any two candidate operation sequences, cluster multiple candidate operation sequences according to the comprehensive semantic similarity, obtain multiple operation sequence groups and generate semantic labels.

[0131] Parameterized generalization: Align each candidate operation sequence in the operation sequence group to obtain an alignment matrix, identify alignment positions (stable points or variable points) where there are differences between each candidate operation sequence in the alignment matrix, obtain the variable type of each alignment position, extract parameters for different variable types, construct parameterized templates for alignment positions, and obtain action block templates.

[0132] Automated verification: Generate test cases for each generated action block template, perform robust verification in a sandbox environment, output verification results and scores, store verified action block templates into the action block library, and optimize or discard action block templates that fail verification.

[0133] The online planning and execution process is described in detail below, such as... Figure 6 As shown, the specific steps include: Receive task execution instructions from the user.

[0134] Emotion perception and tool loading: The system acquires real-time environmental status information of the task execution environment through a multimodal perception system. Based on the task execution instructions and environmental status information, it dynamically retrieves and loads the appropriate and optimal action block template set from the action block library.

[0135] Macro-planning: Based on task execution instructions, environmental status information, and action block template sets, construct prompt text and input it into a large language model for reasoning and planning to generate an executable macro plan.

[0136] Robust execution: Analyze the above macro plan, execute the action block sequence in the task execution environment, record and track the execution information of each atomic operation, and update the task execution status of successfully executed action blocks in real time until the task execution is completed; When an action block fails to execute, multi-level diagnosis is performed on the failed action block based on its execution trajectory. Execution monitoring and optimization: When an action block in the action block sequence fails to execute, analyze the execution trajectory of the action block; obtain the atomic operations that failed to execute in the action block, establish recoverable breakpoints, and mark the exact location where the task execution was paused.

[0137] Precise replanning: Based on fine-grained failure parameter information of failed action blocks, a replanning strategy is implemented to accurately restore the execution of failed action blocks from the saved breakpoint state.

[0138] The following section details the continuous evolution process, such as... Figure 7 As shown, it specifically includes the following: During task execution, the large language model simultaneously monitors and optimizes the task execution process, updates the action block library, and can passively learn through user demonstrations, as well as actively explore, verify, and solidify new skills, forming a self-improving and self-expanding intelligent agent system. Specifically, there are three methods: Method 1, Demonstration Learning: When encountering a task that has never been handled before and there is no directly available action block template in the action block library, the user completes the task through this graphical operation (or provides screen recording), and initiates a single demonstration learning session. The system identifies the objects being operated on and changes in the UI state during screen recording, analyzes the logic and dependencies between operation steps, automatically synthesizes an action block draft, and then quickly verifies it in a test environment before storing it in the action block library.

[0139] Method 2, Block Combination: When the frequency of continuous execution of multiple action block templates exceeds a preset threshold and the execution success rate exceeds a preset success rate, the multiple action block templates are combined and encapsulated into a super block template, and the super block template is stored in the action block library; In addition, when the number of execution failures of an atomic operation in an action block module exceeds a preset number, the action block module is split into multiple sub-action block templates, and the multiple sub-action block templates are stored in the action block library. Method 3, Active Exploration and Verification: When the action block library does not contain action block templates that match the task execution instructions and environment status information, multiple candidate paths are generated based on the task execution instructions and environment status information; Execute each candidate path in the test environment to obtain a comprehensive score for each candidate path; generalize the parameters of the candidate path with the highest comprehensive score to obtain an action block template, and save the action block template to the action block library.

[0140] Based on the same technical concept, this application provides a schematic diagram of the structure of a task execution device, such as... Figure 8 As shown, the device 800 includes: The receiving module 801 is used to receive the task execution command input by the user and obtain the environmental status information of the task execution environment; The acquisition module 802 is used to acquire a matching action block template set from the action block library based on the task execution instruction and the environment state information. Each action block template in the action block template set includes: multiple pre-encapsulated atomic operations. The execution module 803 is used to construct a prompt text based on the task execution instruction, the environment status information, and the action block template set; use the prompt text as a prompt instruction, input it into a large language model for reasoning, and obtain task execution planning information, which includes a sequence of action blocks to be executed; execute the action block sequence in the task execution environment to obtain the task execution result.

[0141] Optionally, the acquisition module 802 is specifically used for: Obtain multiple candidate operation sequences from log data; Feature extraction is performed on each candidate operation sequence to obtain the corresponding semantic vector; Based on the semantic vectors of the multiple candidate operation sequences, the multiple candidate operation sequences are clustered to obtain multiple operation sequence groups; For each operation sequence group, the common structure of each candidate operation sequence in the operation sequence group is extracted, and the parameters of the common structure are constructed to obtain the action block template; Save the action block template to the action block library.

[0142] Optionally, the acquisition module 802 is specifically used for: Obtain multiple raw operation sequences from the log data; For each original operation sequence, each atomic operation in the original operation sequence is mapped to its corresponding operation type to obtain a mapped operation sequence; The mapping operation sequence that meets the preset screening conditions is taken as the candidate operation sequence. The preset screening conditions include: the product of the occurrence frequency and the success rate is greater than a preset threshold.

[0143] Optionally, the acquisition module 802 is specifically used for: Determine the comprehensive semantic similarity between the semantic vectors of any two candidate operation sequences, wherein the semantic vectors include at least one of the following: operation intention features, contextual environment features, target state change features, and temporal relationship features; Based on the obtained comprehensive semantic similarity, the multiple candidate operation sequences are clustered to obtain multiple operation sequence groups.

[0144] Optionally, the acquisition module 802 is specifically used for: Align each candidate operation sequence in the operation sequence group to obtain a corresponding alignment matrix. The alignment matrix includes multiple alignment positions, and the multiple alignment positions form the common structure. For each alignment position, the variable type of the alignment position is obtained based on the differences between the candidate operation sequences at that alignment position; Construct a parameterized template for each alignment position according to the variable type of each alignment position; The action block template is obtained based on the parameterized templates corresponding to the multiple alignment positions.

[0145] Optionally, the execution module 803 is further configured to: When one action block in the action block sequence fails to execute, the execution information of each atomic operation in the action block is obtained; Based on the execution information of each atomic operation in the action block, the atomic operations that failed to execute in the action block are obtained, and the atomic operations are marked as breakpoints where the task is paused.

[0146] Optionally, the execution module 803 is specifically used for: Use at least one of the following recovery strategies to recover failed action blocks: Adjust the remaining atomic operations in the failed action block and continue executing the adjusted remaining atomic operations from the breakpoint; The failed action block is replaced with a substitute block, and the remaining action blocks in the action block sequence are continued from the substitute block. Re-plan the remaining action blocks in the action block sequence that follow the failed action block, and re-execute the updated action block sequence; Revert to the previous stable checkpoint and continue execution of the action block sequence from the previous stable checkpoint, where the stable checkpoint is an atomic operation or an action block.

[0147] Optionally, the execution module 803 is further configured to: When the frequency of consecutive execution of multiple action block templates exceeds a preset threshold and the execution success rate exceeds a preset success rate, the multiple action block templates are encapsulated into a super block template and the super block template is stored in the action block library. When the number of execution failures of an atomic operation in an action block module exceeds a preset number, the action block module is split into multiple sub-action block templates, and the multiple sub-action block templates are stored in the action block library.

[0148] Optionally, the execution module 803 is further configured to: When the action block library does not contain an action block template that matches the task execution instruction and the environment state information, multiple candidate paths are generated based on the task execution instruction and the environment state information. Each candidate path includes at least one of the following: atomic operation and basic action block. Execute each candidate path in the test environment and obtain the corresponding execution results; Based on the execution results of each candidate path, a comprehensive score is obtained for each candidate path; The candidate path with the highest comprehensive score is generalized to obtain an action block template, and the action block template is saved to the action block library.

[0149] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0150] Based on the same technical concept, embodiments of this application provide a computer device, which can be... Figure 1 The terminal devices and / or servers shown, such as Figure 9 As shown, it includes at least one processor 901 and a memory 902 connected to at least one processor. In this embodiment, the specific connection medium between the processor 901 and the memory 902 is not limited. Figure 9 Taking the connection between processor 901 and memory 902 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0151] In this embodiment of the application, the memory 902 stores instructions that can be executed by at least one processor 901. By executing the instructions stored in the memory 902, at least one processor 901 can perform the steps of the above-described task execution method.

[0152] The processor 901 is the control center of the computer device. It can connect to various parts of the computer device using various interfaces and lines, and performs tasks by running or executing instructions stored in the memory 902 and calling data stored in the memory 902. Optionally, the processor 901 may include one or more processing units. The processor 901 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 902 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0153] The processor 901 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0154] Memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 902 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 902 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. In the embodiments of this application, memory 902 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0155] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described task execution method.

[0156] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described task execution method.

[0157] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0162] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A task execution method, characterized in that, include: Receive task execution instructions from the user and obtain environmental status information of the task execution environment; Based on the task execution instructions and the environment state information, a matching action block template set is obtained from the action block library. Each action block template in the action block template set includes: multiple pre-encapsulated atomic operations. A prompt text is constructed based on the task execution instruction, the environmental status information, and the action block template set; Using the prompt text as a prompt instruction, inputting it into a large language model for reasoning, and obtaining task execution planning information, the task execution planning information includes a sequence of action blocks to be executed; The action block sequence is executed in the task execution environment to obtain the task execution result.

2. The method as described in claim 1, characterized in that, Also includes: Obtain multiple candidate operation sequences from log data; Feature extraction is performed on each candidate operation sequence to obtain the corresponding semantic vector; Based on the semantic vectors of the multiple candidate operation sequences, the multiple candidate operation sequences are clustered to obtain multiple operation sequence groups; For each operation sequence group, the common structure of each candidate operation sequence in the operation sequence group is extracted, and the parameters of the common structure are constructed to obtain the action block template; Save the action block template to the action block library.

3. The method as described in claim 2, characterized in that, The step of obtaining multiple candidate operation sequences from log data includes: Obtain multiple raw operation sequences from the log data; For each original operation sequence, each atomic operation in the original operation sequence is mapped to its corresponding operation type to obtain a mapped operation sequence; The mapping operation sequence that meets the preset screening conditions is taken as the candidate operation sequence. The preset screening conditions include: the product of the occurrence frequency and the success rate is greater than a preset threshold.

4. The method as described in claim 2, characterized in that, The step of clustering the multiple candidate operation sequences according to their respective semantic vectors to obtain multiple operation sequence groups includes: Determine the comprehensive semantic similarity between the semantic vectors of any two candidate operation sequences, wherein the semantic vectors include at least one of the following: operation intention features, contextual environment features, target state change features, and temporal relationship features; Based on the obtained comprehensive semantic similarity, the multiple candidate operation sequences are clustered to obtain multiple operation sequence groups.

5. The method as described in claim 2, characterized in that, The step of extracting the common structure of each candidate operation sequence in the operation sequence group and constructing parameters for the common structure to obtain an action block template includes: Align each candidate operation sequence in the operation sequence group to obtain a corresponding alignment matrix. The alignment matrix includes multiple alignment positions, and the multiple alignment positions form the common structure. For each alignment position, the variable type of the alignment position is obtained based on the differences between the candidate operation sequences at that alignment position; Construct a parameterized template for each alignment position according to the variable type of each alignment position; The action block template is obtained based on the parameterized templates corresponding to the multiple alignment positions.

6. The method according to any one of claims 1 to 5, characterized in that, After executing the action block sequence in the task execution environment and obtaining the task execution result, the process further includes: When one action block in the action block sequence fails to execute, the execution information of each atomic operation in the action block is obtained; Based on the execution information of each atomic operation in the action block, the atomic operations that failed to execute in the action block are obtained, and the atomic operations are marked as breakpoints where the task is paused.

7. The method as described in claim 6, characterized in that, Use at least one of the following recovery strategies to recover failed action blocks: Adjust the remaining atomic operations in the failed action block and continue executing the adjusted remaining atomic operations from the breakpoint; The failed action block is replaced with a substitute block, and the remaining action blocks in the action block sequence are continued from the substitute block. Re-plan the remaining action blocks in the action block sequence that follow the failed action block, and re-execute the updated action block sequence; Revert to the previous stable checkpoint and continue execution of the action block sequence from the previous stable checkpoint, where the stable checkpoint is an atomic operation or an action block.

8. The method as described in claim 6, characterized in that, Also includes: When the frequency of consecutive execution of multiple action block templates exceeds a preset threshold and the execution success rate exceeds a preset success rate, the multiple action block templates are encapsulated into a super block template and the super block template is stored in the action block library. When the number of execution failures of an atomic operation in an action block module exceeds a preset number, the action block module is split into multiple sub-action block templates, and the multiple sub-action block templates are stored in the action block library.

9. The method as described in claim 1, characterized in that, Also includes: When the action block library does not contain an action block template that matches the task execution instruction and the environment state information, multiple candidate paths are generated based on the task execution instruction and the environment state information. Each candidate path includes at least one of the following: atomic operation and basic action block. Execute each candidate path in the test environment and obtain the corresponding execution results; Based on the execution results of each candidate path, a comprehensive score is obtained for each candidate path; The candidate path with the highest comprehensive score is generalized to obtain an action block template, and the action block template is saved to the action block library.

10. A task execution device, characterized in that, include: The receiving module is used to receive the task execution instructions input by the user and to obtain the environmental status information of the task execution environment; The acquisition module is used to acquire a matching action block template set from the action block library based on the task execution instruction and the environment state information. Each action block template in the action block template set includes: multiple pre-encapsulated atomic operations. The execution module is used to construct prompt text based on the task execution instruction, the environment state information, and the action block template set; use the prompt text as a prompt instruction, input it into a large language model for reasoning, and obtain task execution planning information, which includes a sequence of action blocks to be executed; execute the action block sequence in the task execution environment to obtain the task execution result.

Images