Method, apparatus, device, medium and program product for task processing

Abstract

Methods, apparatuses, devices, media, and program products for task processing are provided. The method includes: in response to at least one interaction operation execution failure of a first task, obtaining an execution failure record of the first task; in response to the execution failure record satisfying a preset condition, obtaining context information associated with the at least one interaction operation, wherein the preset condition includes at least one of the first task execution failure number reaching a first threshold or an interaction element positioning failure associated with the first task execution; and based on the context information, determining a first operation sequence to re-execute the first task.

Description

Technical Field

[0001] The examples in this article generally relate to the field of artificial intelligence, and in particular to methods, apparatuses, devices, media, and program products for task processing. Background Technology

[0002] In AI-driven browser automation scenarios, the industry has placed higher demands on the stability and efficiency of automated task execution. However, during browser operation, when interactive operations fail consecutively, automatically analyzing the reasons for failure, replanning the operation path, and continuing to complete the task still faces significant technical challenges. Summary of the Invention

[0003] In a first aspect of this paper, a task processing method is provided, comprising: in response to the failure of at least one interactive operation of a first task, obtaining an execution failure record of the first task; in response to the execution failure record satisfying a preset condition, obtaining context information associated with at least one interactive operation, wherein the preset condition includes at least one of the following: the number of execution failures of the first task reaches a first threshold or the location of an interactive element associated with the execution of the first task fails; and based on the context information, determining a first operation sequence to re-execute the first task.

[0004] In a second aspect of this document, a task processing apparatus is provided, comprising: a failure record acquisition module configured to acquire an execution failure record of the first task in response to at least one interactive operation of the first task failing to execute; a context acquisition module configured to acquire context information associated with at least one interactive operation in response to the execution failure record satisfying a preset condition, wherein the preset condition includes at least one of the following: the number of execution failures of the first task reaches a first threshold or the location of an interactive element associated with the execution of the first task fails; and a sequence determination module configured to determine a first operation sequence based on the context information to re-execute the first task.

[0005] In a third aspect of this document, an electronic device is provided, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to the first aspect when executed by the at least one processing unit.

[0006] In a fourth aspect of this document, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executable by a processor to implement the method according to the first aspect.

[0007] In a fifth aspect of this document, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to the first aspect.

[0008] In this way, by automatically acquiring the failure record when a task fails and re-determining the operation sequence based on context information, adaptive recovery and re-execution of the task can be achieved, thereby improving the stability and success rate of automated task execution.

[0009] It should be understood that the content described in this section is not intended to limit the key or important features of the examples in this article, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the various examples herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figure 2 Flowcharts of methods for task processing in some scenarios are shown; Figure 3 Flowcharts for dynamic reprogramming processes are shown in some scenarios; Figure 4 The flowcharts show some scenarios of the process for locating interactive elements based on image information; Figure 5 Flowcharts are shown for some scenarios of processes used to perform multiple operation steps in batches; Figure 6 The diagram illustrates some scenarios of interfaces used to present the task execution status; Figure 7 Flowcharts of methods for task processing in some scenarios are shown; Figure 8 Block diagrams of apparatuses for task processing in several scenarios are shown; and Figure 9 A block diagram of an electronic device capable of implementing multiple illustrative scenarios is shown. Detailed Implementation

[0011] The examples in the text will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0012] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0013] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0015] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] The term "interactive action" as used in this article refers to an action performed by a user or system on a page that can trigger a change in the page state, including but not limited to click actions, input actions, scrolling actions, key presses, waiting actions, etc.

[0017] The term "interactive element" as used in this article refers to elements on a page that can respond to interactive operations, including but not limited to page components that can receive user actions such as buttons, input boxes, links, drop-down menus, checkboxes, and sliders.

[0018] The term "interactive element identifier" as used in this article refers to a short string or code used to uniquely identify an interactive element on a page, such as an element reference identifier (ref), which allows the corresponding interactive element to be precisely located in the page structure.

[0019] The term "interactive element reference mapping" as used in this article refers to a mapping data structure that uses the interactive element identifier as the key and the interactive element's detailed information as the value. It is used to store and retrieve complete attribute information of interactive elements, including but not limited to element type, position, and status.

[0020] The term "replanning processing entity" as used in this article refers to a processing unit that can perform multi-step logical reasoning based on contextual information and generate new operational plans, including but not limited to Large Language Model (LLM), thought chain reasoning engine, or other artificial intelligence processing modules with analysis and planning capabilities.

[0021] The term "operation sequence" as used in this article refers to a set of operations consisting of one or more operation steps arranged in a specific order, used to guide the system to complete a specified task in sequence. The operation steps in the operation sequence may include, but are not limited to, element operations, coordinate clicks, key inputs, page scrolling, waiting, etc.

[0022] The term "contextual information" as used in this article refers to comprehensive information associated with the task execution status, including but not limited to current page status information, execution failure records, task objectives, etc., which is used to help determine subsequent operation plans.

[0023] The term "page state information" as used in this article refers to the descriptive information of interactive elements and their attributes on a page at a specific moment, including but not limited to the status data such as the identifier, type, position, visibility, and interactivity of interactive elements.

[0024] The term "set of interactive elements" used in this article refers to a subset of interactive elements obtained by filtering from all interactive elements contained on a page. This filtering can be based on attributes such as interactivity and visibility of elements to reduce the amount of data related to page state information.

[0025] The term "reference information set" used in this article refers to a collection of operational knowledge and strategy documents pre-organized for a specific platform or application scenario, including but not limited to page structure feature descriptions, interaction rule descriptions, and operation strategy guidelines, which are stored in the form of structured documents and loaded on demand.

[0026] The term "Vision Language Model" (VLM) used in this paper refers to an artificial intelligence model that possesses both image understanding and natural language processing capabilities. It can receive image information and output analysis results on the image content, including but not limited to the position coordinates of target elements in the image and the type judgment of target elements.

[0027] The term “Large Language Model” (LLM) used in this paper refers to a natural language processing model trained on large-scale data, which has the ability to understand, reason, and generate text, and can be used as a decision engine in the execution of automated tasks.

[0028] The term "token" used in this article refers to the basic unit of text processing in large language models. The amount of tokens consumed directly affects the computational cost of model calls and the efficiency of context window utilization.

[0029] The term "intelligent agent" as used in this article refers to a software entity that has the ability to make autonomous decisions and perform tasks, and can automatically plan and execute a series of operational steps to complete a specified task based on user requests.

[0030] As mentioned above, in AI-driven browser automation task execution scenarios, the task execution process often involves multiple consecutive interactive operation steps. In traditional solutions, when an interactive operation fails during task execution, the solution is usually to simply retry a fixed number of times or terminate the process directly, without automatically analyzing the cause of the failure and adjusting subsequent operation strategies accordingly.

[0031] Furthermore, in traditional solutions, when the interactive element location method based on page structure information fails (e.g., the target interactive element is obscured by other page elements, or the page document object model changes dynamically), there is a lack of effective alternative location methods to ensure the continuity of task execution. At the same time, for multiple deterministic interactive operation steps, traditional solutions typically employ a step-by-step invocation of the decision engine, requiring model inference for each operation step, resulting in high computational resource consumption and response latency.

[0032] A task processing scheme is proposed. According to this scheme, when at least one interactive operation of a first task fails, the execution failure record of the first task is obtained. When the execution failure record meets preset conditions, context information associated with at least one interactive operation is obtained, wherein the preset conditions include at least one of the following: the number of execution failures of the first task reaches a first threshold or the location of an interactive element associated with the execution of the first task fails. Then, based on the context information, a first operation sequence is determined to re-execute the first task. Through this scheme, when a task fails, a new operation sequence can be automatically determined based on the execution failure record and context information to re-execute the task, achieving adaptive recovery during task execution. Since the system can automatically trigger replanning after detecting consecutive failures or element location failures, the task can continue to be completed without manual intervention, thereby reducing the interruption rate of task execution. Simultaneously, by determining an operation sequence containing multiple operation steps for batch execution, the number of interaction rounds with the decision engine is reduced, lowering the consumption of computing resources and the bandwidth usage of edge-cloud communication.

[0033] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0034] Figure 1 A schematic diagram of example environment 100 is shown. (e.g.) Figure 1 As shown, the example environment 100 includes an electronic device 120. The electronic device 120 can run an agent 110 that supports task processing. A user 130 can interact with the agent 110 through the electronic device 120. The electronic device 120 can present an interface 140 through the agent 110.

[0035] Electronic device 120 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some cases, electronic device 120 may also support any type of user-facing interface (such as "wearable" circuitry).

[0036] The intelligent agent 110 can be implemented as a software module running on the electronic device 120, or as a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. The intelligent agent 110 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. The intelligent agent 110 can provide background services for applications supporting task processing in the electronic device 120.

[0037] A communication connection can be established between the intelligent agent 110 and the electronic device 120. This communication connection can be established via wired or wireless means. The communication connection can include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections. In some cases, the intelligent agent 110 and the electronic device 120 can interact via their communication connection.

[0038] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0039] The following description of the example will continue with reference to the accompanying drawings.

[0040] Figure 2 A flowchart of an example process 200 for task processing according to some scenarios is shown. Process 200 can be implemented at electronic device 120 / smart agent 110. See below for reference. Figure 2 To describe process 200.

[0041] like Figure 2 As shown in box 205, user 130 requests to perform a first task through electronic device 120. The first task can be any type of automated task, such as performing information queries, data entry, product purchases, ticket reservations, etc. on a web platform. After receiving the request from user 130, electronic device 120 can begin to execute the interactive operation corresponding to the first task through intelligent agent 110.

[0042] In box 210, the electronic device 120 / agent 110 can detect task execution failure during the execution of the first task and trigger a replanning process. Specifically, the electronic device 120 / agent 110 can maintain an execution failure record associated with the first task, which may include a consecutive failure count and detailed information about the failed operation. When the execution failure record meets preset conditions (e.g., the number of consecutive failures reaches a first threshold or the interactive element fails to locate), the electronic device 120 / agent 110 can trigger a replanning process.

[0043] In box 215, electronic device 120 / agent 110 can provide contextual information to the replanning processing entity to obtain a first operation sequence. The contextual information may include current page state information, execution failure records, and the task objective corresponding to the first task. The replanning processing entity analyzes and reasons based on the contextual information to determine a new first operation sequence.

[0044] In box 220, electronic device 120 / intelligent agent 110 can obtain reference content associated with the platform. In some cases, electronic device 120 / intelligent agent 110 can determine the reference content corresponding to the current task execution scenario from a pre-organized set of reference information based on the page content, in order to assist in the accurate execution of subsequent operations. The reference content associated with the platform mentioned here may include reference documents of websites or applications, and each document may contain operations or processing procedures such as page structure features, login detection strategies, form filling rules, data extraction schemes, and anomaly handling.

[0045] For example, electronic device 120 / intelligent agent 110 can determine the first reference content corresponding to the first task from a set of reference information based on the page content. In some embodiments, the page content is associated with the distribution characteristics of interactive elements in the page. For example, electronic device 120 / intelligent agent 110 can determine which platform the page belongs to (e.g., e-commerce platform, online ticketing platform, etc.) based on the position information of each interactive element in the page.

[0046] In some embodiments, the electronic device 120 / intelligent agent 110 can also match the platform according to the user's intent, thereby loading reference content corresponding to the platform and filtering out irrelevant reference content. For example, if the user's intent is to purchase an item, the electronic device 120 / intelligent agent 110 can retain only the reference content associated with the shopping platform.

[0047] In some embodiments, the electronic device 120 / smart agent 110 may also pre-store general reference content (e.g., including domain name pattern matching and scene decision tree) or general reference documents, which can serve as an alternative strategy when no platform-specific reference content or reference document is matched.

[0048] In box 225, electronic device 120 / smart agent 110 can re-execute the first task based on the first operation sequence and reference content. In this way, automatic recovery after task execution failure can be achieved without manual user intervention. Because the system automatically acquires context information and determines a new operation sequence based on the analysis results, it avoids simple retrying of the same failed operation, reducing the consumption of computing resources by invalid operations. Simultaneously, by loading platform-specific reference content on demand instead of loading all reference information at once, the context window occupancy and token consumption of the decision engine are reduced, thereby reducing computational overhead and data transmission volume in edge-cloud communication.

[0049] Figure 3 A flowchart of an example process 300 for dynamic reprogramming based on certain scenarios is shown. Process 300 can be implemented at electronic device 120 / agent 110. See below for reference. Figure 3 To describe process 300.

[0050] like Figure 3 As shown, in box 305, electronic device 120 / smart agent 110 performs a first task. For example, electronic device 120 performs a series of interactive operations on the page corresponding to the first task through smart agent 110.

[0051] In box 310, when the number of failed executions of the first task reaches a first threshold, the electronic device 120 / agent 110 can obtain context information. In some cases, the electronic device 120 / agent 110 can maintain a consecutive failure counter, resetting the counter to zero each time an interaction operation succeeds, and incrementing the counter and recording the operation type and error information of each failed interaction operation. When the consecutive failure count reaches the first threshold (which can be set to an appropriate number of times, such as 2 times, 3 times, etc.), a replanning process is triggered.

[0052] In box 315, the electronic device 120 / agent 110 can determine whether the number of execution failures or replanning attempts is less than a second threshold. In some cases, the electronic device 120 / agent 110 can maintain a replanning attempt counter to record the number of replanning attempts performed. When the number of replanning attempts is less than the second threshold, the replanning operation in box 325 is executed; when the number of replanning attempts reaches or exceeds the second threshold, box 320 returns a failure message. In this way, the system can be prevented from infinitely looping replanning in the same failure scenario. By limiting the maximum number of replanning attempts, the waste of computing resources caused by repeated calls to the decision engine is avoided, and the overall computational overhead of the system is reduced.

[0053] In box 320, after the number of replanning attempts reaches the second threshold, the electronic device 120 / agent 110 finally returns a failure message. The failure message may include a complete report of the failed path and the attempted solutions, so that the user can understand the detailed status of the task execution.

[0054] In box 325, electronic device 120 / smart agent 110 performs replanning. Electronic device 120 / smart agent 110 can obtain the latest state information of the current page, providing a data foundation for subsequent replanning analysis.

[0055] In box 330, electronic device 120 / agent 110 can acquire the set of interactive elements and other contextual information, and use the replanning processing entity to determine a first sequence of operations. For example, electronic device 120 / agent 110 can provide the set of interactive elements and other contextual information to the replanning processing entity.

[0056] In some embodiments, the electronic device 120 / agent 110 may store an interactive element reference map corresponding to each interactive element in the set of interactive elements. The electronic device 120 / agent 110 may provide the replanning processing entity with information on the number of interactive elements associated with the set of interactive elements and information on the range of interactive element references.

[0057] In this way, interactive elements can be quickly located by storing interactive element reference maps locally on the device, and information about the interactive elements involved in the first task can be conveyed with less data by replanning entity mapping relationships.

[0058] In addition, when transmitting context information, the context information can be compressed to reduce the data size and encrypted using appropriate encryption methods.

[0059] In some scenarios, electronic device 120 / intelligent agent 110 can provide page state information, execution failure records, and task objectives to the replanning processing entity. In some examples, page state information can be obtained through page snapshots. For instance, electronic device 120 / intelligent agent 110 can obtain page state information based on the state of each interactive element on the current page.

[0060] In some embodiments, when acquiring a page snapshot, the electronic device 120 / smart agent 110 may return only interactive elements and ignore non-interactive elements, thereby reducing the amount of information that needs to be processed. In some embodiments, the electronic device 120 / smart agent 110 may remove empty structural elements or page elements that do not contain any content, thereby improving the recognition rate of interactive elements.

[0061] In some embodiments, the replanning processing entity may be, for example, a large language model-driven thought chain reasoning engine that can analyze the cause of failure based on the current page state and formulate a new operation plan accordingly. For instance, the replanning processing entity can determine which interactive element failed based on the interactive elements on the page, thereby identifying the cause of the task failure and attempting to replan the operation sequence.

[0062] In some embodiments, the replanning entity can automatically identify and transform different parameters (e.g., converting parameters with the same function but different names into parameters with the same name), and can also automatically fill in default values ​​for missing fields. Furthermore, the replanning entity can also allow retries of certain operation steps after detecting errors in some parameters.

[0063] In this way, the system can generate targeted recovery solutions and provide corresponding operation sequences for different failure scenarios, rather than simply repeating previously failed operations. Because the replanning processing entity only receives a filtered set of interactive elements rather than all elements on the page, the amount of data input to the decision engine (e.g., the replanning processing entity) is reduced, thereby decreasing the decision engine's token consumption and inference time.

[0064] In box 335, electronic device 120 / agent 110 can re-execute the first task based on the first operation sequence. Electronic device 120 / agent 110 can execute the operation steps in the first operation sequence sequentially until the task is completed or replanning is triggered again.

[0065] Figure 4 A flowchart of an example process 400 for locating interactive elements based on image information is shown, depending on certain circumstances. Process 400 can be implemented at electronic device 120 / smart agent 110. See below for reference. Figure 4 To describe process 400.

[0066] like Figure 4 As shown in box 405, electronic device 120 / smart agent 110 detects that the interactive element localization has failed. In some cases, when the interactive element localization process based on the interactive element identifier fails (e.g., the target interactive element is unreachable in the page document object model, is obscured by other elements, or the original identifier becomes invalid due to dynamic changes in the page structure), the interactive element localization process based on image information is triggered.

[0067] In box 410, the electronic device 120 / smart agent 110 triggers the localization of interactive elements based on image information and obtains initial parameters such as the maximum number of iterations. The maximum number of iterations is used to control the number of execution rounds of the visual inference loop (e.g., 5 or 6 times) to prevent infinite iteration.

[0068] In some embodiments, the initial parameters such as the maximum number of iterations can be preset or determined by the replanning processing entity (e.g., an LLM model).

[0069] In box 415, the electronic device 120 / smart agent 110 determines whether the localization of the interactive element has reached the maximum number of iterations. If the maximum number of iterations has not been reached, the image acquisition step in box 425 is continued; if the maximum number of iterations has been reached, box 420 is executed to return the current element localization result.

[0070] In box 420, after reaching the maximum number of iterations, the electronic device 120 / agent 110 returns the current element location result. This result indicates whether the location was successful and the current known location information of the target element.

[0071] In box 425, electronic device 120 / intelligent agent 110 acquires image information of the page, such as taking a screenshot of the current page. Electronic device 120 / intelligent agent 110 can call the page screenshot function to obtain complete visual or image information of the current page for subsequent visual reasoning and analysis.

[0072] In some embodiments, the electronic device 120 / smart agent 110 may acquire and process only image information of a portion of the interactive elements associated with the first task, based on description information associated with the first task. For example, if the first task only requires operation on the upper half of the page, the electronic device 120 / smart agent 110 may acquire only image information or screenshots of the upper half of the page.

[0073] In box 430, electronic device 120 / agent 110 can infer the location information of interactive elements based on image information. In some cases, electronic device 120 / agent 110 can provide a screenshot of the page along with task description information to the visual language model, which analyzes the screenshot and outputs the coordinates of the target interactive element on the page. In some cases, the task description information can guide the visual language model to focus on a specific target element, rather than performing a global scan of the entire page, thereby improving the accuracy of element location.

[0074] If the inference fails, in box 432, electronic device 120 / agent 110 can re-perform the inference and increment the iteration count, returning to box 415 to determine the iteration count. Through this iterative closed-loop mechanism, when the localization result of a single visual inference is insufficient, the system can re-perform the inference based on the updated page screenshot. Since visual inference is only triggered when structured localization fails, rather than being the default localization method for every operation, the system still uses the less computationally expensive structured localization method in most scenarios, switching to visual inference only when necessary. This ensures a high localization success rate while controlling overall computational resource consumption.

[0075] In box 435, electronic device 120 / intelligent agent 110 can determine and execute the interactive operation corresponding to the interactive element. In some cases, electronic device 120 / intelligent agent 110 can determine the operation type corresponding to the target interactive element based on task description information. Operation types include click-type operations or input-type operations. When the operation type is a click-type operation, electronic device 120 / intelligent agent 110 can perform a click operation at the location indicated by the location information. When the operation type is an input-type operation, electronic device 120 / intelligent agent 110 can perform content input at the location indicated by the location information. In some cases, when performing an input-type operation, electronic device 120 / intelligent agent 110 can first click on the target location to give the target input box focus, then clear the existing content and input the target text character by character, to ensure compatibility with controlled components in different types of front-end frameworks.

[0076] In some embodiments, the electronic device 120 / smart agent 110 may specify the target text to be filled in. This target text may be specified by the user, or automatically generated based on the task objectives of the first task.

[0077] In box 440, electronic device 120 / smart agent 110 can wait for a preset duration to allow time for page changes before performing subsequent operations. In some cases, when an interaction causes a page change, electronic device 120 / smart agent 110 can wait for a preset time period before continuing with subsequent steps to ensure that subsequent operations are based on the latest page state that has been loaded.

[0078] In box 445, electronic device 120 / smart agent 110 can execute the next step. Through the above-described visual reasoning closed-loop process, when conventional structured element localization fails, the system can automatically switch to image-based localization, utilizing the image understanding capabilities of the visual language model to locate the target interactive element. Since visual reasoning directly obtains element coordinates through screenshot analysis, without relying on the structured information of the page document object model, it can handle scenarios such as canvas elements and dynamically rendered content that cannot be located using structured methods. Simultaneously, by limiting the maximum number of iterations, the visual reasoning loop is prevented from executing indefinitely due to continuous failures, thereby controlling the number of calls to the visual language model and the corresponding computational resource consumption.

[0079] Figure 5 A flowchart of an example process 500 for batch execution of multiple operation steps is shown, depending on certain scenarios. Process 500 can be implemented at electronic device 120 / smart agent 110. See below for reference. Figure 5 To describe process 500.

[0080] like Figure 5 As shown in box 505, electronic device 120 / agent 110 can acquire multiple operation steps for interactive elements. In some cases, these multiple operation steps can be acquired from a replanning processing entity or determined by agent 110 during the initial task planning phase. Each of the multiple operation steps can include different types of operations such as element manipulation, coordinate click, key input, page scrolling, and waiting.

[0081] In some embodiments, multiple operation steps can be acquired simultaneously (i.e., multiple operation steps can be acquired at once), thereby reducing the number of accesses to the replanning processing entity and improving operational efficiency.

[0082] In box 510, electronic device 120 / intelligent agent 110 can sequentially execute multiple operation steps. Electronic device 120 / intelligent agent 110 can execute all steps sequentially on the server according to the order of the operation steps, without the need for step-by-step participation of the decision engine. In this way, multiple deterministic operations are packaged into a single batch submission. Compared to step-by-step calls to the decision engine, this significantly reduces the number of interaction rounds with the decision engine, lowers the data transmission volume in edge-cloud communication, and reduces the number of inferences performed by the decision engine, thereby shortening the overall response latency of multi-step operations.

[0083] In box 515, the electronic device 120 / agent 110 can determine whether to stop execution if any step fails. In some cases, the error control strategy is automatically determined by the decision engine based on the task scenario.

[0084] In box 520, when the option to stop execution upon failure of any step is selected, the electronic device 120 / agent 110 can terminate execution and return the result and error information upon detecting any failed interaction. In this way, when the execution result of an operation step depends on subsequent steps, execution can be stopped immediately to prevent subsequent steps from continuing based on an error state, reducing the computational resource consumption of invalid operations.

[0085] In box 525, when the option to not stop execution upon step failure is selected, the electronic device 120 / agent 110 can continue executing subsequent steps even if any interaction operation fails. The electronic device 120 / agent 110 records the error information corresponding to the failed step and continues to execute subsequent operation steps in sequence. In this way, for scenarios where operation steps are independent of each other, the failure of one step will not block the execution of other steps, improving the overall completion rate of batch operations.

[0086] In box 530, if the execution of a step causes a page change, the electronic device 120 / intelligent agent 110 can reacquire the page information. In some cases, when the electronic device 120 / intelligent agent 110 detects a page change during the execution of multiple operation steps, it can reacquire the information of the interactive elements on the page after a preset time period (e.g., 500 milliseconds or 600 milliseconds) to ensure that subsequent steps can be executed correctly based on the updated page state. The detection of page changes can be based on the type attribute of the interactive element being operated on. For example, when the interactive element being operated on is a button, link, tab, or other element that may cause the page to jump or refresh, a waiting period and reacquisition of page information are automatically triggered. Through this intelligent page change detection mechanism, the failure of subsequent operations due to the page not being fully loaded is avoided, invalid operation retries caused by inconsistent page states are reduced, and the overall consumption of computing resources is reduced.

[0087] In box 535, the batch execution of steps is complete. Through the above batch operation process, multiple deterministic operation steps are packaged into a single submission and executed sequentially, eliminating the need for the decision engine to participate in the decision-making process of each operation step step by step. As a result, the number of calls to the decision engine is reduced from once per step to a single batch call, significantly reducing token consumption and model inference time, while also reducing the number of round trips and bandwidth usage in edge-cloud communication.

[0088] Figure 6 A schematic diagram of an interface 600 for presenting task execution status according to certain scenarios is shown. See below for reference. Figure 6 This describes the interface presentation during task execution.

[0089] like Figure 6As shown, interface 600 includes an input box 605, a prompt message 610, and task status information 615. In some cases, user 130 can send a task request to agent 110 through input box 605 to perform a first task. For example, user 130 can enter "Help me buy item B on platform A" in input box 605. After receiving this input, electronic device 120 / agent 110 can obtain reference content corresponding to platform A and attempt to perform the purchase task. If electronic device 120 / agent 110 detects task execution failure, electronic device 120 / agent 110 can use the above combination... Figures 2 to 4 The described replanning process attempts to execute the task again.

[0090] When an execution failure is detected during the execution of the first task and a rescheduling is triggered, the electronic device 120 can display a prompt message 610 on the interface 600. This prompt message 610 indicates that the first task is being rescheduled and re-executed. The prompt message 610 can also indicate whether the first task was successfully re-executed.

[0091] In addition, the interface can also display task status information 615, which shows the current execution progress and status of the task. By displaying the task execution status and replanning prompts in real time on the interface, users can understand the progress of task execution in a timely manner, reducing the number of times users repeatedly initiate query requests due to uncertainty about the task status. As a result, the computing resources required for the electronic device 120 to process repeated query requests and the amount of communication data between the electronic device 120 and the intelligent agent 110 are reduced.

[0092] Figure 7 A flowchart of an example process 700 for task processing according to some scenarios is shown. Process 700 can be implemented at electronic device 120. See below for reference. Figure 7 To describe process 700.

[0093] like Figure 7 As shown in block 710, electronic device 120 obtains a record of execution failure of the first task in response to at least one interactive operation of the first task failing.

[0094] In box 720, in response to the execution failure record meeting preset conditions, electronic device 120 obtains context information associated with at least one interactive operation, wherein the preset conditions include at least one of the following: the number of execution failures of the first task reaches a first threshold or the location of an interactive element associated with the execution of the first task fails.

[0095] In box 730, electronic device 120 determines a first sequence of operations based on context information to re-execute the first task.

[0096] In this way, when an interactive operation failure is detected during task execution, the system automatically retrieves the failure record and determines a new operation sequence based on context information to re-execute the task. This allows the task to adaptively recover after a failure, reducing the number of task interruptions and the frequency of manual intervention. Because the system comprehensively considers context information such as page state, failure records, and task objectives when determining the new operation sequence, the generated operation sequence is targeted, avoiding simple repetition of failed operations. This reduces the number of invalid operations that call the decision engine and the corresponding computational resource consumption, while also reducing the bandwidth usage for transmitting duplicate requests in edge-cloud communication.

[0097] In some situations, electronic device 120 can determine the first operation sequence based on context information in response to the fact that the number of times the first task is re-executed is less than a second threshold. By setting an upper limit on the number of re-executions in this way, the system can be prevented from repeatedly replanning in the same failure scenario. Because the maximum number of replanning attempts is limited, the number of calls to the decision engine is effectively controlled, avoiding the waste of computing resources and the continuous increase in token consumption caused by infinite loop replanning.

[0098] In some scenarios, contextual information may include at least one of the following: first page state information, indicating the state of the page associated with the execution of the first task when at least one interactive operation fails; an execution failure record, wherein the execution failure record includes the operation type and error information corresponding to the operation that failed in at least one interactive operation; or a task objective corresponding to the first task. In this way, contextual information describes the complete state when a task fails from multiple dimensions, enabling the replanning processing entity to analyze and make decisions based on sufficient information. Because the contextual information covers page state, failure details, and task objectives, the decision engine can more accurately pinpoint the cause of failure and generate effective recovery solutions, reducing multiple rounds of invalid reasoning due to insufficient information, and lowering the number of reasoning iterations and token consumption of the decision engine.

[0099] In some scenarios, the electronic device 120 can provide the first page state information, execution failure records, and task objectives to the replanning processing entity, and receive the first operation sequence determined by the replanning processing entity based on the first page state information, execution failure records, and task objectives. In this way, the replanning processing entity can comprehensively analyze the current page state and historical failure information to formulate new operation plans. By delegating the tasks of analysis and planning to a dedicated replanning processing entity, task execution and task planning are decoupled, improving the system's modularity and maintainability. Simultaneously, by receiving complete context information and outputting the operation sequence at once, the replanning processing entity reduces multi-round interactions with the decision engine, lowering communication overhead and response latency.

[0100] In some scenarios, electronic device 120 can acquire a set of interactive elements corresponding to a page, which is obtained by filtering from all interactive elements contained in the page. Electronic device 120 can determine the information associated with each interactive element in the set as the first page state information to provide for the replanning processing entity. In this way, the page state information only includes the filtered interactive elements, rather than all elements in the page. Because non-interactive display elements are filtered out, the amount of page information data provided to the decision engine is significantly reduced, decreasing the token consumption required for the decision engine to process each page snapshot, shortening the inference time of the decision engine, and improving the system's response speed.

[0101] In some scenarios, electronic device 120 can store an interactive element reference map corresponding to each interactive element in the interactive element set, and provide the replanning processing entity with information on the number of interactive elements associated with the interactive element set and the scope of interactive element references. In this way, the complete interactive element reference map is stored on the server, and only a summary of the number of elements and the scope of references is passed to the decision engine. Thus, the amount of data passed to the decision engine is reduced from a complete mapping table to lightweight summary information, significantly reducing the token consumption per page snapshot and the data transmission volume of end-to-cloud communication, while also reducing the storage read / write overhead on the server side.

[0102] In some scenarios, the first interactive element targeted by the first operation in at least one interactive operation is determined by an interactive element identifier, and interactive element location failure includes failure of the interactive element location process based on the interactive element identifier. In this way, when the structured element location method (location based on interactive element identifier) ​​fails, the failure event is included in the preset condition judgment of the execution failure record, which can trigger subsequent alternative location processes. Because the judgment conditions for interactive element location failure are clearly defined, the system can accurately identify scenarios requiring a change in location strategy, avoiding unnecessary activation of computationally expensive visual reasoning processes due to misjudgment, thereby controlling overall computational resource consumption while ensuring the location success rate.

[0103] In some scenarios, electronic device 120 can, in response to determining that the failure of interactive element localization is associated with an interactive element identifier, acquire image information of the page associated with the execution of the first task. Electronic device 120 can determine the position information of the first interactive element on the page based on the image information. Electronic device 120 can determine a first operation sequence based on the position information, the first operation sequence including operations on the first interactive element. In this way, when structured localization based on the interactive element identifier fails, the system automatically switches to visual inference localization based on image information, achieving automatic degradation from structured localization to visual localization. Since visual inference is only triggered when structured localization fails, the system still uses lightweight structured localization in most scenarios, switching to computationally more expensive visual inference only when necessary, thereby controlling the overall frequency of visual language model calls and computational resource consumption.

[0104] In some scenarios, electronic device 120 can determine the operation type corresponding to the first interactive element based on task description information, where the operation type includes click-type operations or input-type operations. In response to an operation type of click-type operation, electronic device 120 can determine that the first operation sequence includes performing a click operation at the location indicated by the location information. In response to an operation type of input-type operation, electronic device 120 can determine that the first operation sequence includes performing content input at the location indicated by the location information. In this way, the system can automatically determine the required operation type for the target element based on the task description, without requiring additional user specification or rule configuration. Since the operation type determination and visual reasoning positioning are completed in the same model call, there is no need to initiate an additional model reasoning request separately for operation type determination, thereby reducing the number of model calls and token consumption.

[0105] In some scenarios, the first operation sequence may include multiple operation steps. The electronic device 120 can execute these multiple operation steps based on their sequential arrangement to complete the first task. In this way, multiple deterministic operation steps are packaged into the same operation sequence and submitted for execution at once, eliminating the need for the decision engine to participate in the decision-making process for each operation step step by step. Consequently, the number of calls to the decision engine is reduced from once per step to once for the entire process, significantly reducing token consumption and model inference time, while also reducing the number of round trips in edge-cloud communication.

[0106] In some scenarios, the electronic device 120 can respond to detecting page changes during the execution of multiple operation steps and continue executing subsequent steps after a preset time period. In this way, when the execution of an operation step causes a page change, the system automatically waits for the page to load completely before continuing with subsequent steps, avoiding subsequent operations that execute on an outdated page state due to an incomplete page load. This reduces operation failures and retries caused by inconsistent page states, lowers the consumption of computing resources by invalid operations, and reduces the additional overhead of edge-cloud communication.

[0107] In some scenarios, executing multiple operation steps to complete a first task may include any of the following: stopping the execution of subsequent operation steps in the first operation sequence in response to the failure of any one of the multiple operation steps; or recording the error information corresponding to that operation step and continuing to execute subsequent operation steps in response to the failure of any one of the multiple operation steps. In this way, the system provides two flexible error control strategies, enabling the selection of an appropriate handling method based on the specific needs of the task scenario. For scenarios where there are strong dependencies between operation steps, timely stopping execution can prevent subsequent steps from running based on an error state, reducing the consumption of computing resources by invalid operations. For scenarios where operation steps are independent of each other, skipping failed steps and continuing execution can improve the overall completion rate of batch operations, reduce the rollback and re-execution of all operations due to the failure of a single step, and reduce overall computational and communication overhead.

[0108] In some scenarios, electronic device 120 can acquire a set of reference information indicating a task execution strategy associated with the first task. Electronic device 120 can determine first reference content corresponding to the first task from the reference information set based on page content, where the page content is associated with the distribution characteristics of interactive elements on the page. Electronic device 120 can re-execute the first task based on the first reference content and a first operation sequence. In this way, the system can load corresponding reference content on demand for different page platforms, rather than providing all platform reference information to the decision engine at once. Because only reference content relevant to the current task scenario is loaded, the context window occupancy and token consumption of the decision engine are reduced. Simultaneously, the platform-specific reference content contains operation strategies for specific page structures and interaction patterns, enabling the decision engine to generate operation plans more accurately, reducing the number of trial and error attempts due to a lack of platform knowledge, thereby reducing the computational overhead of repetitive reasoning and the data transmission volume of edge-cloud communication.

[0109] In some situations, the electronic device 120 can display a prompt message on the interface indicating whether the first task was successfully re-executed. This allows users to promptly understand the result of the task re-execution, reducing the number of times users repeatedly initiate query requests due to uncertainty about the task status. Consequently, this reduces the computing resources required by the electronic device 120 to process recurring status query requests, while also lowering the end-to-cloud communication overhead for transmitting recurring queries.

[0110] A corresponding apparatus for implementing the above methods or processes is also provided.

[0111] Figure 8 A block diagram of a task processing device 800 is shown for some scenarios. Device 800 can be implemented as or included in electronic device 120. The various modules / components in device 800 can be implemented by hardware, software, firmware, or any combination thereof.

[0112] like Figure 8 As shown, the device 800 includes a failure record acquisition module 810, configured to acquire an execution failure record of the first task in response to the failure of at least one interactive operation of the first task; a context acquisition module 820, configured to acquire context information associated with at least one interactive operation in response to the execution failure record meeting preset conditions, wherein the preset conditions include at least one of the following: the number of execution failures of the first task reaches a first threshold or the location of an interactive element associated with the execution of the first task fails; and a sequence determination module 830, configured to determine a first operation sequence based on the context information to re-execute the first task.

[0113] In some cases, the sequence determination module 830 can also be configured to determine the first operation sequence based on context information in response to the number of times the first task is re-executed being less than a second threshold.

[0114] In some cases, the context acquisition module 820 may also be configured to acquire context information including at least one of the following: first page state information, which indicates the state of the page associated with the execution of the first task when at least one interactive operation fails; execution failure record, wherein the execution failure record includes the operation type and error information corresponding to the operation that failed in at least one interactive operation; or the task target corresponding to the first task.

[0115] In some cases, the sequence determination module 830 may also be configured to provide the first page state information, execution failure records and task objectives to the replanning processing entity, and receive the first operation sequence determined by the replanning processing entity based on the first page state information, execution failure records and task objectives.

[0116] In some cases, the context acquisition module 820 can also be configured to acquire a set of interactive elements corresponding to the page, which is obtained by filtering from all interactive elements contained in the page, and determine the information associated with each interactive element in the set of interactive elements as the first page state information to provide the re-planning processing entity.

[0117] In some cases, the context acquisition module 820 can also be configured to store an interactive element reference mapping corresponding to each interactive element in the interactive element set, and provide the replanning processing entity with information on the number of interactive elements associated with the interactive element set and the interactive element reference range information.

[0118] In some cases, the failure record acquisition module 810 may also be configured to determine that the first interactive element targeted by the first operation in at least one interactive operation is determined by an interactive element identifier, and interactive element location failure includes failure of the interactive element location process based on the interactive element identifier.

[0119] In some cases, the sequence determination module 830 may also be configured to, in response to determining that the failure to locate an interactive element is associated with an interactive element identifier, acquire image information of the page associated with the execution of the first task, determine the position information of the first interactive element on the page based on the image information, and determine a first operation sequence based on the position information, the first operation sequence including operations on the first interactive element.

[0120] In some cases, the sequence determination module 830 can also be configured to determine the operation type corresponding to the first interactive element based on the task description information, wherein the operation type includes a click operation or an input operation. In response to the operation type being a click operation, the first operation sequence is determined to include performing a click operation on the location indicated by the location information. In response to the operation type being an input operation, the first operation sequence is determined to include performing content input on the location indicated by the location information.

[0121] In some cases, the sequence determination module 830 can also be configured to execute multiple operation steps to complete the first task based on the order of the multiple operation steps in the first operation sequence when the first operation sequence includes multiple operation steps.

[0122] In some cases, the sequence determination module 830 can also be configured to continue executing subsequent steps of the multiple operation steps after a preset time period in response to detecting a page change during the execution of multiple operation steps.

[0123] In some cases, the sequence determination module 830 may also be configured to perform any of the following: in response to the failure of any one of the multiple operation steps, stop the execution of subsequent operation steps in the first operation sequence; or in response to the failure of any one of the multiple operation steps, record the error information corresponding to that operation step and continue to execute subsequent operation steps.

[0124] In some cases, the context acquisition module 820 can also be configured to acquire a set of reference information, which indicates a task execution strategy associated with the first task, and determine the first reference content corresponding to the first task from the set of reference information based on the page content, wherein the page content is associated with the distribution characteristics of interactive elements in the page; the sequence determination module 830 can also be configured to re-execute the first task based on the first reference content and the first operation sequence.

[0125] In some cases, device 800 can also be configured to display a prompt message on the interface indicating whether the first task was successfully re-executed.

[0126] The modules included in device 800 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 800 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0127] Figure 9 A block diagram of an electronic device 900 in which one or more examples can be implemented is shown. It should be understood that... Figure 9 The electronic device 900 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 9 The illustrated electronic device 900 can be used to implement the electronic device 120 discussed above.

[0128] like Figure 9 As shown, electronic device 900 is in the form of a general-purpose electronic device. Components of electronic device 900 may include, but are not limited to, one or more processing units or processors 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processor 910 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 920. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 900.

[0129] Electronic device 900 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 900.

[0130] Electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 9 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various examples.

[0131] The communication unit 940 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 900 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 900 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0132] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 900 can also communicate with one or more external devices (not shown) via communication unit 940 as needed. External devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 900, or with any device that enables electronic device 900 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0133] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0134] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0136] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0137] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0138] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A task processing method, comprising: In response to the failure of at least one interactive operation of the first task, obtain the execution failure record of the first task; In response to the execution failure record meeting a preset condition, context information associated with the at least one interactive operation is obtained, wherein the preset condition includes at least one of the following: the number of execution failures of the first task reaches a first threshold or the location of an interactive element associated with the execution of the first task fails. as well as Based on the context information, a first operation sequence is determined to re-execute the first task.

2. The method of claim 1, wherein determining the first operation sequence to re-execute the first task comprises: In response to the fact that the number of times the first task is re-executed is less than a second threshold, the first operation sequence is determined based on the context information.

3. The method of claim 1, wherein the context information includes at least one of the following: First page status information, indicating the status of the page associated with the execution of the first task when the at least one interactive operation fails. The execution failure record, wherein the execution failure record includes the operation type and error information corresponding to the operation that failed to be executed in the at least one interactive operation, or The task objective corresponding to the first task.

4. The method of claim 3, wherein determining the first operation sequence to re-execute the first task based on the context information comprises: The first page status information, the execution failure record, and the task objective are provided to the re-planning and processing entity. as well as Receive the first operation sequence determined by the replanning processing entity based on the first page status information, the execution failure record, and the task objective.

5. The method according to claim 4, further comprising: Obtain a set of interactive elements corresponding to the page, wherein the set of interactive elements is obtained by filtering from all interactive elements contained in the page; as well as The information associated with each interactive element in the set of interactive elements is determined as the first page state information and provided to the replanning processing entity.

6. The method of claim 5, wherein the method further comprises: Store an interactive element reference mapping corresponding to each interactive element in the set of interactive elements; as well as The replanning processing entity is provided with information on the number of interactive elements associated with the set of interactive elements and information on the scope of interactive element references.

7. The method of claim 1, wherein the first interactive element targeted by the first operation in the at least one interactive operation is determined by an interactive element identifier, and the interactive element location failure includes an interactive element location process failure based on the interactive element identifier.

8. The method according to claim 7, further comprising: In response to determining that the failure to locate the interactive element is associated with the interactive element identifier, image information of the page associated with the execution of the first task is obtained; Based on the image information, the position information of the first interactive element in the page is determined; as well as The first operation sequence is determined based on the location information, and the first operation sequence includes operations on the first interactive element.

9. The method of claim 8, wherein determining the first operation sequence based on the location information comprises: The operation type corresponding to the first interactive element is determined based on the task description information, wherein the operation type includes click-type operations or input-type operations; In response to the operation type being a click-type operation, it is determined that the first operation sequence includes performing a click operation on the location indicated by the location information; as well as In response to the operation type being an input operation, it is determined that the first operation sequence includes performing content input at the location indicated by the location information.

10. The method of claim 1, wherein the first operation sequence comprises a plurality of operation steps, and the first task is re-executed based on the first operation sequence, comprising: Based on the order of the multiple operation steps in the first operation sequence, the multiple operation steps are executed to complete the first task.

11. The method of claim 10, wherein performing the plurality of operational steps to complete the first task includes: In response to a page change detected during the execution of the plurality of operation steps, subsequent steps in the plurality of operation steps are executed after a preset time period.

12. The method of claim 10, wherein performing the plurality of operational steps to complete the first task comprises any one of the following: In response to the failure of any of the plurality of operation steps, the execution of subsequent operation steps in the first operation sequence is stopped, or In response to the failure of any of the plurality of operation steps, the error information corresponding to the failure of the failure step is recorded and the subsequent operation steps are executed.

13. The method according to claim 1, wherein the method further comprises: Obtain a set of reference information, which indicates a task execution strategy associated with the first task; Based on the page content, a first reference content corresponding to the first task is determined from the reference information set, wherein the page content is associated with the distribution characteristics of interactive elements in the page; as well as The first task is re-executed based on the first reference content and the first operation sequence.

14. The method according to claim 1, wherein the method further comprises: A prompt message is displayed on the interface, indicating whether the first task has been successfully re-executed.

15. A task processing apparatus, comprising: The failure record acquisition module is configured to acquire the execution failure record of the first task in response to the failure of at least one interactive operation of the first task. The context acquisition module is configured to acquire context information associated with the at least one interactive operation in response to the execution failure record meeting a preset condition, wherein the preset condition includes at least one of the following: the number of execution failures of the first task reaches a first threshold or the location of an interactive element associated with the execution of the first task fails. as well as The sequence determination module is configured to determine a first operation sequence based on the context information to re-execute the first task.

16. An electronic device comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processing unit.

17. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 14.

18. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 14.

Images