Task execution method and device, equipment, storage medium and program product

Abstract

The invention relates to a task execution method and device, equipment, a storage medium and a program product. The method can comprise the steps of obtaining a task instruction input by a user, wherein the task instruction is used for instructing a third-party application to execute a target task; in response to the task instruction, screen interface information of the third-party application is obtained, and multiple interactive operations are sequentially executed for the third-party application based on the screen interface information until it is confirmed that the target task is completed; wherein the screen interface information on which each interactive operation is based comprises a current screen interface corresponding to the interactive operation. According to the technical scheme, the current screen interface of the third-party application is dynamically obtained, and the next interactive operation is executed based on the current screen interface, so that the target task is gradually propelled until the target task is completed, a complex task can be effectively completed, and the execution capability on a high-level user instruction is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a task execution method, device, equipment, storage medium and program product. BACKGROUND

[0002] With the evolution of terminal human-computer interaction mode, users increasingly expect to drive terminals to complete complex operations autonomously through simple instructions. However, the related art still has obvious limitations in realizing automatic response to high-level user instructions. SUMMARY

[0003] The present disclosure provides a task execution method, device, equipment, storage medium and program product to solve the deficiencies in the related art.

[0004] According to a first aspect of an embodiment of the present disclosure, a task execution method is provided, which includes: obtaining a task instruction input by a user, the task instruction being used to instruct a target task to be executed on a third-party application; in response to the task instruction, obtaining screen interface information of the third-party application, and sequentially executing a plurality of interaction operations on the third-party application based on the screen interface information until it is confirmed that the target task has been completed; wherein the screen interface information on which each interaction operation is based includes a current screen interface corresponding to the interaction operation.

[0005] Embodiments of the present disclosure can effectively complete complex tasks and significantly improve the execution capability of high-level user instructions by dynamically obtaining the current screen interface of the third-party application and autonomously deciding and executing the next interaction operation based on the current screen interface to gradually push the target task until completion.

[0006] In some possible implementation manners, the obtaining of the task instruction input by the user includes: obtaining a voice instruction input by the user, the voice instruction being used to instruct a first page of the third-party application to be opened.

[0007] Embodiments of the present disclosure can directly enter the deep page of the third-party application without the need for the user to navigate step by step by obtaining the voice instruction input by the user and autonomously executing a series of interface perception and interaction operations based on the voice instruction. That is, embodiments of the present disclosure can realize direct access to the deep page by only one voice instruction, significantly simplifying the operation process and improving the task execution efficiency and user experience.

[0008] In some possible implementation manners, the obtaining of the screen interface information of the third-party application and the sequentially executing of the plurality of interaction operations on the third-party application based on the screen interface information until it is confirmed that the target task has been completed includes: Before confirming the completion of the target task, the following process is executed repeatedly: Obtain a screenshot of the third-party application; Based on the task instructions and the screenshot, determine the interactive operation to be performed for the third-party application; The interactive operation is performed on the third-party application, and after the interactive operation is performed, a new screenshot of the third-party application is generated.

[0009] The embodiments disclosed herein ensure that each interactive operation is strictly based on the latest interface of the third-party application through this loop mechanism, thereby ensuring the accuracy of the determined interactive operation.

[0010] In some possible implementations, determining the interactive operation to be performed on the third-party application based on the task instruction and the screenshot includes: Obtain supplementary information input by the user; Based on the task instructions, the screenshot, and the supplementary information, the interactive operation to be performed on the third-party application is determined.

[0011] In this embodiment, the user can input supplementary information during the execution of the target task. After obtaining the supplementary information, the system uses it, along with the original task instruction and the current screenshot, as the basis for decision-making to determine the next interactive operation. This mechanism can effectively compensate for the deficiencies in semantic completeness or operational details of task instructions, significantly improving the accuracy of interactive operations and the fit of task execution, thereby improving the overall completion quality of the target task and user satisfaction.

[0012] In some possible implementations, obtaining supplementary information input by the user includes at least one of the following: The system obtains first information actively input by the user during the execution of the target task, parses the first information, and determines the first information as supplementary information input by the user if the first information is related to the target task. After performing an interactive operation to obtain additional information required for task execution, the system obtains second information input by the user through the feedback interface and identifies the second information as supplementary information input by the user.

[0013] This embodiment of the disclosure monitors and acquires user-initiated information in real time during the execution of the target task, performs semantic analysis and relevance judgment on the information, and only incorporates it as supplementary information into the task execution process when it is confirmed that it is related to the current task target. This enables the system to dynamically respond to the user's temporary intention adjustment or personalized needs during the task process, effectively avoids irrelevant voice interference from erroneously affecting task execution, and improves the robustness and accuracy of the system. This mechanism gives the task execution process higher interactive flexibility and intelligent adaptability.

[0014] The embodiments disclosed herein can autonomously identify missing information nodes in task execution even in the absence of complete initial instructions, and complete the information completion through guided interaction, thereby ensuring the smooth progress of multi-step tasks.

[0015] In some possible implementations, the method further includes: after determining the interactive operation to be performed for the third-party application, recording the interactive operation in the execution record corresponding to the task instruction; The step of determining the interactive operation to be executed for the third-party application based on the task instruction and the screenshot includes: obtaining the interactive operation that has been executed in response to the task instruction from the execution record corresponding to the task instruction; and determining the interactive operation to be executed for the third-party application based on the task instruction, the screenshot, and the interactive operation that has been executed in response to the task instruction.

[0016] This disclosure uses executed interactive operations as reference information for decision-making, enabling a more accurate understanding of the current task stage during decision-making. This allows for the generation of the next interactive operation that better meets the needs of task advancement, effectively avoiding repetitive operations, logical conflicts, or path loops, thereby improving the accuracy of task execution.

[0017] In some possible implementations, performing the interactive operation on the third-party application, and generating a new screenshot of the third-party application after performing the interactive operation, includes: An operation instruction for instructing the interactive operation is sent to the corresponding client, so that the client performs the interactive operation on the third-party application according to the operation instruction, and then takes a screenshot of the screen interface of the third-party application to obtain a new screenshot of the third-party application.

[0018] The embodiments disclosed herein complete decision-making through the cloud and execute interactive operations through the client, giving full play to the powerful processing capabilities of the cloud and avoiding the problem of limited terminal processing resources affecting execution efficiency.

[0019] In some possible implementations, determining the interactive operation to be performed on the third-party application based on the task instruction and the screenshot includes: Input the task instructions and the screenshot into the pre-trained operation prediction model; Obtain the interactive operations to be executed for the third-party application, output by the operation prediction model.

[0020] This disclosure introduces a model to jointly analyze the screen interface information and task instructions of third-party applications, predicting the most likely interactive operations to be executed in the next stage. This enables the system to autonomously generate operation sequences that conform to user intent and application logic without pre-configuring operation rules or relying on manual scripts. This mechanism significantly improves the intelligence and generalization ability of the task execution process. For different applications, different interface layouts, and even interface changes after version updates, the system can dynamically adjust the operation strategy based on the semantic understanding of the model, avoiding the problem that traditional methods based on fixed path matching or control ID recognition are prone to failure due to interface changes.

[0021] In some possible implementations, the processing of the operational prediction model includes: Visual semantic understanding is performed on the screenshot to obtain semantic information that represents the interface content of the screenshot; Based on the semantic information and the task instructions, the interactive operation to be performed on the third-party application is determined.

[0022] In this embodiment of the disclosure, the operation prediction model first performs visual semantic understanding on the screenshot, converting it into semantic information that can represent the content of the interface, and then combines it with the task instructions to determine the next interactive operation. This mechanism enables the model to go beyond the shallow perception of the original image and truly understand the functional roles and task relevance of each element in the current interface, thereby improving the interpretability and accuracy of the decision.

[0023] In some possible implementations, the training process of the operation prediction model includes: Acquire sample data, which includes sample task instructions, multiple sample interaction operations arranged in execution sequence, and sample screenshots on which each sample interaction operation is based. According to the execution sequence of the multiple sample interaction operations, the sample screenshots on which the sample interaction operations are based and the sample task instructions are sequentially input into the operation prediction model to obtain the predicted interaction operations output by the operation prediction model. The operation prediction model is trained based on the differences between each predicted interaction operation and the corresponding sample interaction operation.

[0024] In this embodiment of the disclosure, the training process of the operation prediction model adopts a timing alignment method consistent with the actual task execution logic, enabling the model to fully learn the dynamic mapping relationship between the task objective and the interface state, accurately capture the context dependency and state evolution law of multi-step tasks, and thus exhibit stronger generalization ability and execution stability in complex and long-sequence tasks.

[0025] In some possible implementations, the interactive operation is characterized by at least one of the following parameters: operation type, operation location, and operation content.

[0026] This disclosure uses the operation location to refer to the object being operated on, so that the system does not need to know the object information of the object being operated on, and can simulate user operation only based on the operation location. Therefore, there is no need for third-party applications to report the control information of the interface control, which can further expand the application scenarios of this disclosure.

[0027] In some possible implementations, the method further includes: parsing the task instruction to obtain a parsing result, the parsing result being used to indicate whether the target task requires multiple interactive operations to complete; The step of obtaining the screen interface information of the third-party application in response to the task instruction includes: when the parsing result indicates that the target task requires multiple interactive operations to complete, obtaining the screen interface information of the third-party application in response to the task instruction.

[0028] This embodiment parses task instructions before execution to determine if they are complex tasks requiring multiple interactions. Only when a task is complex is the acquisition of third-party application screen interface information and subsequent interface-based interaction processes initiated. This ensures that interface perception and interaction determination are triggered only when necessary. This mechanism effectively avoids redundant interface analysis processes for simple tasks requiring no multiple steps, significantly reducing the consumption of device computing resources and improving overall task scheduling efficiency and system response performance.

[0029] In some possible implementations, confirming that the target task has been completed includes: In response to the current screen interface of the third-party application matching the task instruction, it is determined that the target task has been completed.

[0030] This embodiment of the disclosure automatically confirms whether the target task has been completed by determining whether the current screen interface of the third-party application matches the task command entered by the user. Regardless of how its specific layout, application version or page path changes, the system can accurately identify whether the task has been completed.

[0031] According to a second aspect of the present disclosure, a task execution apparatus is provided, the apparatus comprising: The acquisition unit is configured to acquire a task instruction input by a user, the task instruction being used to instruct a third-party application to perform a target task. The execution unit is configured to, in response to the task instruction, obtain the screen interface information of the third-party application, and perform multiple interactive operations on the third-party application in sequence based on the screen interface information, until it is confirmed that the target task has been completed; The screen interface information on which each interactive operation is based includes the current screen interface corresponding to the interactive operation.

[0032] In some possible implementations, the acquisition unit is configured to acquire a voice command input by the user, the voice command being used to instruct the opening of the first page of the third-party application.

[0033] In some possible implementations, the execution unit is configured to repeatedly perform the following processing before confirming that the target task has been completed: Obtain a screenshot of the third-party application; Based on the task instructions and the screenshot, determine the interactive operation to be performed for the third-party application; The interactive operation is performed on the third-party application, and after the interactive operation is performed, a new screenshot of the third-party application is generated.

[0034] In some possible implementations, the execution unit is configured to acquire supplementary information input by the user; and based on the task instruction, the screenshot, and the supplementary information, determine the interactive operation to be performed for the third-party application.

[0035] In some possible implementations, the execution unit is configured to perform at least one of the following: The system obtains first information actively input by the user during the execution of the target task, parses the first information, and determines the first information as supplementary information input by the user if the first information is related to the target task. After performing an interactive operation to obtain additional information required for task execution, the system obtains second information input by the user through the feedback interface and identifies the second information as supplementary information input by the user.

[0036] In some possible implementations, the device further includes: The recording unit is configured to record the interaction operation to be performed on the third-party application into the execution record corresponding to the task instruction after determining the interaction operation to be performed on the third-party application. The execution unit is configured to obtain interactive operations in response to the executed task instruction from the execution record corresponding to the task instruction; and to determine interactive operations to be executed for the third-party application based on the task instruction, the screenshot, and the interactive operations in response to the executed task instruction.

[0037] In some possible implementations, the execution unit is configured to send an operation instruction to the corresponding client to instruct the interactive operation, so that after the client performs the interactive operation on the third-party application according to the operation instruction, it takes a screenshot of the screen interface of the third-party application to obtain a new screenshot of the third-party application.

[0038] In some possible implementations, the execution unit is configured to input the task instruction and the screenshot into a pre-trained operation prediction model; and obtain the interactive operation to be executed for the third-party application output by the operation prediction model.

[0039] In some possible implementations, the processing of the operation prediction model includes: performing visual semantic understanding on the screenshot to obtain semantic information representing the interface content of the screenshot; and determining the interactive operation to be performed on the third-party application based on the semantic information and the task instruction.

[0040] In some possible implementations, the device further includes: The training unit is configured to acquire sample data, which includes sample task instructions, multiple sample interaction operations arranged in execution order, and sample screenshots on which each sample interaction operation is based. According to the execution order of the multiple sample interaction operations, the sample screenshots on which the sample interaction operations are based and the sample task instructions are sequentially input into the operation prediction model to obtain the predicted interaction operations output by the operation prediction model. The operation prediction model is trained based on the differences between each predicted interaction operation and the corresponding sample interaction operation.

[0041] In some possible implementations, the interactive operation is characterized by at least one of the following parameters: operation type, operation location, and operation content.

[0042] In some possible implementations, the device further includes: The parsing unit is configured to parse the task instruction and obtain a parsing result, which is used to indicate whether the target task requires multiple interactive operations to complete. The execution unit is configured to, in response to the task instruction, obtain the screen interface information of the third-party application when the parsing result indicates that the target task requires multiple interactive operations to complete.

[0043] In some possible implementations, the execution unit is configured to determine that the target task has been completed in response to a match between the current screen interface of the third-party application and the task instruction.

[0044] According to a third aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method described in the embodiments of the first aspect above.

[0045] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the embodiments of the first aspect above.

[0046] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the embodiments of the first aspect above.

[0047] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: As can be seen from the above embodiments, the embodiments of this disclosure can effectively complete complex tasks and significantly improve the ability to execute high-level user instructions by dynamically obtaining the current screen interface of the third-party application and making autonomous decisions and executing the next interactive operation based on the current screen interface, gradually advancing the target task until completion.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of a task execution system according to an embodiment of the present disclosure.

[0051] Figure 2 This is a schematic flowchart illustrating a task execution method according to an embodiment of the present disclosure.

[0052] Figure 3This is a schematic flowchart illustrating a task execution method according to an embodiment of the present disclosure.

[0053] Figure 4 This is a schematic block diagram of a task execution device according to an embodiment of the present disclosure.

[0054] Figure 5 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0055] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0056] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0057] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0058] For the sake of brevity and ease of understanding, this document uses the terms "greater than" or "less than", "higher than" or "lower than" to describe size relationships. However, it will be understood by those skilled in the art that the term "greater than" also includes the meaning of "greater than or equal to", and "less than" also includes the meaning of "less than or equal to"; the term "higher than" also includes the meaning of "higher than or equal to", and "lower than" also includes the meaning of "lower than or equal to".

[0059] As human-computer interaction modes evolve, users increasingly expect terminals to autonomously complete complex operations through concise commands. However, related technologies still have significant limitations in automating responses to high-level user commands. High-level user commands are those expressing operational intentions, which do not contain specific steps and require multiple steps to complete.

[0060] Currently, terminal responses to user commands typically rely on pre-configured customized interfaces in the application. Upon receiving a user command, the terminal queries the pre-configured interfaces for a matching customized interface and triggers a specific function or redirects to the corresponding interface based on that interface. For example, if the user command is "Open the payment interface of application A," and application A has pre-configured a customized interface for "payment interface," the terminal can directly access application A's payment interface through that interface.

[0061] While this method enables rapid response, its execution logic relies entirely on pre-configured settings on the application side. If the user command involves scenarios where customized interfaces are not provided or are incompletely covered, the terminal cannot respond effectively. Furthermore, this method cannot achieve cross-application control. For example, if the user command is "search for popular songs in the short video application and play those popular songs in the audio playback application," this command involves the cooperation of two applications. Configuring customized interfaces in either the short video application or the audio playback application will not achieve the desired result.

[0062] When user commands cannot be implemented directly through customized interfaces (i.e., the user commands mentioned above involve scenarios where customized interfaces are not provided or are not fully covered, or user commands involve cross-application execution scenarios), users have to break down the original high-level user commands into multiple low-level user commands corresponding to specific operation steps, and input these commands in sequence to guide the system to gradually complete the entire task process.

[0063] For example, if a user wants to access a return code in an e-commerce application, and the application provides a customized interface for return codes, the user can directly enter the command "open the return code for the e-commerce application" into the current system. The system will then directly call the customized interface corresponding to the return code to access the e-commerce application's return code. However, if the e-commerce application does not provide a customized interface for return codes, the user needs to manually enter multiple commands in sequence: "open the e-commerce application," "open my e-commerce interface," "open the return / after-sales interface," and finally "open the return code." This instructs the system to control the e-commerce application step by step according to the entered commands to display the return code within the application.

[0064] This implementation method is not only cumbersome, but also requires users to clearly understand the complete execution path of the task within the third-party application and precisely break down each operation step. For users unfamiliar with third-party applications, this process is difficult to complete, greatly reducing the convenience of interaction. Furthermore, this implementation method significantly extends the task completion time, especially in scenarios with many operation steps, where the efficiency loss is particularly noticeable.

[0065] In view of this, this disclosure proposes a task instruction method that can autonomously make decisions and execute corresponding interactive operations based on task instructions and the current screen interface, simulating user operations to complete target tasks. It is not limited by whether third-party applications provide corresponding customized interfaces, greatly expanding the scope of supported applications and functional scenarios, and enabling more complex operation control. It also eliminates the need for users to break down high-level user instructions into multiple low-level user instructions, significantly reducing the usage threshold and improving the level of automation in task execution and user experience.

[0066] The following describes one or more embodiments of this disclosure in detail.

[0067] The embodiments of this disclosure can be applied to electronic devices with display screens, including but not limited to smartphones, desktop computers, tablets, laptops, e-book readers, smartwatches, smart bracelets, and in-vehicle terminals. One or more embodiments of this disclosure are not limited thereto. During operation, the electronic device can run a task execution system to perform tasks. The application program for this task execution system can be pre-installed on the electronic device, allowing the system to be started and run. Alternatively, when using technologies such as HTML5, the task execution system can be obtained and run without installing the corresponding application program on the electronic device.

[0068] In another embodiment, the task execution system may include, for example, Figure 1 The diagram shows a server 11, a network 12, and an electronic device 13. During operation, the server 11 can run server-side programs of the task execution system to perform data processing and other functions. Meanwhile, the electronic device 13 can run client-side programs of the task execution system to perform data acquisition, human-computer interaction, and other functions. This collaboration between the server 11 and the electronic device 13 enables the task execution scheme. It should be noted that the server 11 can also communicate with multiple electronic devices to implement fingerprint recognition on multiple devices.

[0069] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted by a host cluster. Electronic device 13 can be any type of device, such as a PC, tablet, laptop, PDA (Personal Digital Assistants), mobile phone, wearable device (such as smart glasses, smartwatches, etc.), or vehicle terminal, etc., and this specification does not limit this to any of the embodiments described herein. The network 12 for interaction between electronic device 13 and server 11 can include various types of wired or wireless networks. In one embodiment, network 12 can include the Public Switched Telephone Network (PSTN) and the Internet.

[0070] The following example illustrates how a terminal executes the task execution method provided in this embodiment of the disclosure. Figure 2 This is a schematic flowchart illustrating a task execution method according to an embodiment of the present disclosure. Figure 2 As shown, the method may include the following steps: S201. Obtain the task instruction input by the user, which is used to instruct a third-party application to perform a target task.

[0071] In some embodiments, task instructions can be any form of instruction input by the user, such as voice instructions, text instructions, gesture instructions, etc. Voice instructions are instructions input by the user through voice, text instructions are instructions input by the user through inputting character information, and gesture instructions are instructions input by the user through gestures or other physical movements. This disclosure does not limit the input method or form of the task instructions.

[0072] In this embodiment of the disclosure, the task instruction is used to instruct a third-party application to perform a target task. This third-party application can be one application or multiple applications; this embodiment of the disclosure does not limit the number of third-party applications. For example, the task instruction is "search for popular songs in the short video application and play the popular song in the audio playback application." In this case, there are two third-party applications: the "short video application" and the "audio playback application."

[0073] In related technologies, if a user wants to instruct the current system to open a deeper interface of a third-party application (where a deeper interface refers to an interface that cannot be directly accessed through the main interface of the application and requires multiple interface jumps to reach, such as an order details interface), it usually depends on whether the third-party application provides a customized interface for the deeper interface. If the third-party application does not provide such a customized interface, the user needs to enter multiple commands in sequence to gradually control the current system to open the deeper interface of the third-party application.

[0074] For example, if a user wants to access a return code in an e-commerce application, and the application provides a customized interface for return codes, the user can directly enter the command "open the return code for the e-commerce application" into the current system. The system will then directly call the customized interface corresponding to the return code to access the e-commerce application's return code. However, if the e-commerce application does not provide a customized interface for return codes, the user needs to manually enter multiple commands in sequence: "open the e-commerce application," "open my e-commerce interface," "open the return / after-sales interface," and finally "open the return code." This instructs the system to control the e-commerce application step by step according to the entered commands to display the return code within the application.

[0075] However, in the above-described situation, if the method provided in this disclosure is used, even if the third-party application does not provide a customized interface, direct access to the page can be achieved through a single command. That is, when a user enters the command "Open the return code for the e-commerce application" in the current system, the current system can sequentially execute multiple interactive operations—"Open the e-commerce application," "Open my e-commerce interface," "Open the return / after-sales interface," and "Open the return code"—based on the screen interface information of the e-commerce application, thereby completing the command.

[0076] In some embodiments, when the method provided in this disclosure is applied to a voice assistant, the task instruction is a voice instruction. In some embodiments, obtaining a task instruction input by the user includes: obtaining a voice instruction input by the user, which is used to instruct the opening of a first page of a third-party application. The first page may be a deeper page of the third-party application.

[0077] In some possible implementations, considering that the method provided in this disclosure can complete high-level user instructions, which typically involve multiple operational steps and have high processing complexity, in order to improve system resource utilization efficiency, the method provided in this disclosure can be used when the task instruction is a high-level user instruction, while other faster methods can be used when the task instruction is not a high-level user instruction.

[0078] In some embodiments, the method further includes: parsing the task instruction to obtain a parsing result, the parsing result indicating whether the target task requires multiple interactive operations to complete. Responding to the task instruction, obtaining the screen interface information of a third-party application includes: if the parsing result indicates that the target task requires multiple interactive operations to complete, in response to the task instruction, obtaining the screen interface information of the third-party application. This disclosure embodiment, by parsing the task instruction before execution to determine whether it belongs to a complex task requiring multiple interactive steps, and only initiating the acquisition of the third-party application's screen interface information and subsequent interface-based interactive operation process when it is a complex task, ensures that processes such as interface perception and interactive operation determination are triggered only when necessary. This mechanism effectively avoids performing redundant interface analysis processes for simple tasks that do not require multiple operations, significantly reduces the consumption of device computing resources, and improves the overall task scheduling efficiency and system response performance.

[0079] For example, if the task instruction is "open the first application", then the method provided in this embodiment of the disclosure is not required, and the task instruction can be executed directly; if the task instruction is "open the refund code in the first application" or "order me a coffee in the first application", then the method provided in this embodiment of the disclosure is used to complete the task instruction.

[0080] S202. In response to the task instruction, obtain the screen interface information of the third-party application, and perform multiple interactive operations on the third-party application in sequence based on the screen interface information until it is confirmed that the target task has been completed; wherein, the screen interface information based on each interactive operation includes the current screen interface corresponding to the interactive operation.

[0081] In some embodiments, the current screen interface corresponding to the interactive operation refers to the latest screen interface of the third-party application before the interactive operation is executed.

[0082] In some embodiments, in response to the task instruction, obtaining the screen interface information of the third-party application includes: in response to the task instruction, opening the third-party application and performing a screenshot operation to obtain a screenshot of the third-party application.

[0083] In some possible implementations, step S202 can be completed in a loop as shown below. Specifically, acquiring the screen interface information of the third-party application and sequentially performing multiple interactive operations on the third-party application based on this screen interface information until the target task is confirmed to be completed includes: before confirming the completion of the target task, cyclically performing the following processes: acquiring a screenshot of the third-party application; determining the interactive operation to be performed on the third-party application based on the task instructions and the screen interface; performing the interactive operation on the third-party application; and generating a new screenshot of the third-party application after performing the interactive operation.

[0084] For example, the task instruction is "Open the return code of the e-commerce application". The system first opens the e-commerce application and obtains a screenshot of the e-commerce application. Based on the task instruction and the screenshot, the system determines that the interactive operation to be performed on the e-commerce application is "Open my e-commerce interface". The system performs this interactive operation on the e-commerce application to open the "my e-commerce interface". Then, the system performs a screenshot operation to obtain a screenshot of the third-party application (this screenshot is the screenshot of the "my e-commerce interface"). Finally, based on the task instruction and the screenshot, the system determines that the interactive operation to be performed on the e-commerce application is "Open the return / after-sales interface". The system executes the interactive operation on the e-commerce application to open the return / after-sales interface. Then, the system takes a screenshot of the third-party application (which is the screenshot of the return / after-sales interface). Based on the task instruction and the screenshot, the system determines that the interactive operation to be performed on the e-commerce application is "open the return code." The system executes this interactive operation on the e-commerce application to open the return code. Then, the system takes a screenshot of the third-party application (which is the screenshot of the interface where the return code is located). Based on the task instruction and the screenshot, the system determines that the target task has been completed.

[0085] In some embodiments, performing an interactive operation on a third-party application can be: the current system simulates user operations such as clicking, swiping, and inputting on the interface of the third-party application, thereby triggering the third-party application to respond to the operation by entering the corresponding interface or completing a specific function. In some embodiments, performing an interactive operation on a third-party application can be: sending an operation instruction to the third-party application to instruct it to perform the operation, thereby causing the third-party application to execute the operation instruction to enter the corresponding interface or complete a specific function.

[0086] In some embodiments, the interactive operation is characterized by at least one of the following parameters: operation type, operation location, and operation content. The operation type can be any of the following: click, long press, wait, follow-up question, swipe, input, or return. The operation location can be coordinates in a screenshot. The operation content corresponds to the operation type; when the operation type is input, the operation content can be the input content; when the operation type is follow-up question, the operation content can be the content of the follow-up question; when the operation type is swipe, the operation content can be the direction of the swipe.

[0087] It should be noted that when the object being operated on is represented by the operation position, the current system does not need to know the object information of the object being operated on (such as the control ID of the interface control, etc.). It can simulate the user operation based solely on the operation position, thereby eliminating the need for third-party applications to report the control information of the interface control, which can further expand the application scenarios of the embodiments of this disclosure.

[0088] In some possible implementations, during the execution of the target task, the user can also input supplementary information. After obtaining the supplementary information, the system incorporates it when determining the next interactive operation to make the determined next interactive operation more accurate, thereby completing the target task more accurately. Specifically, determining the interactive operation to be performed on the third-party application based on the task instruction and screenshot includes: obtaining the supplementary information input by the user; and determining the interactive operation to be performed on the third-party application based on the task instruction, screenshot, and supplementary information.

[0089] In some embodiments, the supplementary information is actively input by the user. Obtaining the supplementary information input by the user includes: acquiring first information actively input by the user during the execution of the target task; parsing the first information; and determining that the first information is related to the target task if it is. By monitoring and acquiring the information actively input by the user in real time during the execution of the target task, and performing semantic parsing and relevance judgment on the information, the system incorporates it as supplementary information only when it is confirmed to be related to the current task objective. This allows the system to dynamically respond to the user's temporary intention adjustments or personalized needs during the task process, effectively avoiding the erroneous impact of irrelevant voice interference (such as background dialogue or accidental triggering commands) on task execution, improving the robustness and accuracy of the system. This mechanism endows the task execution process with higher interactive flexibility and intelligent adaptability.

[0090] For example, when a user inputs the command "search for popular songs in a short video app and play the popular song in an audio playback app", the system executes the command. While searching for popular songs in the short video app, the user inputs the first piece of information "the number of popular songs is ten". At this time, the system parses the first piece of information, determines that the first piece of information is related to the target task, and identifies the first piece of information as supplementary information input by the user. Subsequently, when the system performs the operation of selecting popular songs, it will select ten popular songs.

[0091] In some embodiments, the supplementary information is passively input by the user. Obtaining the supplementary information input by the user includes: after performing an interactive operation to obtain additional information required for task execution, obtaining second information input by the user through a feedback interface, and identifying the second information as the supplementary information input by the user. This mechanism enables the system to autonomously identify missing information nodes in task execution even in the absence of complete initial instructions, and to complete the information through guided interaction, thereby ensuring the smooth progress of multi-step tasks.

[0092] In some cases, user-inputted task instructions lack necessary information, preventing the system from completing the target task. In such situations, the system will execute an interactive operation to obtain the additional information required for task execution. For example, if the user's task instruction is "Order me a coffee from merchant A in the food delivery app," the system will navigate to merchant A's store and open the coffee order page. Upon finding that it needs to select whether to add sugar, the system will perform a query (i.e., an interactive operation to obtain the additional information required for task execution), asking the user if they want sugar in their coffee. The user replies "Add sugar," and the system obtains the user's input "Add sugar" (i.e., the second piece of information) through a feedback interface, using it as supplementary information. Based on the task instruction, screenshot, and supplementary information, the system determines that the interactive operation to be performed for the third-party application is to select the "Add sugar" option.

[0093] In some possible implementations, considering that the method provided in this disclosure embodiment executes multiple interactive operations sequentially through an instruction control system, in order to more accurately determine the interactive operation to be executed, this disclosure embodiment also records the already executed interactive operations and refers to the already executed interactive operations when determining the interactive operation to be executed. The method further includes: after determining the interactive operation to be executed for a third-party application, recording the interactive operation in the execution record corresponding to the task instruction. Determining the interactive operation to be executed for the third-party application based on the task instruction and a screenshot includes: obtaining the interactive operation already executed in response to the task instruction from the execution record corresponding to the task instruction; and determining the interactive operation to be executed for the third-party application based on the task instruction, the screenshot, and the interactive operation already executed in response to the task instruction.

[0094] In some embodiments, the system includes a cloud and a client. The client uploads screenshots and task instructions to the cloud, whereby the cloud determines the interactive operation to be performed on the third-party application. The client then performs the interactive operation and, after performing the operation, obtains the current screenshot to generate a new screenshot of the third-party application. Specifically, performing the interactive operation on the third-party application and generating a new screenshot afterward includes: sending an operation instruction to the corresponding client to instruct the client to perform the interactive operation on the third-party application according to the operation instruction, and then taking a screenshot of the third-party application's screen interface to obtain a new screenshot of the third-party application.

[0095] In some possible implementations, the interactive operations to be executed are predicted by a model. Specifically, determining the interactive operations to be executed for the third-party application based on task instructions and screenshots includes: inputting the task instructions and screenshots into a pre-trained operation prediction model; and obtaining the interactive operations to be executed for the third-party application output by the operation prediction model. By introducing a model to jointly analyze the screen interface information of the third-party application and the task instructions, the system predicts the interactive operations most likely to be executed in the next stage. This allows the system to autonomously generate operation sequences that conform to user intent and application logic without pre-configuring operation rules or relying on manual scripts. This mechanism significantly improves the intelligence and generalization ability of the task execution process. For different applications, different interface layouts, and even interface changes after version updates, the system can dynamically adjust the operation strategy based on the semantic understanding of the model, avoiding the problem that traditional methods based on fixed path matching or control ID recognition are prone to failure due to interface changes.

[0096] In some embodiments, the operation prediction model first performs visual semantic understanding on the screenshot, converting it into semantic information that can represent the interface content. Then, it combines this information with the task instruction to determine the next interactive operation. This mechanism allows the model to go beyond a superficial perception of the original image and truly understand the functional roles and task relevance of each element in the current interface, thereby improving the interpretability and accuracy of the decision. The operation prediction model's processing includes: performing visual semantic understanding on the screenshot to obtain semantic information representing the interface content of the screenshot; and determining the interactive operation to be executed for the third-party application based on the semantic information and the task instruction.

[0097] In some embodiments, the semantic information used to characterize the interface content of the screenshot includes at least one of the following: the position of each control in the interface, the type of each control in the interface, the functional semantics of each control in the interface, and the page type.

[0098] The types of controls can include, but are not limited to, buttons, input boxes, drop-down menus, and icons. The functional semantics of a control describe its role in the task context, including but not limited to confirmation, payment entry, payment, search, order submission, and return. The page type indicates the business scenario of the current interface, including but not limited to login page, order page, payment page, search results page, and user center page.

[0099] In some embodiments, the training process of the operation prediction model includes: acquiring sample data, which includes sample task instructions, a plurality of sample interaction operations arranged in execution order, and sample screenshots on which each sample interaction operation is based; inputting the sample screenshots and sample task instructions on which the sample interaction operations are based into the operation prediction model in sequence according to the execution order of the plurality of sample interaction operations, to obtain the predicted interaction operations output by the operation prediction model; and training the operation prediction model based on the differences between each predicted interaction operation and the corresponding sample interaction operation.

[0100] Specifically, training the operation prediction model based on the differences between each predicted interaction operation and the corresponding sample interaction operation means training the operation prediction model based on the differences between the first predicted interaction operation and the first sample interaction operation, the differences between the second predicted interaction operation and the second sample interaction operation, and so on, as well as the differences between the last predicted interaction operation and the last sample interaction operation.

[0101] In some embodiments, confirming that a target task has been completed includes: determining that a target task has been completed in response to a match between the current screen interface of a third-party application and a task instruction.

[0102] In some embodiments, when the interactive operation is an operation for interacting with a user, the operation content determined by the system can be refined using a large language model.

[0103] Next, this disclosure will use the following embodiments: Figure 3 For example, the task execution method provided in this disclosure embodiment will be described by way of example: The user inputs a voice command in the client. The client retrieves the voice command and uploads it to the cloud. The cloud parses the voice command and determines it to be an intent-level command (i.e., a command requiring multiple steps to complete). At this point, the cloud instructs the client to capture and upload a screenshot. After receiving the uploaded screenshot, the cloud uses an operation prediction model to process the voice command and screenshot to obtain the interaction to be executed. The operation command to execute this interaction is sent to the client, and the interaction is recorded in the execution record corresponding to the task command. The client executes the operation command and then captures and uploads a screenshot again. The cloud continues to use the operation prediction model to determine the interaction to be executed based on the screenshot, voice command, and execution record, and sends the operation command to execute this interaction to the client. This process is repeated until the screenshot returned by the client matches the voice command. At this point, the cloud outputs a completion message to the client, which can then display a completion notification.

[0104] Corresponding to the aforementioned embodiments of the task execution method, this disclosure also provides embodiments of the task execution apparatus.

[0105] Please see Figure 4 , Figure 4 This is a schematic block diagram of a task execution device provided in an exemplary embodiment. The device may include: an acquisition unit 401 and an execution unit 402. The acquisition unit 401 is configured to acquire a task instruction input by a user, the task instruction being used to instruct a third-party application to perform a target task. The execution unit 402 is configured to, in response to the task instruction, obtain the screen interface information of the third-party application, and perform multiple interactive operations on the third-party application in sequence based on the screen interface information, until it is confirmed that the target task has been completed. The screen interface information on which each interactive operation is based includes the current screen interface corresponding to the interactive operation.

[0106] In some possible implementations, the acquisition unit 401 is configured to acquire a voice command input by a user, the voice command being used to instruct the opening of the first page of the third-party application.

[0107] In some possible implementations, the execution unit 402 is configured to repeatedly execute the following process before confirming that the target task has been completed: Obtain a screenshot of the third-party application; Based on the task instructions and the screenshot, determine the interactive operation to be performed for the third-party application; The interactive operation is performed on the third-party application, and after the interactive operation is performed, a new screenshot of the third-party application is generated.

[0108] In some possible implementations, the execution unit 402 is configured to acquire supplementary information input by the user; and based on the task instruction, the screenshot, and the supplementary information, determine the interactive operation to be performed for the third-party application.

[0109] In some possible implementations, the execution unit 402 is configured to perform at least one of the following: The system obtains first information actively input by the user during the execution of the target task, parses the first information, and determines the first information as supplementary information input by the user if the first information is related to the target task. After performing an interactive operation to obtain additional information required for task execution, the system obtains second information input by the user through the feedback interface and identifies the second information as supplementary information input by the user.

[0110] In some possible implementations, the device further includes: The recording unit is configured to record the interaction operation to be performed on the third-party application into the execution record corresponding to the task instruction after determining the interaction operation to be performed on the third-party application. The execution unit 402 is configured to obtain interactive operations in response to the executed task instruction from the execution record corresponding to the task instruction; and determine interactive operations to be executed for the third-party application based on the task instruction, the screenshot, and the interactive operations in response to the executed task instruction.

[0111] In some possible implementations, the execution unit 402 is configured to send an operation instruction to the corresponding client to instruct the interactive operation, so that after the client performs the interactive operation on the third-party application according to the operation instruction, it takes a screenshot of the screen interface of the third-party application to obtain a new screenshot of the third-party application.

[0112] In some possible implementations, the execution unit 402 is configured to input the task instruction and the screenshot into a pre-trained operation prediction model; and obtain the interactive operation to be executed for the third-party application output by the operation prediction model.

[0113] In some possible implementations, the processing of the operation prediction model includes: performing visual semantic understanding on the screenshot to obtain semantic information representing the interface content of the screenshot; and determining the interactive operation to be performed on the third-party application based on the semantic information and the task instruction.

[0114] In some possible implementations, the device further includes: The training unit is configured to acquire sample data, which includes sample task instructions, multiple sample interaction operations arranged in execution order, and sample screenshots on which each sample interaction operation is based. According to the execution order of the multiple sample interaction operations, the sample screenshots on which the sample interaction operations are based and the sample task instructions are sequentially input into the operation prediction model to obtain the predicted interaction operations output by the operation prediction model. The operation prediction model is trained based on the differences between each predicted interaction operation and the corresponding sample interaction operation.

[0115] In some possible implementations, the interactive operation is characterized by at least one of the following parameters: operation type, operation location, and operation content.

[0116] In some possible implementations, the device further includes: The parsing unit is configured to parse the task instruction and obtain a parsing result, which is used to indicate whether the target task requires multiple interactive operations to complete. The execution unit 402 is configured to, in response to the task instruction, obtain the screen interface information of the third-party application when the parsing result indicates that the target task requires multiple interactive operations to complete.

[0117] In some possible implementations, the execution unit 402 is configured to determine that the target task has been completed in response to a match between the current screen interface of the third-party application and the task instruction.

[0118] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant methods, and will not be elaborated upon here.

[0119] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0120] Figure 5 This is a schematic block diagram illustrating an electronic device according to embodiments of the present disclosure. For example, the electronic device 500 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0121] Reference Figure 5 The electronic device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0122] Processing component 502 typically controls the overall operation of electronic device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0123] Memory 504 is configured to store various types of data to support the operation of electronic device 500. Examples of this data include instructions for any application or method operating on electronic device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0124] Power supply component 506 provides power to various components of electronic device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 500.

[0125] Multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0126] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when electronic device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0127] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0128] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of electronic device 500. For example, sensor assembly 514 may detect the on / off state of electronic device 500, the relative positioning of components such as the display and keypad of electronic device 500, changes in position of electronic device 500 or a component of electronic device 500, the presence or absence of user contact with electronic device 500, orientation or acceleration / deceleration of electronic device 500, and temperature changes of electronic device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0129] Communication component 516 is configured to facilitate wired or wireless communication between electronic device 500 and other devices. Electronic device 500 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0130] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described in any of the above embodiments.

[0131] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of an electronic device 500 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0132] In an exemplary embodiment, this disclosure also provides a computer program product including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0133] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0134] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A task execution method characterized by, The method comprises: obtaining a task instruction input by a user, the task instruction being used to instruct a target task to be executed on a third-party application; in response to the task instruction, obtaining screen interface information of the third-party application, and sequentially executing a plurality of interaction operations on the third-party application based on the screen interface information until it is confirmed that the target task has been completed; wherein the screen interface information on which each interaction operation is based comprises a current screen interface corresponding to the interaction operation.

2. The method of claim 1, wherein: the obtaining of the task instruction input by the user comprises obtaining a voice instruction input by the user, the voice instruction being used to instruct a first page of the third-party application to be opened.

3. The method of claim 1, wherein, the obtaining of the screen interface information of the third-party application and the sequentially executing of the plurality of interaction operations on the third-party application based on the screen interface information until it is confirmed that the target task has been completed comprises: before it is confirmed that the target task has been completed, the following processing is performed in a loop: obtaining a screenshot of the third-party application; based on the task instruction and the screenshot, determining an interaction operation to be executed on the third-party application; executing the interaction operation on the third-party application, and after the execution of the interaction operation, generating a new screenshot of the third-party application.

4. The method of claim 3, wherein, the determining of the interaction operation to be executed on the third-party application based on the task instruction and the screenshot comprises: obtaining supplementary information input by the user; based on the task instruction, the screenshot and the supplementary information, determining the interaction operation to be executed on the third-party application.

5. The method of claim 4, wherein, the obtaining of the supplementary information input by the user comprises at least one of the following: obtaining first information input by the user actively in a process of executing the target task, analyzing the first information, and if the first information is related to the target task, determining the first information as the supplementary information input by the user; after an interaction operation for obtaining additional information required for task execution is executed, obtaining second information input by the user through a feedback interface, and determining the second information as the supplementary information input by the user.

6. The method of claim 3, wherein: the method further comprises, after the determining of the interaction operation to be executed on the third-party application, recording the interaction operation to an execution record corresponding to the task instruction; the determining of the interaction operation to be executed on the third-party application based on the task instruction and the screenshot comprises: obtaining, from the execution record corresponding to the task instruction, an interaction operation that has been executed in response to the task instruction; and based on the task instruction, the screenshot and the interaction operation that has been executed in response to the task instruction, determining the interaction operation to be executed on the third-party application.

7. The method of claim 3, wherein, the executing of the interaction operation on the third-party application, and after the execution of the interaction operation, the generating of a new screenshot of the third-party application comprises: send operation instructions for indicating the interaction operation to a corresponding client, so that the client performs the interaction operation on the third-party application according to the operation instructions, and then performs screenshot on a screen interface of the third-party application to obtain a new screenshot of the third-party application.

8. The method of claim 3, wherein, The determining the interaction operation to be performed on the third-party application based on the task instruction and the screenshot comprises: inputting the task instruction and the screenshot into a pre-trained operation prediction model; obtaining the interaction operation to be performed on the third-party application output by the operation prediction model.

9. The method of claim 8, wherein, The training process of the operation prediction model comprises: obtaining sample data, the sample data comprising a sample task instruction, a plurality of sample interaction operations arranged in execution time sequence, and a sample screenshot on which each sample interaction operation is based; inputting the sample screenshot on which each sample interaction operation is based and the sample task instruction into the operation prediction model in sequence according to the execution time sequence of the plurality of sample interaction operations, to obtain a predicted interaction operation output by the operation prediction model; training the operation prediction model according to differences between each predicted interaction operation and a corresponding sample interaction operation.

10. The method of claim 3, wherein, The interaction operation is characterized by at least one of the following parameters: operation type, operation position, and operation content.

11. The method of claim 1, wherein: the method further comprises: parsing the task instruction to obtain a parsing result, the parsing result being used to indicate whether the target task needs a plurality of interaction operations to be completed; the obtaining the screen interface information of the third-party application in response to the task instruction comprises: in a case where the parsing result indicates that the target task needs a plurality of interaction operations to be completed, obtaining the screen interface information of the third-party application in response to the task instruction.

12. The method of claim 1, wherein, The confirming that the target task is completed comprises: determining that the target task is completed in response to the current screen interface of the third-party application matching the task instruction.

13. A task execution apparatus characterized by comprising: The apparatus comprises: an obtaining unit configured to obtain a task instruction input by a user, the task instruction being used to indicate a target task to be performed on a third-party application; an executing unit configured to obtain screen interface information of the third-party application in response to the task instruction, and sequentially perform a plurality of interaction operations on the third-party application based on the screen interface information until it is confirmed that the target task is completed; wherein the screen interface information on which each interaction operation is based comprises a current screen interface corresponding to the interaction operation.

14. The apparatus of claim 13, wherein: the executing unit is configured to, before it is confirmed that the target task is completed, cyclically perform the following processing: obtaining a screenshot of the third-party application; determining an interaction operation to be performed on the third-party application based on the task instruction and the screenshot; performing the interaction operation on the third-party application, and generating a new screenshot of the third-party application after the interaction operation is performed.

15. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; a memory for storing processor-executable instructions; wherein the processor implements the method of any of claims 1-12 by executing the executable instructions.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements the steps of the method of any of claims 1-12.

17. A computer program product, characterised in that, including computer program / instructions, which when executed by the processor implement the steps of the method of any of claims 1-12.

Images