Interaction method, intelligent agent training method and device, equipment and medium

Abstract

The invention provides an interaction method, an agent training method and device, equipment and a medium, and relates to the field of artificial intelligence, and the interaction method comprises the steps: obtaining task description information associated with a to-be-executed interaction task; the calling agent determines execution information under the corresponding time step according to the task description information and the visual data under the at least one time step and executes the execution information so as to complete the interaction task; wherein the visual data under each time step is associated with the target user interface under the corresponding time step, and the execution information is used for indicating a to-be-operated interface element in the target user interface under the corresponding time step and an execution action corresponding to the to-be-operated interface element. Therefore, the intelligent agent does not need to depend on a predefined API interface, direct operation is carried out on the human-computer interface level through understanding and semantic analysis of the visual content of the user interface, and the universality and the adaptive capacity of the intelligent agent in heterogeneous equipment and a closed ecological environment are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an interaction method and a training method, apparatus, device and medium for intelligent agents. Background Technology

[0002] With the rapid development of Large Language Model (LLM) and Multimodal Large Language Model (MLLM) technologies, intelligent agents based on large models have made significant progress in task planning, task orchestration, and external tool invocation. In these technologies, intelligent agents interact with target applications or services by calling Application Programming Interfaces (APIs) to respond to and execute user commands.

[0003] APIs, as a standard mechanism for inter-system communication, rely on interface definitions provided by external application vendors or operating systems. Their functional scope is limited by the openness and version compatibility of these interfaces. In practice, some applications do not provide complete API support or only open limited functional interfaces, making it difficult for intelligent agents to complete complex tasks involving deep operations. In other words, the limited scope of API definitions restricts the capabilities of intelligent agents to some extent. Summary of the Invention

[0004] This application proposes an interaction method and a training method, apparatus, device, and medium for intelligent agents, in order to at least partially solve one of the technical problems in the related art.

[0005] One embodiment of this application proposes an interaction method, comprising: acquiring task description information associated with an interaction task to be executed; invoking an agent to determine execution information for a corresponding time step based on the task description information and visual data at at least one time step, and executing the agent to complete the interaction task; wherein, the visual data for each time step is associated with a target user interface for the corresponding time step; wherein, the execution information is used to indicate the interface element to be operated in the target user interface for the corresponding time step, and the execution action corresponding to the interface element to be operated.

[0006] Another embodiment of this application proposes a training method for an intelligent agent, comprising: acquiring first training data; wherein the first training data includes first task description information associated with a first interactive task and first visual data associated with a first user interface at a first time step; invoking a first intelligent agent to execute the first interactive task according to the first task description information and the first visual data at at least one time step to obtain execution trajectory information; wherein the first visual data at each time step is associated with the first user interface at the corresponding time step, the execution trajectory information is used to indicate the execution information of the first intelligent agent at at least one time step, and the execution information is used to indicate the interface element to be operated in the first user interface at the corresponding time step and the execution action corresponding to the interface element to be operated; and performing supervised training on the first intelligent agent according to the labeled trajectory information associated with the first training data and the execution trajectory information to obtain a trained second intelligent agent.

[0007] Another embodiment of this application proposes an interactive device, comprising: an acquisition module for acquiring task description information associated with an interactive task to be executed; and a processing module for invoking an agent to determine and execute execution information at a corresponding time step based on the task description information and visual data at at least one time step, thereby completing the interactive task; wherein the visual data at each time step is associated with a target user interface at the corresponding time step; and wherein the execution information is used to indicate an interface element to be operated in the target user interface at the corresponding time step, and the execution action corresponding to the interface element to be operated.

[0008] In another embodiment of this application, a training apparatus for an intelligent agent is proposed, comprising: a first acquisition module for acquiring first training data; wherein the first training data includes first task description information associated with a first interactive task and first visual data associated with a first user interface at a first time step; a first execution module for invoking a first intelligent agent to execute the first interactive task according to the first task description information and the first visual data at at least one time step to obtain execution trajectory information; wherein the first visual data at each time step is associated with the first user interface at the corresponding time step, and the execution trajectory information is used to indicate the execution information of the first intelligent agent at at least one time step, and the execution information is used to indicate the interface element to be operated in the first user interface at the corresponding time step and the execution action corresponding to the interface element to be operated; and a first training module for performing supervised training on the first intelligent agent according to the labeled trajectory information associated with the first training data and the execution trajectory information to obtain a trained second intelligent agent.

[0009] Another embodiment of this application proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the interaction method described in the foregoing aspect, or the training method for an intelligent agent described in the foregoing aspect.

[0010] In another aspect of this application, a non-transitory computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the interaction method as described in the foregoing aspect, or implement the agent training method as described in the foregoing aspect.

[0011] Another aspect of this application proposes a computer program product having a computer program stored thereon, which, when executed by a processor, implements the interaction method as described in the foregoing aspect, or implements the agent training method as described in the foregoing aspect.

[0012] The interaction method and training method, apparatus, device, and medium proposed in this application acquire task description information associated with the interactive task to be executed, and call the intelligent agent to determine and execute the execution information at the corresponding time step based on the task description information and visual data at at least one time step. The visual data at each time step is associated with the target user interface at that time step; the execution information at each time step includes the interface elements to be operated in the target user interface at that time step and their corresponding execution actions (such as click, swipe, long press, input, etc.), thereby realizing end-to-end interactive tasks. In summary, this application does not rely on predefined API interfaces, but instead performs direct operation at the human-computer interface level through understanding and semantic parsing of the visual content of the user interface, significantly improving the versatility and adaptability of the intelligent agent in heterogeneous devices and closed ecosystems. Specifically, the intelligent agent can recognize the visual features, layout structure, and semantic meaning of interface elements, and combine this with task description information in natural language form to map intent to action, achieving non-intrusive operation of the application. This overcomes the limitations of traditional API call modes in terms of functional coverage and system compatibility, enhancing the task execution boundaries and scalability of the intelligent agent. Furthermore, by generating execution information step-by-step over time, this mechanism supports dynamic planning and context-aware execution of complex, multi-step tasks, exhibiting good fault tolerance and path correction capabilities. Optionally, this mechanism is applicable to user interface environments of different operating systems and device forms (such as mobile phones, tablets, in-vehicle systems, and smart home devices). It only needs to provide the intelligent agent with visual data including the user interface and task description information input by the user to drive automated interaction, reducing dependence on the underlying system architecture and improving deployment flexibility.

[0013] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0014] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating an interactive method provided for an exemplary embodiment of this application; Figure 2 A flowchart illustrating another interaction method provided for an exemplary embodiment of this application; Figure 3 A schematic diagram illustrating the process of an intelligent agent performing an interactive task, provided as an exemplary embodiment of this application; Figure 4 A flowchart illustrating a method for training an agent, provided as an exemplary embodiment of this application; Figure 5 A flowchart illustrating another method for training an intelligent agent provided as an exemplary embodiment of this application; Figure 6 A flowchart illustrating yet another method for training an intelligent agent, provided as an exemplary embodiment of this application; Figure 7 A schematic diagram of the structure of an interactive device provided for an exemplary embodiment of this application; Figure 8 A schematic diagram of the structure of a training device for an intelligent agent provided for an exemplary embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0015] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0016] It should be noted that the acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations and do not violate public order and good morals.

[0017] It should also be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0018] In view of at least one of the problems existing in the above-mentioned related technologies, this application proposes an interaction method and a training method, apparatus, device and medium for intelligent agents.

[0019] The following description, with reference to the accompanying drawings, illustrates an interaction method and a corresponding intelligent agent training method, apparatus, device, and medium provided in embodiments of this application. Before detailing the embodiments of this application, for ease of understanding, commonly used technical terms in this application will first be introduced: An intelligent agent that interacts within a user interface environment: Used on electronic devices with displays (such as mobile phones, smart cockpits, televisions, and personal computers), this agent is no longer limited to API calls. It understands the user's interaction needs semantically and, based on screen understanding, directly and automatically interacts with the user interface, completing the user's interaction task in one step. This not only improves the user experience but also avoids accidental user interface operations, thus increasing interaction efficiency. This intelligent agent must possess at least the following core capabilities: 1. User interface awareness: Possesses user interface design paradigms and interaction conventions, and is able to identify and understand the visual features and functional semantics of common interface elements such as buttons, input boxes, lists, and labels; 2. Positioning capability: Possesses pixel-level or sub-pixel-level interface element positioning capability to accurately complete fine-grained user interface operations such as clicking, swiping, long pressing, and text input; 3. Planning and reasoning ability: Combining task description information in natural language form with the current user interface state, it can realize intent parsing, multi-step task planning and cross-modal reasoning, has good scene generalization ability, and can pay attention to task context and operation details during execution; 4. Decision-making and reflection capabilities: Possess adaptive decision-making and reflection capabilities in dynamic or unexpected environments, be able to identify and handle abnormal situations such as pop-up ads, permission prompts, network latency, and operation failures, and adjust subsequent behavioral strategies through feedback mechanisms to ensure the robustness and completion rate of tasks.

[0020] Time Step: In user interface interaction tasks, a time step refers to a discrete unit of time or step used to divide the interaction process between the agent and the user interface environment. Each time step represents a specific point in time or time period during which the agent can perceive the environmental state, make decisions, and perform actions.

[0021] Page source files (such as Extensible Markup Language (XML) files): are used to describe the structure, layout, and component properties of a graphical user interface (GUI).

[0022] Figure 1 This is a flowchart illustrating an interaction method provided for an exemplary embodiment of this application. It should be noted that the interaction method of this application embodiment can be applied to an interaction device. In some possible embodiments, the interaction device can be configured in an electronic device to enable the electronic device to perform interactive functions. Additionally, in some possible embodiments, the interaction device can also be software within the electronic device.

[0023] The electronic devices mentioned include, but are not limited to, terminals and personal computers. A terminal is a user-side entity used to receive or transmit signals, such as a mobile phone. Terminals can also be called terminal equipment, user equipment (UE), mobile station (MS), mobile terminal equipment (MT), etc. Terminals can be communication-enabled vehicles, smart cars, mobile phones, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of this application do not limit the specific technology or device form used in the terminal.

[0024] like Figure 1 As shown, the interaction method may include the following steps S101 to S102: Step S101: Obtain the task description information associated with the interactive task to be executed.

[0025] Interactive tasks are those used to interact with the user interface. Examples of interactive tasks include, but are not limited to, navigation tasks, ticket booking / purchase tasks, and food ordering tasks. It should be noted that there are no restrictions on the domain to which interactive tasks belong. Examples of interactive tasks include, but are not limited to, fields such as autonomous driving, smart homes, and smart assistants.

[0026] The task description information can be a description of the user's task requirements in natural language for the interactive task input. The input methods for the task description information include, but are not limited to: touch input (such as swiping, clicking, etc.), keyboard input, voice input, etc.

[0027] In any embodiment of this application, the task description information is obtained, for example, by: in response to a triggering operation on a target interface element in a user interface (referred to as the first user interface in this application) associated with the agent, the task description information associated with the target interface element is obtained.

[0028] The first user interface is the user interaction entry associated with the intelligent agent. It is an interactive interface used to start or wake up the intelligent agent. It is configured to receive user-triggered operations and switch the intelligent agent from an inactive state to a running state in response to the triggering operation. The first user interface is also configured to receive task description information input by the user, which is used to instruct the intelligent agent to automatically execute the corresponding interactive task.

[0029] The triggering operations include, but are not limited to: click operations, drag operations, voice commands, and gesture operations. Target interface elements include interactive interface elements in the first user interface.

[0030] As an example, taking the user interface of a navigation application (APP) or map APP as an example, a user can drag a marker on the first user interface to the target location, and thus determine the task description information as: navigate to the location indicated by the marker based on the drag result of the marker.

[0031] In any embodiment of this application, the task description information is obtained, for example, by using the target description information as the task description information in response to a selection operation of at least one natural language description information recommended in the first user interface. For example, the natural language description information displayed in the first user interface may include: common or frequently used task description information, task description information entered historically by the user, personalized recommended task description information, etc. The user can directly select the task description information associated with the interactive task to be performed from the various natural language description information displayed in the first user interface, thereby reducing the user's operation steps.

[0032] In any embodiment of this application, the task description information is obtained, for example, by obtaining the input task description information in response to an input operation on an interactive control of the first user interface. The interactive control includes, but is not limited to, voice input controls, text input controls, dialog windows, text fields, rich text editors, drop-down lists (allowing users to select one or more options from a predefined list of options as input natural language description information), etc.

[0033] It should be understood that the above-mentioned method of obtaining task description information is only an illustrative example. Other methods can also be used to obtain task description information in natural language form. For example, users can input task description information by dragging and dropping controllable controls in the first user interface. This application embodiment does not limit this.

[0034] In summary, different methods can be used to obtain task description information associated with the interactive task input by the user, thereby improving the flexibility and applicability of this method.

[0035] Step S102: The agent is invoked to determine and execute the execution information for the corresponding time step based on the task description information and visual data at at least one time step, so as to complete the interactive task. The visual data for each time step is associated with the target user interface for the corresponding time step, and the execution information is used to indicate the interface elements to be operated in the target user interface for the corresponding time step, as well as the execution actions corresponding to the interface elements to be operated.

[0036] Here, an agent refers to an intelligent agent that is capable of interacting in a user interface environment, and this agent is a trained agent.

[0037] The visual data includes, but is not limited to, images and videos (i.e., image sequences). The visual data displays or includes its associated target user interface, which includes the user interface (such as the homepage, home screen, or other pages) and desktop of any app. For example, if the visual data is an image, a screenshot or caption can be taken of the target user interface to obtain the visual data. Similarly, if the visual data is a video, the screen of the target user interface can be recorded to obtain the visual data.

[0038] The execution information at each time step indicates the interface elements to be operated on in the target user interface at that time step, and the corresponding action to be performed on those elements. The interface elements to be operated on refer to the visual interface elements that the agent needs to interact with or operate at the corresponding time step. Interface elements are the basic units of human-computer interaction, including but not limited to: buttons, input boxes, sliders, icons, links, cards, etc.

[0039] Among them, the execution action refers to the action that the intelligent agent needs to perform at the corresponding time step, including but not limited to: opening the APP, clicking, swiping, long pressing, inputting, and jumping across APPs.

[0040] The target user interface at the first time step includes the user interface currently displayed on the electronic device, the first user interface associated with the agent (such as the user interface for obtaining task information), and the desktop. The first time step is the first time step among the various time steps between the start and end of the interactive task.

[0041] The target user interface at a time step other than the first time step (such as the i-th time step, where i is a positive integer greater than 1) can be the user interface to which the agent jumps or updates when executing the execution information at the (i-1)-th time step. The i-th time step refers to the non-first time step among the various time steps between the start and end of the interactive task.

[0042] In this embodiment, an intelligent agent can be invoked to determine and execute execution information for a given time step based on task description information and visual data associated with the target user interface at at least one time step, thereby completing the interactive task. The execution information indicates the interface elements to be operated on in the target user interface at the corresponding time step, and the corresponding execution actions for those elements.

[0043] As an example, for the j-th time step (j is a positive integer) in at least one time step, the agent can be invoked to determine the execution information for the j-th time step based on the task description information and the visual data associated with the target user interface at the j-th time step. The execution information for the j-th time step is used to indicate the interface element to be operated in the target user interface at the j-th time step, and the corresponding execution action of that interface element. Thus, in this application, the agent can execute the corresponding execution action on the interface element to be operated in the target user interface at the j-th time step based on the execution information for the j-th time step, to obtain and display the user interface that the j-th time step jumps to or updates to, until the interactive task is completed.

[0044] The interaction method of this application embodiment obtains task description information associated with the interaction task to be executed, and calls an intelligent agent to determine and execute execution information at the corresponding time step based on the task description information and visual data at at least one time step. The visual data at each time step is associated with the target user interface at that time step; the execution information at each time step includes the interface elements to be operated in the target user interface at that time step and their corresponding execution actions (such as clicking, swiping, long pressing, inputting, etc.), thereby realizing end-to-end interaction tasks. In summary, this application does not rely on predefined API interfaces, but instead performs direct operation at the human-computer interface level through understanding and semantic parsing of the visual content of the user interface, significantly improving the versatility and adaptability of the intelligent agent in heterogeneous devices and closed ecosystems. The intelligent agent can recognize the visual features, layout structure, and semantic meaning of interface elements, and combine this with task description information in natural language form to map intent to action, achieving non-intrusive operation of the application. This breaks through the limitations of traditional API call modes in terms of functional coverage and system compatibility, enhancing the task execution boundaries and scalability of the intelligent agent. Furthermore, by generating execution information step-by-step over time, this mechanism supports dynamic planning and context-aware execution of complex, multi-step tasks, exhibiting good fault tolerance and path correction capabilities. Optionally, this mechanism is applicable to user interface environments of different operating systems and device forms (such as mobile phones, tablets, in-vehicle systems, and smart home devices). It only needs to provide the intelligent agent with visual data including the user interface and task description information input by the user to drive automated interaction, reducing dependence on the underlying system architecture and improving deployment flexibility.

[0045] It should be noted that in any embodiment of this application, by automatically performing interactive tasks through an intelligent agent, powerful support can be provided for accessibility services, thereby developing accessibility application programs (APPs) for disabled users or users who are temporarily unable to use electronic devices. This technology can significantly improve the user experience of these users, enabling them to access and operate various digital content and services more conveniently and efficiently.

[0046] In other words, intelligent agents can help develop apps with accessibility features, specifically designed to provide accessibility for users with disabilities or those who are temporarily unable to interact with electronic devices. This not only improves the ease of use and accessibility of applications, but also provides these users with a more friendly and inclusive digital environment.

[0047] As one possible implementation method, Figure 2This is a flowchart illustrating another interaction method provided for an exemplary embodiment of this application. It should be noted that this interaction method can be executed alone, or it can be executed together with any embodiment or possible implementation thereof in this application, or it can be executed together with any technical solution in related technologies. This application does not impose any limitations on this.

[0048] like Figure 2 As shown, the interaction method may include the following steps S201 to S202: Step S201: Obtain the task description information associated with the interactive task to be executed.

[0049] It should be noted that the explanation of step S201 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0050] Step S202: The agent is invoked to determine and execute the execution information for the corresponding time step based on the task description information and visual data at at least one time step, in order to complete the interactive task. The visual data for each time step is associated with the target user interface for the corresponding time step. The execution information includes the execution action, execution parameters, and subtask description information for the corresponding time step in the execution path associated with the interactive task. The execution parameters are used to indicate the interface elements to be operated in the target user interface for the corresponding time step.

[0051] It should be noted that the explanations of visual data, execution actions, and interface elements to be operated in the foregoing embodiments also apply to this embodiment, and will not be repeated here.

[0052] The execution information at each time step includes the execution action and execution parameters output by the agent at that time step, as well as the subtask description information at that time step in the execution path associated with the interactive task.

[0053] The execution path includes subtask description information output by the agent at at least one time step. For example, taking the interactive task of "buying movie tickets in a shopping app (hereinafter referred to as APP1)" as an example, the execution path includes: opening APP1 → selecting "Movies" or "Movie Shows" (tab) → entering the movie name → selecting to buy tickets → selecting the cinema → selecting the date and showtime → selecting seats and making payment.

[0054] The execution parameters at each time step are used to indicate the interface elements to be operated on in the target user interface of the agent at that time step.

[0055] As one possible implementation, the execution parameters at each time step may include at least one of the following: the identification information of the interface element to be operated, the page position information, and the text description information output by the agent at that time step.

[0056] The identification information includes, but is not limited to: the display name, icon, and unique identifier (such as ID) of the interface element to be operated on in the target user interface. The page position information indicates the coordinate position of the interface element to be operated on in the target user interface. The text description information indicates the natural language semantic content carried or associated with the interface element to be operated on, including but not limited to: the functional intent, operational consequences, status description, prompts, or contextual descriptions related to the interactive task. In other words, the text description information provides semantic understanding support, enabling the agent to associate visual input with task intent and achieve context-based reasoning and decision-making.

[0057] In this embodiment, an intelligent agent can be invoked to determine and execute the execution information for a given time step based on task description information and visual data at at least one time step, thereby completing the interactive task. The visual data for each time step is associated with the target user interface for that time step, and the execution information for each time step includes the execution action, execution parameters, and subtask description information for that time step within the execution path associated with the interactive task.

[0058] In any embodiment of this application, the interactive task can be performed using steps A to F: Step A: For the first time step in at least one time step, the first user interface associated with the agent is taken as the target user interface under the first time step, and the agent is called to determine the execution information under the first time step based on the task description information and the visual data associated with the target user interface under the first time step.

[0059] The first time step is the first time step among all the time steps between the start and end of the interactive task.

[0060] The explanations of the execution information at each time step in the aforementioned embodiments also apply to this embodiment, and will not be repeated here.

[0061] Step B: Determine whether the target user interface in the first time step contains the interface element to be operated indicated by the execution information in the first time step. If yes, execute steps C-D; otherwise, execute steps E-F and step D.

[0062] Step C: In response to the fact that the target user interface under the first time step contains the interface element to be operated indicated by the execution information under the first time step, the corresponding execution action is performed on the interface element to be operated in the target user interface under the first time step according to the execution information under the first time step, so as to obtain and display the second user interface to which the first time step jumps or updates.

[0063] In this embodiment, the second user interface to which the user jumps or updates in the first time step can be the same user interface as the target user interface in the first time step, or it can be a different user interface. This application does not impose any restrictions on this. It should be noted that when the second user interface and the target user interface in the first time step are the same user interface, the interface elements displayed in these two user interfaces are not exactly the same.

[0064] Step D: Determine if the task termination condition is met. If so, confirm that the interactive task is completed and end the calling agent.

[0065] The task termination condition is a pre-defined condition used to determine whether to end the interactive task.

[0066] For example, the task termination conditions include, but are not limited to, at least one of the following: The first aspect is that the second user interface, which is accessed or updated at each time step, includes a first interface element associated with the interactive task. This first interface element is a pre-defined element used to indicate the completion of the interactive task. For example, the first interface element includes, but is not limited to, "Order Successful," "Payment Completed," and "Network Timeout."

[0067] The second item is that the second user interface to which each time step jumps or updates includes a second interface element whose attribute state changes to a specified state. The specified state is a pre-defined attribute state used to indicate the completion of the interactive task.

[0068] As an example, taking a second interface element that includes a progress bar to indicate the progress of task execution as the interactive task is nearing completion, the fill ratio of the second interface element will gradually approach 100%. When the fill ratio of the second interface element reaches 100%, it can be determined that the interactive task has been completed and the task termination condition has been met.

[0069] For example, taking a file upload task as an interactive task, when the upload progress bar in the file upload interface reaches 100%, it can be determined that the file has been uploaded and the file upload task can be ended.

[0070] As another example, taking a second interface element including a button element as an example, during the execution of an interactive task, the second interface element may change from a "disabled" state to an "available" state. This change in attribute state can be used to indicate that the task has been completed.

[0071] The third condition is that the total number of executed time steps exceeds the set threshold. The threshold is the pre-set maximum number of executions.

[0072] It's important to note that setting a threshold for the number of time steps is necessary to prevent interactive tasks from getting stuck in infinite loops or stagnating for extended periods. If the total number of time steps executed by the agent exceeds this threshold, the task will be considered complete even without an explicit completion marker. This prevents system resources from being wasted due to program bugs or abnormal situations causing the interactive task to run indefinitely.

[0073] The fourth item is that the content similarity between the second user interface jumped to or updated to in two adjacent time steps is higher than the set similarity threshold. The similarity threshold is a relatively large threshold, such as 95% or 97%.

[0074] Understandably, when the content of the user interface jumped to or updated by two adjacent time steps is highly similar, it may mean that the interactive task is stagnant and has not made any substantial progress. For example, in a navigation task, if the remaining driving distance, estimated arrival time, and other information displayed on the user interface jumped to or updated by several consecutive time steps remain almost unchanged, and the similarity exceeds the set similarity threshold, it can be determined that the navigation task may have encountered an anomaly (such as location loss), and thus the task should be terminated.

[0075] It should be noted that the above task termination conditions are merely illustrative examples, but this application is not limited thereto. Personalized task termination conditions can be set according to the application requirements in different application scenarios, and this application embodiment does not impose any restrictions on this.

[0076] In summary, setting multiple different task termination conditions can improve the flexibility and applicability of task completion. That is, different task termination conditions allow interactive tasks to flexibly adjust completion criteria based on real-time execution conditions, balancing task execution efficiency and resource consumption.

[0077] In this embodiment, it can be determined whether the first time step meets the task termination condition. If so, the interactive task is determined to be completed and the calling agent is terminated. If not, the second user interface that the first time step jumps to or updates to is taken as the target user interface under the second time step. Under the second time step, the calling agent determines the execution information under the second time step based on the task description information and the visual data associated with the target user interface under the second time step and executes it to complete the interactive task.

[0078] Step E: In response to the fact that the target user interface in the first time step does not contain the interface element to be operated as indicated by the execution information in the first time step, jump to the desktop and update the target user interface in the first time step according to the desktop.

[0079] In this embodiment of the application, if the target user interface at the first time step does not contain the interface element to be operated indicated by the execution information at the first time step, the user can jump to the desktop and update the target user interface at the first time step according to the desktop.

[0080] Step F: The agent is invoked to redetermine the execution information for the first time step based on the task description information and the visual data associated with the target user interface in the updated first time step, and then executes it to obtain and display the second user interface to which the first time step jumps or is updated.

[0081] For example, taking the interactive task of "buying movie tickets in APP1" as an example, if "APP1" is not in the first user interface associated with the agent, it can jump to the desktop and use the desktop as the target user interface in the first time step, so that the agent can effectively identify and locate the specific location of "APP1" from the desktop, and then click on "APP1" to enter the homepage of APP1 to buy movie tickets.

[0082] In summary, in scenarios where agents interact with user interfaces, if the target user interface at the first time step does not contain the interface element predicted by the agent, the system automatically jumps to the device desktop and re-predicts the execution information at the first time step. This significantly improves the agent's adaptability, robustness, and task completion rate. Specifically, by introducing a "desktop" as a system-level neutral interface, the agent can proactively reset its execution context when encountering operational deadlocks, re-perceive the current device state, and plan task paths, thus constructing an adaptive closed loop of "perception-decision-correction." This mechanism not only enhances fault tolerance for abnormal scenarios but also gives the agent human-like operational flexibility: when unable to execute as expected, it can proactively return to the global view (desktop), integrate cross-application resources, identify available entry points, and organize discrete interface operations into a coherent task flow. Therefore, the failure of a single-step operation no longer leads to task termination but triggers a system-level recovery strategy, improving task completion rate and strengthening the agent's generalization ability and interaction continuity in open, heterogeneous user interface environments.

[0083] In any embodiment of this application, the interactive task may also be performed using steps G to J: Step G: In response to the fact that the task termination condition is not met at the (i-1)th time step, the second user interface that the (i-1)th time step jumps to or updates to is taken as the target user interface at the ith time step; where i is a positive integer greater than 1.

[0084] Here, the i-th time step refers to the non-first time step among the various time steps between the start and end of the interactive task.

[0085] Step H: Continue to call the agent to determine the execution information at the i-th time step based on the task description information and the visual data associated with the target user interface at the i-th time step.

[0086] Step 1: Based on the execution information at the i-th time step, perform the corresponding execution actions on the interface elements to be operated in the target user interface at the i-th time step, so as to obtain and display the second user interface that the i-th time step jumps to or updates to.

[0087] Step J: Determine whether the task termination condition is met at the i-th time step. If so, determine that the interactive task is completed and end the calling agent.

[0088] In this embodiment, it can be determined whether the task termination condition is met at the i-th time step. If yes, the interactive task is determined to be completed and the calling agent is terminated. If no, the second user interface that the i-th time step jumps to or updates to is taken as the target user interface at the (i+1)-th time step. At the (i+1)-th time step, the calling agent determines the execution information at the (i+1)-th time step based on the task description information and the visual data associated with the target user interface at the (i+1)-th time step and executes it to complete the interactive task.

[0089] In summary, after the agent predicts and executes the execution information at each time step, it can dynamically determine whether the interactive task is completed based on the task termination condition. This allows the agent to dynamically adapt to complex environments, improve the quality of task completion, and avoid ineffective computation and resource waste.

[0090] For example, let's take the interactive task as "buy a movie ticket in APP1" and the visual data as images. Assume the user inputs a task description in natural language form as "buy a movie ticket for today's XXXX (movie name) in APP1," and the target user interface at the first time step is as follows: Figure 3 (a) shows the desktop. Based on the task description information and the desktop screenshot, the agent predicts the execution parameters in the execution information of the first time step, including the name or coordinates of "APP1", the execution action as "open", and the subtask description information as "open APP1". The agent executes the execution information of the first time step (clicks APP1 on the desktop), and can jump to the following... Figure 3 (b) shows the homepage of APP1, and serves as the target user interface at the second time step.

[0091] The agent, based on the task description information and Figure 3 (b) shows a screenshot of the user interface. The execution parameters in the predicted execution information for the second time step include the name or coordinates of "movie performance," the execution action is "click," and the subtask description information is "select movie." The agent executes the execution information for the second time step (click). Figure 3 (b) The “Movie Shows” tab shown in area 31 can be used to jump to, for example... Figure 3 (c) shows the user interface, and serves as the target user interface at the third time step.

[0092] The agent, based on the task description information and Figure 3 (c) shows a screenshot of the user interface. The execution parameters in the predicted execution information at the third time step include the name or coordinates of the "input box," the execution action is "input," and the subtask description information is "input movie name." The agent executes the execution information at the third time step (in...). Figure 3(c) Enter the movie name "XXXX" in the input box shown in area 32 and search. This will take you to a page like... Figure 3 The user interface shown in (d) is used as the target user interface in the fourth time step.

[0093] The agent, based on the task description information and Figure 3 (d) shows a screenshot of the user interface. The execution parameters in the predicted execution information at the fourth time step include the name or coordinates of the "Buy Tickets" button, the execution action is "Click", and the subtask description information is "Select Buy Tickets". The agent executes the execution information at the fourth time step (Click). Figure 3 (d) The “Buy Tickets” button shown in area 33 can jump to, for example, Figure 3 The user interface shown in (e) serves as the target user interface at the fifth time step.

[0094] The agent, based on the task description information and Figure 3 (e) shows a screenshot of the user interface. The execution parameters in the predicted execution information at the fifth time step include the coordinates of "FFFF Cinema", the execution action is "click", and the subtask description information is "select cinema". The agent executes the execution information at the fifth time step (click). Figure 3 (e) Area 34 shows "FFFF Cinema", which can be used to jump to such as Figure 3 The user interface shown in (f) is used as the target user interface at the sixth time step.

[0095] The agent, based on the task description information and Figure 3 (f) shows a screenshot of the user interface. The execution parameters in the predicted execution information at the sixth time step include the coordinates of the ticket purchase button corresponding to the movie ticket for "XXXX" at a certain time today. The execution action is "click", and the subtask description information is "select date and showtime". The agent executes the execution information at the sixth time step (click). Figure 3 (f) The “Buy Tickets” button shown in area 35 can jump to, for example, Figure 3 (g) shows the user interface, and serves as the target user interface at the seventh time step.

[0096] The agent, based on the task description information and Figure 3(g) shows a screenshot of the user interface. The execution parameters in the execution information at the seventh time step predicted include the coordinates of the "a certain seat" and the "pay" button. The execution action is "click" and the subtask description information is "select seat and pay". When the agent executes the execution information at the seventh time step, it can jump to the payment interface, so that the user can manually pay in the payment interface to buy the movie ticket for "XXXX". Alternatively, in the case of password-free payment, the agent can automatically buy the movie ticket for "XXXX".

[0097] The interaction method in this application embodiment includes, in the execution information at each time step, the execution action and execution parameters at that time step, as well as the subtask description information at that time step in the execution path associated with the interaction task. This can enhance the transparency and interpretability of task execution, enabling the intelligent agent to have strong decision-making ability, adaptability and interpretability in complex environments.

[0098] The above are various embodiments corresponding to the application method (i.e., interaction method) of the intelligent agent. This application also proposes a training method for the intelligent agent.

[0099] Figure 4 This is a flowchart illustrating a method for training an intelligent agent, provided as an exemplary embodiment of this application. It should be noted that this method for training the intelligent agent can be executed alone, or it can be executed in conjunction with any embodiment of this application or any possible implementation thereof, or it can be executed in conjunction with any technical solution in related technologies. This application does not impose any limitations on this method.

[0100] like Figure 4 As shown, the training method for this intelligent agent may include the following steps S401 to S403: Step S401: Obtain first training data; wherein, the first training data includes first task description information associated with the first interactive task and first visual data associated with the first user interface at the first time step.

[0101] The method of obtaining the first training sample is not limited. For example, the first training sample can be obtained from the training set or the test set. Alternatively, the first training sample can be generated manually. Or, information can be collected from a real user interface environment to obtain the first training sample, etc. The embodiments of this application do not limit this.

[0102] Step S402: Invoke the first intelligent agent to execute the first interactive task based on the first task description information and the first visual data at at least one time step, so as to obtain execution trajectory information; wherein, the first visual data at each time step is associated with the first user interface at the corresponding time step, the execution trajectory information is used to indicate the execution information of the first intelligent agent at at least one time step, and the execution information is used to indicate the interface elements to be operated in the first user interface at the corresponding time step and the execution actions corresponding to the interface elements to be operated.

[0103] It should be noted that the implementation principle of step S402 is similar to that of step S102, and will not be elaborated here.

[0104] In any embodiment of this application, the execution information at each time step may include the execution action, execution parameters, and subtask description information at the corresponding time step in the predicted execution path associated with the first interactive task; wherein, the execution parameters are used to indicate the interface elements to be operated at the corresponding time step.

[0105] The execution parameters include at least one of the following: the identifier information of the interface element to be operated at the corresponding time step, the page position information, and the text description information.

[0106] Step S403: Based on the labeled trajectory information and execution trajectory information associated with the first training data, supervised training is performed on the first intelligent agent to obtain the trained second intelligent agent.

[0107] The content of the labeled trajectory information is similar to that of the execution trajectory information, and will not be elaborated here.

[0108] In this application embodiment, the value of the loss function in the supervised training stage can be determined based on the difference between the labeled trajectory information and the execution trajectory information associated with the first training data. In this application, this loss function is referred to as the first loss value. The first loss value is positively correlated with the difference mentioned above. That is, the larger the difference, the larger the first loss value, and vice versa. Thus, in this application, the first agent can be supervised and trained based on the first loss value to minimize the first loss value and obtain the trained second agent.

[0109] It should be noted that the above example only uses the termination condition of supervised training as the first loss value minimization, but this application is not limited to this. For example, the termination condition may also include: the training time reaches a set time, the training rounds reach a set number of rounds, etc. The embodiments of this application do not limit this.

[0110] In any embodiment of this application, the method for obtaining the first training sample and the labeled trajectory information is as follows: First, the labeled execution path associated with the first interactive task is obtained. Then, the description information of each subtask in the labeled execution path is executed, and the interface elements and execution actions operated when executing each subtask description information are recorded to obtain the labeled trajectory information and the first visual data associated with each subtask description information. Then, the first task description information associated with the labeled trajectory information can be obtained, and the first training data is generated based on the first visual data associated with each subtask description information and the first task description information.

[0111] In this embodiment, the first visual data associated with the description information of the first subtask can be used as the first visual data associated with the first user interface in the first time step, or all the first visual data associated with the description information of the subtasks can be used as the first visual data associated with the first user interface in the first time step. This application does not limit this.

[0112] The first task description information can be manually configured or generated by calling a large model; this embodiment does not impose any limitations on this. That is, in response to a user's configuration operation triggered by the labeled trajectory information, the first task description information manually configured by the user is obtained, or the first task description information is generated by calling a large model based on the labeled trajectory information. Therefore, different methods can be used to obtain the first task description information associated with the first interactive task, which can improve the flexibility and applicability of the method.

[0113] As an example, training data can be automatically collected using Android's debugging tools: For each first interactive task, a labeled execution path is set. The subtask descriptions of each step in the labeled execution path are executed, and screenshots of the current user interface and screen element information obtained by the Android debugging tools are recorded. This process continues until all steps in the labeled execution path are completed, resulting in a labeled trajectory. This trajectory can then be manually annotated to obtain the first task description information. Alternatively, a large model can be used to generate the first task description information based on the labeled trajectory information.

[0114] In summary, it can quickly generate a large amount of labeled trajectory information, reducing the cost of training data collection.

[0115] In any embodiment of this application, the method for obtaining the first training sample and the labeled trajectory information is as follows: First, the first task description information associated with the first interactive task can be obtained. Then, in response to the user-triggered interactive operation adapted to the first task description information, the interface elements operated by the interactive operation and the execution action can be recorded to obtain the labeled trajectory information and the first visual data associated with the interactive operation. Then, the first training data can be generated based on the first visual data and the first task description information.

[0116] The first task description information can be manually input, or it can be diverse task description information generated by a large model. This application embodiment does not limit this.

[0117] As an example, manual operations can be tracked to collect training data: For each first interaction task, diverse first task description information is generated manually or by a large model. Each first task description requires the user to perform manual operations on an electronic device. Simultaneously, screenshots of the current user interface and screen element information obtained by Android debugging tools are recorded, along with the user's interactive operations, including the actions performed, the interface elements operated on, and the operation area (or position). Each completed first task description yields a labeled trajectory.

[0118] In summary, diverse labeled trajectory information can be obtained to cope with anomalies and changes in various applications and improve the task completion capabilities of intelligent agents.

[0119] The training method for the intelligent agent provided in this application can improve the task completion ability of the intelligent agent by using a first training sample to conduct supervised training.

[0120] As one possible implementation, this application also proposes a method for training an agent.

[0121] Figure 5 This is a schematic flowchart illustrating another method for training an intelligent agent, provided as an exemplary embodiment of this application. It should be noted that this method for training the intelligent agent can be executed alone, or it can be executed in conjunction with any embodiment of this application or any possible implementation thereof, or it can be executed in conjunction with any technical solution in related technologies. This application does not impose any limitations on this method.

[0122] like Figure 5 As shown, in Figure 4 Based on the illustrated embodiment, the training method for this agent may further include the following steps S501 to S504: Step S501: Obtain second training data; wherein, the second training data includes second task description information associated with the second interactive task and second visual data associated with the second user interface at the first time step.

[0123] The second training data consists of training data without labeled trajectory information. The method for obtaining the second training data is similar to that for the first training data, and will not be elaborated upon here.

[0124] Step S502: Invoke the second agent to perform the second interactive task based on the second task description information and the second visual data at at least one time step to obtain an image sequence; wherein the second visual data at each time step is associated with the second user interface at the corresponding time step, and the image sequence includes visual images associated with the user interface updated by the second agent performing the execution action at at least one time step.

[0125] For example, taking a screenshot of a user interface associated with a visual image as an example, at each time step, the second agent performs the corresponding execution action on the interface element to be operated in the second user interface at that time step according to the execution information at that time step. Then, it can obtain the user interface to which it jumps or updates at that time step, and perform a screenshot operation on the user interface to obtain a screenshot of the user interface at the current time step. By sorting the screenshots of the user interfaces at all time steps, an image sequence can be obtained.

[0126] Step S503: Determine the target reward value based on the image sequence; wherein the target reward value is used to indicate the performance quality of the second agent in the second interactive task.

[0127] It should be noted that reward functions or reward models are generally divided into step rewards and task completion rewards. Step rewards calculate the reward for each step of execution, while task completion rewards calculate the reward for the entire trajectory after the interactive task is completed. Because step rewards are too sparse and difficult to determine—for example, if the agent makes an error at a certain time step and can revert to the next step to continue correctly—it is difficult to determine the impact of the current time step on the final result. To address these issues, this application uses task completion rewards to calculate the target reward value.

[0128] In any embodiment of this application, calculating the target reward value based on the task completion reward includes: in response to the task type to which the second interactive task belongs being the first task type, obtaining the set interface element associated with the first task type, and determining the target reward value for the second agent to perform the second interactive task based on the difference between the interface element information in the last visual image in the image sequence and the set interface element.

[0129] The first type of interactive task includes tasks whose user interface, as navigated to by the last time step, contains the same interface elements. Here, "defined interface elements" refers to the same interface elements that appear in the user interface navigated to by the last time step after the first type of interactive task has been successfully executed, including but not limited to: "Order Successful," "Payment Completed," etc.

[0130] As an example, the target reward value is relatively high when the interface element information of the last visual image in the image sequence includes the target interface element, and relatively low when the interface element information of the last visual image does not include the target interface element.

[0131] In summary, considering that different task completion states can be determined by different interface elements, the interface element information in the final visual image required for discrimination can be found from the image sequence according to the rules. Compared with using all visual images and interface element information in the image sequence, filtering the final visual image and interface element information is more accurate and faster inference.

[0132] In any embodiment of this application, calculating the target reward value based on the task completion reward includes: in response to the task type to which the second interactive task belongs being the second task type, extracting keyframes from the image sequence; and calling a large model to predict the target reward value of the second agent performing the second interactive task based on the keyframes and / or the interface element information in the keyframes.

[0133] The second type of interactive task includes tasks where the user interface to which the last time step jumps or updates contains different or dynamic interface elements, that is, the user interface page to which the last time step jumps or updates is dynamically changing.

[0134] For example, in the case where the large model includes a large language model, the large language model can be invoked to predict the target reward value of the second agent performing the second interactive task based on the interface element information in the keyframe.

[0135] For example, when the large model includes a multimodal large model, the multimodal large model can be invoked to predict the target reward value of the second agent performing the second interactive task based on the key frame and the interface element information in the key frame.

[0136] In summary, by using a large model to predict the reward value of an agent performing tasks under a dynamic user interface, the method can intelligently infer the agent's performance quality without relying on fixed rules when interface elements change frequently, thus improving the flexibility and applicability of the method.

[0137] Step S504: Based on the target reward value, perform reinforcement learning on the second agent to obtain a trained third agent.

[0138] In the embodiments of this application, reinforcement learning techniques can be used to perform reinforcement learning on the second agent based on the target reward value, thereby obtaining a trained third agent.

[0139] In any embodiment of this application, third training data can also be obtained; wherein, the third training data includes third task description information associated with the third interactive task and third visual data associated with the third user interface at the first time step; the third agent is invoked to execute the third interactive task according to the third task description information and the third visual data at at least one time step to obtain a first execution result, and the set agent is invoked to execute the third interactive task according to the third task description information and the third visual data at at least one time step to obtain a second execution result. Thus, in this application, knowledge distillation can be performed on the set agent according to the difference between the first execution result and the second execution result to obtain a trained target agent.

[0140] In this context, the third visual data at each time step is associated with the third user interface at the corresponding time step; the number of parameters of the third intelligent agent is higher than that of the set intelligent agent, that is, the third intelligent agent can be a large-sized intelligent agent, and the set intelligent agent can be a small-sized intelligent agent. For example, the third intelligent agent can be a teacher intelligent agent, and the set intelligent agent can be a student intelligent agent.

[0141] In summary, knowledge distillation technology distills a large-scale third-party agent into a smaller-scale target agent, improving the inference speed of the target agent while maintaining its prediction accuracy or positive evaluation metrics. Optionally, the target agent can be deployed in an electronic device, avoiding the uploading of visual data associated with the user interface on the user device to the cloud and thus preventing infringement on user privacy.

[0142] The training method for intelligent agents provided in this application effectively avoids the tedious process of manual annotation by performing reinforcement learning on the intelligent agents, providing strong support for convenient task expansion, and enhancing the generalization ability of intelligent agents in diverse scenarios.

[0143] As one possible implementation, this application also proposes a method for training an agent.

[0144] Figure 6This is a flowchart illustrating another method for training an intelligent agent, provided as an exemplary embodiment of this application. It should be noted that this method for training the intelligent agent can be executed alone, or it can be executed in conjunction with any embodiment of this application or any possible implementation thereof, or it can be executed in conjunction with any technical solution in related technologies. This application does not impose any limitations on this method.

[0145] like Figure 6 As shown, the training method for this intelligent agent may include the following steps S601 to S604: Step S601: Using fourth training data associated with at least one pre-training mechanism, the initial agent is pre-trained to obtain the first agent.

[0146] In any embodiment of this application, at least one pre-training mechanism includes a first pre-training mechanism, wherein the fourth training data includes: a first sample image displaying a sample user interface, and fourth task description information associated with the first sample image; wherein the fourth task description information is used to indicate the target interface element to be operated in the sample user interface, for example, taking the fourth task description information as "click the search button in the current user interface" as an example, the target interface element is "search button".

[0147] In this application, an initial agent can be used to predict the predicted position of the target interface element in the first sample image based on the fourth task description information, and the initial agent can be pre-trained based on the difference between the predicted position and the actual position of the target interface element to obtain the first agent; wherein, the actual position is determined based on the page source file of the sample user interface.

[0148] For example, the value of the loss function in the pre-training stage can be determined based on the difference between the predicted position and the actual position of the target interface element. In this application, this value is referred to as the second loss value. The second loss value is positively correlated with the difference mentioned above. That is, the larger the difference, the larger the second loss value, and vice versa. Thus, in this application, the initial agent can be pre-trained based on the second loss value to minimize the second loss value and obtain the first agent.

[0149] It should be noted that the above example only uses the minimization of the second loss value as the termination condition of the pre-training stage, but this application is not limited to this. For example, the termination condition may also include: the training time reaches a set time, the training rounds reach a set number of rounds, etc. The embodiments of this application do not limit this.

[0150] In summary, using the fourth training data to pre-train the initial agent can enhance the agent's ability to perceive, reason, and locate interface elements.

[0151] In any embodiment of this application, at least one pre-training mechanism includes a second pre-training mechanism, and the fourth training data includes image-text data pairs. In this application, an initial agent can encode sample text in the image-text data pair to obtain text features, and an initial agent can encode a second sample image in the image-text data pair to obtain image features. Thus, in this application, the initial agent can be semantically aligned based on the semantic similarity between text features and image features to obtain a first agent. Therefore, a large number of image-text data pairs can be used to align visual and linguistic spaces, significantly improving the agent's comprehensive capabilities in multimodal perception, cross-modal reasoning, and task execution.

[0152] It should be understood that the above example only uses one pre-training mechanism to pre-train the initial agent to obtain the first agent. In actual applications, multiple pre-training mechanisms can be combined to pre-train the initial agent. For example, the first pre-training mechanism can be used to pre-train the initial agent, and then the second pre-training mechanism can be used to pre-train the pre-trained initial agent again to obtain the first agent; or, the second pre-training mechanism can be used to pre-train the initial agent, and then the first pre-training mechanism can be used to pre-train the pre-trained initial agent again to obtain the first agent. The embodiments of this application do not limit this.

[0153] Step S602: Obtain first training data; wherein, the first training data includes first task description information associated with the first interactive task and first visual data associated with the first user interface at the first time step.

[0154] Step S603: Invoke the first intelligent agent to execute the first interactive task based on the first task description information and the first visual data at at least one time step, so as to obtain the execution trajectory information; wherein, the first visual data at each time step is associated with the first user interface at the corresponding time step.

[0155] The execution trajectory information is used to indicate the execution information of the first agent at at least one time step, and the execution information is used to indicate the interface element to be operated in the first user interface at the corresponding time step and the execution action corresponding to the interface element to be operated.

[0156] Step S604: Based on the labeled trajectory information and execution trajectory information associated with the first training data, supervised training is performed on the first intelligent agent to obtain the trained second intelligent agent.

[0157] It should be noted that the explanations of steps S602 to S604 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0158] The training method for the agent in this application uses fourth training data associated with at least one pre-training mechanism to pre-train the initial agent, which can significantly improve the agent's basic capabilities, generalization performance and training efficiency, while reducing the dependence on labeled data.

[0159] In any embodiment of this application, in order to reduce the cost of manual annotation and provide generalization of the agent at low cost, the training process of the agent can be divided into the following stages: pre-training, continued pre-training, supervised fine-tuning training with few samples, reinforcement training, and knowledge distillation training; wherein, the input data of the agent is: visual data + user instructions (i.e., task description information), and the output is the execution path, execution action and execution parameters obtained by path planning. The execution action includes common actions of operating a screen device: opening an APP, clicking, swiping, long pressing, inputting, jumping across APPs, etc., and the execution parameters include: APP name, coordinates, text, etc.

[0160] The first stage, pre-training: Based on text and image data, the agent is trained with semantic alignment to learn general knowledge and improve generalization ability. That is, a large number of text and image data pairs are used to align the visual space and the linguistic space of LLM.

[0161] In the second stage, pre-training continues: taking visual data as an example, the agent is further trained on screenshot images of the user interface and interactive tasks based on the pre-trained data. This improves the agent's semantic understanding of the screenshot images and enhances the agent's ability to understand, perceive, reason, and locate interface elements in the screenshot images.

[0162] The third stage, supervised fine-tuning training with a small number of samples: The agent is trained in a supervised manner using a small amount of training data with labeled trajectory information, enabling the agent to acquire basic execution capabilities. This training stage requires only a small amount of manually labeled high-quality training data, resulting in lower costs.

[0163] The core of supervised training is how to quickly and cost-effectively acquire labeled trajectory information. These trajectories are used to indicate a series of screenshots and actions required to complete an interactive task.

[0164] For example, the methods for collecting and labeling training data mainly include the following two: The first type is an automatic data collection tool based on Android debugging tools.

[0165] An execution path is defined for each interactive task. Each step is executed, and the current screenshot and UI element information obtained from the Android debugging tool are recorded until the task path ends. Each completed path constitutes a trajectory. Based on the trajectory, user-input task description information is manually annotated, or external large-scale models can be used to generate task description information.

[0166] For example: Task definition: Purchase movie tickets in APP1; Execution path definition: Open APP1 → Select "Movies" or "Movie Shows" (tab) → Enter movie name → Select "Buy Tickets" → Select cinema → Select date and showtime → Select seat and pay.

[0167] The advantage of this method is that it can quickly generate a large number of trajectories, making it suitable for obtaining a large amount of labeled data during the cold start phase of agent training.

[0168] The second type is a data collection tool based on manual trajectory tracking.

[0169] For each interactive task, diverse task description information is first generated manually or through a large model. Each task description requires manual operation by the user on the device. Simultaneously, the current screenshot and interface element information obtained by the Android debugging tool, along with the user's interaction, are recorded, including the actions performed and the area of ​​operation. Each completed task description constitutes a trajectory.

[0170] For example: Task definition: Purchase movie tickets in APP1; User request: Purchase a movie ticket for today's XXXX movie in APP1; User execution path: Open APP1 → Select "Movies" or "Movie Shows" (tab) → Enter movie name → Select ticket purchase → Select cinema → Select date and showtime → Select seat and payment.

[0171] The advantage of this method is that the trajectory can be diversified, it can cope with anomalies and changes in various applications, and it is suitable for supplementing training data for a certain type of poorly performing application scenario.

[0172] It should be noted that the cost of labeling supervised training data is relatively high. Therefore, a small amount of supervised training data can be labeled on a small number of tasks to perform supervised fine-tuning training of the agent.

[0173] The fourth stage, reinforcement training: In reinforcement learning, only the task completion state needs to be defined. Rules and a large model are used to help determine whether the task is completed, thereby guiding the agent to explore and utilize resources. That is, a reward model for task completion is constructed based on rules and a large model, and online or offline reinforcement learning algorithms are used to improve the agent's generalization ability. This training stage does not require manual annotation and performs reinforcement training on the agent after supervised fine-tuning, which can improve training efficiency.

[0174] It's important to note that during supervised fine-tuning training, the agent tends to memorize training data, making it difficult to explore new interactive tasks and resulting in insufficient generalization ability. Furthermore, supervised fine-tuning training data cannot cover all real-world interactive scenarios, such as application redesigns, pop-ups, and advertisements. Therefore, after supervised fine-tuning training, reinforcement learning can be applied to the agent to expand and generalize to meet the needs of more interactive tasks.

[0175] The enhanced training simulation environment is built on an Android emulator. After training for a period of time, it can be switched to a real device simulation environment with different system versions and models. The Android emulator can be expanded to multiple versions at low cost, which helps improve training efficiency, while real device simulation can cover more system versions and models, which helps improve task completion rates in various scenarios.

[0176] The reward model in reinforcement learning is generally divided into step rewards and task completion rewards. Step rewards calculate the reward for each step of execution, while task completion rewards calculate the reward for the entire trajectory after the task is completed. It should be noted that step rewards are too sparse and difficult to determine. For example, if the agent makes an error at a certain time step and can revert to the next step to continue correctly, it is difficult to determine the impact of the current time step on the final result. To address these issues, this application employs task completion rewards. These rewards can be constructed using rules and a large model.

[0177] First, different task completion states can be determined by different screenshots. Therefore, based on rules, key screenshots and interface element information needed for identification are found from the trajectory. Compared to using screenshots and interface element information from the entire trajectory, selecting key screenshots and interface elements for identification is more accurate and faster inference.

[0178] Secondly, some failure states can be determined by whether the agent outputs the action "Complete". For example, some failure states can be determined by UI element information, such as whether the UI element information includes "Order Successful" or "Payment Completed". Compared with using external large models for determination, the above rule-based determination is faster, more accurate, and less costly.

[0179] Furthermore, for trajectories that cannot be determined by rules, a large model request is constructed to request an external large language model (using interface element information) or a multimodal large model (using screenshot images and interface element information) for determination.

[0180] In summary, real-time interaction can be achieved through Android emulators and real device environments, allowing for the acquisition of richer trajectories. The reward value can be calculated by the task completion reward model, and the agent can be iteratively updated through reinforcement learning algorithms (such as proximal policy optimization and online policy gradient methods) to improve the agent's generalization ability.

[0181] The fifth stage, knowledge distillation training, distills large-sized agents into smaller ones, improving response speed. In addition, it can provide edge-side agents without uploading screenshots to the cloud, thus protecting user privacy.

[0182] At each time step, the agent outputs execution information in the form of a sequence of character tokens, which includes the probability distribution of each token. Larger agents generate soft labels, i.e., the probability distribution of each category, while smaller agents adjust their parameters based on their own original labels and the soft labels output by the larger agents.

[0183] In summary, the technical solution provided in this application has at least the following advantages: 1. It requires only a small amount of manually labeled supervised training data, effectively reducing labeling costs. 2. Reinforcement learning uses a task-level reward model, avoiding manual labeling, facilitating task expansion, and improving the generalization ability of the agent. 3. Through knowledge distillation technology, large-sized agents are distilled into small-sized agents, improving inference speed while basically ensuring that the performance indicators do not decrease, and providing edge-side agents, thereby avoiding the uploading of user screenshots to the cloud and avoiding infringement of user privacy.

[0184] To implement the above embodiments, this application also proposes an interactive device.

[0185] Figure 7 This is a schematic diagram of the structure of an interactive device provided for an exemplary embodiment of this application.

[0186] like Figure 7 As shown, the interactive device 700 may include an acquisition module 710 and a processing module 720.

[0187] The acquisition module 710 is used to acquire task description information associated with the interactive task to be executed; the processing module 720 is used to call the agent to determine the execution information at the corresponding time step based on the task description information and visual data at at least one time step, and execute it to complete the interactive task; wherein, the visual data at each time step is associated with the target user interface at the corresponding time step; wherein, the execution information is used to indicate the interface elements to be operated in the target user interface at the corresponding time step, and the execution action corresponding to the interface elements to be operated.

[0188] In one implementation of this application, the execution information includes the execution action and execution parameters at the corresponding time step, as well as the subtask description information at the corresponding time step in the execution path associated with the interactive task; wherein, the execution parameters are used to indicate the interface elements to be operated in the target user interface at the corresponding time step; wherein, the execution parameters include at least one of the identification information, page position information, and text description information of the interface elements to be operated at the corresponding time step.

[0189] In one implementation of this application, the acquisition module 710 is configured to perform any of the following: in response to a triggering operation on a target interface element in a first user interface associated with the agent, acquire task description information associated with the target interface element; in response to a selection operation on target description information among at least one natural language description information recommended in the first user interface, use the target description information as task description information; in response to an input operation on an interactive control of the first user interface, acquire input task description information.

[0190] In one implementation of this application, the interactive control includes at least one of the following: a voice input control, a text input control, a dialog window, a text field, a rich text editor, and a drop-down list.

[0191] In one implementation of this application, the processing module 720 is configured to: for the first time step in at least one time step, take the first user interface associated with the agent as the target user interface under the first time step, and call the agent to determine the execution information under the first time step based on the task description information and the visual data associated with the target user interface under the first time step; in response to the target user interface under the first time step containing the interface element to be operated indicated by the execution information under the first time step, perform the corresponding execution action on the interface element to be operated in the target user interface under the first time step according to the execution information under the first time step, so as to obtain and display the second user interface to which the first time step jumps or updates; in response to the task end condition being met, determine that the interactive task is completed, and end the calling of the agent.

[0192] In one implementation of this application, the processing module 720 is further configured to: jump to the desktop in response to the fact that the target user interface in the first time step does not contain the interface element to be operated indicated by the execution information in the first time step; update the target user interface in the first time step according to the desktop; and call the agent to redetermine the execution information in the first time step and execute it according to the task description information and the visual data associated with the updated target user interface in the first time step, so as to obtain and display the second user interface to which the first time step jumps or is updated.

[0193] In one implementation of this application, the processing module 720 is further configured to: respond to the fact that the task termination condition is not met at the (i-1)th time step, take the second user interface that the (i-1)th time step jumps to or updates to as the target user interface at the i-th time step; where i is a positive integer greater than 1; continue to call the agent to determine the execution information at the i-th time step based on the task description information and the visual data associated with the target user interface at the i-th time step; according to the execution information at the i-th time step, perform corresponding execution actions on the interface elements to be operated in the target user interface at the i-th time step to obtain and display the second user interface that the i-th time step jumps to or updates to; and respond to the fact that the task termination condition is met, determine that the interactive task is completed and end the calling of the agent.

[0194] In one implementation of this application, the task termination condition includes at least one of the following: the second user interface to which the user jumps or updates includes a first interface element associated with the interactive task; the second user interface to which the user jumps or updates includes a second interface element whose attribute state has changed to a specified state; the total number of executed time steps is greater than a set number threshold; and the content similarity between the second user interfaces to which two adjacent time steps jump or update is higher than a set similarity threshold.

[0195] It should be noted that the foregoing explanation of any embodiment of the interaction method also applies to the interaction device of that embodiment, and will not be repeated here.

[0196] In the interactive device of this application embodiment, task description information associated with the interactive task to be executed is obtained, and an intelligent agent is invoked to determine and execute the execution information at the corresponding time step based on the task description information and visual data at at least one time step. The visual data at each time step is associated with the target user interface at the corresponding time step; the execution information at each time step includes the interface elements to be operated in the target user interface at that time step and their corresponding execution actions (such as clicking, swiping, long pressing, inputting, etc.), thereby realizing end-to-end interactive tasks. In summary, this application does not rely on predefined API interfaces, but instead performs direct operation at the human-computer interface level through understanding and semantic parsing of the visual content of the user interface, significantly improving the versatility and adaptability of the intelligent agent in heterogeneous devices and closed ecosystems. The intelligent agent can recognize the visual features, layout structure, and semantic meaning of interface elements, and combine task description information in natural language form to map intent to action, achieving non-intrusive operation of the application. This breaks through the limitations of traditional API call modes in terms of functional coverage and system compatibility, enhancing the task execution boundaries and scalability of the intelligent agent. Furthermore, by generating execution information step-by-step over time, this mechanism supports dynamic planning and context-aware execution of complex, multi-step tasks, exhibiting good fault tolerance and path correction capabilities. Optionally, this mechanism is applicable to user interface environments of different operating systems and device forms (such as mobile phones, tablets, in-vehicle systems, and smart home devices). It only needs to provide the intelligent agent with visual data including the user interface and task description information input by the user to drive automated interaction, reducing dependence on the underlying system architecture and improving deployment flexibility.

[0197] To implement the above embodiments, this application also proposes a training device for intelligent agents.

[0198] Figure 8 This is a schematic diagram of the structure of a training device for an intelligent agent provided for an exemplary embodiment of this application.

[0199] like Figure 8 As shown, the training device 800 for the intelligent agent may include: a first acquisition module 810, a first execution module 820, and a first training module 830.

[0200] The system includes a first acquisition module 810 for acquiring first training data, which includes first task description information associated with a first interactive task and first visual data associated with a first user interface at the first time step. A first execution module 820 is used to invoke a first agent to execute a first interactive task based on the first task description information and the first visual data at at least one time step, to obtain execution trajectory information. The first visual data at each time step is associated with the first user interface at the corresponding time step, and the execution trajectory information indicates the execution information of the first agent at at least one time step. The execution information indicates the interface elements to be operated in the first user interface at the corresponding time step and the corresponding execution actions. A first training module 830 is used to perform supervised training on the first agent based on the labeled trajectory information and execution trajectory information associated with the first training data, to obtain a trained second agent.

[0201] In one implementation of this application, the execution information includes the execution action and execution parameters at the corresponding time step, as well as the subtask description information at the corresponding time step in the predicted execution path associated with the first interactive task; wherein, the execution parameters are used to indicate the interface elements to be operated at the corresponding time step.

[0202] In one implementation of this application, the first acquisition module 810 is configured to: acquire the annotation execution path associated with the first interactive task; execute the description information of each subtask in the annotation execution path, and record the interface elements and execution actions operated when executing each subtask description information, so as to obtain the annotation trajectory information and the first visual data associated with each subtask description information; acquire the first task description information associated with the annotation trajectory information, and generate the first training data based on the first visual data associated with each subtask description information and the first task description information.

[0203] In one implementation of this application, the first acquisition module 810 is configured to perform any of the following: in response to a configuration operation triggered by the labeled trajectory information, acquire the configured first task description information; and call the large model to generate the first task description information based on the labeled trajectory information.

[0204] In one implementation of this application, the first acquisition module 810 is configured to: acquire first task description information associated with the first interactive task; in response to an interactive operation adapted to the first task description information, record the interface elements operated by the interactive operation and the execution action to obtain labeled trajectory information and first visual data associated with the interactive operation; and generate first training data based on the first visual data and the first task description information.

[0205] In one implementation of this application, the training device 800 for the intelligent agent further includes: a second acquisition module for acquiring second training data; wherein the second training data includes second task description information associated with the second interactive task and second visual data associated with the second user interface at the first time step; a second execution module for invoking the second intelligent agent to execute the second interactive task according to the second task description information and the second visual data at at least one time step to obtain an image sequence; wherein the second visual data at each time step is associated with the second user interface at the corresponding time step, and the image sequence includes visual images associated with the user interface updated by the second intelligent agent executing the execution action at at least one time step; a determination module for determining a target reward value according to the image sequence; wherein the target reward value is used to indicate the execution quality of the second intelligent agent on the second interactive task; and a second training module for performing reinforcement learning on the second intelligent agent based on the target reward value to obtain a trained third intelligent agent.

[0206] In one implementation of this application, the determining module is configured to: in response to the task type to which the second interactive task belongs being the first task type, obtain a set interface element associated with the first task type; and determine the target reward value for the second agent to perform the second interactive task based on the difference between the interface element information in the last visual image in the image sequence and the set interface element.

[0207] In one implementation of this application, the determining module is configured to: extract keyframes from the image sequence in response to the second interactive task being a second task type; and call a large model to predict the target reward value of the second agent performing the second interactive task based on the keyframes and / or the interface element information in the keyframes.

[0208] In one implementation of this application, the training device 800 for the intelligent agent further includes: a third acquisition module for acquiring third training data; wherein the third training data includes third task description information associated with the third interactive task and third visual data associated with the third user interface at the first time step; a third execution module for invoking the third intelligent agent to execute the third interactive task according to the third task description information and the third visual data at at least one time step, and obtaining a first execution result; wherein the third visual data at each time step is associated with the third user interface at the corresponding time step; a fourth execution module for invoking a set intelligent agent to execute the third interactive task according to the third task description information and the third visual data at at least one time step, and obtaining a second execution result; wherein the number of parameters of the third intelligent agent is higher than the number of parameters of the set intelligent agent; and a third training module for performing knowledge distillation on the set intelligent agent according to the difference between the first execution result and the second execution result, so as to obtain a trained target intelligent agent.

[0209] In one implementation of this application, the training device 800 for the agent further includes: a fourth training module, used to pre-train the initial agent using fourth training data associated with at least one pre-training mechanism to obtain a first agent.

[0210] In one implementation of this application, at least one pre-training mechanism includes a first pre-training mechanism, and the fourth training data includes: a first sample image displaying a sample user interface, and fourth task description information associated with the first sample image; wherein the fourth task description information is used to indicate the target interface element to be operated in the sample user interface; and a fourth training module is used to: use an initial agent to predict the predicted position of the target interface element in the first sample image based on the fourth task description information; and pre-train the initial agent based on the difference between the predicted position and the actual position of the target interface element to obtain a first agent; wherein the actual position is determined based on the page source file of the sample user interface.

[0211] In one implementation of this application, at least one pre-training mechanism includes a second pre-training mechanism, and the fourth training data includes image-text data pairs; the fourth training module is used to: encode sample text in the image-text data pairs using an initial agent to obtain text features; encode a second sample image in the image-text data pairs using an initial agent to obtain image features; and perform semantic alignment on the initial agent based on the semantic similarity between the text features and the image features to obtain a first agent.

[0212] It should be noted that the foregoing explanation of the training method embodiment for any intelligent agent also applies to the training device for the intelligent agent in that embodiment, and will not be repeated here.

[0213] In the training device for the intelligent agent in this application embodiment, the intelligent agent's task completion ability can be improved by using a first training sample to conduct supervised training.

[0214] To implement the above embodiments, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the interaction method or the training method of the intelligent agent as described in any of the foregoing embodiments.

[0215] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example, the electronic device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0216] Reference Figure 9The electronic device 900 may include one or more of the following components: processing component 902, memory 904, power component 906, multimedia component 908, audio component 910, input / output (I / O) interface 912, sensor component 914, and communication component 916.

[0217] Processing component 902 typically controls the overall operation of electronic device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0218] Memory 904 is configured to store various types of data to support the operation of electronic device 900. Examples of this data include instructions for any application or method operating on electronic device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0219] Power component 906 provides power to various components of electronic device 900. Power component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 900.

[0220] Multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0221] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when electronic device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0222] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0223] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of electronic device 900. For example, sensor assembly 914 can detect the on / off state of electronic device 900, the relative positioning of components such as the display and keypad of electronic device 900, changes in position of electronic device 900 or a component of electronic device 900, the presence or absence of user contact with electronic device 900, orientation or acceleration / deceleration of electronic device 900, and temperature changes of electronic device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0224] Communication component 916 is configured to facilitate wired or wireless communication between electronic device 900 and other devices. Electronic device 900 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), Bluetooth, and other technologies.

[0225] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0226] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of an electronic device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0227] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the interaction method or the training method of the intelligent agent as described in any of the foregoing method embodiments.

[0228] To implement the above embodiments, this application also proposes a computer program product on which a computer program is stored, which, when executed by a processor, implements the interaction method or the training method of the intelligent agent as described in any of the foregoing method embodiments.

[0229] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0230] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0231] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0232] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0233] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0234] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0235] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0236] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An interaction method, characterized in that, include: Retrieve task description information associated with the interactive task to be executed; The intelligent agent determines and executes the execution information for the corresponding time step based on the task description information and visual data at at least one time step, in order to complete the interactive task; wherein, the visual data for each time step is associated with the target user interface for the corresponding time step. The execution information is used to indicate the interface element to be operated in the target user interface at the corresponding time step, and the execution action corresponding to the interface element to be operated.

2. The method according to claim 1, characterized in that, The execution information includes the execution action and execution parameters at the corresponding time step, as well as the subtask description information at the corresponding time step in the execution path associated with the interactive task; The execution parameters are used to indicate the interface elements to be operated in the target user interface at the corresponding time step.

3. The method according to claim 2, characterized in that, The execution parameters include at least one of the following: the identifier information of the interface element to be operated at the corresponding time step, the page position information, and the text description information.

4. The method according to claim 1, characterized in that, The acquisition of task description information associated with the interactive task to be executed includes any one of the following: In response to a trigger operation on a target interface element in a first user interface associated with the agent, task description information associated with the target interface element is obtained; In response to a selection operation of target description information from at least one natural language description information recommended in the first user interface, the target description information is used as the task description information; In response to an input operation to the interactive control of the first user interface, the input task description information is obtained.

5. The method according to claim 1, characterized in that, The calling agent determines and executes the execution information for the corresponding time step based on the task description information and visual data at at least one time step, in order to complete the interactive task, including: For the first time step in the at least one time step, the first user interface associated with the agent is taken as the target user interface under the first time step, and the agent is invoked to determine the execution information under the first time step based on the task description information and the visual data associated with the target user interface under the first time step; In response to the fact that the target user interface under the first time step contains the interface element to be operated indicated by the execution information under the first time step, according to the execution information under the first time step, the corresponding execution action is performed on the interface element to be operated in the target user interface under the first time step to obtain and display the second user interface updated to the first time step. In response to the fulfillment of the task termination condition, the interactive task is determined to be completed, and the invocation of the intelligent agent is terminated.

6. The method according to claim 5, characterized in that, The calling agent determines and executes the execution information for the corresponding time step based on the task description information and visual data at at least one time step to complete the interactive task, and further includes: If the target user interface at the first time step does not contain the interface element to be operated as indicated by the execution information at the first time step, the user will be redirected to the desktop. Based on the desktop, update the target user interface at the first time step; The agent is invoked to redetermine the execution information for the first time step based on the task description information and the visual data associated with the target user interface at the updated first time step, and then executes the process to obtain and display the second user interface updated to the first time step.

7. The method according to claim 5 or 6, characterized in that, The calling agent determines and executes the execution information for the corresponding time step based on the task description information and visual data at at least one time step to complete the interactive task, and further includes: In response to the fact that the task termination condition is not met at the (i-1)th time step, the second user interface updated at the (i-1)th time step is taken as the target user interface at the ith time step; where i is a positive integer greater than 1. The agent continues to be invoked to determine the execution information at the i-th time step based on the task description information and the visual data associated with the target user interface at the i-th time step. Based on the execution information at the i-th time step, perform corresponding execution actions on the interface elements to be operated in the target user interface at the i-th time step to obtain and display the second user interface updated at the i-th time step; In response to the fulfillment of the task termination condition, the interactive task is determined to be completed, and the invocation of the intelligent agent is terminated.

8. The method according to claim 7, characterized in that, The task termination condition includes at least one of the following: The updated second user interface includes a first interface element associated with the interactive task; The updated second user interface includes a second interface element whose attribute state changes to a specified state; The total number of time steps executed exceeds the set threshold. The content similarity between the second user interface updated at two adjacent time steps is higher than the set similarity threshold.

9. A method for training an intelligent agent, characterized in that, include: Acquire first training data; wherein, the first training data includes first task description information associated with the first interactive task and first visual data associated with the first user interface at the first time step; The first intelligent agent is invoked to execute the first interactive task based on the first task description information and the first visual data at at least one time step, so as to obtain execution trajectory information; wherein, the first visual data at each time step is associated with the first user interface at the corresponding time step, and the execution trajectory information is used to indicate the execution information of the first intelligent agent at the at least one time step, and the execution information is used to indicate the interface element to be operated in the first user interface at the corresponding time step and the execution action corresponding to the interface element to be operated. Based on the labeled trajectory information associated with the first training data and the execution trajectory information, the first agent is subjected to supervised training to obtain a trained second agent.

10. The method according to claim 9, characterized in that, The execution information includes the execution action and execution parameters at the corresponding time step, as well as the subtask description information at the corresponding time step in the predicted execution path associated with the first interactive task; The execution parameters are used to indicate the interface elements to be operated at the corresponding time step.

11. The method according to claim 9, characterized in that, The first training data and the labeled trajectory information were obtained in the following manner: Obtain the annotation execution path associated with the first interactive task; Execute the description information of each subtask in the annotation execution path, and record the interface elements and execution actions operated when executing each subtask description information, so as to obtain the annotation trajectory information and the first visual data associated with each subtask description information; Obtain first task description information associated with the labeled trajectory information, and generate the first training data based on first visual data associated with each subtask description information and the first task description information.

12. The method according to claim 11, characterized in that, The step of obtaining the first task description information associated with the labeled trajectory information includes any one of the following: In response to a configuration operation triggered by the labeled trajectory information, the configured first task description information is obtained; The large model is invoked to generate the first task description information based on the labeled trajectory information.

13. The method according to claim 9, characterized in that, The first training data and the labeled trajectory information were obtained in the following manner: Obtain the description information of the first task associated with the first interactive task; In response to an interactive operation adapted to the first task description information, the interface elements operated by the interactive operation and the actions performed are recorded to obtain the labeled trajectory information and the first visual data associated with the interactive operation. The first training data is generated based on the first visual data and the first task description information.

14. The method according to claim 9, characterized in that, The method further includes: Acquire second training data; wherein the second training data includes second task description information associated with the second interactive task and second visual data associated with the second user interface at the first time step; The second agent is invoked to execute the second interactive task based on the second task description information and the second visual data at at least one time step, so as to obtain an image sequence; wherein the second visual data at each time step is associated with the second user interface at the corresponding time step, and the image sequence includes a visual image associated with the user interface updated by the second agent executing the execution action at at least one time step; Based on the image sequence, a target reward value is determined; wherein the target reward value is used to indicate the performance quality of the second agent in performing the second interactive task; Based on the target reward value, reinforcement learning is performed on the second agent to obtain a trained third agent.

15. The method according to claim 14, characterized in that, Determining the target reward value based on the image sequence includes: In response to the fact that the task type to which the second interactive task belongs is the first task type, the setting interface element associated with the first task type is obtained; Based on the difference between the interface element information in the last visual image of the image sequence and the set interface element, the target reward value for the second agent to perform the second interactive task is determined.

16. The method according to claim 14, characterized in that, Determining the target reward value based on the image sequence includes: In response to the fact that the task type to which the second interactive task belongs is the second task type, keyframes are extracted from the image sequence; The large model is invoked to predict the target reward value for the second agent to perform the second interactive task based on the keyframe and / or the interface element information in the keyframe.

17. The method according to claim 15, characterized in that, The method further includes: Acquire third training data; wherein the third training data includes third task description information associated with the third interactive task and third visual data associated with the third user interface at the first time step; The third intelligent agent is invoked to execute the third interactive task based on the third task description information and the third visual data at at least one time step, and a first execution result is obtained; wherein, the third visual data at each time step is associated with the third user interface at the corresponding time step; The designated intelligent agent is invoked to execute the third interactive task based on the third task description information and the third visual data at at least one time step, and a second execution result is obtained; wherein, the number of parameters of the third intelligent agent is higher than the number of parameters of the designated intelligent agent; Based on the difference between the first execution result and the second execution result, knowledge distillation is performed on the set agent to obtain the trained target agent.

18. The method according to any one of claims 9-17, characterized in that, The first intelligent agent is obtained through the following steps: The first agent is obtained by pre-training an initial agent using fourth training data associated with at least one pre-training mechanism.

19. The method according to claim 18, characterized in that, The at least one pre-training mechanism includes a first pre-training mechanism, and the fourth training data includes: a first sample image displaying a sample user interface, and fourth task description information associated with the first sample image; wherein, the fourth task description information is used to indicate the target interface element to be operated in the sample user interface. The step of pre-training the initial agent using fourth training data associated with at least one pre-training mechanism to obtain the first agent includes: The initial agent predicts the predicted position of the target interface element in the first sample image based on the fourth task description information. Based on the difference between the predicted and actual positions of the target interface elements, the initial agent is pre-trained to obtain the first agent. The actual location is determined based on the page source file of the sample user interface.

20. The method according to claim 18, characterized in that, The at least one pre-training mechanism includes a second pre-training mechanism, and the fourth training data includes image-text data pairs; The step of pre-training the initial agent using fourth training data associated with at least one pre-training mechanism to obtain the first agent includes: An initial intelligent agent is used to encode the sample text in the image-text data pair to obtain text features; The initial intelligent agent is used to encode the second sample image in the image-text data pair to obtain image features; Based on the semantic similarity between the text features and the image features, the initial agent is semantically aligned to obtain the first agent.

21. An interactive device, characterized in that, The apparatus for implementing the interaction method as described in any one of claims 1-8 includes: The acquisition module is used to acquire task description information associated with the interactive task to be executed; The processing module is used to call the intelligent agent to determine and execute the execution information at the corresponding time step based on the task description information and visual data at at least one time step, so as to complete the interactive task; wherein, the visual data at each time step is associated with the target user interface at the corresponding time step. The execution information is used to indicate the interface element to be operated in the target user interface at the corresponding time step, and the execution action corresponding to the interface element to be operated.

22. A training device for an intelligent agent, characterized in that, For implementing the method of training an agent as described in any one of claims 9-20, the apparatus comprises: The first acquisition module is used to acquire first training data; wherein, the first training data includes first task description information associated with the first interactive task and first visual data associated with the first user interface at the first time step; The first execution module is used to call the first intelligent agent to execute the first interactive task according to the first task description information and the first visual data at at least one time step, so as to obtain execution trajectory information; wherein, the first visual data at each time step is associated with the first user interface at the corresponding time step, and the execution trajectory information is used to indicate the execution information of the first intelligent agent at at least one time step, and the execution information is used to indicate the interface element to be operated in the first user interface at the corresponding time step and the execution action corresponding to the interface element to be operated. The first training module is used to perform supervised training on the first agent based on the labeled trajectory information associated with the first training data and the execution trajectory information, so as to obtain a trained second agent.

23. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the method as claimed in any one of claims 1 to 8, or implements the steps of the method as claimed in any one of claims 9 to 20.

24. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method according to any one of claims 1 to 8, or implement the steps of the method according to any one of claims 9 to 20.

25. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8, or implements the steps of the method according to any one of claims 9 to 20.

Images