Training method and task processing method and device of intelligent agent

Abstract

The present disclosure relates to a method for training an agent, a task processing method and device. The method for training an agent can include: obtaining task input information of a plurality of sample tasks corresponding to a current training round; the task input information of each sample task includes a first multimedia resource; obtaining first reward data corresponding to task trajectories of the plurality of sample tasks based on the agent processing the first multimedia resource corresponding to each of the plurality of sample tasks; the first reward data corresponding to each task trajectory represents the mastery degree of the agent to the corresponding sample task in the current training round; determining a target task from the plurality of sample tasks based on the first reward data corresponding to the task trajectories of the plurality of sample tasks; determining first loss information corresponding to the current training round based on the task trajectory of the target task; and adjusting parameters of the agent based on the first loss information. The present disclosure improves the training efficiency and stability.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for training an intelligent agent, a method for processing tasks, and an apparatus. Background Technology

[0002] With the rapid development of large language model technology, intelligent agents capable of understanding and manipulating devices have become a research hotspot, aiming to achieve complex task processing capabilities similar to those of human users. This technological field is evolving from relying on large-scale manually labeled data to autonomous learning. Early training paradigms mainly relied on supervised fine-tuning (SFT), which involves collecting a large amount of operational trajectory data from human experts to train the model. However, this approach is not only costly in terms of data acquisition, but also results in agents with poor generalization ability when facing the dynamically changing mobile application ecosystem, making it difficult to adapt to unfamiliar applications. To overcome these limitations, research has gradually shifted towards annotation-free and reinforcement learning (RL) methods, aiming to enable agents to generate data and optimize policies through self-exploration and interaction with the environment.

[0003] In related technologies, reinforcement learning-based agent training methods train agents using static training samples, which leads to low training efficiency. Summary of the Invention

[0004] This disclosure provides a method for training an intelligent agent, a task processing method, and an apparatus to at least solve the problem in related technologies where reinforcement learning-based intelligent agent training methods rely on static training samples, resulting in low training efficiency. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a method for training an intelligent agent is provided, comprising: Obtain the task input information of multiple sample tasks corresponding to the current training round; the task input information of each sample task includes the first multimedia resource; Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the first reward data corresponding to the task trajectory of the multiple sample tasks is obtained; the first reward data corresponding to each task trajectory represents the degree of mastery of the intelligent agent on the corresponding sample task in the current training round. Based on the first reward data corresponding to the task trajectories of the multiple sample tasks, a target task is determined from the multiple sample tasks; the target task is a task other than the first task and the second task among the multiple sample tasks, the agent's mastery of the first task is greater than the agent's mastery of the target task, and the agent's mastery of the second task is less than the agent's mastery of the target task. The first loss information corresponding to the current training round is determined based on the task trajectory of the target task. The parameters of the agent are adjusted based on the first loss information.

[0005] In one exemplary embodiment, determining the target task from the plurality of sample tasks based on the first reward data corresponding to the task trajectories of the plurality of sample tasks includes: Obtain the target reward data range; Based on the data matching between the first reward data corresponding to the task trajectory of the multiple sample tasks and the target reward data interval, the sample task corresponding to the first reward data and the target reward data interval is determined as the target task.

[0006] In an exemplary embodiment, the first reward data corresponding to each of the plurality of sample tasks includes the first reward data corresponding to the plurality of task trajectories of each task; The step of matching the first reward data corresponding to the task trajectory of the multiple sample tasks with the target reward data interval, and determining the sample task corresponding to the first reward data and the target reward data interval as the target task, includes: Data fusion is performed based on the first reward data corresponding to multiple task trajectories of each task to obtain the fused reward data corresponding to each task; The fusion reward data corresponding to each task is matched with the target reward data range, and the task whose fusion reward data falls within the target reward data range is determined as the target task.

[0007] In one exemplary embodiment, the task trajectory of each sample task includes at least one action; The process of processing the first multimedia resources corresponding to each of the multiple sample tasks based on the intelligent agent to obtain the first reward data corresponding to the task trajectories of the multiple sample tasks includes: Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the task trajectories of the multiple sample tasks are obtained. Based on a multimodal large language model, trajectory analysis is performed on the task trajectories of the multiple sample tasks to obtain first reward data corresponding to the task trajectories of the multiple sample tasks, and second reward data corresponding to each action in the task trajectory of each sample task; the second reward data corresponding to each action represents the correctness of the agent in performing the corresponding action. The step of determining the first loss information corresponding to the current training round based on the task trajectory of the target task includes: Based on the second reward data corresponding to each action in the task trajectory of the target task, a set of positive samples in the task trajectory of the target task is determined; the set of positive samples includes a first state and the successful action corresponding to the first state; Collect multiple candidate actions output by the agent in response to the first state; The first loss information is determined based on the matching information of the multiple candidate actions and the successful actions.

[0008] In one exemplary embodiment, the task trajectory includes screenshots of the user interface and operation logs; The method involves performing trajectory analysis on the task trajectories of the multiple sample tasks based on a multimodal large language model to obtain first reward data corresponding to the task trajectories of the multiple sample tasks, and second reward data corresponding to each action in the task trajectory of each sample task, including: Information processing is performed on the screenshots of the operation interface and the operation log based on a lightweight visual language model to obtain semantic description information corresponding to the task trajectory; The semantic description information, the multiple sample tasks, and the screenshot of the operation interface are input into the multimodal large language model for trajectory analysis to obtain the first reward data corresponding to the task trajectory of the multiple sample tasks, and the second reward data corresponding to each action in the task trajectory of each sample task.

[0009] In an exemplary embodiment, determining the first loss information based on the matching information of the plurality of candidate actions and the successful action includes: Based on the matching information between the multiple candidate actions and the successful action, the third reward data corresponding to each of the multiple candidate actions is determined; The advantage score for each of the multiple candidate actions is determined based on the third reward data corresponding to each of the multiple candidate actions. The first loss information is determined based on the advantage scores corresponding to each of the multiple candidate actions.

[0010] In an exemplary embodiment, determining the third reward data corresponding to each of the plurality of candidate actions based on the matching information between the plurality of candidate actions and the successful action includes: Determine the first matching information between the action type of each candidate action and the action type of the successful action; Determine the second matching information between the action parameters of each candidate action and the action parameters of the successful action; The first matching information and the second matching information are fused to obtain the third reward data for each candidate action.

[0011] In one exemplary embodiment, the method further includes: Based on the second reward data corresponding to each action in the task trajectory of the target task, a negative sample set in the task trajectory of the target task is determined; the negative sample set includes a second state and the failed action corresponding to the second state; Determine the first probability distribution information of the failed action of the agent corresponding to the previous training round of the current training round, given the second state; Determine the second probability distribution information of the agent's output action given the second state in the current training round; A second loss information is generated based on the first probability distribution and the second probability distribution; the second loss information aims to maximize the difference between the first probability distribution and the second probability distribution. The step of adjusting the parameters of the agent based on the first loss information includes: The parameters of the agent are adjusted based on the first loss information and the second loss information.

[0012] In an exemplary embodiment, after adjusting the parameters of the agent based on the first loss information, the method further includes: The sample task for the next training round is determined based on the second task and the new task that the agent has not yet processed.

[0013] In an exemplary embodiment, before obtaining the task input information of multiple sample tasks corresponding to the current training round, the method further includes: Obtain prior information about the application structure of the target application; Based on the interaction exploration performed by the agent according to the prior information of the application structure, at least one interaction trajectory between the agent and the target application is obtained; The at least one interaction trajectory is processed based on a multimodal large language model to obtain multiple sample tasks for training the agent.

[0014] In one exemplary embodiment, the multiple sample tasks used to train the agent each have their own corresponding task difficulty level; Before obtaining the task input information of multiple sample tasks corresponding to the current training round, the method further includes: Determine the current difficulty level corresponding to the current training round; Select sample tasks with the current difficulty level from the multiple samples used to train the agent, and use them as the multiple sample tasks corresponding to the current training round.

[0015] According to a second aspect of the present disclosure, a task processing method is provided, comprising: Obtain the task input information of the task to be processed; the task input information of the task to be processed includes a second multimedia resource; The second multimedia resource is input into the intelligent agent for task processing to obtain the task processing result corresponding to the task to be processed. The agent is trained using the training method described above.

[0016] According to a third aspect of the present disclosure, a training apparatus for an intelligent agent is provided, comprising: The sample task acquisition unit is configured to acquire task input information for multiple sample tasks corresponding to the current training round; the task input information for each sample task includes a first multimedia resource. The first task processing unit is configured to perform processing on the first multimedia resources corresponding to each of the plurality of sample tasks based on the intelligent agent, and obtain the first reward data corresponding to the task trajectory of the plurality of sample tasks; the first reward data corresponding to each task trajectory represents the degree of mastery of the intelligent agent on the corresponding sample task in the current training round. The target task determination unit is configured to execute first reward data corresponding to the task trajectories of the plurality of sample tasks to determine the target task from the plurality of sample tasks; the target task is a task other than the first task and the second task among the plurality of sample tasks, the agent's mastery of the first task is greater than the agent's mastery of the target task, and the agent's mastery of the second task is less than the agent's mastery of the target task. The loss information determination unit is configured to perform task trajectory determination based on the target task to determine the first loss information corresponding to the current training round. The parameter adjustment unit is configured to adjust the parameters of the agent based on the first loss information.

[0017] In one exemplary embodiment, the target task determination unit is configured to perform: Obtain the target reward data range; Based on the data matching between the first reward data corresponding to the task trajectory of the multiple sample tasks and the target reward data interval, the sample task corresponding to the first reward data and the target reward data interval is determined as the target task.

[0018] In an exemplary embodiment, the first reward data corresponding to each of the plurality of sample tasks includes first reward data corresponding to multiple task trajectories of each task; the target task determination unit is configured to execute: Data fusion is performed based on the first reward data corresponding to multiple task trajectories of each task to obtain the fused reward data corresponding to each task; The fusion reward data corresponding to each task is matched with the target reward data range, and the task whose fusion reward data falls within the target reward data range is determined as the target task.

[0019] In one exemplary embodiment, the task trajectory of each sample task includes at least one action; The first task processing unit is configured to execute: Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the task trajectories of the multiple sample tasks are obtained. Based on a multimodal large language model, trajectory analysis is performed on the task trajectories of the multiple sample tasks to obtain first reward data corresponding to the task trajectories of the multiple sample tasks, and second reward data corresponding to each action in the task trajectory of each sample task; the second reward data corresponding to each action represents the correctness of the agent in performing the corresponding action. The loss information determination unit is configured to execute: Based on the second reward data corresponding to each action in the task trajectory of the target task, a set of positive samples in the task trajectory of the target task is determined; the set of positive samples includes a first state and the successful action corresponding to the first state; Collect multiple candidate actions output by the agent in response to the first state; The first loss information is determined based on the matching information of the multiple candidate actions and the successful actions.

[0020] In one exemplary embodiment, the task trajectory includes screenshots of the user interface and operation logs; The first task processing unit is configured to execute: Information processing is performed on the screenshots of the operation interface and the operation log based on a lightweight visual language model to obtain semantic description information corresponding to the task trajectory; The semantic description information, the multiple sample tasks, and the screenshot of the operation interface are input into the multimodal large language model for trajectory analysis to obtain the first reward data corresponding to the task trajectory of the multiple sample tasks, and the second reward data corresponding to each action in the task trajectory of each sample task.

[0021] In an exemplary embodiment, the loss information determination unit includes: An action matching unit is configured to perform matching information based on the plurality of candidate actions and the successful action to determine the third reward data corresponding to each of the plurality of candidate actions; The advantage score determination unit is configured to determine the advantage score corresponding to each of the multiple candidate actions based on the third reward data corresponding to each of the multiple candidate actions. The first loss determination unit is configured to determine the first loss information based on the advantage scores corresponding to each of the plurality of candidate actions.

[0022] In one exemplary embodiment, the action matching unit is configured to perform: Determine the first matching information between the action type of each candidate action and the action type of the successful action; Determine the second matching information between the action parameters of each candidate action and the action parameters of the successful action; The first matching information and the second matching information are fused to obtain the third reward data for each candidate action.

[0023] In one exemplary embodiment, the apparatus further includes a second loss determination unit configured to perform: Based on the second reward data corresponding to each action in the task trajectory of the target task, a negative sample set in the task trajectory of the target task is determined; the negative sample set includes a second state and the failed action corresponding to the second state; Determine the first probability distribution information of the failed action of the agent corresponding to the previous training round of the current training round, given the second state; Determine the second probability distribution information of the agent's output action given the second state in the current training round; A second loss information is generated based on the first probability distribution and the second probability distribution; the second loss information aims to maximize the difference between the first probability distribution and the second probability distribution. The parameter adjustment unit is configured to perform: The parameters of the agent are adjusted based on the first loss information and the second loss information.

[0024] In one exemplary embodiment, the apparatus further includes a first sample task determination unit configured to perform: The sample task for the next training round is determined based on the second task and the new task that the agent has not yet processed.

[0025] In one exemplary embodiment, the apparatus further includes a sample task creation unit configured to perform: Obtain prior information about the application structure of the target application; Based on the interaction exploration performed by the agent according to the prior information of the application structure, at least one interaction trajectory between the agent and the target application is obtained; The at least one interaction trajectory is processed based on a multimodal large language model to obtain multiple sample tasks for training the agent.

[0026] In one exemplary embodiment, the multiple sample tasks used to train the agent each have their own corresponding task difficulty level; The device further includes a second sample task determination unit, configured to perform: Determine the current difficulty level corresponding to the current training round; Select sample tasks with the current difficulty level from the multiple samples used to train the agent, and use them as the multiple sample tasks corresponding to the current training round.

[0027] According to a fourth aspect of the present disclosure, a task processing apparatus is provided, comprising: The task acquisition unit is configured to acquire task input information of the task to be processed; the task input information of the task to be processed includes a second multimedia resource. The second task processing unit is configured to input the second multimedia resource into the intelligent agent for task processing and obtain the task processing result corresponding to the task to be processed. The agent is trained using the training method described above.

[0028] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the agent training method or task processing method as described above.

[0029] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the agent training method or task processing method as described above.

[0030] According to a seventh aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the device to perform the above-described intelligent agent training method or task processing method.

[0031] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: In the training process of the agent, this disclosure obtains task trajectories for multiple sample tasks by processing first multimedia resources corresponding to multiple sample tasks in the current training round, determines first reward data corresponding to each task trajectory, and then determines the target task from multiple sample tasks based on the first reward data. Since the first reward data corresponding to each task trajectory can characterize the agent's mastery of the corresponding sample task in the current training round, the target task determined from multiple sample tasks based on the first reward data is the task that meets the mastery level condition. The agent's mastery of the target task can be between the first task and the second task. The first task can be a simple task that the agent has already mastered in the current training round, and the second task can be a task that the agent finds difficult to solve in the current training round. This invention addresses the challenge of difficult tasks by selecting a target task with a suitable level of mastery to determine the first loss information for the current training round. Based on this first loss information, the agent's parameters are adjusted. This breaks away from related technologies that use static training samples to adjust the agent's parameters, resulting in the agent wasting significant computational resources on simple tasks it has already mastered, or repeatedly failing on difficult tasks that it cannot currently solve without learning anything. Instead, this invention dynamically selects sample tasks for adjusting the model's parameters based on the agent's actual performance on multiple sample tasks in the current training round. This allows the agent to learn tasks within its current capability boundary, which are neither fully mastered known tasks nor unattainable tasks far beyond its capabilities, thus avoiding ineffective training and improving training efficiency.

[0032] Furthermore, this disclosure dynamically selects the target task for determining the first loss information from multiple sample tasks, uses a portion of the sample tasks to determine the first loss information, and updates the agent parameters based on the first loss information. The corresponding sample parameters are moderate, which can avoid drastic fluctuations in parameter updates, thereby improving training stability.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0035] Figure 1 This is a schematic diagram of an implementation environment according to an exemplary embodiment.

[0036] Figure 2 This is a flowchart illustrating an agent training method according to an exemplary embodiment.

[0037] Figure 3 This is a flowchart illustrating a task processing method according to an exemplary embodiment.

[0038] Figure 4 This is a schematic diagram illustrating a fully automated GUI intelligent agent closed-loop self-evolution technology framework according to an exemplary embodiment.

[0039] Figure 5 This is a schematic diagram illustrating a targeted reinforcement learning update process for dynamically selected samples according to an exemplary embodiment.

[0040] Figure 6 This is a block diagram of a training device for an intelligent agent according to an exemplary embodiment.

[0041] Figure 7 This is a block diagram of a task processing apparatus according to an exemplary embodiment.

[0042] Figure 8 This is a schematic diagram of an electronic device structure according to an exemplary embodiment.

[0043] Figure 9 This is a schematic diagram of another electronic device structure according to an exemplary embodiment. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0045] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0046] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0047] Please see Figure 1 The illustration shows an implementation environment provided by an embodiment of the present disclosure. The implementation environment may include at least one terminal 110 and a server 120, which can communicate with each other via a network.

[0048] Optionally, terminal 110 may include: smartphones, tablets, laptops, digital assistants, smart wearable devices, in-vehicle terminals, servers, etc. Terminal 110 has an interactive application installed and running. In this embodiment, the interactive application may be an application that provides task processing functions, such as a social application, instant messaging application, live streaming application, game application, e-commerce application, virtual reality (VR) application, augmented reality (AR) application, etc. This application embodiment does not limit the specific type of application.

[0049] Optionally, server 120 provides background services for the interactive application installed on terminal 110. Terminal 110 can communicate with server 120 based on browser / server (B / S) mode or client / server (C / S) mode. The operating system running on terminal 110 in this embodiment may include, but is not limited to, Android, iOS, Linux, Windows, etc.

[0050] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0051] To address the issue of low training efficiency in reinforcement learning-based agent training methods that rely on static training samples, this disclosure provides an agent training method. The executing entity can be the aforementioned terminal, server, or a system based on a terminal and server. Optionally, please refer to... Figure 2 The method may include: S210. Obtain the task input information of multiple sample tasks corresponding to the current training round; the task input information of each sample task includes the first multimedia resource.

[0052] In this embodiment, the task input information for each sample task may include at least one form of first multimedia resource such as text, audio, image, or video; the first multimedia resource corresponding to each sample task is a resource that carries at least one of the following information: task description information, task requirement information, and input basis required for task execution. Optionally, the task description information, task requirement information, and input basis required for task execution may all be presented in at least one form such as text, audio, image, or video.

[0053] In one optional embodiment, in an intelligent customer service scenario, the task input information for a corresponding customer service task may include at least one form of question information such as text, audio, image, or video. In a code generation scenario, the task input information for a corresponding code generation task may include at least one form of code requirements or code function descriptions such as text, audio, image, or video. In a mathematical calculation scenario, the task input information for a corresponding mathematical calculation task may include at least one form of mathematical problem such as text, audio, image, or video.

[0054] S220. Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the first reward data corresponding to the task trajectory of the multiple sample tasks is obtained; the first reward data corresponding to each task trajectory represents the degree of mastery of the intelligent agent on the corresponding sample task in the current training round.

[0055] An intelligent agent is a closed-loop artificial intelligence system that can perceive its environment through sensors, autonomously plan its actions through decision-making mechanisms, and take actions through actuators to achieve a preset goal. Unlike the passive response of traditional artificial intelligence, intelligent agents possess autonomy, adaptability, and long-term optimization capabilities. An intelligent agent can include elements such as perception, decision-making, action, and optimization. Perception involves acquiring environmental information through multimodal inputs such as vision, text, APIs (Application Programming Interfaces), and GUIs (Graphical User Interfaces). Decision-making relies on model reasoning, task decomposition, memory, and tool calls to generate action plans. Actions are executed through API calls, GUI operations, and mechanical control, and feedback is obtained. Optimization involves iteratively strategizing based on feedback to improve long-term task performance. Intelligent agents can use LLM (Large Language Model) / VLM (Vision Language Model) as a base model, leveraging the reasoning, planning, and natural language understanding capabilities provided by LLM / VLM.

[0056] The intelligent agent in this embodiment can be a software intelligent agent, such as an intelligent customer service representative, a personal assistant (automatic schedule management, batch photo album cleanup, intelligent message reply), a recommendation system, automated office (mobile approval processes, automatic report generation and sharing), a GUI agent, code generation, or edge automation (phone factory testing, APP compatibility verification, user behavior simulation), or a physical intelligent agent, such as a robot or an IoT device. Taking a GUI intelligent agent as an example, a GUI intelligent agent is an autonomous task system with a multimodal large model at its core, capable of simulating human gestures to operate mobile application interfaces, and realizing a closed-loop process of "perception → planning → execution → feedback → error correction"; it replaces / assists humans in completing complex tasks in applications (such as ordering takeout, hailing a ride, managing photo albums, etc.) directly through visual understanding and gesture operation of the interface.

[0057] Multiple sample tasks corresponding to the current training round can serve as training samples for the agent in the current training round. The task trajectory of a sample task can be a temporal sequence formed based on the environmental state, agent actions, action feedback, and task progress during the entire process from task initiation to task termination, i.e., when the agent processes the first multimedia resource corresponding to the sample task. In reinforcement learning, the task trajectory can include state and action, along with key information such as reward and next state, completely recording a continuous process of agent-environment interaction. For example, a complete task trajectory can be:

[0058] in, It is the environmental state at time t, which is the environmental information perceived by the agent; It is the action performed by agent t at time t; It is the reward obtained after performing the action at time t, which is a quantitative feedback on the task objective; It is to perform an action The state that is then transitioned to.

[0059] In this embodiment, the first reward data corresponding to each task trajectory can be the reward data obtained by the agent in performing the corresponding sample task based on the task trajectory. The first reward data corresponding to each task trajectory represents the degree of mastery of the agent in the corresponding sample task in the current training round. The first reward data is positively correlated with the degree of mastery of the agent in the corresponding sample task. That is, the higher the first reward data, the higher the degree of mastery of the agent in the corresponding sample task in the current training round. Conversely, the lower the first reward data, the lower the degree of mastery of the agent in the corresponding sample task in the current training round.

[0060] S230. Based on the first reward data corresponding to the task trajectories of the plurality of sample tasks, determine the target task from the plurality of sample tasks; the target task is a task other than the first task and the second task among the plurality of sample tasks, the agent's mastery of the first task is greater than the agent's mastery of the target task, and the agent's mastery of the second task is less than the agent's mastery of the target task.

[0061] Based on the first reward data corresponding to the task trajectories of multiple sample tasks, these tasks can be dynamically divided into a first task, a target task, and a second task. The agent's mastery of the first task is greater than its mastery of the target task; that is, the first reward data corresponding to the task trajectory of the first task is greater than that corresponding to the task trajectory of the target task. The first task can be a simple task that the agent has already mastered in the current training round. The agent's mastery of the second task is less than its mastery of the target task; that is, the first reward data corresponding to the task trajectory of the second task is less than that corresponding to the task trajectory of the target task. The second task can be a difficult task that the agent cannot solve in the current training round. The target task can be a task within the optimal learning zone, meaning a task of moderate difficulty for the agent in the current training round. There can be one or more first tasks and one or more second tasks. The number of first and second tasks can be determined based on the first reward data corresponding to each sample task and the task selection criteria.

[0062] In an optional embodiment, the task selection criteria can be conditions associated with a preset reward data range, thereby classifying multiple sample tasks through the preset reward data range. Based on the first reward data corresponding to the task trajectory of each sample task, a suitable reward data range is determined, thus determining the type of each sample task. Accordingly, determining the target task from the multiple sample tasks based on the first reward data corresponding to the task trajectories of the multiple sample tasks includes: Obtain the target reward data range; Based on the data matching between the first reward data corresponding to the task trajectory of the multiple sample tasks and the target reward data interval, the sample task corresponding to the first reward data and the target reward data interval is determined as the target task.

[0063] The preset reward data range may include a first reward data range corresponding to the first task, a target reward data range corresponding to the target task, and a second reward data range corresponding to the second task; wherein the lower limit of the first reward data range is greater than the upper limit of the target reward data range, and the upper limit of the second reward data range is less than the lower limit of the target reward data range. In one example, the first reward data range is... The target reward data range is The second reward data range is .

[0064] In one optional embodiment, the agent can execute multiple sample tasks once, and obtain the first reward data corresponding to a task trajectory of each sample task. By matching the first reward data corresponding to the task trajectories of multiple sample tasks with the first reward data interval, the target reward data interval, and the second reward data interval, the reward data interval to which the first reward data corresponding to the task trajectory of each sample task belongs can be determined, thereby determining the task category of each sample task.

[0065] In this embodiment, by pre-setting a reward data interval for dynamically dividing sample tasks, and then obtaining the first reward data corresponding to the task trajectories of multiple sample tasks, the sample task whose first reward data falls within the target reward data interval can be identified as the target task, thereby improving the efficiency of target task identification. Furthermore, since the target reward data interval is located between the first reward data interval and the second reward data interval, the corresponding target reward data interval can correspond to a task with a moderate level of mastery for the agent. Thus, the target task that matches the target reward data interval can be a task of moderate difficulty for the agent. A task of moderate difficulty allows the agent to learn a task that is "within reach with a little effort," thereby avoiding ineffective training and improving training efficiency.

[0066] In another optional embodiment, the agent can execute multiple sample tasks multiple times, thereby obtaining first reward data corresponding to multiple task trajectories for each sample task. Then, based on the first reward data corresponding to each of the multiple task trajectories, the fused reward data for each sample task can be determined. Thus, by matching the fused reward data corresponding to the task trajectories of multiple sample tasks with a first reward data interval, a target reward data interval, and a second reward data interval, the reward data interval to which the fused reward data corresponding to the task trajectory of each sample task belongs can be determined, thereby determining the task category of each sample task. Accordingly, the first reward data for each task in the multiple sample tasks includes the first reward data corresponding to the multiple task trajectories of each task. The step of matching the first reward data corresponding to the task trajectory of the multiple sample tasks with the target reward data interval, and determining the sample task corresponding to the first reward data and the target reward data interval as the target task, includes: Data fusion is performed based on the first reward data corresponding to multiple task trajectories of each task to obtain the fused reward data corresponding to each task; The fusion reward data corresponding to each task is matched with the target reward data range, and the task whose fusion reward data falls within the target reward data range is determined as the target task.

[0067] In this embodiment, the agent can attempt multiple sample tasks corresponding to the current training round M times, resulting in M ​​task trajectories. The trajectory set formed by the M task trajectories can be... , mission trajectory The corresponding first reward data is ..., mission trajectory The corresponding first reward data is The corresponding fusion reward data for each task can be calculated using the following formula: (1) The fused reward data for each task can be obtained by averaging the first reward data corresponding to the M task trajectories for each task. In each task's M task trajectories, the first reward data for each trajectory is either 0 or 1, corresponding to the fused reward data for that task. For example, if M=5, the agent makes 5 attempts to complete task 1. The first reward data corresponding to the 5 task trajectories are 0, 0, 1, 1, 1. The fusion reward data corresponding to the task trajectory of task 1 is (0+0+1+1+1) / 5=0.6.

[0068] In this embodiment, by making multiple attempts on each of the multiple sample tasks, first reward data corresponding to multiple task trajectories for each task is obtained. Then, data fusion processing is performed based on the first reward data corresponding to multiple task trajectories for each task to obtain fused reward data for each task. Thus, by comprehensively considering the results of multiple attempts on the same task, the fused reward data corresponding to the task can be determined, avoiding the randomness of determining the reward data by only trying the task once, and improving the stability and accuracy of the reward data determination.

[0069] In another optional embodiment, the task selection criteria can also be based on the sorting results of the first reward data. When the first reward data corresponding to the task trajectories of multiple sample tasks is obtained, the multiple sample tasks can also be sorted based on the first reward data. For example, the multiple sample tasks can be sorted in descending order of the first reward data. Accordingly, the top m consecutive sample tasks can be determined as the first task, the next n consecutive tasks after the m sample tasks can be determined as the target task, and the remaining tasks can be determined as the second task. The specific values ​​of m and n can be determined based on the actual implementation.

[0070] S240. Determine the first loss information corresponding to the current training round based on the task trajectory of the target task.

[0071] The target task can be a task within the optimal learning zone, that is, a task of moderate difficulty for the agent in the current training round. Accordingly, the first loss information corresponding to the current training round can be determined based on the task trajectory of the target task.

[0072] As can be seen from the above, the task trajectory can include information such as state, action, reward, and next state. Accordingly, in the process of determining the first loss information corresponding to the current training round based on the task trajectory of the target task, the loss can be minimized by gradient descent based on the state, action, and reward data in the task trajectory of the target task, combined with the target loss function, so that the agent can update in the direction of "generating the optimal trajectory".

[0073] S250. Adjust the parameters of the agent based on the first loss information.

[0074] The initial loss information can be used as a basis for adjusting the parameters of the agent in the current training round. The parameters of the agent can be adjusted based on this initial loss information to obtain the trained agent for the current training round. It should be noted that as the number of training rounds increases, the agent's task processing capability also continuously improves, and the difficulty of the tasks it can handle also continuously increases.

[0075] In the training process of the agent, this disclosure determines the first reward data corresponding to each task trajectory based on the task trajectory obtained by the agent processing multiple sample tasks in the current training round. Then, based on the first reward data, a target task can be determined from multiple sample tasks. Since the first reward data corresponding to each task trajectory can characterize the agent's mastery of the corresponding sample task in the current training round, the target task determined from multiple sample tasks based on the first reward data is the task that meets the mastery level condition. The agent's mastery of the target task can be between the first task and the second task. The first task can be a simple task that the agent has already mastered in the current training round, and the second task can be a task that the agent has mastered in the current training round. This invention addresses challenging tasks by selecting a target task with a suitable level of mastery to determine the first loss information for the current training round. Based on this first loss information, the agent's parameters are adjusted. This approach breaks away from related technologies that use static training samples to adjust agent parameters, leading to wasted computational resources on simple tasks already mastered or repeated failures on difficult tasks beyond the agent's current capabilities, resulting in no learning. Instead, this invention dynamically selects sample tasks for model parameter adjustment based on the agent's actual performance on multiple sample tasks in the current training round. This allows the agent to learn tasks that are "within reach," avoiding ineffective training and improving training efficiency. Furthermore, the dynamic selection of target tasks from multiple sample tasks to determine the first loss information, using a subset of sample tasks to determine the first loss information and updating agent parameters based on it, results in suitable sample parameters that avoid drastic fluctuations in parameter updates, thus improving training stability.

[0076] In another optional embodiment, the task trajectory of each sample task includes at least one action; trajectory analysis can be performed on the task trajectories of multiple sample tasks based on a multimodal large language model to obtain the first reward data corresponding to the task trajectories of multiple sample tasks, and the second reward data corresponding to each action in each task trajectory, that is, both trajectory-level reward data and step-level (action-level) reward data can be obtained simultaneously; correspondingly, the step of processing the first multimedia resources corresponding to each of the multiple sample tasks based on the intelligent agent to obtain the first reward data corresponding to the task trajectories of the multiple sample tasks includes: Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the task trajectories of the multiple sample tasks are obtained. Based on a multimodal large language model, trajectory analysis is performed on the task trajectories of the multiple sample tasks to obtain the first reward data corresponding to the task trajectories of the multiple sample tasks, and the second reward data corresponding to each action in the task trajectory of each sample task; the second reward data corresponding to each action represents the correctness of the agent in performing the corresponding action.

[0077] The agent processes multiple sample tasks corresponding to the current training round, resulting in task trajectories for each sample task. Each sample task's trajectory can be one or more. One trajectory is obtained by the agent executing each sample task once, while multiple trajectories are obtained by the agent executing each sample task multiple times. These task trajectories are then input into a multimodal large language model for trajectory analysis. Leveraging the powerful reasoning and analytical capabilities of the multimodal large language model, the model yields first reward data corresponding to the task trajectories and second reward data corresponding to each action within each sample task's trajectory. The first reward data is trajectory-level, representing the agent's mastery of the corresponding sample task in the current training round. The second reward data is step-level (action-level), representing the correctness of the agent's actions performed in the corresponding state. The second reward data for each step can be 0 or 1.

[0078] In this embodiment, the target task can be determined from the multiple sample tasks based on the first reward data corresponding to the task trajectories of the multiple sample tasks; further, based on the second reward data of each action in the task trajectory of the target task, the set of positive samples in the task trajectory of the target task can be determined; the set of positive samples may include a first state and the successful action corresponding to the first state, the second reward data corresponding to the successful action can be 1, and the successful action corresponding to the first state can be the action that should be performed in a given first state.

[0079] For each first state in the positive sample set and the corresponding successful action, given the first state, multiple candidate actions output by the agent for the first state are collected. These candidate actions are then matched with the successful action corresponding to the first state, and the first loss information is determined based on the matching information. Optionally, each action has corresponding action attributes. By matching the action attributes of each candidate action with the action attributes of the successful action, the matching information between the candidate actions and the successful action corresponding to the first state can be determined. The action attributes of each action may include action type, action parameters, action duration, etc.

[0080] The agent outputs multiple candidate actions in response to a first state. These candidate actions can be actions performed by the agent in response to the same first state, and they are independent of each other. In other words, from the perspective of the source of the multiple candidate actions, all of them are actions performed by the agent in response to the same first state; from the perspective of the relationship between the multiple candidate actions, they are multiple independent parallel generation results of the same agent in response to the same state, with equal status and mutual independence; from the perspective of the function of the multiple candidate actions, they have the same function and can all represent the agent's response to the first state. In this embodiment, when the agent performs an action in a given state, it can use probabilistic sampling rather than deterministic output, thereby enabling the agent to output multiple actions for the same first state. The randomness of the agent's output can be achieved by at least one method, such as dynamically adjusting the sampling temperature, combining kernel sampling or Top-K sampling, or using an initial random seed.

[0081] In this embodiment, the agent processes multiple sample tasks in the current training round. The resulting task trajectories of these sample tasks are input into a multimodal large language model for trajectory analysis. This yields first reward data corresponding to the task trajectories of the multiple sample tasks and second reward data corresponding to each action within the task trajectories. The second reward data for each action characterizes the correctness of the agent's execution of the corresponding action. Thus, the first reward data can be used to determine whether the corresponding sample task is completed as a whole, while the second reward data enables dense step-level rewards to identify whether each step is correct and effective. This allows for both overall rewards for the task trajectory and step-level rewards for the operation steps within the task trajectory, enriching the reward data for the task trajectory. Furthermore, the first reward data can be used to determine the target task from the multiple sample tasks, and the second reward data can be used to determine the set of positive samples from the task trajectories of the target task for constructing loss information. Therefore, constructing loss information based on dense step-level reward data can improve the accuracy of agent training.

[0082] In one optional embodiment, taking a GUI intelligent agent as an example, the GUI intelligent agent is an autonomous task system with a multimodal large model as its core, capable of simulating human gestures to operate mobile application interfaces, and realizing a closed-loop process of "perception → planning → execution → feedback → error correction"; it replaces / assists humans in completing complex tasks in applications (such as ordering takeout, hailing a ride, managing photo albums, etc.) directly through visual understanding and gesture operation of the interface. Accordingly, the task trajectory includes screenshots of the operation interface and operation logs; The method involves performing trajectory analysis on the task trajectories of the multiple sample tasks based on a multimodal large language model to obtain first reward data corresponding to the task trajectories of the multiple sample tasks, and second reward data corresponding to each action in the task trajectory of each sample task, including: Information processing is performed on the screenshots of the operation interface and the operation log based on a lightweight visual language model to obtain semantic description information corresponding to the task trajectory; The semantic description information, the multiple sample tasks, and the screenshot of the operation interface are input into the multimodal large language model for trajectory analysis to obtain the first reward data corresponding to the task trajectory of the multiple sample tasks, and the second reward data corresponding to each action in the task trajectory of each sample task.

[0083] This embodiment provides an evaluation engine to assess the performance of an agent during the execution of sample tasks. Evaluation is performed by inputting the task trajectories of multiple sample tasks executed by the agent into the evaluation engine. The evaluation engine employs a two-layer architecture. The first layer is a semantic step description, which uses a lightweight visual language model to process information from screenshots of the operation interface and operation logs to obtain semantic description information corresponding to each task trajectory. The lower-level operation interface screenshots and logs can be transformed into higher-level natural language semantic descriptions to reduce computational costs and improve interpretability. The second layer is an overall trajectory adjudication, which uses a multimodal large language model to perform logical reasoning and consistency verification on task instructions, the semantic description information output from the first layer, and the operation interface screenshots. Ultimately, it outputs two types of feedback signals: one is a trajectory-level reward R(τ)∈{0,1}, used to determine whether the task is completed as a whole; the other is a dense step-level reward rt∈{0,1}, used to identify whether each step is correct and effective.

[0084] In this embodiment, a multi-level trajectory evaluation mechanism is established. First, a lightweight visual language model processes the screenshots of the operation interface and operation logs in the task trajectory to obtain the semantic description information corresponding to the task trajectory. This reduces the understanding cost of subsequent models and improves interpretability and processing efficiency. Then, based on a multimodal large language model, logical reasoning and consistency verification are performed on the task-related semantic description information, sample tasks, operation interface screenshots, etc., to obtain trajectory-level rewards and dense step-level rewards. That is, by leveraging the powerful data processing capabilities of the lightweight visual language model and the multimodal large language model, the efficiency and accuracy of reward data determination can be improved.

[0085] In an optional embodiment, determining the first loss information based on the matching information of the plurality of candidate actions and the successful action includes: Based on the matching information between the multiple candidate actions and the successful action, the third reward data corresponding to each of the multiple candidate actions is determined; The advantage score for each of the multiple candidate actions is determined based on the third reward data corresponding to each of the multiple candidate actions. The first loss information is determined based on the advantage scores corresponding to each of the multiple candidate actions.

[0086] The matching information between multiple candidate actions and successful actions can characterize the degree of matching between multiple candidate actions and successful actions, or the similarity between multiple candidate actions and successful actions; thus, given the second reward data of successful actions, the third reward data of candidate actions can be determined. For example, the second reward data of successful actions, the matching information of candidate actions, and the second reward data can be fused to obtain the third reward data of candidate actions.

[0087] Based on the third reward data corresponding to each of the multiple candidate actions, the advantage score corresponding to each of the multiple candidate actions can be determined. The third reward data of each candidate action can characterize the superiority of the candidate action relative to other candidate actions, that is, the degree of deviation between the third reward data corresponding to each candidate action and the average reward data corresponding to multiple candidate actions. Essentially, it is a quantitative indicator of how much better each candidate action is than other candidate actions in the same group.

[0088] In reinforcement learning, reward data for candidate actions can be used to guide the agent in policy optimization. In this embodiment, by further processing the third reward data of candidate actions, the advantage score corresponding to the candidate action is obtained. This allows for the determination of the corresponding loss information based on the advantage score, which represents relative performance, instead of using absolute reward data. This avoids the interference of absolute reward data fluctuations on training. Determining loss information based on relative advantage scores can improve the stability and efficiency of agent training. Furthermore, by positively incentivizing the agent with the first state and successful actions in the positive sample set, the agent can be guided to update towards successful actions.

[0089] In this embodiment, by matching the action attributes corresponding to each of the multiple candidate actions with the action attributes of the successful action, the matching information between the multiple candidate actions and the successful action corresponding to the first state can be determined. The action attributes of each action may include action type, action parameters, action duration, etc. Accordingly, determining the third reward data corresponding to each of the multiple candidate actions based on the matching information between the multiple candidate actions and the successful action includes: Determine the first matching information between the action type of each candidate action and the action type of the successful action; Determine the second matching information between the action parameters of each candidate action and the action parameters of the successful action; The first matching information and the second matching information are fused to obtain the third reward data for each candidate action.

[0090] The first matching information between the action type of each candidate action and the action type of the successful action can be the action type matching score between the action type of the candidate action and the action type of the successful action; the second matching information between the action parameters of each candidate action and the action parameters of the successful action can be the similarity score between the action parameters of each candidate action and the action parameters of the successful action; the information fusion of the first matching information and the second matching information can be either a direct summation of the first matching information and the second matching information, or a weighted summation of the first matching information and the second matching information, the weights of which can be determined according to the actual implementation.

[0091] For each first state in the positive sample set, given the first state, collect N candidate actions that the agent would perform in response to the first state. Each candidate action is calculated based on action type matching and parameter similarity matching. Fine-grained third reward data The calculation formula is as follows: (2) in, This is the successful action corresponding to the first state. Match scores to action types. A similarity score is assigned to the action parameters. For example, action types may include click, long press, drag, etc., and action parameters may include click location (coordinates), long press duration, drag distance, etc. When calculating the action type matching score, if... and Action type matching, =1, if and Action type mismatch =0; in and In cases where action types do not match, there is no need to calculate the similarity score of action parameters, thus saving computational resources and allowing for direct determination. =0, in and If the action type matches, it can be generated first. The feature vector corresponding to the action parameters and The similarity between the feature vectors corresponding to the action parameters is calculated to obtain a similarity score for the action parameters.

[0092] Once the third reward data for each candidate action is determined, the calculation of each candidate action can be performed. The advantage score, and the formula for calculating the advantage score is as follows: (3) in, Actions for candidates The third reward data, For N candidate actions The average of the third reward data, For N candidate actions The standard deviation of the third reward data, To find the minimum value, avoid having a denominator of 0.

[0093] The formula for determining the first loss information based on the advantage scores corresponding to multiple candidate actions is as follows: (4) in, It is the ratio of the probability of the new strategy to the probability of the old strategy within the group. The j-th action within group g, clip(.) clipping function, restricts... In [1] ,1+ ]between, To find the minimum value, avoid a denominator of 0. Let be the advantage score for the j-th action.

[0094] In this embodiment, during the process of determining the third reward data corresponding to each of the multiple candidate actions, the action type matching information and action parameter matching information of each candidate action and the successful action can be determined separately. Then, the third reward data of each candidate action can be determined together based on the matching information of multiple dimensions, thereby improving the accuracy of the determination of the third reward data.

[0095] Furthermore, based on the second reward data corresponding to each action in the task trajectory of the target task, while determining the positive sample set in the task trajectory of the target task, a negative sample set can also be determined. The positive sample set may include a first state and the successful action corresponding to the first state. The second reward data corresponding to the successful action can be 1. The successful action corresponding to the first state can be the action that should be executed given the first state. Correspondingly, the negative sample set may include a second state and the failed action corresponding to the second state. The second reward data corresponding to the failed action can be 0. The failed action corresponding to the second state can be the action that should not be executed given the second state, or the action that was not completed given the second state, etc. Accordingly, second loss information can be constructed based on the negative sample set. The second loss information can be aimed at suppressing the agent from repeating mistakes. Optionally, this embodiment provides a method for generating second loss information, which may include: Based on the second reward data corresponding to each action in the task trajectory of the target task, a negative sample set in the task trajectory of the target task is determined; the negative sample set includes a second state and the failed action corresponding to the second state; Determine the first probability distribution information of the failed action of the agent corresponding to the previous training round of the current training round, given the second state; Determine the second probability distribution information of the agent's output action given the second state in the current training round; A second loss information is generated based on the first probability distribution and the second probability distribution; the second loss information aims to maximize the difference between the information of the first probability distribution and the second probability distribution.

[0096] Based on the second reward data corresponding to each action in the task trajectory of the target task, actions with a second reward data of 0 can be identified as failed actions. Each failed action corresponds to a second state; that is, given a second state, the agent performed a failed action. A failed action can be an erroneous action or an incomplete action, resulting in a second reward data of 0. In each training round, the agent's parameters are updated. The agent for the current training round is obtained by updating the parameters of the agent for the previous training round. This allows us to obtain the first probability distribution information of the failed action output by the agent in the previous training round given a second state. We can also obtain the second probability distribution information of the action output by the agent in the current training round given a second state. Since the first probability distribution information of the failed action output by the agent in the previous training round given a second state is already known, to avoid the agent repeatedly outputting erroneous actions given a second state, i.e., to prevent the agent from repeating mistakes, negative suppression can be achieved by constructing second loss information. The formula for calculating the second loss information is as follows: (5) in, The policy corresponding to the agent in the current training round. The reference policy can be the policy corresponding to the agent in the previous training round. The above formula maximizes the current policy. Distribution of failed actions (from reference strategy) The difference between the generated failure samples helps to prevent the model from repeating mistakes.

[0097] Accordingly, adjusting the parameters of the agent based on the first loss information includes: The parameters of the agent are adjusted based on the first loss information and the second loss information.

[0098] Optionally, by combining positive incentives and negative inhibition, the overall optimization objective function... Defined as: (6) in, The above is the first loss information. This is the second type of lost information. To balance the coefficients, the agent is positively incentivized using the first state and successful actions from the positive sample set, guiding the agent to update in the direction of successful actions; negative inhibition is applied using the negative sample set to maximize the current policy. Distribution of failed actions (from reference strategy) The difference between the generated failure samples is used to suppress the model from repeating mistakes; finally, by minimizing the total loss, the agent can simultaneously consolidate successful experiences and avoid known errors, achieving rapid iteration and improvement of the policy.

[0099] For each training round, the multiple sample tasks may include at least new tasks that the agent has not processed and tasks that the agent has not fully mastered; optionally, in this embodiment, after adjusting the agent's parameters based on the first loss information, the method further includes: The sample task for the next training round is determined based on the second task and the new task that the agent has not yet processed.

[0100] Based on the first reward data corresponding to the task trajectories of multiple sample tasks, these tasks can be dynamically divided into a first task, a target task, and a second task. The agent's mastery of the first task is greater than its mastery of the target task; that is, the first reward data corresponding to the task trajectory of the first task is greater than the first reward data corresponding to the task trajectory of the target task. The first task can be a simple task that the agent has already mastered in the current training round. The agent's mastery of the second task is less than its mastery of the target task; that is, the first reward data corresponding to the task trajectory of the second task is less than the first reward data corresponding to the task trajectory of the target task. The second task can be a difficult task that the agent cannot solve in the current training round. In other words, the second task that the agent has not mastered in the current training round can continue to be used as a sample task in the next training round. Each round is defined. Active task pool Its construction formula is That is, the new tasks of the current stage The task that was not mastered in the previous round The combined system ensures intelligent physical fitness review and helps overcome difficult problems.

[0101] In this embodiment, difficult tasks that the agent failed to overcome in previous training rounds can be used as sample tasks in subsequent training rounds for the agent to train. By dynamically selecting sample tasks for targeted reinforcement learning updates, the agent's ability to handle difficult tasks can be further improved.

[0102] In an optional embodiment, an agent can also perform interactive exploration, and the results of the interactive exploration can be processed using a multimodal large language model to obtain multiple sample tasks for training the agent; optionally, before obtaining the task input information of the multiple sample tasks corresponding to the current training round, the method further includes: Obtain prior information about the application structure of the target application; Based on the interaction exploration performed by the agent according to the prior information of the application structure, at least one interaction trajectory between the agent and the target application is obtained; The at least one interaction trajectory is processed based on a multimodal large language model to obtain multiple sample tasks for training the agent.

[0103] The target application can be an application installed and running on a mobile device or an application installed and running on a non-mobile device. Prior information about the target application's structure can include screenshots of each page, elements contained within each page, a tree structure diagram of the target application, and the topology and dependencies between modules of the target application. Using this prior information as an exploration anchor, the agent is guided to explore based on either depth-first or breadth-first search. During the exploration process, the system meticulously records every interaction trajectory between the agent and the target application, including screenshot sequences, operation logs, and UI state changes, thereby collecting a massive amount of raw interaction data. Subsequently, using a multimodal large language model as a generator, the system takes the aforementioned raw trajectories and application context information (which exists when switching between applications) as input and outputs a structured task set C={T1,...,TN}. During the generation process, four principles are followed: executability, clarity, progressive difficulty, and coverage. This ensures that the generated task instructions are clear, parameters are based on real screen content, and the task difficulty covers everything from simple single-step operations to complex multi-step planning.

[0104] In this embodiment, by guiding the agent to explore based on the corresponding exploration strategy, and generating sample tasks for training based on the exploration results, the unstructured interaction logs are transformed into structured, multi-modal large model generation capabilities using functionally perceptual exploration and multimodal large model generation capabilities. This provides a high-quality data foundation for the agent's cold-start learning, completely eliminates the dependence on expensive manually labeled data, greatly reduces development and maintenance costs, and enables the agent to be scaled up to massive applications at low cost.

[0105] In another optional embodiment, the multiple sample tasks used to train the agent each have their own corresponding task difficulty level; Before obtaining the task input information of multiple sample tasks corresponding to the current training round, the method further includes: Determine the current difficulty level corresponding to the current training round; Select sample tasks with the current difficulty level from the multiple samples used to train the agent, and use them as the multiple sample tasks corresponding to the current training round.

[0106] By leveraging functional awareness exploration and the generation capabilities of multimodal large models, unstructured interaction logs are transformed into structured, multi-difficulty training tasks, providing a high-quality data foundation for the agent's cold-start learning. This allows for the determination of the task difficulty level for each sample task, enabling the agent to be trained in each training round based on matching sample tasks. As the number of training rounds for the agent increases, the task difficulty level of the sample tasks used for training also adaptively increases; for example, in the first training round, sample tasks of the first difficulty level are used to train the agent, and in the second training round, sample tasks of the second difficulty level are used. The first training round is the training round preceding the second training round, and the difficulty level of the first task is lower than that of the second task.

[0107] In this embodiment, by assigning sample tasks of appropriate difficulty levels to each training round for training the agent, it is possible to avoid the agent wasting a lot of computing resources on simple tasks that it has already mastered in the current training round, or repeatedly failing on difficult tasks that it cannot solve at present and thus learning nothing, resulting in low training efficiency. Therefore, by training the agent based on task difficulty levels that are appropriate for the current training round, computing resources can be saved and training efficiency can be improved.

[0108] The agent trained using the above-described agent training method can be applied to specific task processing scenarios. Optionally, please refer to [link to relevant documentation]. Figure 3 This embodiment also provides a task processing method, which may include: S310. Obtain the task input information of the task to be processed; the task input information of the task to be processed includes a second multimedia resource; S320. The second multimedia resource is input into the intelligent agent for task processing to obtain the task processing result corresponding to the task to be processed; the intelligent agent is trained based on the training method described above in this embodiment.

[0109] In this embodiment, the task input information for the task to be processed may include at least one form of second multimedia resource, such as text, audio, image, or video; the second multimedia resource corresponding to the task to be processed is a resource that carries at least one of the following information: task description information, task requirement information, and input basis required for task execution. Optionally, the task description information, task requirement information, and input basis required for task execution may all be presented in at least one form, such as text, audio, image, or video.

[0110] In one optional embodiment, in an intelligent customer service scenario, the task input information for a corresponding customer service task may include at least one form of question information such as text, audio, image, or video. In a code generation scenario, the task input information for a corresponding code generation task may include at least one form of code requirements or code function descriptions such as text, audio, image, or video. In a mathematical calculation scenario, the task input information for a corresponding mathematical calculation task may include at least one form of mathematical problem such as text, audio, image, or video.

[0111] The intelligent agent in this embodiment can be a software intelligent agent, such as an intelligent customer service agent, a personal assistant (automatic schedule management, batch photo album cleanup, intelligent message reply), a recommendation system, an automated office system (mobile approval process, automatic report generation and sharing), a GUI agent, a code generation agent, or an edge automation agent (phone factory testing, APP compatibility verification, user behavior simulation), etc. It can also be a physical intelligent agent, such as a robot or an IoT device. Taking a GUI intelligent agent as an example, a GUI intelligent agent is an autonomous task system with a multimodal large model at its core, capable of simulating human gestures to operate mobile application interfaces, and achieving a closed-loop process of "perception → planning → execution → feedback → error correction." It replaces / assists humans in completing complex tasks in applications (such as ordering takeout, hailing a ride, managing photo albums, etc.) directly through visual understanding and gesture operation of the interface. Accordingly, the intelligent agent trained in this embodiment can be used to handle customer service tasks, personal assistant tasks, recommendation tasks, automated variable tasks, GUI tasks, code generation tasks, edge automation tasks, etc., and also to handle robot tasks, IoT device tasks, etc.

[0112] By dynamically selecting sample tasks for adjusting model parameters based on the agent's actual performance on multiple sample tasks in the current training round, the agent learns tasks that are "within reach," thus avoiding ineffective training and improving training efficiency. Furthermore, the sample parameters corresponding to the method of updating agent parameters based on the first loss information are moderate, avoiding drastic fluctuations in parameter updates, thereby improving training stability and ultimately enhancing the agent's performance in task processing scenarios. Consequently, using an agent generated based on this training method for task processing can improve the accuracy of task processing.

[0113] It should be noted that any of the methods in this embodiment can be combined based on the actual implementation situation and have corresponding beneficial effects, which will not be elaborated here.

[0114] To address the technical problems of existing technologies, such as heavy reliance on expensive manually labeled data, poor model generalization ability in unseen applications, and low training efficiency of traditional reinforcement learning methods in long-sequence GUI tasks due to reward sparsity and curriculum rigidity, please refer to [link to relevant documentation]. Figure 4 This embodiment proposes a fully automated GUI agent closed-loop self-evolution technology framework. Its improvement lies in abandoning the traditional "manual data collection-supervised fine-tuning" paradigm and instead establishing a self-learning closed loop without human intervention. Specifically, firstly, random exploration behavior is transformed into ordered training courses through autonomous curriculum generation, thus solving the problem of "where does the training data come from" with zero human cost; secondly, a fine-grained meta-evaluation engine utilizes the visual reasoning capabilities of a multimodal large model to replace manual judgment, providing denser and more accurate step-level rewards than traditional environmental feedback, solving the problem of "how to distinguish good from bad in the absence of truth values"; finally, through difficulty-adaptive dynamic strategy optimization, the limitations of traditional static courses are broken. Training samples are dynamically selected based on the agent's real-time success rate, allowing the agent to learn only tasks that are currently "within reach," thus solving the problem of how to avoid ineffective training and maximize learning efficiency, establishing a continuously iterative self-improving closed loop.

[0115] Step 1: Self-managed curriculum generation This step aims to address the "what to learn" question, namely, building a high-quality training task library without human guidance. The process is divided into two sub-steps: exploration and generation. First, a function-aware exploration strategy is employed, using the structural prior information of the mobile application as exploration anchors to guide the agent in a depth-first search (DFS). During exploration, the system meticulously records every interaction trajectory between the agent and the app, including screenshot sequences, operation logs, and UI state changes, thereby collecting a massive amount of raw interaction data. For example, for the recipe creation task, the agent can interact with the app and will experience changes from interface a1 to interface a2, interface a3, and interface a4. Subsequently, a large language model is used as a generator, inputting the aforementioned raw trajectories and app context information, to output a structured set of course tasks. During the generation process, four principles are followed: executability, clarity, progressive difficulty, and coverage. This ensures that the generated task instructions are clear, the parameters are based on real screen content, and the task difficulty ranges from simple single-step operations to complex multi-step planning. The generated tasks are also assigned an initial difficulty rating for subsequent phased training.

[0116] Step 2: Agent Execution and Fine-Grained Meta-Evaluation This step aims to address the question of "how well it has learned," that is, accurately evaluating the agent's performance in the absence of human ground truth. The agent then applies the current policy... Perform tasks in the course to generate new interaction trajectories. These trajectories are then fed into an evaluation engine for automated scoring. This engine employs a two-layer architecture: the first layer is a semantic step description (Observer), which uses a distilled, lightweight visual language model to process each step in the trajectory in parallel, transforming the lower-level operation screenshots and logs into a higher-level natural language semantic description. The first layer reduces computational costs and improves interpretability; the second layer is the overall trajectory adjudicator, which uses a multimodal large model to perform logical reasoning and consistency verification on the semantic descriptions of the entire task instructions, all steps, and the final screenshots. Finally, the adjudicator outputs two types of feedback signals: one is trajectory-level reward. First, it is used to determine whether the task has been completed as a whole; second, it provides dense step-by-step rewards. This is used to identify whether each operation is correct and effective. For example, during the execution of a task, it goes through changes from interface b1 → interface b2 → interface b3 → interface b4 → interface b5 → interface b6 → interface b7 → interface b8. Interfaces b1, b2, b3, b5, b7, and b8 correspond to successful actions, and rewards are given accordingly. =1; Interfaces b4 and b6 correspond to failure actions. =0.

[0117] Step 3: Difficulty Adaptive Dynamic Strategy Optimization This phase aims to address the question of "how to learn efficiently" by dynamically selecting samples for targeted reinforcement learning updates. The process is as follows: Figure 5 As shown. The entire optimization process is carried out in multiple evolutionary rounds. First, each round is defined. Active task pool Its construction formula is That is, the new tasks of the current stage The task that was not mastered in the previous round This process is combined to ensure the agent can review and tackle challenging problems. In each evolutionary round, a dynamic task triage mechanism is first implemented. The agent then tackles each task in the active task pool. Conduct multiple rounds ( (Number) attempts to collect trajectory sets The average rate of return for this task was then calculated. .

[0118] Based on this average rate of return and preset threshold and The system dynamically divides the task into three areas: If If it is in the Mastered area, it will be archived directly; if If it is not mastered, it will be classified as an "Unmastered" zone and postponed to the next round; only if... When a task is identified as being in the Optimal Learning Zone, the system selects only the task trajectory within the Optimal Learning Zone for the current policy update, thereby maximizing learning efficiency.

[0119] Secondly, a comparative strategy is employed to optimize the algorithm for updating the model. The step-level rewards obtained during the evaluation phase are utilized. The trajectory data of the optimal learning area is split into a set of positive samples. (Successful actions) and negative sample set (Failed Actions). For positive samples, Group Relative Policy Optimization (GRPO) is used for positive incentives. For negative samples, an Adversarial Imitation loss function is introduced, which maximizes the difference between the current policy and the distribution of failed actions (determined by the reference policy). The differences between generated failure samples are used to suppress the model from repeating mistakes. Finally, by combining positive incentives and negative inhibition to determine the total loss, the agent can simultaneously consolidate successful experiences and avoid known errors, thus achieving rapid iteration and improvement of the strategy.

[0120] This embodiment proposes a difficulty-adaptive dynamic policy optimization mechanism. Based on a dynamic triage method using task success rate, this mechanism defines the agent's optimal learning zone in real time and constructs a composite contrastive loss function by combining GRPO and adversarial imitation learning. This addresses the core challenges of uneven training data quality and unstable policy updates during self-evolution. Secondly, a fine-grained meta-evaluation mechanism is proposed. This mechanism, through a two-stage design, cleverly utilizes the capabilities of large models to achieve automated, low-cost, and high-precision trajectory evaluation, providing crucial step-level dense reward signals for reinforcement learning and breaking through the evaluation bottleneck under conditions without human truth values. Finally, an autonomous curriculum generator is proposed. This component leverages functional-aware exploration and the generation capabilities of multimodal large models to transform unstructured interaction logs into structured, difficulty-divided training courses, providing a high-quality data foundation for the agent's cold-start learning.

[0121] Figure 6 This is a block diagram illustrating a training apparatus for an intelligent agent according to an exemplary embodiment. (Refer to...) Figure 6 The device includes: The sample task acquisition unit 610 is configured to acquire task input information of multiple sample tasks corresponding to the current training round; the task input information of each sample task includes a first multimedia resource. The first task processing unit 620 is configured to perform processing on the first multimedia resources corresponding to each of the plurality of sample tasks based on the intelligent agent, and obtain the first reward data corresponding to the task trajectory of the plurality of sample tasks; the first reward data corresponding to each task trajectory represents the degree of mastery of the intelligent agent on the corresponding sample task in the current training round. The target task determination unit 630 is configured to execute first reward data corresponding to the task trajectories of the plurality of sample tasks to determine a target task from the plurality of sample tasks; the target task is a task other than the first task and the second task among the plurality of sample tasks, the agent's mastery of the first task is greater than the agent's mastery of the target task, and the agent's mastery of the second task is less than the agent's mastery of the target task. The loss information determination unit 640 is configured to perform task trajectory determination based on the target task to determine the first loss information corresponding to the current training round; The parameter adjustment unit 650 is configured to adjust the parameters of the agent based on the first loss information.

[0122] In one exemplary embodiment, the target task determination unit is configured to perform: Obtain the target reward data range; Based on the data matching between the first reward data corresponding to the task trajectory of the multiple sample tasks and the target reward data interval, the sample task corresponding to the first reward data and the target reward data interval is determined as the target task.

[0123] In an exemplary embodiment, the first reward data corresponding to each of the plurality of sample tasks includes first reward data corresponding to multiple task trajectories of each task; the target task determination unit is configured to execute: Data fusion is performed based on the first reward data corresponding to multiple task trajectories of each task to obtain the fused reward data corresponding to each task; The fusion reward data corresponding to each task is matched with the target reward data range, and the task whose fusion reward data falls within the target reward data range is determined as the target task.

[0124] In one exemplary embodiment, the task trajectory of each sample task includes at least one action; The first task processing unit is configured to execute: Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the task trajectories of the multiple sample tasks are obtained. Based on a multimodal large language model, trajectory analysis is performed on the task trajectories of the multiple sample tasks to obtain first reward data corresponding to the task trajectories of the multiple sample tasks, and second reward data corresponding to each action in the task trajectory of each sample task; the second reward data corresponding to each action represents the correctness of the agent in performing the corresponding action. The loss information determination unit is configured to execute: Based on the second reward data corresponding to each action in the task trajectory of the target task, a set of positive samples in the task trajectory of the target task is determined; the set of positive samples includes a first state and the successful action corresponding to the first state; Collect multiple candidate actions output by the agent in response to the first state; The first loss information is determined based on the matching information of the multiple candidate actions and the successful actions.

[0125] In one exemplary embodiment, the task trajectory includes screenshots of the user interface and operation logs; The first task processing unit is configured to execute: Information processing is performed on the screenshots of the operation interface and the operation log based on a lightweight visual language model to obtain semantic description information corresponding to the task trajectory; The semantic description information, the multiple sample tasks, and the screenshot of the operation interface are input into the multimodal large language model for trajectory analysis to obtain the first reward data corresponding to the task trajectory of the multiple sample tasks, and the second reward data corresponding to each action in the task trajectory of each sample task.

[0126] In an exemplary embodiment, the loss information determination unit includes: An action matching unit is configured to perform matching information based on the plurality of candidate actions and the successful action to determine the third reward data corresponding to each of the plurality of candidate actions; The advantage score determination unit is configured to determine the advantage score corresponding to each of the multiple candidate actions based on the third reward data corresponding to each of the multiple candidate actions. The first loss determination unit is configured to determine the first loss information based on the advantage scores corresponding to each of the plurality of candidate actions.

[0127] In one exemplary embodiment, the action matching unit is configured to perform: Determine the first matching information between the action type of each candidate action and the action type of the successful action; Determine the second matching information between the action parameters of each candidate action and the action parameters of the successful action; The first matching information and the second matching information are fused to obtain the third reward data for each candidate action.

[0128] In one exemplary embodiment, the apparatus further includes a second loss determination unit configured to perform: Based on the second reward data corresponding to each action in the task trajectory of the target task, a negative sample set in the task trajectory of the target task is determined; the negative sample set includes a second state and the failed action corresponding to the second state; Determine the first probability distribution information of the failed action of the agent corresponding to the previous training round of the current training round, given the second state; Determine the second probability distribution information of the agent's output action given the second state in the current training round; A second loss information is generated based on the first probability distribution and the second probability distribution; the second loss information aims to maximize the difference between the first probability distribution and the second probability distribution. The parameter adjustment unit is configured to perform: The parameters of the agent are adjusted based on the first loss information and the second loss information.

[0129] In one exemplary embodiment, the apparatus further includes a first sample task determination unit configured to perform: The sample task for the next training round is determined based on the second task and the new task that the agent has not yet processed.

[0130] In one exemplary embodiment, the apparatus further includes a sample task creation unit configured to perform: Obtain prior information about the application structure of the target application; Based on the interaction exploration performed by the agent according to the prior information of the application structure, at least one interaction trajectory between the agent and the target application is obtained; The at least one interaction trajectory is processed based on a multimodal large language model to obtain multiple sample tasks for training the agent.

[0131] In one exemplary embodiment, the multiple sample tasks used to train the agent each have their own corresponding task difficulty level; The device further includes a second sample task determination unit, configured to perform: Determine the current difficulty level corresponding to the current training round; Select sample tasks with the current difficulty level from the multiple samples used to train the agent, and use them as the multiple sample tasks corresponding to the current training round.

[0132] Figure 7 This is a block diagram of a task processing apparatus according to an exemplary embodiment. (Refer to...) Figure 7 The device includes: The task acquisition unit 710 is configured to acquire task input information of the task to be processed; the task input information of the task to be processed includes a second multimedia resource. The second task processing unit 720 is configured to input the second multimedia resource into the intelligent agent for task processing and obtain the task processing result corresponding to the task to be processed. The agent is trained using the training method described above.

[0133] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0134] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform any of the methods described above.

[0135] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the device to perform any of the methods described above.

[0136] Figure 8 This is a block diagram illustrating an electronic device for training or task processing of an intelligent agent according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for training an intelligent agent or processing tasks. Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] Figure 9 This is a block diagram illustrating an electronic device for training or task processing of an intelligent agent according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the device may include an RF (Radio Frequency) circuit 910, a memory 920 including one or more computer-readable storage media, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a WiFi (Wireless Fidelity) module 970, a processor 980 including one or more processing cores, and a power supply 990, among other components. Those skilled in the art will understand that... Figure 9The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The RF circuit 910 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 980 for processing; additionally, it transmits uplink data to the base station. Typically, the RF circuit 910 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. Furthermore, the RF circuit 910 can also communicate wirelessly with networks and other terminals. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0138] The memory 920 can be used to store software programs and modules. The processor 980 executes various functional applications and data processing by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for the functions, etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 920 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 920 may also include a memory controller to provide access to the memory 920 for the processor 980 and the input unit 930.

[0139] The input unit 930 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 930 may include a touch-sensitive surface 931 and other input devices 932. The touch-sensitive surface 931, also known as a touch display screen or touchpad, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface 931), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 931 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 980, and can receive and execute commands from the processor 980. In addition, the touch-sensitive surface 931 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 931, the input unit 930 may also include other input devices 932. Specifically, other input devices 932 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc. The display unit 940 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the terminal. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit 940 may include a display panel 941, which may optionally be configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or similar display. Further, a touch-sensitive surface 931 may cover the display panel 941. When the touch-sensitive surface 931 detects a touch operation on or near it, it transmits the information to the processor 980 to determine the type of touch event. Subsequently, the processor 980 provides corresponding visual output on the display panel 941 according to the type of touch event. The touch-sensitive surface 931 and the display panel 941 can be two independent components to implement input and output functions; however, in some embodiments, the touch-sensitive surface 931 and the display panel 941 can be integrated to achieve both input and output functions.

[0140] The terminal may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 941 according to the ambient light level, and the proximity sensor can turn off the display panel 941 and / or backlight when the terminal is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that identify the terminal's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured on the terminal, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0141] Audio circuitry 960, speaker 961, and microphone 962 provide an audio interface between the user and the terminal. Audio circuitry 960 converts received audio data into electrical signals, which are then transmitted to speaker 961, where they are converted into sound signals for output. Conversely, microphone 962 converts collected sound signals into electrical signals, which are received by audio circuitry 960, converted back into audio data, and then processed by processor 980 before being transmitted via RF circuitry 910 to, for example, another terminal, or output to memory 920 for further processing. Audio circuitry 960 may also include an earphone jack to facilitate communication between a peripheral headset and the terminal.

[0142] WiFi is a short-range wireless transmission technology. This terminal, through the WiFi module 970, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 9 The WiFi module 970 is shown, but it is understood that it is not a necessary component of the terminal and can be omitted as needed without changing the nature of the invention.

[0143] The processor 980 is the control center of the terminal, connecting various parts of the terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 920, and calls data stored in the memory 920, to perform various functions and process data, thereby enabling overall monitoring of the terminal. Optionally, the processor 980 may include one or more processing cores; preferably, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 980.

[0144] The terminal also includes a power supply 990 (such as a battery) to power various components. Preferably, the power supply can be logically connected to the processor 980 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 990 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0145] Although not shown, the terminal may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the terminal is a touch screen display, and the terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors of the instructions in the method embodiment of the present invention.

[0146] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0147] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training an intelligent agent, characterized in that, The method includes: Obtain the task input information of multiple sample tasks corresponding to the current training round; the task input information of each sample task includes the first multimedia resource; Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the first reward data corresponding to the task trajectory of the multiple sample tasks is obtained; the first reward data corresponding to each task trajectory represents the degree of mastery of the intelligent agent on the corresponding sample task in the current training round. Based on the first reward data corresponding to the task trajectories of the multiple sample tasks, a target task is determined from the multiple sample tasks; the target task is a task other than the first task and the second task among the multiple sample tasks, the agent's mastery of the first task is greater than the agent's mastery of the target task, and the agent's mastery of the second task is less than the agent's mastery of the target task. The first loss information corresponding to the current training round is determined based on the task trajectory of the target task. The parameters of the agent are adjusted based on the first loss information.

2. The method according to claim 1, characterized in that, The process of determining the target task from the multiple sample tasks based on the first reward data corresponding to the task trajectories of the multiple sample tasks includes: Obtain the target reward data range; Based on the data matching between the first reward data corresponding to the task trajectory of the multiple sample tasks and the target reward data interval, the sample task corresponding to the first reward data and the target reward data interval is determined as the target task.

3. The method according to claim 2, characterized in that, The first reward data for each of the multiple sample tasks includes the first reward data corresponding to the multiple task trajectories of each task. The step of matching the first reward data corresponding to the task trajectory of the multiple sample tasks with the target reward data interval, and determining the sample task corresponding to the first reward data and the target reward data interval as the target task, includes: Data fusion is performed based on the first reward data corresponding to multiple task trajectories of each task to obtain the fused reward data corresponding to each task; The fusion reward data corresponding to each task is matched with the target reward data range, and the task whose fusion reward data falls within the target reward data range is determined as the target task.

4. The method according to claim 1, characterized in that, Each sample task's trajectory includes at least one action; The process of processing the first multimedia resources corresponding to each of the multiple sample tasks based on the intelligent agent to obtain the first reward data corresponding to the task trajectories of the multiple sample tasks includes: Based on the processing of the first multimedia resources corresponding to each of the multiple sample tasks by the intelligent agent, the task trajectories of the multiple sample tasks are obtained. Based on a multimodal large language model, trajectory analysis is performed on the task trajectories of the multiple sample tasks to obtain first reward data corresponding to the task trajectories of the multiple sample tasks, and second reward data corresponding to each action in the task trajectory of each sample task; the second reward data corresponding to each action represents the correctness of the agent in performing the corresponding action. The step of determining the first loss information corresponding to the current training round based on the task trajectory of the target task includes: Based on the second reward data corresponding to each action in the task trajectory of the target task, a set of positive samples in the task trajectory of the target task is determined; the set of positive samples includes a first state and the successful action corresponding to the first state; Collect multiple candidate actions output by the agent in response to the first state; The first loss information is determined based on the matching information of the multiple candidate actions and the successful actions.

5. The method according to claim 4, characterized in that, The task trajectory includes screenshots of the operation interface and operation logs; The method involves performing trajectory analysis on the task trajectories of the multiple sample tasks based on a multimodal large language model to obtain first reward data corresponding to the task trajectories of the multiple sample tasks, and second reward data corresponding to each action in the task trajectory of each sample task, including: Information processing is performed on the screenshots of the operation interface and the operation log based on a lightweight visual language model to obtain semantic description information corresponding to the task trajectory; The semantic description information, the multiple sample tasks, and the screenshot of the operation interface are input into the multimodal large language model for trajectory analysis to obtain the first reward data corresponding to the task trajectory of the multiple sample tasks, and the second reward data corresponding to each action in the task trajectory of each sample task.

6. The method according to claim 4, characterized in that, Determining the first loss information based on the matching information of the multiple candidate actions and the successful actions includes: Based on the matching information between the multiple candidate actions and the successful action, the third reward data corresponding to each of the multiple candidate actions is determined; The advantage score for each of the multiple candidate actions is determined based on the third reward data corresponding to each of the multiple candidate actions. The first loss information is determined based on the advantage scores corresponding to each of the multiple candidate actions.

7. The method according to claim 6, characterized in that, The step of determining the third reward data corresponding to each of the multiple candidate actions based on the matching information between the multiple candidate actions and the successful action includes: Determine the first matching information between the action type of each candidate action and the action type of the successful action; Determine the second matching information between the action parameters of each candidate action and the action parameters of the successful action; The first matching information and the second matching information are fused to obtain the third reward data for each candidate action.

8. The method according to claim 1, characterized in that, The method further includes: Based on the second reward data corresponding to each action in the task trajectory of the target task, a negative sample set in the task trajectory of the target task is determined; the negative sample set includes a second state and the failed action corresponding to the second state; Determine the first probability distribution information of the failed action of the agent corresponding to the previous training round of the current training round, given the second state; Determine the second probability distribution information of the agent's output action given the second state in the current training round; A second loss information is generated based on the first probability distribution and the second probability distribution; the second loss information aims to maximize the difference between the first probability distribution and the second probability distribution. The step of adjusting the parameters of the agent based on the first loss information includes: The parameters of the agent are adjusted based on the first loss information and the second loss information.

9. The method according to claim 1, characterized in that, After adjusting the parameters of the agent based on the first loss information, the method further includes: The sample task for the next training round is determined based on the second task and the new task that the agent has not yet processed.

10. The method according to claim 1, characterized in that, Before obtaining the task input information of multiple sample tasks corresponding to the current training round, the method further includes: Obtain prior information about the application structure of the target application; Based on the interaction exploration performed by the agent according to the prior information of the application structure, at least one interaction trajectory between the agent and the target application is obtained; The at least one interaction trajectory is processed based on a multimodal large language model to obtain multiple sample tasks for training the agent.

11. The method according to claim 10, characterized in that, The multiple sample tasks used to train the agent each have their own corresponding task difficulty level; Before obtaining the task input information of multiple sample tasks corresponding to the current training round, the method further includes: Determine the current difficulty level corresponding to the current training round; Select sample tasks with the current difficulty level from the multiple samples used to train the agent, and use them as the multiple sample tasks corresponding to the current training round.

12. A task processing method, characterized in that, The method includes: Obtain the task input information of the task to be processed; the task input information of the task to be processed includes a second multimedia resource; The second multimedia resource is input into the intelligent agent for task processing to obtain the task processing result corresponding to the task to be processed. The agent is trained based on the training method described in any one of claims 1-11.

13. A training device for an intelligent agent, characterized in that, The device includes: The sample task acquisition unit is configured to acquire task input information for multiple sample tasks corresponding to the current training round; the task input information for each sample task includes a first multimedia resource. The first task processing unit is configured to perform processing on the first multimedia resources corresponding to each of the multiple sample tasks based on the intelligent agent, and obtain the first reward data corresponding to the task trajectory of the multiple sample tasks; the first reward data corresponding to each task trajectory represents the degree of mastery of the intelligent agent on the corresponding sample task in the current training round. The target task determination unit is configured to execute first reward data corresponding to the task trajectories of the plurality of sample tasks to determine the target task from the plurality of sample tasks; the target task is a task other than the first task and the second task among the plurality of sample tasks, the agent's mastery of the first task is greater than the agent's mastery of the target task, and the agent's mastery of the second task is less than the agent's mastery of the target task. The loss information determination unit is configured to perform task trajectory determination based on the target task to determine the first loss information corresponding to the current training round. The parameter adjustment unit is configured to adjust the parameters of the agent based on the first loss information.

14. A task processing device, characterized in that, The device includes: The task acquisition unit is configured to acquire task input information of the task to be processed; the task input information of the task to be processed includes a second multimedia resource. The second task processing unit is configured to input the second multimedia resource into the intelligent agent for task processing and obtain the task processing result corresponding to the task to be processed. The agent is trained based on the training method described in any one of claims 1-11.

15. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the agent training method as described in any one of claims 1 to 11, or the task processing method as described in claim 12.

16. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method for an agent as described in any one of claims 1 to 11, or the task processing method as described in claim 12.

17. A computer program product, characterized in that, The computer program product includes a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from and executes the computer program, causing the device to perform the training method for an agent as described in any one of claims 1 to 11, or the task processing method as described in claim 12.

Images