Agent training and task processing method and apparatus, agent, and related product
By calculating action information entropy and dynamically allocating credit weights in GUI scenarios, the difficulty of credit allocation for agents in sparse reward environments is solved, achieving efficient training and improved stability of agents in GUI tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2026-03-18
- Publication Date
- 2026-07-24
Smart Images
Figure CN122452606A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of GUI, large models, and intelligent agents, and specifically to an intelligent agent training and task processing method, apparatus, intelligent agent, device, medium, and product. Background Technology
[0002] In a graphical user interface (GUI) scenario, GUI agents can be used to complete GUI tasks. Summary of the Invention
[0003] This disclosure provides a method, apparatus, agent, device, medium, and product for training and processing intelligent agents and tasks.
[0004] According to one aspect of this disclosure, a method for training an intelligent agent is provided, comprising: acquiring an exploration trajectory generated by the intelligent agent, the exploration trajectory including samples of multiple steps, each step including: GUI interaction actions; determining the action information entropy of each step based on the GUI interaction actions of each step; determining the credit weight of each step based on the action information entropy, wherein the credit weight and the action information entropy have a positive mapping relationship; determining the target credit value of each step based on the credit weight; and adjusting the model parameters of the intelligent agent based on the target credit value.
[0005] According to another aspect of this disclosure, an agent-based task processing method is provided, comprising: acquiring a GUI task; and employing a GUI agent to execute the GUI task; wherein the GUI agent is trained using the method described in any of the preceding claims.
[0006] According to another aspect of this disclosure, an agent training apparatus is provided, comprising: an acquisition module for acquiring an exploration trajectory generated by an agent, the exploration trajectory including samples of multiple steps, each sample including a GUI interaction action; a first determination module for determining the action information entropy of each step based on the GUI interaction action of each step; a second determination module for determining the credit weight of each step based on the action information entropy, wherein the credit weight and the action information entropy have a positive mapping relationship; a third determination module for determining the target credit value of each step based on the credit weight; and an adjustment module for adjusting the model parameters of the agent based on the target credit value.
[0007] According to another aspect of this disclosure, an agent-based task processing apparatus is provided, comprising: an acquisition module for acquiring a GUI task; and an execution module for executing the GUI task using a GUI agent; wherein the GUI agent is trained using the method described in any of the preceding claims.
[0008] According to another aspect of this disclosure, an intelligent agent is provided, comprising: an input module for acquiring a GUI task; a processing module for processing the GUI task to obtain a task processing result; and an output module for outputting the task processing result; wherein the GUI intelligent agent is trained using the method described in any of the preceding claims.
[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the preceding aspects.
[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.
[0011] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.
[0012] According to embodiments of this disclosure, the reinforcement learning effect of intelligent agents can be improved.
[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0015] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;
[0016] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;
[0017] Figure 3 This is a schematic diagram of the architecture of the agent training phase according to an embodiment of this disclosure;
[0018] Figure 4 This is a schematic diagram according to the third embodiment of the present disclosure;
[0019] Figure 5 This is a schematic diagram according to the fourth embodiment of the present disclosure;
[0020] Figure 6 This is a schematic diagram according to the fifth embodiment of the present disclosure;
[0021] Figure 7 This is a schematic diagram according to the sixth embodiment of the present disclosure;
[0022] Figure 8 This is a schematic diagram of an electronic device used to implement the agent training method or agent-based task processing method of the embodiments of this disclosure. Detailed Implementation
[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0024] To better understand this disclosure, the relevant terms are explained below:
[0025] A graphical user interface (GUI) is a computer user interface that uses graphics to display operations.
[0026] An intelligent agent is an entity capable of perceiving its environment and autonomously taking actions to achieve specific goals. It can be software, hardware, or a system. An intelligent agent is not merely a simple tool, but an intelligent agent that proactively adapts to its environment and learns to optimize its strategies. Current development of intelligent agents is based on large models, such as the Large Language Model (LLM), as its core intelligent foundation. Its architecture typically includes modules for memory, planning, tool usage, and action execution.
[0027] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. They are capable of generating natural language text or understanding the meaning of language text, and their parameter scale typically reaches billions to trillions. LLMs can perform operations based on prompts.
[0028] Prompts are text instructions entered by the user into the LLM (Large Model for Learning) to guide the model in generating specific, accurate, or expected outputs. They are the core way humans interact with large models, determining the direction in which the model understands the task and generates content. Specifically, they can include tasks, which explicitly tell the model what to do; context, which provides relevant information to help the model understand the scenario or conditions in which the task occurs; examples, which indicate the expected format or style of input and output; and roles, which indicate the role the model should play.
[0029] A GUI agent is an artificial intelligence system capable of understanding natural language instructions, perceiving GUI information, and automatically performing operations (such as clicking and inputting) to complete tasks. It primarily runs on desktop operating systems (such as Windows and Ubuntu) or the GUI interfaces of desktop applications, relying on visual / structured information such as screenshots and interface element coordinates to perceive the environment. By traversing the corners of the interface and accumulating GUI interaction sequences, it can achieve functions such as task discovery (e.g., automatically generating software operation flows), interface testing (e.g., detecting functional anomalies), or automated office work (e.g., batch processing of documents).
[0030] GUI agents are used to handle GUI tasks.
[0031] A GUI task refers to an operation performed on a GUI (Graphical User Interface). Examples include opening a browser and then clicking a button within it; or opening a folder in the operating system and then copying files from that folder.
[0032] GUI agents can be trained through reinforcement learning. Reinforcement learning policy optimization algorithms include, for example, proximal policy optimization (PPO) and group relative policy optimization (GRPO).
[0033] In reinforcement learning, exploration trajectories can be collected. These trajectories are typically sequences of trajectory points from multiple steps. Each trajectory point usually includes three types of information: state s, action a, and reward r.
[0034] For example, if the total number of steps is represented by T (a positive integer, usually greater than 1), then the exploration trajectory can be represented as: (s1, a1, r1, s2, a2, r2, ..., s T ,a T ,r T ), where (s k ,a k ,r k) represents the trajectory point of the k-th step (k is greater than or equal to 1 and less than or equal to T).
[0035] In GUI scenarios, the GUI agent interacts directly with the GUI environment and typically employs a sparse reward mechanism. This means that a positive reward (e.g., 1) is only given when the agent completes the final task node (e.g., "successfully placed an order and made payment"), while intermediate steps or failed trajectories are given zero rewards.
[0036] Based on the aforementioned sparse reward mechanism, there are problems such as the agent having difficulty learning effective strategies, the model failing to converge, and difficulties in credit assignment.
[0037] To address the problems caused by sparse reward mechanisms, heuristic step optimization or data filtering mechanisms can be introduced, mainly including:
[0038] When processing an exploration trajectory, calculate the action entropy of the agent at each step when generating instructions.
[0039] Set a static hard threshold, for example, when calculating the policy gradient and advantage function, directly mask and discard the 20% of trajectories with the lowest information entropy in the same batch.
[0040] Only high-entropy steps (i.e. steps with a flat probability distribution and high uncertainty in model prediction) are retained for the optimization and updating of model parameters, thereby forcing the reinforcement learning engine to concentrate its computing power on key decision bifurcation points.
[0041] The above method is a credit allocation mechanism based on hard entropy truncation, which has the following problems:
[0042] This disrupts the coherence of the Markov Decision Process (MDP): the success of a GUI task is determined by a series of consecutive actions. The above approach directly discards low-entropy steps (i.e., routine operations where the model exhibits high determinism, such as "click to confirm" after input, or simple "page scrolling"), severing the causal chain of state-action pairs. This prevents the model from performing reasonable gradient backpropagation on the fundamental environmental interactions essential for task success.
[0043] It is prone to "forgetting basic operations" and instability in the later stages of training: Because low-entropy routine operations are excluded from the optimization objective for a long time and do not receive positive or negative relative advantage guidance, the model is prone to the phenomenon of "only knowing the complex decision of the bifurcation point, but making mistakes when executing deterministic basic instructions" in long sequence complex tasks, causing the policy to converge to the second-best point, which seriously affects the robustness of online deployment.
[0044] Static truncation thresholds lack dynamic adaptability: the methods described above use a fixed percentage (e.g., hard-dropping 20%) for step filtering. However, in the actual training cycle of reinforcement learning, the agent's cognition evolves dynamically: the early action probability distribution generally tends to be flat (high entropy), while the later distribution tends to be sharp (low entropy). Fixed mask strategies cannot adapt to the agent's fine-grained credit allocation needs for "exploration" and "exploitation" at different training stages.
[0045] To address the aforementioned problems, namely to improve the reinforcement learning performance of intelligent agents, this disclosure provides the following embodiments.
[0046] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure. This embodiment provides a method for training an intelligent agent. Figure 1 As shown, the method includes:
[0047] 101. Obtain the exploration trajectory generated by the intelligent agent, the exploration trajectory includes samples of multiple steps, and the samples of each step include: GUI interaction actions.
[0048] 102. Based on the GUI interaction actions of each step, determine the action information entropy of each step.
[0049] 103. Based on the action information entropy, determine the credit weight of each step, and the credit weight and the action information entropy have a positive mapping relationship.
[0050] 104. Based on the credit weight, determine the credit value for each step.
[0051] 105. Based on the credit value, adjust the model parameters of the agent.
[0052] The method in this embodiment can be applied in a GUI scenario, and the aforementioned intelligent agent can be specifically referred to as a GUI intelligent agent.
[0053] An exploration trajectory refers to a trajectory generated by an agent to describe the execution process of a target task. Specifically, the target task can be carried in a prompt message and input into the GUI agent. The agent generates an exploration trajectory to execute the target task based on the prompt message.
[0054] In a GUI scenario, the aforementioned target tasks are specifically GUI tasks, such as operating system interface navigation tasks, cross-application graphical interface operation tasks, or, for complex software, such as the interface tasks of the intelligent guidance module of an office suite or enterprise resource planning (ERP) system.
[0055] Assuming the total number of steps in the target task is T (a positive integer greater than 1), the exploration trajectory is a sample of T steps. Each step's sample includes actions, and may also include states, reward values, etc.
[0056] Taking a sample for each step, including both state and action, as an example, the exploration trajectory can be represented as:
[0057] (s1,a1,s2,a2,...,s T ,a T ).
[0058] In a GUI scenario, the samples corresponding to state s and action a are: GUI environment state and GUI interaction action, respectively. GUI environment state includes, for example, basic interface information (such as the currently active window, page path, etc.), control state (such as whether an input box is clickable, etc.), and / or interface element layout information (such as the coordinates and size of key controls, etc.). GUI interaction actions include, for example, clicking, inputting, and / or, dropdown, etc.
[0059] For each step, the action information entropy of each step can be calculated.
[0060] Action information entropy is used to characterize the uncertainty of an action. That is, for a certain step, the higher the uncertainty of the action in that step, the greater the action information entropy.
[0061] Among them, the motion information entropy can be obtained according to the preset calculation formula of motion information entropy.
[0062] After obtaining the action information entropy for each step, the credit weight of each step is calculated based on the action information entropy, and then the target credit value for the corresponding step is determined according to the information weight.
[0063] Credit weights are used to assign credit to each action. A higher credit weight indicates that more credit is assigned to the corresponding action.
[0064] In reinforcement learning, credit refers to the degree of causal contribution of a certain action (or state) in a time-series decision-making process to the final reward outcome; that is, a certain action (or state) leads to the final success or failure to multiple degrees.
[0065] The target credit score refers to the final credit score used to characterize the credit of an action. The higher the credit of an action (i.e., the greater its contribution to the final result), the higher the target credit score.
[0066] In reinforcement learning, an objective function can be constructed based on the reward value, and the model parameters can be adjusted based on the objective function; or, some reinforcement learning algorithms (such as PPO or GRPO) construct an objective function based on the advantage value, and then adjust the model parameters.
[0067] Based on this, the target credit score can specifically be the target reward score or the target advantage score.
[0068] Taking the target credit score as the target advantage score as an example, the initial advantage score can be calculated using a common algorithm (such as PPO or GRPO), and then the initial advantage score can be weighted using credit weights to obtain the target advantage score.
[0069] After obtaining the target credit score, the model parameters of the agent can be adjusted using the target credit score.
[0070] For example, an objective function can be constructed based on the target advantage value, and the model parameters of the agent can be adjusted by maximizing the objective function.
[0071] Specifically, the model parameters can be the model parameters of a large model in the agent, such as a large language model (LLM).
[0072] In this embodiment, the credit weight of the corresponding step is determined based on the action information entropy of each step, which can realize the dynamic allocation of credit for each step, rather than hard truncation or equal distribution, thus achieving more flexible and effective credit allocation. This can ensure the coherence of the Markov decision process, improve robustness and more refined credit allocation, thereby improving the reinforcement learning effect of the agent and thus improving the performance of the agent, such as improving the success rate of the agent in executing GUI tasks.
[0073] Figure 2 This is a schematic diagram based on the second embodiment of the present disclosure, which provides a method for training an intelligent agent. The relevant steps are described in detail in this embodiment.
[0074] like Figure 2 As shown, the method includes:
[0075] 201. Obtain the exploration trajectory generated by the intelligent agent, the exploration trajectory includes samples of multiple steps, and the samples of each step include: GUI interaction actions.
[0076] In addition to the actions mentioned above, the samples for each step may also include the state and the initial credit value. Taking the initial credit value as the initial advantage value as an example, assuming there are T steps in total, the exploration trajectory can be represented as (s1, a1, A1, s2, a2, A2, ..., s T ,a T A T ), s k a k A k It represents the GUI environment state, GUI interaction actions, and initial advantage value at the k-th step.
[0077] Specifically, prompts for the target task can be input into the intelligent agent; the intelligent agent then executes the target task based on the prompts to output the exploration trajectory.
[0078] For example, if the target task is a navigation task, such as "find a specific hidden file in the file manager and modify its read and write permissions", the target task can be included in a prompt message and input to the agent. The agent will then perform the corresponding operation based on the prompt message, and the output will be the exploration trajectory. This can include the actions performed in each step. For example, the first step is to open the graphical interface of the system file manager, the second step is to show hidden files in the file manager, the third step is to click on the hidden file and a menu will pop up, the fourth step is to click on "properties" in the menu, and the fifth step is to modify the read and write permissions based on the properties.
[0079] In this way, the agent can explore based on the prompts and thus obtain the exploration trajectory.
[0080] Furthermore, the intelligent agent can be used to execute the target task multiple times in parallel based on the prompting information, so as to output multiple exploration trajectories in parallel.
[0081] For example, an agent can control multiple replicas of the environment in parallel, executing the target task in each replica in parallel, thereby obtaining multiple exploration trajectories in parallel. For instance, G (positive integers greater than 1) exploration trajectories can be generated in parallel. Then, based on these G exploration trajectories, a target function can be constructed using the GRPO method, and model parameters can be adjusted.
[0082] In this way, by generating multiple exploration trajectories in parallel, processing efficiency can be improved.
[0083] 202. Based on the GUI interaction actions of each step, determine the action information entropy of each step.
[0084] Among them, action information entropy is used to characterize the uncertainty of an action; that is, for a certain step k, if the action a in that step... k The higher the uncertainty, the greater the action information entropy. Generally speaking, routine actions (such as "click to confirm" after input or simple "page scrolling") have higher certainty and lower corresponding action information entropy. Conversely, actions with difficult decision-making (such as clicking on attributes of multi-level menus) have higher uncertainty and higher corresponding action information entropy.
[0085] Specifically, the tokens contained in the GUI interaction actions of each step can be obtained; the entropy of the action information can be determined based on the probability that the tokens belong to each token in a preset vocabulary.
[0086] Among them, the token is the basic unit of large model processing. Each action can be converted into one or more tokens. Specifically, the tokens contained in each action can be obtained according to the preset relationship between actions and tokens.
[0087] Each action can include one or more lexical units. For each lexical unit, the probability that the lexical unit belongs to each lexical unit in the preset vocabulary can be obtained, thereby determining the action information entropy.
[0088] In this way, the entropy of action information can be obtained simply and efficiently by measuring the probability that the lexical elements contained in the action belong to the lexical elements in the preset vocabulary.
[0089] Furthermore, a preset function is used to calculate the probability to obtain the action information entropy; wherein the preset function indicates that the probability and the action information entropy have an inverse mapping relationship.
[0090] Based on this, the higher the probability mentioned above, the lower the action information entropy, that is, the stronger the action certainty, the lower the corresponding action information entropy. Conversely, the stronger the action uncertainty, the higher the corresponding action information entropy, thus obtaining the action information entropy more reasonably.
[0091] Action information entropy can be calculated using a preset formula, which may be:
[0092]
[0093] Among them, H k It is the entropy of the action information in the k-th step;
[0094] N is the action a in the k-th step. kThe number of tokens contained, where i represents the i-th token; where a token is the basic unit of processing in the large model, each action can be converted into one or more tokens, and the number of tokens contained in each action can be obtained according to the preset relationship between actions and tokens;
[0095] V is the vocabulary size, which is the total number of tokens in the vocabulary used by the large model. It is a known fixed value, and v represents the v-th token in the vocabulary.
[0096] This represents the probability that the i-th token generated in the k-th step is the v-th token in the vocabulary;
[0097] log represents the logarithmic operation.
[0098] Therefore, based on the above calculation formula, the action information entropy of each step (k=1~T) can be obtained.
[0099] In this way, the "decision uncertainty" of a GUI agent when facing a specific interactive interface can be precisely quantified mathematically through action information entropy. High entropy indicates that the step is a complex decision-making fork (such as searching for a specific deep menu), while low entropy indicates that the step is a deterministic routine operation (such as clicking the confirmation button).
[0100] 203. Using a preset mapping function, determine the credit weight based on the action information entropy; the mapping function indicates that the credit weight and the action information entropy have a positive mapping relationship.
[0101] Among them, a mapping function can be pre-constructed, such as w=f(x), where w is the credit weight, x is the action information entropy, and f indicates that w and x have a positive mapping relationship, that is, the larger x is, the larger w is, and the smaller x is, the smaller w is. The credit weight can be calculated based on f(x).
[0102] Based on this, for a certain step k, the greater its action information entropy, that is, the higher the uncertainty of the action ak in that step, the greater the credit weight of that step, that is, assigning a higher weight to actions with high uncertainty.
[0103] In this embodiment, a preset mapping function is used to determine the credit weight based on the action information entropy, which can obtain the credit weight simply and efficiently.
[0104] Furthermore, the mapping function includes a temperature coefficient, and the temperature coefficient has an inverse mapping relationship with the current iteration number.
[0105] That is, the mapping function mentioned above not only includes the action information entropy x, but also the current iteration number t, and w and t have an inverse mapping relationship.
[0106] Based on action information entropy and temperature coefficient, the index value corresponding to each step can be determined based on the ratio of action information entropy to temperature coefficient; the index mean can be determined based on the index value corresponding to each step; and for the current step among the multiple steps, the ratio of the index value corresponding to the current step to the index mean can be determined as the credit weight of the current step.
[0107] That is, the average exponent can be calculated first based on the action information entropy and temperature coefficient of each step; the current step mentioned above can be any of multiple steps, that is, if the current step is represented by k, the credit weight of step k can be calculated based on the exponent value corresponding to k and the average exponent mentioned above. When k is different, such as k=1...T, the credit weight of each step can be obtained.
[0108] In this way, by determining credit weights based on the index mean, we can consider exploring the overall information of the trajectory, and then reallocate credit at each step based on the overall information, thereby improving flexibility and dynamism.
[0109] The specific formula for calculating the mapping function can be:
[0110]
[0111] in, It is the credit weight of the k-th step; exp is the exponential operation;
[0112] It is the entropy of the action information in the k-th step. It is the entropy of the action information in the j-th step;
[0113] L is the length of each exploration track (as T mentioned above), or it can be the total length of multiple exploration tracks in each batch;
[0114] It is a temperature coefficient, which can be set to a fixed value, or it can be determined based on the current iteration number t, and has an inverse mapping relationship with the current iteration number t, that is, the larger t is, the higher the temperature coefficient. The smaller the value, the more specific the reverse mapping relationship can be set according to the actual situation.
[0115] In related technologies, processing is usually based on a fixed entropy threshold (such as filtering out 20% of low-entropy steps), which can cause the Markov decision process to break down.
[0116] This embodiment utilizes the entropy of L steps for exponential mapping, preserving the gradient of the entire sequence and achieving smooth weight allocation. Furthermore, the aforementioned temperature coefficient has an inverse mapping relationship with the current iteration number, enabling fine-grained credit weighting, such as in the early stages of training (…). Larger weights and smoother weight distribution encourage the model to consolidate all basic GUI operations; in the later stages of training ( (Reducing), the weights rapidly converge on the high-entropy "critical decision steps," guiding computing power to concentrate on overcoming difficult nodes.
[0117] When obtaining credit weights, the initial credit value can be weighted using credit weights to obtain the target credit value.
[0118] Taking the target credit score as the target advantage score as an example, the following can be executed:
[0119] 204. Using the credit weight, the initial advantage values of each step are weighted to obtain the target advantage value.
[0120] The target advantage value can be obtained by multiplying the credit weight by the initial advantage value.
[0121] The calculation formula can be as follows:
[0122]
[0123] in, It is the target advantage value for the k-th step;
[0124] A k This is the initial advantage value for the k-th step. The initial advantage value can be obtained using a general algorithm, such as the common method of PPO to obtain the initial advantage value for each step; as mentioned above, the exploration trajectory contains the initial advantage value for each step, so the initial advantage value can be obtained from the exploration trajectory.
[0125] In this way, initial advantage values can be obtained efficiently based on the exploration trajectory, thus improving processing efficiency.
[0126] w k It is the credit weight over k steps.
[0127] The target advantage value can be obtained based on the above formula.
[0128] In this way, the target advantage value is obtained by weighting. Unlike the traditional "shared advantage on the same trajectory" or "advantage decays linearly over time", this embodiment can accurately and proportionally transmit the delayed reward caused by the long sequence back to the GUI operation steps that really play a decisive role in the success of the task.
[0129] 205. Based on the target advantage value, adjust the model parameters of the agent.
[0130] Specifically, this may include:
[0131] Based on the target advantage value and the pruning function, construct the target function;
[0132] Based on the objective function, the model parameters of the agent are adjusted.
[0133] Furthermore, the sample for each step also includes: the GUI environment state; based on this, the action probability ratio of the GUI interaction action in the GUI environment state can be obtained; a first parameter is determined based on the action probability ratio and the target advantage value; the action probability ratio is truncated using the pruning function, and a second parameter is determined based on the truncated ratio and the target advantage value; the target function is constructed based on the first parameter and the second parameter.
[0134] In this way, when constructing the objective function, the action probability ratio and objective advantage value mentioned above can be referenced, which makes it easier for the agent to learn more effective strategies.
[0135] The objective function can be formulated as follows:
[0136]
[0137] in, It is the objective function;
[0138] This refers to the adjusted model (such as the large model in an agent). These are its model parameters; This represents the model before adjustment;
[0139] It is the importance sampling ratio, which represents the ratio of the action probability of the new strategy to that of the old strategy;
[0140] These are the action and state of the k-th step, respectively.
[0141] It is the target advantage value for the k-th step;
[0142] `clip` is the clipping function. It is a preset cutoff threshold;
[0143] min is the minimum value operation;
[0144] E is the expected value calculation. Specifically, for the current iteration, the mean of the relevant data calculated based on the above formula for all steps of all exploration trajectories in the current iteration can be used as the expected value.
[0145] After constructing the objective function described above, the model parameters can be adjusted by maximizing the objective function.
[0146] In this way, through the above pruning function, the parameter updates of the policy network (agent) are still restricted within the trust region after dynamic information entropy redistribution, ensuring the stability and monotonic convergence of online training for long sequence GUI tasks.
[0147] In addition, regarding the specific architecture, Figure 3 This is a schematic diagram of the architecture of the training phase of an agent according to an embodiment of this disclosure.
[0148] like Figure 3 As shown, during the training phase, compared to the usual training method, in addition to the model parameter adjustment module 301, the training system 300 introduces a credit dynamic allocation module 302 in this embodiment. The credit dynamic allocation module 302 can determine the credit weight of each step based on the action information entropy of each step, and determine the target credit value of each step based on the credit weight, thereby realizing the dynamic allocation of action credit for each step. Thus, the model parameter adjustment module 301 can construct an objective function based on the target credit value of each step, and adjust the model parameters according to the objective function.
[0149] In this embodiment, credit is dynamically allocated at each step, demonstrating high versatility and generalization ability. This solves the credit allocation problem in long-sequence, sparse reward environments without requiring an additional state value evaluation network, making it particularly suitable for complex automated control scenarios driven by various multimodal large models.
[0150] The training process described above will be explained below using a specific GUI task as an example:
[0151] Objective: Cross-OS and Desktop GUI Navigation.
[0152] Application scenario: General-purpose computer-use agents that need to execute complex cross-application commands within a real desktop operating system (such as Ubuntu).
[0153] Based on the above training methods, including:
[0154] Suppose an agent is performing a navigation task: "locating a specific hidden file in a file manager and modifying its read / write permissions." This interaction sequence often involves dozens of steps. Simple mouse movements or scrolling through a file list exhibit high determinism and extremely low information entropy in the action distribution. However, accurately identifying and clicking the "properties-permissions" option in a multi-level menu introduces high uncertainty in the policy network, resulting in a sudden increase in information entropy. Applying the algorithm of this invention, the system can precisely focus on and proportionally amplify the global sparse reward for successfully modifying permissions through continuous annealing soft weights, feeding it back to high-entropy key decision steps such as "clicking an attribute." Simultaneously, the system assigns smaller non-zero weights to low-entropy steps such as "scrolling through the list," preserving the coherence of the Markov decision process, thereby significantly improving the model's convergence efficiency and generalization ability in large operating system navigation tasks.
[0155] Figure 4 This is a schematic diagram based on a third embodiment of the present disclosure, which provides a task processing method based on an intelligent agent. For example... Figure 4 As shown, the method includes:
[0156] 401. Obtain GUI task.
[0157] 402. Use a GUI agent to execute the GUI task.
[0158] The GUI agent is trained using any of the above embodiments.
[0159] For example, GUI tasks can be input into a GUI agent. The GUI agent can execute the action of the current step based on the state of the current step (such as a screenshot of the GUI environment provided by other services). After the action of the current step is executed, a new state can be generated, which will serve as the state of the next step. In this way, the GUI task can be completed step by step.
[0160] In this embodiment, since the GUI agent described above has good performance, the success rate of GUI task execution can be improved when the GUI agent is used to perform GUI tasks.
[0161] Figure 5 This is a schematic diagram based on the fourth embodiment of the present disclosure, which provides an intelligent agent training device. For example... Figure 5 As shown, the device 500 includes: an acquisition module 501, a first determination module 502, a second determination module 503, a third determination module 504, and an adjustment module 505.
[0162] The acquisition module 501 is used to acquire the exploration trajectory generated by the agent, the exploration trajectory includes samples of multiple steps, and the samples of each step include: GUI interaction actions; the first determination module 502 is used to determine the action information entropy of each step based on the GUI interaction actions of each step; the second determination module 503 is used to determine the credit weight of each step based on the action information entropy, and the credit weight and the action information entropy have a positive mapping relationship; the third determination module 504 is used to determine the target credit value of each step based on the credit weight; the adjustment module 505 is used to adjust the model parameters of the agent based on the target credit value.
[0163] In this embodiment, the credit weight of the corresponding step is determined based on the action information entropy of each step, which can realize the dynamic allocation of credit for each step, rather than hard truncation or equal distribution, thus achieving more flexible and effective credit allocation. This can ensure the coherence of the Markov decision process, improve robustness and more refined credit allocation, thereby improving the reinforcement learning effect of the agent and thus improving the performance of the agent, such as improving the success rate of the agent in executing GUI tasks.
[0164] In some embodiments, the second determining module 503 is further configured to:
[0165] The credit weight is determined based on the action information entropy using a preset mapping function; the mapping function indicates that the credit weight and the action information entropy have a positive mapping relationship.
[0166] In this embodiment, a preset mapping function is used to determine the credit weight based on the action information entropy, which can obtain the credit weight simply and efficiently.
[0167] In some embodiments, the mapping function includes a temperature coefficient, and the temperature coefficient is inversely mapped to the current iteration number.
[0168] This embodiment utilizes the entropy of L steps for exponential mapping, preserving the gradient of the entire sequence and achieving smooth weight distribution. Furthermore, the aforementioned temperature coefficient has an inverse mapping relationship with the current iteration number, enabling fine-grained credit weighting. For example, in the early stages of training, the weight distribution is smooth, encouraging the model to consolidate all basic GUI operations; in the later stages of training, the weights rapidly converge towards the high-entropy "critical decision steps," guiding computing power to concentrate on overcoming difficult nodes.
[0169] In some embodiments, determining the credit weight based on the action information entropy using a preset mapping function includes:
[0170] Based on the ratio of the action information entropy of each step to the temperature coefficient, the index value corresponding to each step is determined.
[0171] Based on the index values corresponding to each of the aforementioned steps, the index mean is determined.
[0172] For the current step among the multiple steps, the ratio of the index value corresponding to the current step to the mean index is determined as the credit weight of the current step.
[0173] In this way, by determining credit weights based on the index mean, we can consider exploring the overall information of the trajectory, and then reallocate credit at each step based on the overall information, thereby improving flexibility and dynamism.
[0174] In some embodiments, the credit score is a target advantage score;
[0175] The third determining module 504 is further used for:
[0176] The initial advantage values of each step are weighted using the credit weight to obtain the target advantage value.
[0177] In this embodiment, the target advantage value is obtained by weighting. Unlike the traditional "shared advantage on the same trajectory" or "advantage decays linearly over time", this embodiment can accurately and proportionally transmit the delayed reward caused by the long sequence back to the GUI operation steps that truly play a decisive role in the success of the task.
[0178] In some embodiments, the sample of each step of the exploration trajectory further includes an initial advantage value; the device further includes an extraction module for obtaining the initial advantage value of each step from the exploration trajectory.
[0179] In this way, initial advantage values can be obtained efficiently based on the exploration trajectory, thus improving processing efficiency.
[0180] In some embodiments, the adjustment module 505 is further configured to:
[0181] Based on the target advantage value and the pruning function, construct the target function;
[0182] Based on the objective function, the model parameters of the agent are adjusted.
[0183] In this embodiment, through the aforementioned pruning function, the parameter updates of the policy network (agent) are still restricted within the Trust Region after dynamic information entropy redistribution, ensuring the stability and monotonic convergence of online training for long-sequence GUI tasks.
[0184] In some embodiments, the sample for each step further includes: GUI environment state; the adjustment module 505 is further configured to:
[0185] Obtain the probability ratio of the GUI interaction action in the GUI environment state;
[0186] The first parameter is determined based on the action probability ratio and the target advantage value;
[0187] The action probability ratio is clipped using the clipping function, and a second parameter is determined based on the clipped ratio and the target advantage value.
[0188] The objective function is constructed based on the first parameter and the second parameter.
[0189] In this way, when constructing the objective function, the action probability ratio and objective advantage value mentioned above can be referenced, which makes it easier for the agent to learn more effective strategies.
[0190] In some embodiments, the acquisition module 501 is further configured to:
[0191] Input prompts for the target task into the intelligent agent;
[0192] The intelligent agent is used to perform the target task based on the prompt information, so as to output the exploration trajectory.
[0193] In this way, the agent can explore based on the prompts and thus obtain the exploration trajectory.
[0194] In some embodiments, the acquisition module 501 is further configured to:
[0195] The intelligent agent is used to execute the target task multiple times in parallel based on the prompt information, so as to output multiple exploration trajectories in parallel.
[0196] In this way, by generating multiple exploration trajectories in parallel, processing efficiency can be improved.
[0197] In some embodiments, the first determining module 502 is further configured to:
[0198] Obtain the lexical units contained in the GUI interaction actions of each step; determine the action information entropy based on the probability that the lexical units belong to each lexical unit in the preset vocabulary.
[0199] In this way, the entropy of action information can be obtained simply and efficiently by measuring the probability that the lexical elements contained in the action belong to the lexical elements in the preset vocabulary.
[0200] Figure 6 This is a schematic diagram based on the fifth embodiment of the present disclosure, which provides a task processing device based on an intelligent agent. For example... Figure 6 As shown, the device 600 includes an acquisition module 601 and an execution module 602.
[0201] The acquisition module 601 is used to acquire GUI tasks; the execution module 602 is used to execute the GUI tasks using a GUI agent; wherein the GUI agent is trained using any of the above embodiments.
[0202] In this embodiment, since the GUI agent described above has good performance, the success rate of GUI task execution can be improved when the GUI agent is used to perform GUI tasks.
[0203] Figure 7 This is a schematic diagram based on the sixth embodiment of the present disclosure, which provides an intelligent agent. For example... Figure 7 As shown, the intelligent agent 700 includes: an input module 701, a processing module 702, and an output module 703.
[0204] The input module 701 is used to acquire GUI tasks; the processing module 702 is used to process the GUI tasks to obtain task processing results; the output module 703 is used to output the task processing results; wherein, the GUI agent is trained using any of the above embodiments.
[0205] For example, a GUI task can be input into a GUI agent. The GUI agent can then execute the action for the current step based on its current state (such as a screenshot of the GUI environment provided by another service). The execution of the current step generates a new state, which serves as the state for the next step. This allows the GUI task to be completed step-by-step, yielding a result such as success or failure. The result can then be fed back to the user, or the actions performed at each step can also be provided to the user.
[0206] In this embodiment, because the aforementioned GUI agent has good performance, the success rate of GUI task execution can be improved when performing GUI tasks based on this GUI agent.
[0207] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.
[0208] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.
[0209] It is understandable that, unless otherwise specified, the order of steps in the process indicates that the temporal relationship between these steps is not limited.
[0210] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.
[0211] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0212] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0213] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0214] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0215] The computing unit 801 can be a variety of general-purpose and / or proprietary processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various proprietary artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as agent training methods or agent-based task processing methods. For example, in some embodiments, the agent training method or agent-based task processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the agent training method or agent-based task processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform an agent training method or an agent-based task processing method.
[0216] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), proprietary integrated circuits (ASICs), proprietary standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a proprietary or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0217] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable task processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0218] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0219] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0220] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0221] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0222] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0223] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for training an agent, comprising: The exploration trajectory generated by the intelligent agent is obtained. The exploration trajectory includes samples of multiple steps, and the samples of each step include: GUI interaction actions; Based on the GUI interaction actions at each step, determine the action information entropy of each step; Based on the action information entropy, the credit weight of each step is determined, and the credit weight and the action information entropy have a positive mapping relationship. Based on the credit weight, the target credit value for each step is determined; The model parameters of the agent are adjusted based on the target credit value.
2. The method according to claim 1, wherein, Determining the credit weight of each step based on the action information entropy includes: The credit weight is determined based on the action information entropy using a preset mapping function; the mapping function indicates that the credit weight and the action information entropy have a positive mapping relationship.
3. The method according to claim 2, wherein, The mapping function includes a temperature coefficient, and the temperature coefficient has an inverse mapping relationship with the current iteration number.
4. The method according to claim 3, wherein, The step of determining the credit weight based on the action information entropy using a preset mapping function includes: Based on the ratio of the action information entropy of each step to the temperature coefficient, the index value corresponding to each step is determined. Based on the index values corresponding to each of the aforementioned steps, the index mean is determined. For the current step among the multiple steps, the ratio of the index value corresponding to the current step to the mean index is determined as the credit weight of the current step.
5. The method according to claim 1, wherein, The target credit score is the target advantage score; The determination of the target credit value for each step based on the credit weight includes: The initial advantage values of each step are weighted using the credit weight to obtain the target advantage value.
6. The method according to claim 5, wherein, The sample for each step of the exploration trajectory also includes: an initial advantage value; The method further includes: obtaining initial advantage values for each step from the exploration trajectory.
7. The method according to claim 5, wherein, The step of using the credit weight to weight the initial advantage values of each step includes: The product of the credit weight and the initial advantage value is taken as the target advantage value.
8. The method according to claim 5, wherein, The step of adjusting the model parameters of the agent based on the target credit value includes: Based on the target advantage value and the pruning function, construct the target function; Based on the objective function, the model parameters of the agent are adjusted.
9. The method according to claim 8, wherein, The sample for each step also includes: GUI environment status; The construction of the objective function based on the target advantage value and the pruning function includes: Obtain the probability ratio of the GUI interaction action in the GUI environment state; The first parameter is determined based on the action probability ratio and the target advantage value; The action probability ratio is clipped using the clipping function, and a second parameter is determined based on the clipped ratio and the target advantage value. The objective function is constructed based on the first parameter and the second parameter.
10. The method according to any one of claims 1-9, wherein, The acquisition of the exploration trajectory generated by the intelligent agent includes: Input prompts for the target task into the intelligent agent; The intelligent agent is used to perform the target task based on the prompt information, so as to output the exploration trajectory.
11. The method according to claim 10, wherein, The step of employing the intelligent agent to perform the target task based on the prompting information and output the exploration trajectory includes: The intelligent agent is used to execute the target task multiple times in parallel based on the prompt information, so as to output multiple exploration trajectories in parallel.
12. The method according to any one of claims 1-9, wherein, The determination of the action information entropy for each step based on the GUI interaction actions at each step includes: Obtain the lexical units contained in the GUI interaction actions of each step; The entropy of the action information is determined based on the probability that the lexical belongs to each lexical in a preset vocabulary.
13. The method according to claim 12, wherein, The step of determining the action information entropy based on the probability that the word belongs to each word in a preset vocabulary includes: A preset function is used to calculate the probability to obtain the action information entropy; wherein, the preset function indicates that the probability and the action information entropy have an inverse mapping relationship.
14. A task processing method based on intelligent agents, comprising: Get GUI tasks; A GUI agent is used to execute the GUI task; The GUI agent is trained using the method described in any one of claims 1-13.
15. An agent training device, comprising: The acquisition module is used to acquire the exploration trajectory generated by the intelligent agent. The exploration trajectory includes samples of multiple steps, and the samples of each step include: GUI interaction actions. The first determining module is used to determine the action information entropy of each step based on the GUI interaction actions of each step. The second determining module is used to determine the credit weight of each step based on the action information entropy, and the credit weight and the action information entropy have a positive mapping relationship. The third determining module is used to determine the target credit value for each step based on the credit weight. An adjustment module is used to adjust the model parameters of the agent based on the target credit value.
16. A task processing device based on intelligent agents, comprising: The acquisition module is used to acquire GUI tasks; The execution module is used to execute the GUI task using a GUI agent; The GUI agent is trained using the method described in any one of claims 1-13.
17. An intelligent agent, comprising: The input module is used to obtain GUI tasks; The processing module is used to process the GUI task to obtain the task processing result; The output module is used to output the task processing results; The agent is trained using the method described in any one of claims 1-13.
18. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.
19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.
20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.