Intelligent agent control method and related devices

Abstract

Embodiments of the present disclosure disclose an agent control method and related equipment. The method comprises: obtaining a first performance index sequence of an agent in a first training window when the agent performs multi-round interaction with an environment module through a communication interface under a first single-task maximum interaction number; determining a first benchmark performance index based on the first performance index sequence, and determining a first threshold according to the first benchmark performance index; obtaining a second performance index sequence of the agent in a second training window when the agent performs multi-round interaction with the environment module under the first single-task maximum interaction number; and adjusting the first single-task maximum interaction number to a second single-task maximum interaction number according to the second performance index sequence and the first threshold. The scheme provided by the embodiments of the present disclosure can match the interaction depth with the real learning progress of the agent, and improve the training stability.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an intelligent agent control method, intelligent agent control device, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Currently, intelligent agents have evolved into multi-turn interactive decision-making systems. In multi-turn interactive tasks, the agent needs to interact with the environment in multiple steps to complete a complete task. The maximum number of interaction turns allowed for each task (i.e., the upper limit of task length) is a key hyperparameter affecting training performance.

[0003] In related technologies, there are two main ways to set the maximum number of interaction rounds: (1) Fixed maximum number of interaction rounds: Most frameworks in related technologies pre-set a fixed maximum number of interaction rounds (usually 10 steps) before training. Regardless of the complexity of the task, all training tasks are performed within this upper limit.

[0004] (2) Static increase of rounds: Some improvement schemes adopt a simple static increase strategy, such as increasing the maximum number of interaction rounds by a fixed value every N (N is a positive integer) training cycles (e.g., increasing 10 rounds every 10 epochs).

[0005] The two methods described above are used to constrain the exploration range of an agent in a single task. However, fixing the maximum number of interaction rounds leads to limited capabilities and wasted resources. A fixed maximum number of interaction rounds (e.g., 10 steps) cannot adapt to tasks of varying complexity: for complex long-range tasks requiring 30-100 steps, the agent cannot learn long-range planning capabilities due to insufficient interaction space; for simple tasks requiring only 3 steps, it generates a large number of invalid interactions, resulting in wasted computational resources. Furthermore, this method cannot generalize to task scenarios of varying complexity. Staticly increasing the number of rounds ignores learning progress, leading to training oscillations. Static strategies (e.g., adding 10 rounds every N epochs) rely solely on time or training cycles, without considering whether the agent has truly mastered the task at the current difficulty level. Forcibly increasing the number of interaction rounds before the agent has mastered basic skills will cause the policy search space to expand abruptly, leading to training oscillations, slow convergence, or even policy collapse; conversely, when the agent has mastered the current difficulty level, static strategies may fail to increase the number of rounds in a timely manner, resulting in low learning efficiency. Summary of the Invention

[0006] This disclosure provides an agent control method, agent control device, computer equipment, computer-readable storage medium, and computer program product, which can improve the efficiency and stability of agent training and save computing resources.

[0007] This disclosure provides an agent control method, comprising: acquiring a first performance index sequence within a first training window when the agent interacts with an environment module through a communication interface in multiple rounds under a first single-task maximum number of interactions; determining a first benchmark performance index based on the first performance index sequence, and determining a first threshold based on the first benchmark performance index; acquiring a second performance index sequence between the agent and the environment module within a second training window under the first single-task maximum number of interactions; and adjusting the first single-task maximum number of interactions between the agent and the environment module to a second single-task maximum number of interactions based on the second performance index sequence and the first threshold.

[0008] This disclosure provides an intelligent agent control device, comprising: a performance index acquisition unit, configured to acquire a first performance index sequence within a first training window when the intelligent agent interacts with an environment module through a communication interface in multiple rounds under a first single-task maximum number of interactions; a benchmark threshold determination unit, configured to determine a first benchmark performance index based on the first performance index sequence, and determine a first threshold based on the first benchmark performance index; the performance index acquisition unit is further configured to acquire a second performance index sequence between the intelligent agent and the environment module within a second training window under the first single-task maximum number of interactions; and an interaction count adjustment unit, configured to adjust the first single-task maximum number of interactions between the intelligent agent and the environment module to a second single-task maximum number of interactions based on the second performance index sequence and the first threshold.

[0009] This disclosure provides a computer device including a processor, a memory, and an input / output interface. The processor is connected to both the memory and the input / output interface. The input / output interface is used to receive and output data. The memory is used to store a computer program. The processor is used to invoke the computer program to cause the computer device containing the processor to execute the method in any embodiment of this disclosure.

[0010] This disclosure provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, such that a computer device having the processor performs the methods of any embodiment of this disclosure.

[0011] This disclosure provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various alternative embodiments of this disclosure. Attached Figure Description

[0012] Figure 1 This is an architecture diagram of a computer system provided in an embodiment of this disclosure.

[0013] Figure 2 This is a flowchart of an intelligent agent control method provided in an embodiment of this disclosure.

[0014] Figure 3 This is a flowchart of another intelligent agent control method provided in the embodiments of this disclosure.

[0015] Figure 4 This is a block diagram of an intelligent agent control device provided in an embodiment of this disclosure.

[0016] Figure 5 This is a schematic diagram of the architecture of an intelligent agent training system provided in an embodiment of this disclosure.

[0017] Figure 6 This is a schematic diagram illustrating the application of the intelligent agent control method provided in this embodiment to the field of robotics.

[0018] Figure 7 This is a schematic diagram illustrating the application of the intelligent agent control method provided in this embodiment to the field of intelligent question answering.

[0019] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0022] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0024] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0025] Figure 1 This is a schematic diagram illustrating the implementation environment of an intelligent agent control method and an intelligent agent application method provided in this application. For example... Figure 1 As shown, the implementation environment includes a computer device 101, which may be a server or a terminal. In this embodiment, the intelligent agent control method of this application can be executed through the computer device 101.

[0026] During the training phase, the agent can interact with one or more environmental modules in multiple rounds, continuously optimizing its strategy under a dynamically variable constraint on the maximum number of interactions per single task. These environmental modules include, but are not limited to, one or more of the following: network navigation environments (such as web browsing, online shopping, software collaboration, etc.), deep search environments (such as multi-round information retrieval and integration), digital game environments (such as text-based sandbox games), embodied task environments (such as grid world navigation and operation), and scientific experimental environments (such as instrument operation and measurement in a virtual laboratory). During training, the method provided in this embodiment adaptively adjusts the maximum number of interactions per single task (e.g., the second maximum number of interactions per single task) for subsequent tasks based on the agent's real-time performance index sequence (e.g., the first performance index sequence and / or the second performance index sequence) at the current interaction depth (e.g., the maximum number of interactions per single task). This allows the agent to gradually encounter more complex tasks after mastering the current difficulty level, thereby training an agent with efficient long-range planning capabilities and a high task success rate.

[0027] In the application phase, the trained agent can be deployed to any of the aforementioned scenarios and autonomously execute multi-round interactive decisions based on user-inputted task instructions or environmental feedback. Simultaneously, the system supports users inputting intervention events or adjusting task objectives at any time. The agent can flexibly control the number of interaction rounds based on the current interaction depth limit, avoiding unnecessary resource consumption or premature termination, significantly improving the agent's adaptability and controllability in practical applications.

[0028] In one application scenario, pre-trained agents can be used to perform automated testing or performance evaluation of target systems (such as e-commerce websites, search engines, game platforms, robot control software, etc.). For example, during the pre-launch testing phase, agents can simulate real user interactions and, within a dynamically adjusted upper limit of interaction depth, explore potential logical vulnerabilities, response anomalies, or performance bottlenecks in the system. This allows for early detection and fix of problems, ensuring system quality and stability.

[0029] In another application scenario, in tasks requiring human-machine collaboration (such as assisted search, intelligent customer service, game companionship, etc.), users can input instructions at any time to adjust the behavior strategy of the intelligent agent. The intelligent agent will respond to the user's intent efficiently, based on the maximum number of interaction rounds allowed at the moment (i.e., the maximum number of interactions), while ensuring the completion of the task, thus achieving flexible and controllable human-machine collaboration.

[0030] The server can be an independent physical server, a cluster of multiple physical servers, or a distributed system. It can also be a cloud server or server cluster providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, in-vehicle terminal, smart speaker, smartwatch, etc. The terminal and server can be directly or indirectly connected via wired or wireless communication. The specific connection method can be determined according to the actual application scenario, and this application does not impose any limitations.

[0031] Figure 2 This is a flowchart of an intelligent agent control method provided in an embodiment of this disclosure. The control method provided in this embodiment can be used to adaptively adjust the interaction depth of the intelligent agent. The intelligent agent in this embodiment refers to an autonomous decision-making system built based on Large Language Model(s) (LLM(s)) or other machine learning algorithms, capable of perceiving environmental states, executing specific actions, and optimizing strategies based on environmental feedback. The intelligent agent can gradually learn the ability to complete complex tasks through multiple rounds of interaction with the environment.

[0032] For example, the intelligent agent in this disclosure embodiment has the following capabilities: Sensing capability: Receives status data from the environment module through a communication interface, including but not limited to task instructions, environmental observations, and interaction history; Decision-making and execution capabilities: Determine the current state based on the received state data, and generate and execute a sequence of actions based on the current state and task objectives. These actions include, but are not limited to, text output, tool calls, API (Application Programming Interface) requests, or physical control commands. Policy optimization capability: In multiple rounds of interaction with the environment, based on the reward signals or performance indicators returned by the environment module, the internal model parameters are updated through reinforcement learning algorithms (such as PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), REINFORCE++, etc.) to maximize the success rate of long-term tasks. Interaction depth adaptation: During training, the agent is constrained by the maximum number of interactions per single task currently configured. This number can be dynamically adjusted based on the agent's real-time performance score, enabling the agent to gradually transition from simple tasks to complex long-term tasks.

[0033] The agent learns the ability to complete complex tasks through multiple rounds of interaction with the environment module. Its learning progress is quantitatively represented by a sequence of performance indicators collected in real time and used to drive the adaptive adjustment of the interaction depth.

[0034] like Figure 2 As shown, this embodiment uses the method applied to computer devices (including terminals and / or servers) as an example for illustration. The method includes the following steps.

[0035] In S110, the sequence of first performance indicators within the first training window is obtained when the agent interacts with the environment module through the communication interface in multiple rounds under the maximum number of interactions in the first single task.

[0036] The maximum number of interactions per task in this embodiment (including the maximum number of interactions per first task, the maximum number of interactions per second task, the maximum number of interactions per third task, etc.) is also called the maximum number of interactions (max_turn), which refers to the upper limit of the number of rounds that the agent is allowed to interact with the environment module (including the environment or external tools) during a single task processing. If this upper limit of rounds is exceeded, the execution of the single task will automatically terminate. In this embodiment, the initial value of this parameter is set to, for example, 10 rounds (this is only for illustrative purposes and is not limited to this), and is dynamically adjusted according to the agent's performance during the agent's training process.

[0037] In this embodiment, a single task (episode) refers to a complete interaction process experienced by the agent from receiving a specific target instruction or initial state until the target is determined to be completed, failed, or the maximum number of interactions for a single task is reached. During this process, the agent interacts with the environment module in multiple rounds (i.e., multiple "observation-thinking-action" cycles). Each interaction (performing an operation or action and receiving feedback from the environment module) is called a round, meaning a single task can contain multiple rounds of interaction. The environment states between different single tasks are usually reset and do not affect each other. During the execution of each single task, the number of rounds of interaction between the agent and the environment module cannot exceed the current maximum number of interactions for that single task. If the task succeeds or fails before reaching the current maximum number of interactions for that single task, the single task ends early; if it does not end after reaching the current maximum number of interactions for that single task, it is forcibly terminated and judged as a failure (or penalized). This embodiment dynamically adjusts the maximum number of interactions for subsequent single tasks by real-time detection of the agent's performance (performance index sequence) in multiple single tasks, achieving progressive training from easy to difficult.

[0038] For example, in a web navigation task, suppose the task instruction is "Buy a mobile phone on an e-commerce website, with a price not exceeding 5000 yuan." The starting point for the agent to execute the task is that the agent opens the browser homepage and sees the search box and product list. The process may include the agent performing a series of operations (clicking the search box, entering keywords, clicking filter conditions, selecting products, adding to the shopping cart, and submitting the order). After each operation / action is performed, the environment module returns a new page state as the current state. If the order is successfully submitted, the page displays "Order generated." If the task fails to complete within a timeout period, enters an infinite loop after clicking an incorrect link, or encounters an error midway, the task fails. Alternatively, if the number of interaction rounds reaches the "maximum number of interactions per task" (e.g., 10 rounds) but the task is not successful, the task execution fails.

[0039] For example, in a deep search task, suppose the task instruction is "Please find out who the first woman to summit Mount Everest was." The agent's task execution begins when it receives the question but has no external information. The process may include the agent generating a search query (e.g., "the first woman to summit Mount Everest"), calling a search engine API, analyzing the search results, possibly refining the search (e.g., "the name of the first woman to summit Mount Everest"), and integrating information to form an answer. If the correct answer "XXX" is output, the task is successful. If an incorrect answer is given or the answer is uncertain, the task fails. Alternatively, if no answer is given after the maximum number of interactions (e.g., 4 rounds), the task fails.

[0040] For example, in a digital game task, suppose the task instruction is "Craft a stone axe". The agent starts by having initial resources (such as wood and stone) in the game world. The agent executes actions such as gathering resources, opening a crafting table, crafting sticks, and crafting a stone axe via text commands. After each command, the environment provides feedback on the current inventory state. If the agent successfully crafts a stone axe and it appears in the inventory, the task is successful. If the required resources are exhausted and cannot be obtained, or if an incorrect crafting recipe is executed, the task fails. Alternatively, if the maximum number of interactions (e.g., 20 rounds) is reached without crafting the axe, the task fails.

[0041] For example, in an embodied task, suppose the task instruction is "Go to the red box and open it." The agent's starting point for executing the task is that the agent is in an initial cell in the grid world, and the red box is in another cell further away. The agent outputs actions such as movement direction (up / down / left / right), opening doors, and pushing objects. The environment updates the agent's position and returns a new observation. If the agent moves to a cell adjacent to the red box and performs the "open" action, and the box is opened, the task is successful. If the agent hits a wall, falls into a trap, or exceeds the time limit, the task fails. Alternatively, if the agent fails to reach the box within the maximum number of interactions (e.g., 20 rounds), the task fails.

[0042] For example, in a scientific experiment task, suppose the task instruction is "measure the temperature of a glass of water." The agent's starting point for executing the task is that it is in a virtual laboratory with a glass of water and a thermometer on a table. The agent outputs commands / actions such as "pick up the thermometer," "put the thermometer in the water," and "read the thermometer reading." The environment returns the temperature reading. If the agent correctly reads the temperature (e.g., "23℃"), the task fails. If the thermometer is not calibrated, the glass of water is spilled, etc., the task fails. Alternatively, if the maximum number of interactions (e.g., 20 rounds) is not reached before completion, the task fails.

[0043] A single task is the basic unit for agent training and evaluation. In this embodiment, the maximum number of interactions per task is the hyperparameter that constrains the upper limit of the process rounds.

[0044] In this embodiment of the disclosure, the first training window refers to the set of consecutive interaction rounds used to collect the initial performance indicators of the agent. Exemplarily, the window length of the first training window is fixed. The first training window is also called the baseline consecutive steps (represented by K, where K is a positive integer), which refers to the initial training step length used to calculate the first baseline performance indicator (e.g., the baseline average score (avg_score)) given the current maximum number of interactions (max_turn, for example, taking the maximum number of interactions for the first single task as an example; after max_turn is updated, for example, the maximum number of interactions for the second single task becomes the current maximum number of interactions). For example, the default K=30 (this is only for illustrative purposes and is not limited to this). This parameter ensures that the calculation of the baseline average score has sufficient statistical stability.

[0045] The first performance index sequence in this embodiment refers to the performance indexes collected within the first training window and arranged in chronological order.

[0046] The performance metrics in this embodiment can also be referred to as performance scores, which are numerical indicators used to quantitatively evaluate the quality of the agent's task completion after each round of interaction between the agent and the environment module. For example, they can be calculated based on task achievement (task progress), accuracy, or environmental reward functions, and the score range can be, for example, [0, 100], with higher values ​​indicating better task completion quality.

[0047] In some embodiments, for tasks with explicit standard answers, the output of the agent after each round of interaction is compared with the standard answer using preset rules to automatically calculate performance metrics. For example, string equality matching can be used: when the agent's output is completely identical to the standard answer, a full score (e.g., 1.0) is awarded; partial matching (e.g., keyword overlap exceeding a threshold) receives a half-score; and no matching receives 0 points. This is suitable for generation tasks with fixed standard answers. For example, string inclusion matching can be used, such as calculating the proportion of keyword intersection and union based on Jaccard similarity to determine a continuous performance metric of 0-1. This is suitable for keyword-hitting generation tasks. For example, a combination of multi-dimensional rules can be used to extract and score instructions, compare answer information, and evaluate response quality across multiple rounds of dialogue. The final evaluation result is obtained by combining multi-dimensional scores, suitable for multi-round dialogue scenarios. For example, trajectory matching can be used to compare the agent's execution trajectory with a reference trajectory, providing four matching modes: strict matching, unordered matching, subset matching, and superset matching. This is suitable for verifying deterministic interaction processes. For example, an LLM can generate a list of evaluation criteria based on predefined criteria, and then quantitatively score task samples based on these criteria, which is suitable for open-ended task evaluation. This approach is computationally efficient, yields highly deterministic results, and is suitable for scenarios where task outcomes are predictable and evaluation criteria can be formalized.

[0048] In other embodiments, a reward model can be used to determine the performance metric. The reward model is a trained or designed neural network that takes state-action pairs from an agent as input and outputs a scalar score as the performance metric. This model can be trained, for example, on human preference data or expert trajectory data. This approach is more flexible and adaptable, suitable for evaluation scenarios with high semantic understanding requirements and those difficult to quantify through rules.

[0049] In some embodiments, performance metrics can be determined based on feedback from the environment module. When the agent interacts with the environment module, the environment module returns raw reward signals or task progress information, which are then converted into scores within the range [0, 100] as performance metrics by designing a mapping function. For example, scores can be mapped linearly or non-linearly based on the degree of task completion, suitable for scenarios where task progress can be quantified (such as game scores or navigation progress). For example, scores can be mapped based on a binary success / failure mechanism, such as rewarding 100 points for success and 0 points for failure, suitable for scenarios where the task outcome is clearly binary. For example, intermediate milestone rewards can be used, awarding a certain score for each completed sub-goal (dense rewards), suitable for complex tasks that can be decomposed into multiple steps. For example, in scientific experiment tasks, the experimental objective is decomposed into several key steps (such as retrieving instruments, measuring, and recording results), and a corresponding score is accumulated for each completed step. For example, reinforcement learning based on scoring anchors (RLVR) can be used to extend scoring-based rewards to open-ended tasks, using scoring criteria as anchors to provide fine-grained style control and performance feedback, suitable for open-ended tasks and generative tasks requiring style control. This approach directly utilizes the feedback signals from the environment itself, eliminating the need for additional model training.

[0050] In S120, a first benchmark performance index is determined based on the first performance index sequence, and a first threshold is determined based on the first benchmark performance index.

[0051] In this embodiment, the first benchmark performance metric represents the agent's baseline performance level under the current task difficulty. The first benchmark performance metric can be the arithmetic mean of a sequence of first performance metrics, also referred to as the benchmark average score (avg_score). The benchmark average score refers to the arithmetic mean of the agent's performance scores within a specific number of training steps in the initial stage under the current maximum number of interactions configuration. It is used to characterize the agent's initial performance level in the current stage and provides a benchmark reference for setting the subsequent first threshold.

[0052] On the one hand, the calculation of the arithmetic mean involves only summation and division, resulting in low time and space complexity (only the accumulated sum and count need to be maintained). During agent training, the length of the first performance metric sequence (e.g., 30 steps in the first training window) is fixed. The arithmetic mean can be incrementally updated after each interaction, eliminating the need to store the entire sequence and reducing memory usage and computational latency. This is particularly important for large-scale distributed training requiring high-frequency sampling (e.g., running hundreds of environments simultaneously to collect trajectories in parallel). The lightweight nature of the arithmetic mean prevents benchmark calculations from becoming a performance bottleneck in the training pipeline. On the other hand, the arithmetic mean assigns equal weight to each performance metric within the first training window, avoiding any artificial temporal bias. In the initial stage of agent training (i.e., the first training window), the agent's strategy is unstable, and scores may fluctuate significantly. The arithmetic mean provides an unbiased estimate of the overall level within the first training window, unaffected by temporal order. This ensures that the first benchmark performance metric truly reflects the agent's steady-state fundamental capabilities at the current difficulty level, rather than being dominated by recent random fluctuations, thus providing a stable and reliable reference benchmark for subsequent first threshold setting. If the avg_score itself fluctuates wildly due to the calculation method, the first threshold will also oscillate wildly, causing the agent to sometimes "meet the standard" (triggering an upgrade) and sometimes "fail to meet the standard," resulting in frequent false triggers or missed triggers. Furthermore, the training process is prone to falling into an unstable cycle of "upgrade-rollback-upgrade," disrupting the orderly nature of the progressive course. Meanwhile, the arithmetic mean, because it smooths the contribution of all performance metrics within the first training window, has strong robustness to outliers (such as exceptionally high or low scores in a particular step). Even if there are individual abnormal scores within the first training window, their impact on the mean is diluted by the window length (the impact is 1 / K). This ensures the smooth evolution of the first baseline performance metric and the first threshold, making the judgment of whether the agent has "mastered" the skill more stable and reliable, and the training curve smoother.

[0053] This disclosure does not limit the arithmetic mean of the first performance index sequence to be used as the first benchmark performance index. In other embodiments, a weighted summation method can also be used for calculation. For example, in the first performance index sequence obtained after 30 training steps, the weight of the index closer to the current time is set to be greater. The following embodiments all use the arithmetic mean as an example, but this disclosure is not limited to this.

[0054] The first threshold in this embodiment refers to the performance threshold used to determine whether the agent has mastered the current interaction depth / current difficulty task.

[0055] In S130, the second performance index sequence of the agent and the environment module within the second training window is obtained under the maximum number of interactions in the first single task.

[0056] In this embodiment of the disclosure, the second training window refers to a set of consecutive interaction rounds used to detect the performance of the agent after the first training window under the maximum number of interactions in the first single task, and the window length is configurable.

[0057] In this embodiment of the disclosure, the second performance index sequence refers to the performance index sequence collected within the second training window.

[0058] In S140, based on the second performance index sequence and the first threshold, the first single-task maximum number of interactions between the agent and the environment module in multiple rounds is adjusted to the second single-task maximum number of interactions.

[0059] For example, the maximum number of interactions for the first single task is less than the maximum number of interactions for the second single task. During agent training, when the system determines that the agent has fully grasped the task difficulty under the current maximum number of interactions, it automatically and appropriately increases the interaction depth, giving the agent the opportunity to encounter more complex and longer-term tasks. The agent first learns short-term tasks (such as simple webpage navigation that can be completed within 10 rounds) and establishes basic skills (such as clicking and filling out forms). After increasing the number of interactions, the agent begins to encounter long-term tasks that require more steps (such as cross-page shopping). That is, this embodiment of the disclosure adopts progressive course learning to realize the cognitive law of "from easy to difficult". Only when the agent has proven that it can stably solve the task of the current difficulty will a longer interaction space be opened to prevent strategy collapse or gradient explosion caused by a sudden expansion of the exploration space. At the same time, the agent will not waste a lot of rounds trying impossible long-term tasks at a low level stage, but will only conduct long-term exploration after it has acquired basic capabilities, resulting in higher utilization of each training sample.

[0060] For example, in the web navigation task, the agent progresses from "clicking the homepage link" (5 rounds) to "searching for products and placing an order" (15 rounds). In the embodied task, it progresses from "moving to the red box" (8 rounds) to "opening the box, taking out the key, and returning to the starting point" (20 rounds). In the scientific experiment task, it progresses from "measuring water temperature" (4 rounds) to "preparing a solution and measuring the pH value" (12 rounds).

[0061] In technologies with a fixed maximum number of interactions, even for simple tasks (such as webpage navigation that can be completed in just 3 rounds), the agent is still forced to interact for a preset 10 rounds, resulting in meaningless operations (such as repeated clicks and timeouts) in the subsequent 7 rounds, wasting CPU / GPU computing and storage resources. In this embodiment, because the maximum number of interactions for the first single task is relatively small, and the interaction depth is only increased after the agent's performance exceeds a first threshold (i.e., it has mastered the current difficulty), the maximum number of interactions for the second single task is greater than that for the first single task. For simple tasks, the agent's performance metrics will quickly reach a high level, and the system will appropriately increase the number of interactions; when the agent has not yet mastered the task, the system will not increase the number of interactions prematurely, thus avoiding a large amount of invalid computation caused by long-term exploration at a low level. Simple tasks maintain a low number of interactions, while complex tasks receive more rounds, directing computing resources to tasks that truly require long-term planning, avoiding invalid interactions caused by a fixed high upper limit.

[0062] For example, the maximum number of interactions in the first task is greater than the maximum number of interactions in the second task. Under certain circumstances, the system may also need to reduce the maximum number of interactions per task, i.e., adjust it from the larger maximum number of interactions in the first task to the smaller maximum number of interactions in the second task. This is an adaptive protection mechanism. Conditions triggering degradation include, but are not limited to: Persistently poor performance: If the agent fails to reach the first threshold for an extended period of time under the current interaction limit (e.g., a success rate of less than 10% for 20 consecutive tasks), it indicates that the current difficulty is too high for the agent and it needs to be reverted to a simpler task environment to rebuild the foundation.

[0063] Resource constraint trigger: If the interaction cost is too high (such as API call fees exceeding the budget) or the user wait time is too long, the system will proactively reduce the interaction limit to save resources.

[0064] Safe rollback: After increasing the number of interactions, if a significant drop in task success rate is observed (e.g., from 80% to 30%), the rollback mechanism is triggered to restore the previous lower limit.

[0065] For example, during the training phase, the agent's `max_turn` was downgraded from 20 to 10. This was because the average success rate of the last 50 tasks dropped from 70% to 25%, and the performance exhibited extremely high variance, indicating that the agent was lost in long-term tasks. After the downgrade, the agent consolidated its foundation in short tasks and attempted to upgrade again later. As another example, if the system detects that a user has had three consecutive unresolved conversations that timed out and required human intervention, it determines that the user may prefer shorter answers and proactively reduces `max_turn` from 15 to 8, reducing user waiting time and improving satisfaction. Yet another example is when a smart assistant deployed on a mobile device detects that the battery level is below 20%, the system automatically reduces the maximum number of interactions per task from 12 to 5 to reduce computational power consumption and extend device battery life.

[0066] On the one hand, when an agent repeatedly fails over long rounds, continuing to provide numerous interaction rounds will only accumulate more negative samples, exacerbating policy bias. Reducing the number of interactions forces the agent to return to tasks within its capabilities, avoiding ineffective exploration and rebuilding confidence and basic policies. On the other hand, excessively high interaction limits may lead to unnecessary long dialogues, increasing server load and user waiting time. Dynamic degradation can respond to resource conditions in real time, saving resources and achieving cost control. The degradation mechanism in this embodiment provides a "negative feedback" adjustment: when performance indicators fall below the lower threshold, the system automatically reduces the difficulty to prevent training from entering an infinite loop that cannot converge. This constitutes a complete closed-loop control (two-way adjustment). At the same time, the demand for task complexity may change at different times or for different user groups (e.g., faster response is needed during peak evening hours), and the degradation mechanism allows the system to flexibly match the complexity requirements of the current scenario.

[0067] The method provided in this disclosure introduces a performance-driven closed-loop feedback mechanism to align the adjustment of the maximum number of interactions per task with the actual learning progress of the agent, thereby achieving dynamic adaptive adjustment of the maximum number of interactions per task. This enables on-demand allocation of computing resources, improved training stability, and enhanced sample efficiency.

[0068] Sample efficiency refers to an agent's ability to learn effective information from each interaction sample. With a fixed number of short interactions (e.g., 10 rounds), the agent will never encounter tasks requiring 15 rounds to complete, thus failing to learn long-term planning. Conversely, with a fixed number of long interactions (e.g., 50 rounds), the agent explores numerous ineffective actions early on, resulting in extremely low sample utilization. This embodiment employs a progressive curriculum: starting with short interactions, the agent first learns short-term skills with high success rates (e.g., correct clicking, form filling), which form the foundation for long-term tasks. Once basic skills are mastered, the depth of interaction is gradually increased, allowing the agent to naturally combine short-term skills into long-term strategies. This "from easy to difficult" curriculum design aligns with cognitive principles; each sample occurs within a difficulty range that the agent can "reach with a little effort," thus maximizing sample utilization.

[0069] For example, determining the first threshold based on the first benchmark performance index includes: obtaining a first incremental index based on the first benchmark performance index; and determining the first threshold based on the first benchmark performance index and the first incremental index.

[0070] For example, the first threshold is a critical value used to determine whether the agent has mastered the task of current difficulty. This threshold is not a fixed constant, but is dynamically calculated based on the agent's actual performance at the current maximum number of interactions. In some embodiments, the first threshold is jointly determined by a first baseline performance metric (e.g., the aforementioned avg_score) and a first incremental metric (score_increment), for example, calculated using the following formula: threshold=avg_score+score_increment(1) In this embodiment of the disclosure, the first incremental metric, also referred to as the score increment, is a dynamic value heuristically calculated based on a first benchmark performance metric (e.g., the benchmark average score avg_score) and used to set the performance improvement target. This increment can be adaptively adjusted according to the Agent's current performance level, ensuring that the learning target is both challenging and achievable.

[0071] For example, obtaining a first incremental index based on the first benchmark performance index includes: inputting the first benchmark performance index into an adaptive incremental generation model to determine a candidate incremental index (temp_score_increment), wherein the adaptive incremental generation model is configured to: if it is determined that the first benchmark performance index is lower than or equal to a first preset threshold, then determine the candidate incremental index based on a benchmark incremental ratio; if it is determined that the first benchmark performance index is higher than the first preset threshold, then determine the candidate incremental index based on the first benchmark performance index and the benchmark incremental ratio; and determine the first incremental index based on the candidate incremental index.

[0072] For example, the adaptive incremental generation model is further configured to: if it is determined that the first benchmark performance index is higher than the first preset threshold and less than or equal to the second preset threshold, then obtain the difference between the preset value and the first benchmark performance index, and determine the candidate incremental index based on the difference and the benchmark incremental ratio; if it is determined that the first benchmark performance index is greater than the second preset threshold, then determine the candidate incremental index based on the difference, the benchmark incremental ratio and the attenuation coefficient.

[0073] In this embodiment of the disclosure, the first preset threshold is less than the second preset threshold. The values ​​of the first preset threshold and the second preset threshold can be set according to actual needs, and this disclosure does not limit them.

[0074] This disclosure provides an adaptive incremental generation model. Based on the numerical range of a first benchmark performance indicator (e.g., the benchmark average score avg_score), the model dynamically determines the incremental value of the maximum number of interactions in the current single task using a piecewise function. Its design follows the principle that "the lower the level of the first benchmark performance indicator, the larger the increment; the higher the level of the first benchmark performance indicator, the smaller the increment," so that the learning objective is always kept within a reasonable range that the agent can "reach with a little effort." This avoids both objectives that are too low and lack challenge, and objectives that are too high and cannot be achieved.

[0075] Reinforcement learning training inherently involves randomness. An agent might achieve unusually high scores in certain rounds of interaction due to chance (such as completing a complex operation by accident), but without truly mastering the skill. If a fixed threshold (e.g., a threshold set to 80) is used, these random fluctuations might be mistakenly judged as "mastery." Increasing the number of interactions at this point would prematurely trigger a difficulty escalation, leading to subsequent training oscillations. This embodiment addresses this by constructing a first threshold based on a first baseline performance indicator and an adaptive increment (e.g., a first increment indicator), where the first increment indicator is adaptively adjusted according to the first baseline performance indicator, ensuring that the first threshold is higher than the agent's normal fluctuation range. Only when the agent consistently (e.g., for multiple consecutive steps or at a high percentage) exceeds this first threshold is it considered to have truly mastered the skill. This mechanism effectively filters out random noise, ensuring the reliability of adjustment decisions.

[0076] For example, determining the first incremental indicator based on the candidate incremental indicator includes: obtaining the maximum value between the candidate incremental indicator and a first preset parameter; obtaining the minimum value between the maximum value and a second preset parameter; and determining the first incremental indicator based on the minimum value.

[0077] In this embodiment, the first preset parameter is less than the second preset parameter. The values ​​of the first and second preset parameters can be set according to actual needs, and this disclosure does not limit them. The following example illustrates the situation with the first preset parameter = 0.1 and the second preset parameter = 10.

[0078] In this embodiment of the disclosure, boundary constraints are applied to the candidate incremental index temp_score_increment to obtain the first incremental index score_increment: score_increment=min(max(0.1,temp_score_increment),10) (2) This constraint ensures that the increment always remains within a reasonable range of [0.1, 10]. The lower bound of 0.1 guarantees that even if the agent is close to perfect (avg_score = 99.9), the increment will not be lower than 0.1, and the first threshold will not exceed 100, thus maintaining the guiding effect for further improvement; the upper bound of 10 prevents the calculation of excessively large increments (e.g., more than 20 according to the linear formula) when the avg_score is extremely low (e.g., 0 points), which would cause the first threshold to exceed the reasonable range, making it impossible for the agent to reach the target. This numerical constraint mechanism improves the robustness of the entire training process.

[0079] For example, the second training window includes a first consecutive step threshold. Adjusting the first maximum number of interactions per single task between the agent and the environment module to the second maximum number of interactions per single task, based on the second performance indicator sequence and the first threshold, includes: if, within the first consecutive step threshold, all performance indicators in the second performance indicator sequence are greater than the first threshold, then the first maximum number of interactions per single task is increased by a first round increment to obtain the second maximum number of interactions per single task.

[0080] In this embodiment of the disclosure, the first consecutive step threshold is used to determine the number of consecutive steps required to achieve "complete mastery". This parameter determines the number of consecutive steps required to trigger a rapid update of the maximum number of interactions, and is an indicator for evaluating the high stability of the agent's capabilities.

[0081] In this embodiment of the disclosure, the first round increment refers to the amount by which the maximum number of interactions per task increases when the "complete mastery" condition is triggered.

[0082] For example, the second training window further includes a second consecutive step threshold. The adjustment of the first maximum single-task interaction count for the agent to interact with the environment module in multiple rounds, based on the second performance indicator sequence and the first threshold, to the second maximum single-task interaction count, further includes: if, within the second consecutive step threshold, the proportion of performance indicators in the second performance indicator sequence that are greater than the first threshold reaches a predetermined proportion threshold, then the first maximum interaction count is increased by a second round increment to obtain the second maximum single-task interaction count; wherein the first round increment is greater than the second round increment, and the second consecutive step threshold is greater than the first consecutive step threshold.

[0083] In this embodiment, the second consecutive step threshold is used to determine the length of the observation window for "partial mastery". This window is larger than the first consecutive step threshold and is used to evaluate the overall achievement rate of the agent over a longer time scale.

[0084] In this embodiment of the disclosure, the predetermined proportion threshold refers to the minimum proportion of the required number of steps within the second consecutive step threshold. It can be set according to actual needs, and this disclosure does not limit its value.

[0085] In this embodiment of the disclosure, the second round increment refers to the amount by which the maximum number of interactions per task increases when the "partial mastery" condition is triggered.

[0086] In this embodiment of the disclosure, the values ​​of the first consecutive step threshold and the second consecutive step threshold can be set according to actual needs, and this disclosure does not limit them.

[0087] In some embodiments, a consecutive steps threshold (consecutive_steps_threshold, n, where n is a positive integer) is set, which refers to the length of the observation window used to determine performance stability. This parameter determines the number of consecutive steps required to trigger the maximum number of interactions (max_turn) update, and is an indicator for evaluating the agent's ability mastery. In some embodiments, a first consecutive steps threshold is set to n, a second consecutive steps threshold is set to 2n, and a first round increment is set to 2Q, and a second round increment is set to Q, but this disclosure is not limited to these.

[0088] In this embodiment of the disclosure, after the agent reaches a certain level of maturity in training at the current maximum number of interactions in a single task, the system decides whether to increase the upper limit and by how much. To this end, this embodiment of the disclosure provides a dual-condition triggering mechanism. This mechanism distinguishes between "complete mastery" and "partial mastery" based on the comparison between the agent's performance in the second training window and the first threshold, and assigns different round increments to each, thereby achieving a balance between training robustness and evolution speed.

[0089] This disclosure employs a two-level max_turn update strategy to balance training robustness and evolution speed: Condition 1: Complete mastery of the condition (Rapid Advance): Triggering logic: During training, if the agent's score is higher than the threshold for n consecutive steps.

[0090] Execution action: max_turn (updated maximum number of interactions per single task, e.g., maximum number of interactions per second single task) = max_turn (previous maximum number of interactions per single task, e.g., maximum number of interactions per first single task) + 2Q.

[0091] Condition 1 applies to the rapid improvement phase where the model performs exceptionally well and the policy is highly stable. Consecutive successes demonstrate that the model / agent has thoroughly solidified its skills at the current depth, its policy is highly stable, and there are no fluctuations or randomness. Directly increasing the difficulty level can significantly improve training efficiency. At this point, using a larger increment allows the agent to quickly enter a higher difficulty task space, improving training efficiency and avoiding wasting computational resources on tasks already mastered. For example, in a web navigation task, if the agent successfully completes the shopping process within 15 steps for 10 consecutive rounds, it indicates that it has mastered all operations at that difficulty level and can tackle complex tasks requiring 30 steps.

[0092] Condition 1 above can be expressed as: (3) Where t is the current training step number (i.e., the sequence number of the current interaction round). Let be the performance metric for the i-th step in the second performance metric sequence, where i is an integer greater than or equal to t-n+1 and less than or equal to t. This formula means that within the most recent n consecutive interaction steps, the performance score in each round is greater than the first threshold.

[0093] Condition 2: Stable Progress (Partial Mastery) Triggering logic: In the most recent 2n steps, the score of 1.5n steps is higher than the threshold.

[0094] Execute the action: max_turn = max_turn + Q.

[0095] Random noise and policy fluctuations (such as environmental randomness and sampling errors) are unavoidable during reinforcement learning training. Partial mastery allows for a certain degree of fault tolerance by setting a longer observation window (2n) and a moderate success rate (e.g., 75%). As long as the agent demonstrates sufficient competence over a longer timescale / window, the difficulty can be steadily increased, enhancing the algorithm's tolerance to training noise and preventing training stagnation due to a single accidental failure. This approach is suitable when the agent can complete the task of the current difficulty in most cases, but occasionally fails due to random factors (such as webpage loading delays or minor changes in search results). In this case, a small increment (Q) is used to steadily advance, ensuring progress while avoiding the risks of overly aggressive escalation.

[0096] Condition 2 above can be expressed by a formula: (4) in This is an indicator function that returns 1 when the condition is true and 0 otherwise. 1.5n is the lower limit of the number of steps required to achieve the target within the 2n window, corresponding to a predetermined percentage threshold of 75%.

[0097] If neither of the above two conditions is met, it indicates that the agent has not yet reached a sufficient level of mastery at the current difficulty. The system maintains max_turn unchanged and continues training at the current interaction depth.

[0098] In this embodiment, the dual-condition collaborative mechanism checks the two conditions in order of priority: first, the complete mastery condition (more stringent) is checked; if not met, the partial mastery condition (less stringent) is checked. If neither condition is met, training continues under the current max_turn. This design ensures that fast learners can quickly progress to higher difficulties, stable learners can continue to improve, and fluctuating learning processes are not prematurely pushed to higher difficulties.

[0099] Common sources of noise in reinforcement learning include environmental randomness (such as changes in the position of web page elements), sampling errors (fluctuations in small batches of data), and model exploration. If a single "continuous mastery" condition is used, a single accidental failure will reset the counter, which may cause the agent to be unable to upgrade for a long time due to random factors, resulting in training stagnation. The "partial mastery" condition in this embodiment effectively addresses noise through the following mechanisms: a longer observation window (2n) smooths out short-term fluctuations; a moderate proportion threshold (75%) tolerates a certain proportion of failures; and no need for continuous mastery, allowing single outliers caused by noise to be compensated by other successful samples within the window. Experiments show that, under the same noise level, the upgrade triggering delay provided by this embodiment is lower than that of a single variance, and the success rate curve of the training process is smoother.

[0100] An agent might perform exceptionally well within a short window (e.g., scoring high in 5 consecutive steps) by chance, but it may not have actually mastered long-term skills. If the number of interaction rounds is immediately increased significantly, the agent is likely to expose its inadequacy in longer sequences, leading to training failure. This disclosed embodiment requires "complete mastery" by achieving the target in all n consecutive steps (e.g., 10 steps), and "partial mastery" by requiring a high percentage of success within 2n steps. Both require a sufficient number of samples, effectively eliminating short-term luck. Statistically, the probability of random fluctuations decreases exponentially with window length; therefore, this scheme significantly reduces the false trigger rate. For example, in a standard web navigation task, using a single consecutive 5-step achievement as the trigger condition results in a high false trigger rate; while using this scheme's n=10 consecutive achievement or 75% achievement within 2n=20 steps reduces the false trigger rate.

[0101] Static scaling methods in related technologies (such as adding 10 epochs every N epochs) fail to perceive the actual capabilities of the agent, leading to either slow upgrades wasting resources or rapid upgrades causing training crashes. This disclosure implements a tiered response through a dual-condition mechanism: agents with excellent and highly stable performance are given large increments (2Q) for rapid advancement and improved efficiency; agents with good performance but normal fluctuations are given small increments (Q) for steady progress and to avoid risks; and agents with insufficient performance are not upgraded to ensure a solid foundation. This tiered strategy allows agents with different learning abilities to obtain training paths that match their progress, improving overall training efficiency.

[0102] This embodiment of the disclosure introduces a dual-condition triggering mechanism of "complete mastery" and "partial mastery," coupled with different round increments (2Q and Q), to achieve refined evaluation and graded response of the agent's learning progress. This mechanism maximizes training efficiency while ensuring training stability, effectively solving the problems of oscillation, stagnation, or resource waste inherent in static increment schemes or single-condition triggering schemes.

[0103] Because the first threshold itself changes dynamically with the agent's performance, the "achievement" judgment in the two-condition mechanism also adjusts adaptively. When the agent progresses rapidly, the first threshold increases, making it more difficult to achieve the goal, and the trigger frequency naturally decreases to avoid over-upgrading; when the agent progresses slowly, the first threshold increases gradually, and the trigger frequency increases appropriately to maintain learning motivation. This inherent adaptive characteristic allows the entire training process to automatically adjust the upgrade pace without manual intervention.

[0104] This disclosure is not limited to the above-described dual-condition triggering mechanism. In other embodiments, the mechanism can be flexibly adjusted according to task complexity and training objectives. More stringent conditions for complete mastery: For tasks requiring extremely high precision (such as medical diagnostic assistance), a larger threshold for the first consecutive steps can be set, and the threshold for the first consecutive steps must be met to ensure that the agent's strategy is absolutely stable before upgrading.

[0105] More lenient control conditions: For noisy environments (such as real-time stock trading simulations), the predetermined percentage threshold can be reduced to 60% (i.e., 1.2n / 2n) to tolerate more random fluctuations.

[0106] Non-integer multiple window: The second consecutive step threshold is not necessarily an integer multiple of the first consecutive step threshold. For example, it can be set as the first consecutive step threshold = 8, the second consecutive step threshold = 15, and the required number of steps to meet the standard is ≥10 (the predetermined proportion threshold is about 67%).

[0107] Dynamically adjusted n value: In the early stages of training, due to the large fluctuations in the policy, a smaller n (such as 5) can be used to facilitate rapid upgrades; in the later stages of training, a larger n (such as 20) can be used to ensure stability.

[0108] In other embodiments, in addition to multiples of 2Q and Q, other incremental mappings can be designed: Under conditions of complete mastery, the increment in the first round is 3Q; under conditions of partial mastery, the increment in the second round is Q.

[0109] Under the condition of complete mastery, the increment in the first round = Q + 2; under the condition of partial mastery, the increment in the second round = Q.

[0110] Under conditions of complete mastery, the increment in the first round = 1.5Q Under conditions of partial mastery, the second round increment = 0.5Q (Adaptive rounding).

[0111] In other embodiments, a third condition (such as "preliminary mastery") can be added to the two conditions: for example, if the success rate reaches 50% within a 4n window, an increment of 0.5Q is given to further improve training stability for scenarios where progress is extremely slow but continuous.

[0112] Static increment strategies in related technologies (such as adding 5 rounds every 1000 steps) ignore the actual capabilities of the agent. When the agent has not yet learned basic operations (such as correctly clicking a button), forcibly increasing the number of interaction rounds will cause the policy search space to expand abruptly: from an exploration space of 10 steps to 15 steps. The agent may accumulate more errors in long sequences, leading to gradient explosion or policy collapse. This application only increases the number of interactions after the agent's continuous or proportional performance exceeds an adaptive threshold, improving training stability and avoiding training oscillations. The first threshold is dynamically generated based on the agent's recent average performance, ensuring that the timing of "increasing difficulty" is precisely when the agent has stably mastered the current difficulty. Experimental data show that after adopting the solution provided by the embodiments of this disclosure, the success rate and reward curve fluctuation amplitude during training are reduced, and the number of iterations required for convergence is reduced.

[0113] For example, performance score acquisition, threshold comparison, and adjustment of the maximum number of interactions are all performed in real time, without interrupting the training process or waiting for offline evaluation. Specifically, after each round of interaction with the environment module, the agent immediately calculates the performance score for that round based on the task completion status of the current round (such as sub-goal achievement, immediate reward, action correctness, etc.), and there is no significant delay in obtaining this performance score. After each round of interaction, the system compares the newly generated performance score with the current dynamic threshold (e.g., the first threshold) and updates the achievement count within the second training window in real time. This process is performed in parallel with the interaction execution, without introducing additional waiting time. When the system detects that the conditions for complete mastery or partial mastery are met, it immediately updates max_turn after the current training step and synchronizes the updated value to the agent and environment module to constrain the number of interaction rounds for the next single task. The entire adjustment process does not cause any interruption to the training process.

[0114] This real-time closed-loop feedback mechanism allows for immediate responses to learning progress. The system can sense and increase the interaction depth the instant the agent's capabilities improve, avoiding training efficiency losses due to delayed adjustments. For example, if the agent performs well in 10 consecutive rounds of interaction, the system immediately increases `max_turn` in the next task, enabling it to tackle longer tasks immediately, rather than waiting for a fixed time period (e.g., every N epochs) before adjustment. Furthermore, it avoids training efficiency losses due to delayed upgrades. The real-time adjustment mechanism triggers an increase in `max_turn` the moment the agent masters the current difficulty, allowing it to immediately tackle more challenging tasks, thereby maximizing the learning value of each training step and shortening the training time required to reach the target interaction depth. In distributed training or real-hardware deployment scenarios, the real-time adjustment mechanism enables the system to adapt to the performance differences of different computing nodes. This embodiment achieves adaptive closed-loop control of the training process by acquiring performance scores in real-time and adjusting the maximum number of interactions per task in real-time. Its adjustment granularity reaches the "per round of interaction" level, improving the timeliness of training response and resource utilization efficiency.

[0115] For example, the method provided in this disclosure further includes: obtaining a third performance index sequence within a third training window when the agent interacts with the environment module through the communication interface in multiple rounds under the second single-task maximum interaction count; determining a second benchmark performance index based on the third performance index sequence, and determining a second threshold based on the second benchmark performance index; obtaining a fourth performance index sequence between the agent and the environment module within a fourth training window under the second single-task maximum interaction count; adjusting the second single-task maximum interaction count of the agent and the environment module in multiple rounds to a third single-task maximum interaction count based on the fourth performance index sequence and the second threshold, until a preset single-task maximum interaction count upper limit is reached.

[0116] The method provided in this disclosure includes an iterative adjustment process for the maximum number of interactions per single task. Specifically, when the agent adjusts to obtain a new maximum number of interactions per single task (e.g., a second maximum number of interactions per single task) through a dual-condition triggering mechanism under the current maximum number of interactions per single task (e.g., the first maximum number of interactions per single task), the system will repeatedly execute similar benchmark calibration, threshold generation, performance detection and adjustment steps based on the new number of interactions, until the preset upper limit of the maximum number of interactions per single task is reached.

[0117] In this embodiment, the length of the third training window can be the same as that of the first training window (e.g., 30 steps), and it is used to collect the initial performance of the agent under the maximum number of interactions in the second single task. Each performance indicator in the third performance indicator sequence is a performance score obtained by the agent after each round of interaction.

[0118] For example, the second benchmark performance index is calculated in the same way as the first benchmark performance index, such as by using the arithmetic mean of a third performance index sequence. Then, the second incremental index is calculated based on the second benchmark performance index using the aforementioned adaptive incremental generation model, and then summed to obtain the second threshold.

[0119] In this embodiment, the length of the fourth training window can be the same as that of the second training window (e.g., 20 or 30 steps), used to collect the performance of the agent in subsequent interactions after the maximum number of interactions in the second single task. Exemplarily, this adjustment also employs the aforementioned dual-condition triggering mechanism: if the complete mastery condition is met within the fourth training window (e.g., the performance score exceeds the second threshold for n consecutive steps), the maximum number of interactions in the second single task is increased by a first-round increment (e.g., 2Q) to obtain the maximum number of interactions in the third single task; if the partial mastery condition is met (e.g., the performance score reaches 75% within 2n steps), the second-round increment (e.g., Q) is increased to obtain the maximum number of interactions in the third single task.

[0120] Repeat the above steps until the current maximum number of interactions per single task reaches the preset maximum number of interactions per single task (finally_turn_max). Through the above iterative process, this disclosure achieves a progressive and adaptive increase in the training difficulty of the agent, enabling the agent to gradually transition from short-term simple tasks to long-term complex tasks. Each difficulty level is only upgraded after the agent has fully mastered it, thereby ensuring the stability and efficiency of the training process.

[0121] To ensure the training process proceeds within a reasonable range and avoids indefinite extension or waste of resources, the embodiments of this disclosure set the following termination conditions. When any one of these termination conditions is met, the system stops further adjusting the maximum number of interactions per task, or terminates the entire training process.

[0122] Condition 1: Depth Achieved. When the current maximum number of interactions per single task reaches the preset maximum number of interactions per single task (finally_turn_max), it is determined that the agent has the ability to handle tasks of the target complexity, and the interaction depth is stopped from increasing. It should be noted that the system does not necessarily terminate training immediately after reaching the preset maximum number of interactions per single task. In one embodiment, the system can continue to fine-tune the policy at the current maximum number of interactions (i.e., only optimize the agent's policy parameters, without increasing max_turn), to further improve the task success rate. In another embodiment, if the total number of training steps has not yet reached the preset maximum number of interactions per single task and the agent's performance still has room for improvement, the system can maintain the current interaction depth and continue training until convergence.

[0123] Condition 2: Resource Exhaustion. When the total number of interaction steps or total training rounds accumulated during training reaches the preset maximum number of training steps, the system automatically terminates training to prevent the training process from extending indefinitely. This condition ensures the rational use of computing resources and avoids invalid computations caused by errors in task complexity estimation or agent learning stagnation. The maximum number of training steps can be preset according to actual computing power budget, time cost, or experimental design, for example, it can be set to 1 million steps or 5 million steps.

[0124] In addition to the two main conditions mentioned above, embodiments of this disclosure may also include the following optional termination conditions: Performance saturation: When the improvement of the agent's baseline average score (avg_score) is lower than a preset threshold (e.g., lower than 1 point) within multiple consecutive training windows (e.g., 10 windows), and the current interaction depth has reached or is close to the upper limit, it can be determined that the performance has saturated and training can be terminated in advance.

[0125] Manual intervention: Receiving interrupt signals from external input, such as termination commands sent by users through a graphical interface or command line.

[0126] Anomaly detection: When a serious anomaly is detected during the training process (such as reward value divergence, gradient explosion, success rate remaining at zero, etc.), training is automatically terminated and logs are saved for debugging purposes.

[0127] The following example, using a complete training process, illustrates the collaborative work between iterative adjustments and termination conditions.

[0128] Preset parameters: Initial maximum number of interactions per task: max_turn=10; Maximum number of interactions per task: finally_turn_max=30; First round increment: 2Q=6, Second round increment: Q=3; Maximum number of training steps: 1,000,000 steps.

[0129] Execution process: The first difficulty stage (max_turn=10): The agent is trained with a maximum of 10 rounds. After several rounds, a dual condition is triggered, and max_turn is adjusted to 13 (partial mastery) or 16 (complete mastery).

[0130] The second difficulty stage (max_turn=13 or 16): The system recalibrates the baseline and threshold, and continues training. When the trigger condition is met again, max_turn is increased again.

[0131] Repeated iterations: max_turn increases gradually: for example, from 10 to 13, then to 16, then to 19, then to 22, then to 25, then to 28, and then to 30.

[0132] Depth achieved: When max_turn reaches 30, the system stops increasing the interaction depth and enters the strategy fine-tuning stage.

[0133] Resource depletion check: If the total number of training steps exceeds 1,000,000 before reaching 30 rounds, training will be terminated early to avoid wasting resources.

[0134] This disclosure achieves a fully automated, progressive transition from the initial interaction depth to the target interaction depth through iterative training steps. The system requires no manual specification of when to increase difficulty or pre-setting the number of difficulty levels; instead, it is entirely driven by the agent's real-time performance, making the training process truly "adaptive." By setting termination conditions in two dimensions—the maximum number of interactions per task and the maximum number of training steps—this disclosure effectively avoids the indefinite extension of the training process. The former controls the upper limit of the task's complexity, while the latter controls the total computational budget. The combination of these two factors allows training to fully explore long-range tasks without exhausting resources due to the inability to complete individual tasks, making it particularly suitable for large-scale distributed training scenarios. Upon reaching the interaction depth limit, the system does not simply terminate but can continue to fine-tune the strategy at the current difficulty level. This design aligns with the cognitive principle of "first expanding the breadth (increasing interaction depth), then refining the precision (optimizing the strategy)." After obtaining sufficient interaction space, the agent still has ample opportunity to further optimize its decision-making quality at the final difficulty level, thereby achieving a higher task success rate.

[0135] The following is through Figure 3 The methods provided in the embodiments of this disclosure are illustrated by examples, but this disclosure is not limited thereto. Figure 3 As shown in the embodiments of this disclosure, the method proposed is an intelligent agent control method based on score heuristic dynamic adjustment of the maximum number of interactions, which adopts a closed-loop feedback control mechanism and may include the following steps.

[0136] In S301, training begins.

[0137] In S302, initialize parameters.

[0138] In the initialization stage, for example, set the maximum number of interactions for initialization max_turn = 10, configure the benchmark consecutive steps K = 30, and set parameters such as the consecutive steps threshold n, the benchmark increment ratio r, and the turn increment Q. The upper limit of the final maximum number of interactions finally_turn_max can also be set.

[0139] In S303, train for K steps to calculate avg_score.

[0140] S303 is the benchmark calculation stage. Under the current max_turn configuration, execute K training steps. Calculate the average performance score avg_score of the agent in these K training steps, and establish the performance benchmark score at the current interaction depth.

[0141] In S304, calculate score_increment; if avg_score < 0, that is, the first preset threshold = 0, then continue to execute S305; if 0 < avg_score ≤ 50, that is, the second preset threshold = 50, then continue to execute S306; if avg_score > 50, then continue to execute S307.

[0142] S304 is the heuristic increment calculation stage. According to the interval where avg_score is located, use a three-segment non-linear function to calculate the score increment score_increment. The increment can also be ensured to be within a reasonable range through boundary constraints.

[0143] S305, S306, and S307 are the three-segment score_increment calculation formulas provided by the embodiments of the present disclosure.

[0144] In order to make the threshold setting adapt to the learning rate of the agent at different stages, the following heuristic formula is used to calculate temp_score_increment: In S305, for the negative score interval (avg_score < 0), use the following formula: temp_score_increment = 100×r(5) When the model performance is extremely poor (the score is negative), set a large fixed increment target to force the model / agent to quickly break away from the wrong strategy area. At this time, do not consider the specific value of the current score, but give a large improvement space to help the model establish a basic understanding of the task.

[0145] In S306, for the middle score interval (0 < avg_score ≤ 50), use the following formula: temp_score_increment=(100-avg_score)×r(6) The increment is proportional to the remaining improvement potential (100 - avg_score), ensuring that the model / agent has sufficient motivation to pursue higher accuracy in moderately difficult tasks. As avg_score increases, the increment gradually decreases, which conforms to the natural law of the learning curve.

[0146] In S307, for high score intervals (avg_score>50), the following formula is used: temp_score_increment=(100-avg_score)×r×0.5 (7) The difficulty of improving scores at higher levels increases exponentially. Therefore, a decay coefficient of 0.5 (for example, but not limited to this) is introduced to moderately relax the requirements. The decay coefficient is a real number between (0,1). This is because: improving from 50 to 60 points is much less difficult than improving from 90 to 100 points; excessively high thresholds can lead to overfitting of the model in the current round; moderately relaxing the requirements encourages the model to move to more complex tasks at higher rounds as early as possible.

[0147] Then, the above boundary constraints are used to determine score_increment, ensuring that the increment always remains within a reasonable range of [0.1, 10].

[0148] In S308, threshold is set to avg_score + score_increment.

[0149] S308 is the dynamic threshold setting stage, which sets a judgment threshold based on the baseline score and the increment. This threshold serves as the standard for subsequent judgments on whether the agent has mastered the current difficulty level.

[0150] In S309, the training steps are performed and the score is recorded.

[0151] In S310, determine whether the condition is met for n consecutive steps where score > threshold. If it is met, continue to S311; otherwise, continue to S312.

[0152] S309-S313 is the performance detection and update phase. During subsequent training, the score obtained by the agent in solving the task is detected in real time.

[0153] In S311, if the "complete mastery" condition is met (score exceeds threshold for n consecutive training steps), max_turn+=2Q is executed, and then S314 is executed.

[0154] In S312, determine whether the target of 2n steps to 1.5n steps is met; if it is met, continue to S313; if it is not met, maintain max_turn and continue training, returning to S309 above.

[0155] In S313, if the "partial mastery" condition is met (in the last 2n training steps, the score exceeds the threshold in 1.5n steps), max_turn+=Q is executed, and then S314 is executed.

[0156] In S314, it is determined whether max_turn ≥ finally_turn_max or the maximum number of training steps has been reached. If it is satisfied, it jumps to S315; if it is not satisfied, it returns to S303 above and recalculates the baseline score avg_score under the new difficulty in the updated max_turn.

[0157] In S315, training terminates if finally_turn_max or the maximum number of training steps is reached.

[0158] This embodiment of the invention, through the aforementioned closed-loop feedback control mechanism, achieves adaptive difficulty adjustment and improves sample efficiency. By real-time detection of the agent's performance score and dynamic calculation of thresholds, the interaction depth is increased only after the agent has mastered the current difficulty, ensuring that each training sample operates within a difficulty range that the agent can "reach with a little effort." Furthermore, the dual-condition triggering mechanism balances robustness and evolution speed. By distinguishing between "complete mastery" and "partial mastery" conditions and assigning different round increments (2Q and Q), a refined, tiered response to the agent's learning progress is achieved. Agents with excellent performance and stable strategies can quickly advance to higher difficulties (large increments), agents with good performance but normal fluctuations can progress steadily (small increments), while agents with insufficient performance are not upgraded. This design effectively avoids false triggering caused by random noise while ensuring that agents with different learning abilities receive training paths that match their progress.

[0159] Furthermore, this embodiment of the disclosure also filters noise through adaptive thresholds to reduce the false trigger rate. The thresholds (including the first threshold and / or the second threshold) in this embodiment of the disclosure are generated based on the benchmark average score and adaptive increments, and the thresholds are always higher than the normal fluctuation range of the agent. Only when the threshold is exceeded multiple times consecutively or in a high proportion is it judged as "true mastery". In this embodiment of the disclosure, different increment calculation formulas are adopted according to the different intervals (negative score, medium score, high score) of avg_score: a fixed large increment is given to the negative score interval to force the model to quickly get rid of the wrong strategy; the increment in the medium score interval is proportional to the remaining improvement space to maintain a moderate challenge; a decay coefficient is introduced in the high score interval to avoid overfitting or frustration caused by excessively high thresholds. This design conforms to the cognitive law of learning - it takes big leaps from nothing to something, and fine adjustments are needed from good to excellent. Boundary constraints further ensure the numerical stability of the increments. By iteratively executing the closed loop of "benchmark calibration - threshold generation - performance detection - difficulty upgrade", a fully automatic and gradual transition from the initial short-term task to the target long-term task is realized. Meanwhile, by setting two termination conditions—the maximum number of interactions per task and the maximum total number of training steps—the infinite extension of the training process is effectively avoided, ensuring the rational use of computing resources.

[0160] It is understood that the parameters used in the embodiments (such as K=30, n=10, r=0.2, Q=3, attenuation coefficient 0.5, boundary interval [0.1, 10], etc.) are all exemplary values. In practical applications, they can be flexibly adjusted according to factors such as task complexity, environmental noise level, and computing power budget. For example, for environments with high noise, n can be increased or the preset ratio threshold can be raised; for tasks that require fast convergence, Q can be increased or the upper bound of the increment can be raised; for tasks with extremely high accuracy requirements, the attenuation coefficient can be reduced. This flexibility makes this solution widely applicable to various scenarios such as web navigation, depth search, digital games, embodied tasks, and scientific experiments.

[0161] The embodiments disclosed herein upgrade open-loop time scheduling to performance-driven adaptive adjustment through score-heuristic closed-loop feedback control, achieving improvements in training stability, sample efficiency, and resource utilization.

[0162] It should be noted that this disclosure is not limited to the above-described three-segment piecewise function. In other embodiments, different numbers of segments and formula forms can be selected according to the characteristics of the task: Two-stage approach: Only distinguishes between "low partitions" (e.g., avg_score ≤ 60) and "high partitions" (e.g., avg_score > 60). Low partitions use a fixed large increment (e.g., 20), while high partitions use a linearly decreasing increment (e.g., 100 - avg_score) × 0.15. This implementation is simpler.

[0163] Four-segment approach: For example, the interval is subdivided into [0,20), [20,40), [40,70), and [70,100], with each segment using a fixed increment of 25, 18, 10, and 5 respectively. This embodiment provides a smoother transition and is suitable for learning tasks that require fine control.

[0164] Example of a continuous function: For example, using the exponential decay function score_increment=20×exp(-0.05×avg_score) makes the increment decrease continuously and smoothly with avg_score.

[0165] In reinforcement learning, the difficulty of credit assignment is closely related to the task length (i.e., the number of interaction rounds per task). For long-duration tasks (e.g., complex operations requiring 30 steps), the reward signal for success or failure needs to propagate backward along the timeline to earlier interaction rounds. As the number of interaction rounds increases, the gradient variance grows exponentially, making it difficult for the agent to accurately identify which intermediate actions played a crucial role in the final success. Related techniques directly set a large fixed maximum number of interactions (e.g., 30 steps) in an attempt to allow the agent to learn long-duration tasks from the beginning. This approach faces a double dilemma: on the one hand, when the agent has not yet mastered the basic skills, credit assignment is almost impossible to perform effectively in the face of sparse rewards in long sequences, leading to training failure; on the other hand, even if success is achieved, the accumulation of noise in long sequences makes gradient estimation extremely unstable and convergence difficult.

[0166] The embodiments disclosed herein effectively alleviate the aforementioned problems by progressively expanding the interaction depth from small to large. The mechanism of action is manifested in the following two aspects: 1. Reduce the time span of credit allocation.

[0167] In the training process disclosed herein, the maximum number of interactions per task starts with a small initial value (e.g., 10 steps) and is gradually increased only after the agent has fully mastered the current difficulty (e.g., to 13 steps, 16 steps, ... up to 30 steps). This means that in the early stages of training, the agent only needs to handle short-range tasks. In short-range tasks, the time interval between the reward signal and the action is short, the credit allocation distance is small, and the gradient variance is low, making it easier for the agent to establish a causal relationship between "action and result". For example, for a task completed within 10 steps, the final reward can be accurately fed back to the key action in the 2nd or 3rd step. As the agent gradually forms a solid basic policy on short-range tasks, the number of interaction rounds is gradually increased. At this point, although the credit allocation distance becomes longer, the agent already has the ability to combine short-range skills into long-range policies, and the contribution of intermediate steps is relatively clearer, thereby reducing the difficulty of long-range credit allocation.

[0168] 2. Provide dense and effective learning signals.

[0169] Under a fixed long-term task setting, most of the agent's early interactions fail to yield positive rewards (because the task is not yet complete), resulting in extremely sparse learning signals. The progressive strategy provided in this disclosure enables the agent to achieve frequent successes at each difficulty level (because the task difficulty matches the current ability), thereby generating dense positive rewards and higher performance scores. These scores themselves can serve as auxiliary dense reward signals, providing more intermediate anchors for credit allocation. For example, when the agent succeeds consecutively in a 10-step task, its score sequence itself reveals which sub-goals have been achieved, which is equivalent to decomposing the sparse final reward into multiple sub-goal rewards, helping the agent to more accurately identify key steps.

[0170] Combining the dynamic interaction number adjustment method provided in this disclosure with credit allocation technology (such as hierarchical reinforcement learning, reward shaping, etc.) can further enhance the synergistic effect and obtain better training results.

[0171] like Figure 4 As shown, this embodiment of the present disclosure provides an intelligent agent control device 400, including: a performance index acquisition unit 410, configured to acquire a first performance index sequence within a first training window when the intelligent agent interacts with an environment module through a communication interface in multiple rounds under a first single-task maximum number of interactions; a benchmark threshold determination unit 420, configured to determine a first benchmark performance index based on the first performance index sequence, and determine a first threshold based on the first benchmark performance index; the performance index acquisition unit 410 is further configured to acquire a second performance index sequence between the intelligent agent and the environment module within a second training window under the first single-task maximum number of interactions; and an interaction number adjustment unit 430, configured to adjust the first single-task maximum number of interactions between the intelligent agent and the environment module to a second single-task maximum number of interactions based on the second performance index sequence and the first threshold.

[0172] For example, the benchmark threshold determination unit 420 is further configured to: obtain a first incremental index based on the first benchmark performance index; and determine the first threshold based on the first benchmark performance index and the first incremental index.

[0173] For example, the benchmark threshold determination unit 420 is further configured to: input the first benchmark performance index into an adaptive incremental generation model to determine a candidate incremental index, wherein the adaptive incremental generation model is configured to: if the first benchmark performance index is determined to be lower than or equal to a first preset threshold, then determine the candidate incremental index based on a benchmark incremental ratio; if the first benchmark performance index is determined to be higher than the first preset threshold, then determine the candidate incremental index based on the first benchmark performance index and the benchmark incremental ratio; and determine the first incremental index based on the candidate incremental index.

[0174] For example, the adaptive incremental generation model is further configured to: if it is determined that the first benchmark performance index is higher than the first preset threshold and less than or equal to the second preset threshold, then obtain the difference between the preset value and the first benchmark performance index, and determine the candidate incremental index based on the difference and the benchmark incremental ratio; if it is determined that the first benchmark performance index is greater than the second preset threshold, then determine the candidate incremental index based on the difference, the benchmark incremental ratio and the attenuation coefficient.

[0175] For example, the benchmark threshold determination unit 420 is further configured to: obtain the maximum value of the candidate incremental index and the first preset parameter; obtain the minimum value between the maximum value and the second preset parameter; and determine the first incremental index based on the minimum value.

[0176] For example, the second training window includes a first consecutive step threshold. The interaction count adjustment unit 430 is further configured to: if, within the first consecutive step threshold, all performance metrics in the second performance metric sequence are greater than the first threshold, then increase the first single-task maximum interaction count by a first round increment to obtain the second single-task maximum interaction count.

[0177] For example, the second training window further includes a second consecutive step threshold. The interaction count adjustment unit 430 is further configured to: if, within the second consecutive step threshold, the proportion of performance indicators in the second performance indicator sequence that are greater than the first threshold reaches a predetermined proportion threshold, then increase the first maximum interaction count by a second round increment to obtain the second single-task maximum interaction count. Wherein, the first round increment is greater than the second round increment, and the second consecutive step threshold is greater than the first consecutive step threshold.

[0178] For example, the performance index acquisition unit 410 is further configured to acquire a third performance index sequence within a third training window when the agent interacts with the environment module through the communication interface in multiple rounds under the second single-task maximum interaction count. The benchmark threshold determination unit 420 is further configured to determine a second benchmark performance index based on the third performance index sequence, and determine a second threshold based on the second benchmark performance index. The performance index acquisition unit 410 is further configured to acquire a fourth performance index sequence between the agent and the environment module within a fourth training window under the second single-task maximum interaction count. The interaction count adjustment unit 430 is further configured to adjust the second single-task maximum interaction count of the agent interacting with the environment module in multiple rounds to a third single-task maximum interaction count based on the fourth performance index sequence and the second threshold, until a preset single-task maximum interaction count upper limit is reached.

[0179] Figure 4 Other aspects of the embodiments are the same as those described above and will not be repeated here.

[0180] Figure 5 This is a schematic diagram of the architecture of an intelligent agent training system provided in this embodiment. This embodiment can be deployed as the core module (dynamic interaction number adjustment module) of an intelligent agent training system. The overall architecture includes an intelligent agent 510, an environment module 520, a policy update module 530, and a dynamic interaction number adjustment module 540. The modules exchange data and transmit control commands through a communication interface, jointly achieving adaptive adjustment of the maximum number of interactions per task based on a score-heuristic approach.

[0181] In this embodiment of the disclosure, agent 510 is an autonomous decision-making entity constructed based on a Large Language Model (LLM) or other machine learning algorithms. Agent 510 can be used for: Decision execution: Based on the current State and Reward signal received from the environment module 520, generate an Action through the internal policy network and send the Action to the environment module 520.

[0182] Score Generation: After each round of interaction, a performance score is calculated based on the task completion status (e.g., sub-goal achievement, final reward signal, etc.), and this score is fed back to the dynamic interaction count adjustment module 540 in real time. The agent 510 also receives the current maximum number of interactions per task, max_turn, from the dynamic interaction count adjustment module 540.

[0183] Policy storage: Maintains current policy parameters (e.g., neural network weights) and receives updated parameters from the policy update module 530 for subsequent decision-making. The agent 510 is also used to send its interaction trajectory data (including state, action, reward, etc.) with the environment module 520 to the policy update module 530 as training data for the policy update module 530.

[0184] In some embodiments, agent 510 encapsulates a reasoning-action loop based on a large language model (LLM). Agent 510 receives observations (i.e., at least a portion of the state) from environment module 520, performs multiple rounds of reasoning, and outputs an action (e.g., calling a predefined API). Agent 510 supports various cueing strategies (such as ReAct, Chain-of-Thought), sampling configurations (temperature, top-p, etc.), and customized reward functions.

[0185] In this embodiment of the disclosure, the environment module 520 is used to provide the external scene required for the intelligent agent 510 to perform its tasks. This environment can be a simulation environment (such as a web navigation simulator, game engine, robot simulator, etc.) or a real physical environment (through sensor and actuator interfaces, etc.). The environment module 520 is used for: State Update: Receive the Action sent by agent 510, update the current state according to environmental dynamics or task logic, generate a new State and return it to agent 510.

[0186] Reward Calculation: Calculate a reward signal (Reward) based on task completion status (e.g., whether a sub-goal was achieved, whether the task was ultimately successful) and return it to agent 510. The reward can be sparse (given only at the end of the task) or dense (given at each step).

[0187] Interaction control: The maximum number of interaction rounds within a single task is limited by adjusting the max_turn set by module 540 (optional, or indirectly passed through agent 510) based on the number of dynamic interactions. When the limit is reached, the current task is forcibly terminated and a failure reward is returned.

[0188] For example, the environment module 520 provides diverse task scenarios, offering services externally through a server-client architecture. For example, each environment is encapsulated as an independent service, supporting the deployment of multiple replicas to process requests in parallel. The environment client communicates with the environment server via the HTTP protocol, exposing the following API interfaces to the agent: observation: to obtain the current observation status of the environment; available_actions: Gets the set of currently available actions; step: Perform an action and return the updated status and reward; reset: Resets the environment to its initial state.

[0189] The method provided in this disclosure can be applied to various types of task environments, including but not limited to the following five categories: (1) Web navigation task: The agent interacts with dynamic websites to perform tasks such as booking flights and extracting structured information. In this type of task, the agent needs to follow instructions, parse text and visual content, operate dynamic interfaces, and plan multi-step action sequences.

[0190] (2) Deep search tasks: Agents use tools such as browsers or Python interpreters to perform multi-step, goal-oriented queries. These tasks require agents to have strong information retrieval, multi-hop reasoning, long-term memory, and cross-source knowledge synthesis capabilities.

[0191] (3) Digital game tasks: intelligent agents explore and solve problems in an interactive game environment, emphasizing real-time decision-making, strategy development and adaptability to complex dynamic scenarios.

[0192] (4) Embodied tasks: Intelligent agents control virtual or physical entities (such as robots) to navigate, operate and perform tasks, which requires goal-oriented planning, spatial reasoning and robust perception-action association capabilities.

[0193] (5) Scientific Experiment Tasks: The agent conducts experiments and solves problems in physical environment simulation or knowledge-intensive scenarios, requiring precise execution, dynamic feedback interpretation, evidence-based reasoning, and iterative hypothesis correction.

[0194] All of the above-mentioned environments support the dynamic adjustment of the maximum number of interactions per task in the method provided in the embodiments of this disclosure. For example, in a web page navigation task, the initial maximum number of interactions per task can be set to 10 steps, and gradually increased to 15 steps, 20 steps or even higher as the agent's performance score improves, enabling the agent to progressively learn long-range navigation strategies.

[0195] In this embodiment of the disclosure, the policy update module 530 is used to collect trajectory data generated by the interaction between the agent 510 and the environment module 520, calculate the policy gradient using reinforcement learning algorithms (such as PPO, GRPO, REINFORCE++, etc.), and update the policy parameters of the agent 510, thereby improving its task completion ability. The policy update module 530 is used for: Data collection: Receive information such as state, action, reward, and score at each time step from agent 510, and cache it to form a trajectory segment.

[0196] Dominance estimation: Calculate the dominance value at each time step using generalized dominance estimation (GAE) or other methods.

[0197] Policy optimization: Calculate the loss function based on a preset RL algorithm (e.g., the pruning target of PPO), and update the agent's policy network parameters through backpropagation.

[0198] Parameter synchronization: The updated policy parameters are sent back to agent 510 for subsequent decision-making.

[0199] In this embodiment, the dynamic interaction count adjustment module 540 is used to detect the performance score of the agent in real time, dynamically adjust the maximum number of interactions per task (max_turn) according to preset heuristic rules (score increment segment calculation, dual-condition triggering, etc.), and synchronize the updated max_turn to the agent 510 and / or the environment module 520, thereby achieving closed-loop adaptive control of training difficulty. The dynamic interaction count adjustment module 540 is used for: Score reception: The performance score for each round is obtained in real time from agent 510.

[0200] Benchmark calculation: At each max_turn level, collect the initial K-step scores and calculate the benchmark average score avg_score.

[0201] Threshold generation: Based on the numerical range of avg_score, a three-segment nonlinear function is used to calculate the score increment score_increment and generate a dynamic threshold threshold.

[0202] Conditional judgment: Detect the subsequent score sequence to determine whether the conditions of "complete mastery" or "partial mastery" are met.

[0203] Parameter adjustment: Based on the conditions met, increase the current max_turn by 2Q (fully mastered) or Q (partially mastered), and check whether the preset upper limit finally_turn_max has been reached.

[0204] Feedback output: The updated max_turn is sent to agent 510 (and optionally to environment module 520) to constrain the interaction rounds of subsequent single tasks.

[0205] This disclosure enables the agent to adaptively increase or decrease the interaction depth according to the learning progress by dynamically adjusting the maximum number of interactions per task, `max_turn`. This mechanism leads to potential differences in the actual number of interaction rounds for different episodes: for example, a simple task might terminate successfully within 3 rounds, while a complex task might require 15 rounds; in the early stages of training, `max_turn` is small, and trajectories are generally short; as training progresses, `max_turn` gradually increases, and the trajectory length also grows. This inconsistency in trajectory length (variable-length sequences) presents engineering challenges for batch-based reinforcement learning training. For example, PPO and GRPO require trajectories in the same batch to have the same length for efficient tensor operations.

[0206] To address the aforementioned issues, this embodiment employs an engineering approach combining padding and masking to unify the processing of variable-length trajectories, enabling them to be efficiently input into the policy update module for batch calculation. The specific implementation is as follows: (1) Track length alignment (Padding) For B trajectories in a batch, first determine the maximum actual trajectory length Lmax = max(L1, L2, ..., LB) in the batch, where Lj is the actual interaction round of the j-th trajectory. For trajectories with a length less than Lmax, fill their end with a preset padding token. The padding token can be an all-zero vector, a special empty action identifier, or a repeated termination state representation, which does not participate in the actual policy gradient calculation.

[0207] For example, suppose a batch contains three trajectories with actual lengths of [5, 12, 8], then Lmax = 12. The trajectory of length 5 is filled with placeholders at positions 6 to 12; the trajectory of length 8 is filled with placeholders at positions 9 to 12. After filling, all trajectories are expanded to a uniform length of 12, forming a dense tensor of shape [B, Lmax, ...], which facilitates parallel computation on a graphics processing unit (GPU) or neural network accelerator (NPU).

[0208] (2) Invalid position mask In the filled trajectory, the data corresponding to the filled positions does not represent real interaction experience. Directly using this data for loss function calculation would introduce noise and mislead policy updates. Therefore, this disclosure introduces a mask matrix M, whose dimension is the same as the filled trajectory tensor, where: For the actual interaction step position, the mask value is set to 1 (valid); For filling placeholder positions, the mask value is set to 0 (invalid).

[0209] When the policy update module calculates the loss function, the mask matrix is ​​multiplied element by element by the loss components (such as policy loss and value function loss), so that the contribution at the filling position is set to zero and thus completely ignored.

[0210] During training, the actual number of interaction rounds for each single task is determined by its natural termination (success / failure) or reaching the current max_turn limit, without the need for a preset fixed length. When collecting batches, trajectories of different lengths are sampled from the experience replay buffer, and the maximum length of the current batch is calculated in real time and dynamic padding is performed. By adopting a variable-length sequence processing method that combines padding and masking, this disclosure achieves dynamic adjustment of max_turn while retaining the efficiency of batch training, enabling the dynamic adjustment mechanism to seamlessly integrate with mainstream reinforcement learning training frameworks (such as PPO and GRPO). By converting variable-length sequences into dense tensors, the parallel computing capabilities of GPUs / NPUs are fully utilized, avoiding iterative loops or fragmented computations caused by varying trajectory lengths. Compared to the static scheme that allocates the maximum possible length for each trajectory (such as a preset finally_turn_max), dynamic padding only allocates GPU memory according to the maximum length within the actual batch, reducing average GPU memory usage (especially when max_turn is small in the early stages of training). The masking mechanism ensures that the padding position does not contribute to the gradient, and the policy update is entirely based on real interaction experience, without introducing bias due to padding.

[0211] like Figure 5 As shown, the dynamic interaction number adjustment module 540 further includes 6 functional modules: initialization module 541, baseline calculation module 542, incremental calculation module 543, threshold setting module 544, performance detection module 545, and update control module 546.

[0212] For example, the dynamic interaction count adjustment module 540 can be set in Figure 4 In the intelligent agent control device 400 of the embodiment.

[0213] To more clearly demonstrate the system's operating mechanism, the module interactions during a training loop are described below in chronological order: Initialization phase: The dynamic interaction count adjustment module 540 sends the initial max_turn (e.g., 10) to the agent 510 and the environment module 520. The policy update module 530 initializes the policy parameters of the agent 510.

[0214] Benchmark calibration phase (at the start of each max_turn level): Agent 510 and environment module 520 interact for K steps (e.g., 30 steps) under the max_turn constraint. Agent 510 sends the score for each step to the dynamic interaction count adjustment module 540 in real time. The dynamic interaction count adjustment module 540 calculates the avg_score and generates the threshold.

[0215] Training and Testing Phase: Agent 510 continues to interact with the environment module 520, performing actions, receiving states, and receiving rewards. Agent 510 continuously sends its scores to the dynamic interaction count adjustment module 540. The dynamic interaction count adjustment module 540 maintains a sliding window to determine whether the conditions for complete or partial mastery are met. Simultaneously, agent 510 sends its interaction trajectory (state, action, reward) to the policy update module 530. The policy update module 530 periodically (e.g., every batch_size steps) calculates gradients, updates the policy parameters of agent 510, and sends the new parameters back to agent 510.

[0216] Difficulty escalation phase (when mastery conditions are met): The dynamic interaction count adjustment module 540 calculates a new max_turn (+2Q or +Q). The new max_turn is sent to the agent 510 and the environment module 520.

[0217] The system returns to the baseline calibration phase and recalculates avg_score and threshold under the new difficulty.

[0218] Termination phase (when max_turn reaches finally_turn_max or the total number of training steps is exhausted): The dynamic interaction number adjustment module 540 stops adjusting, and the system can continue to fine-tune the strategy or end the training.

[0219] In some implementations, the policy update module 530 can also provide the current policy evaluation metrics (such as policy entropy and value loss) to the dynamic interaction number adjustment module 540 as an auxiliary judgment condition. For example, when the policy entropy is too low (indicating that the agent may be trapped in a local optimum), even if the score meets the target, the increase of max_turn can be temporarily suspended to encourage more exploration.

[0220] In large-scale training scenarios Figure 5 The modules can be deployed in a distributed manner: multiple agent instances 510 run in parallel on different computing nodes, with each instance interacting with a replica of the environment module 520. The dynamic interaction number adjustment module 540 acts as a central controller, collecting the scores of all agents and maintaining a unified global value of max_turn (or maintaining an independent value for each agent). The policy update module 530 uses a distributed reinforcement learning framework to aggregate trajectory data from multiple agents for global policy updates.

[0221] In online inference scenarios (where no policy update is required), the policy update module 530 and / or the dynamic interaction count adjustment module 540 can be omitted.

[0222] Figure 5 The system architecture shown achieves modular design by decoupling the dynamic interaction count adjustment module from the agent, environment, and policy update modules. Each module has a clear responsibility and can be developed, tested, and replaced independently. For example, different RL algorithms (policy update module) or different task environments (environment module) can be replaced without modifying the dynamic interaction count adjustment module. The score flows from the agent to the dynamic interaction count adjustment module, and the adjusted max_turn flows back to the agent and environment modules, forming a complete negative feedback loop that automatically matches the training difficulty to the agent's capabilities. This embodiment also supports distributed training and multi-agent scenarios; simply increasing the number of agent instances does not change the dynamic interaction count adjustment module's unified management. This architecture can be easily integrated into reinforcement learning training frameworks as a pluggable component for interaction depth control.

[0223] Continue to refer to Figure 5 The dynamic interaction count adjustment module 540 may further include an initialization module 541, a baseline calculation module 542, an incremental calculation module 543, a threshold setting module 544, a performance detection module 545, and an update control module 546. For example, the dynamic interaction count adjustment module 540 may be set in... Figure 4 In the intelligent agent control device 400 of the embodiment.

[0224] In this embodiment of the disclosure, the initialization module 541 is used for parameter configuration and environment preparation before training begins. The output of the initialization module 541 includes the completed initialization of the training environment and parameter configuration. Key parameters include, but are not limited to: Set the initial maximum number of interactions, max_turn (adjust according to task complexity; default max_turn=10). Set the round increment to Q (adjust according to task complexity, the default is Q=1); Set the baseline number of consecutive steps K (adjust according to task complexity, default is K=30); Set the threshold for consecutive steps n (adjust according to task complexity, default is n=3); Set the baseline increment ratio r (adjust according to task complexity, default is 0.1); Set the maximum number of final interactions finally_turn_max (adjust according to task complexity, default is 50).

[0225] In this embodiment, the round increment Q refers to the specific value by which the maximum number of interactions increases when the update condition is triggered. This disclosure designs a two-level incremental mechanism: when the Agent has complete control, max_turn = max_turn + 2Q; when it has partial control, max_turn = max_turn + Q.

[0226] In this embodiment, the final maximum number of interactions (finally_turn_max) refers to a preset training termination limit, i.e., the highest interaction depth that the agent is allowed to reach. This parameter is set according to the complexity of the target task to prevent the training process from being extended indefinitely.

[0227] In this embodiment, the benchmark calculation module 542 is used to calculate the performance benchmark at the current interaction depth after each max_turn level update. K-step training is performed, and the benchmark average score avg_score is calculated and output.

[0228] In this embodiment of the disclosure, the incremental calculation module 543 is used to calculate the score increment score_increment based on the benchmark average score avg_score using a three-segment nonlinear function. For example, the score increment score_increment is calculated and output using the corresponding formula in the three-segment nonlinear function according to different benchmark scores.

[0229] In this embodiment, the threshold setting module 544 is used to set and output the judgment threshold for the current stage based on the baseline average score avg_score and the score increment score_increment. For example, threshold = avg_score + score_increment. The threshold dynamically drifts with the agent's initial performance in each stage, ensuring that the goal is both challenging and achievable.

[0230] In this embodiment, the performance detection module 545 is used to detect the performance score of the agent during the training process in real time and maintain an observation window. It records the score for each training step, maintains the score history for the most recent n steps and 2n steps, counts the number of steps that meet the target (the number of steps with a score > threshold), and outputs the real-time score sequence and target statistics.

[0231] In this embodiment of the disclosure, the update control module 546 is used to determine whether to trigger a max_turn update based on the performance detection result. For example, the determination logic is as follows: Full mastery condition: if the threshold is exceeded for n consecutive steps, then max_turn += 2Q; Partial mastery condition: If the threshold is exceeded in 1.5n out of 2n steps, then max_turn+=Q.

[0232] The training terminates when `finally_turn_max` is reached or the maximum number of training steps is reached. The updated `max_turn` or the training termination signal is output.

[0233] This disclosure establishes a heuristic dynamic feedback control mechanism based on the real-time learning performance of an agent, aiming to solve the fundamental problem of the disconnect between interaction depth and learning progress in traditional control methods. Traditional agent control methods suffer from two major drawbacks: first, the use of a fixed maximum number of interactions limits the agent's capabilities when facing complex tasks; second, the use of a static round-increasing strategy ignores the agent's actual learning progress, easily leading to training oscillations or slow convergence. This disclosure achieves closed-loop adaptive control of the training process by introducing a score-based heuristic dynamic adjustment mechanism. The system monitors the agent's performance score in real time and automatically determines when and how to increase the number of interaction rounds based on its mastery level at the current interaction depth. This mechanism ensures on-demand allocation of computing resources, enabling the agent to evolve efficiently along the optimal learning path.

[0234] This disclosure provides a heuristic dynamic adjustment mechanism: abandoning the approach of presetting a fixed maximum number of interaction rounds or simply increasing them linearly, it adaptively determines when and how to increase the number of interaction rounds by detecting the agent's performance score at the current interaction depth in real time. This mechanism includes: Real-time feedback: The score at each training step is used as a feedback signal to evaluate the learning status of the agent; Adaptive threshold: Dynamically set the judgment threshold based on the current benchmark performance to ensure that the goal is both challenging and achievable; On-demand allocation: Only increase the number of interactions after the agent has fully mastered the ability to solve the task of current difficulty, so as to avoid wasting resources.

[0235] This disclosure provides a progressive learning path: by starting with low interaction counts (simple tasks) and gradually increasing interaction counts (complex tasks) only after the agent has mastered the current difficulty, a learning curve that conforms to cognitive laws and progresses from easy to difficult is constructed. This design has the following advantages: Reduce the search space: initially limit the number of interactions to reduce the complexity of strategy search; Stable training process: Avoid training breakdowns caused by directly facing complex tasks when the agent's capabilities are insufficient; Enhancing generalization ability: Through progressive learning, the agent can build more solid basic skills; This disclosure provides a dual-condition triggering mechanism: it designs two evaluation dimensions, "complete mastery" and "partial mastery," achieving a balance between training robustness and evolution speed. Complete Path Mastery (Continuous Target Achievement): Requires exceeding the threshold for n consecutive steps, corresponding to a larger round span (+2Q), suitable for rapid growth phase; Partial Mastery Path (Proportion Achievement): Allows for certain fluctuations (achieving the target percentage within 2n steps), corresponding to a smaller round span (+Q), enhancing tolerance to training noise.

[0236] On the one hand, dynamic benchmark calibration is provided: a benchmark score is independently calculated for each interaction level (maximum number of interactions), eliminating evaluation bias when crossing difficulty levels. On the other hand, a three-stage adaptive increment is designed: a non-linear increment formula is designed for different performance ranges to scientifically balance learning motivation and convergence difficulty. In addition, a two-level update strategy is provided: through a dual-track system of "continuous achievement (complete mastery)" and "proportional achievement (partial mastery)," a balance between robustness and speed is achieved. Closed-loop automatic control is realized: the entire training process requires no manual intervention, and the system automatically completes a smooth transition from simple to complex tasks.

[0237] This disclosure presents a novel solution: 1. Establish a dynamic mapping between performance and interaction depth: By detecting the performance score of the agent in real time, a dynamic mapping relationship between learning progress and interaction depth is established to achieve true adaptive adjustment.

[0238] 2. Achieve closed-loop automatic control: The entire training process requires no manual intervention. The system automatically adjusts the training parameters based on the actual performance of the agent, ensuring that the training process is always in the optimal state.

[0239] 3. Optimize resource allocation: Through a progressive learning path, ensure that computing resources are concentrated on tasks within the agent's current capabilities, avoiding wasting resources on tasks that are too difficult or too easy.

[0240] 4. Improve training stability: The dual-condition triggering mechanism ensures that the number of interactions is increased only after the agent has fully grasped the current difficulty, effectively avoiding training oscillations.

[0241] The intelligent agent control method provided in this disclosure can be widely applied in the field of robot control. Figure 6 A schematic diagram illustrating the application of the intelligent agent control method provided in this disclosure embodiment to the field of robotics is shown. Figure 6 As shown, in order to train and apply the intelligent agent, the system provided in this application embodiment includes: a robot 100, an intelligent agent 510 (which can be deployed in the controller of the robot 100 or in an independent computing unit) and a server 300.

[0242] Server 300 establishes a data connection with robot 100 through a communication interface to obtain environmental interaction data of robot 100 during task execution. The environmental interaction data includes, but is not limited to: Status data: Robot 100 collects its own status data through its onboard sensors (such as inertial measurement unit, joint angle sensor, force sensor, vision camera, LiDAR, etc.), including but not limited to joint angle, joint angular velocity, end effector position, body posture, load torque, etc. Motion data: Control command data corresponding to the actions performed by robot 100, such as joint motor torque, desired position, speed command, etc. Reward value: The immediate or delayed reward value that Robot 100 receives after performing an action, based on the degree of task completion; Target value: The pre-set task objectives that the robot is to complete, such as "reaching a designated location", "grabbing a specific object", "assembling parts", etc.

[0243] In addition, the server 300 also obtains the actual target value that the robot 100 actually achieves after performing the action (such as the actual end position reached, whether the object was successfully grasped, etc.) for subsequent reward correction and performance evaluation.

[0244] In this application scenario, the agent's performance score can be calculated based on the actual effect of the robot performing the task. For example, server 300 determines the task achievement degree of robot 100 after performing the action based on the state parameters, action data, and actual target value of the first of two adjacent time points, and maps this achievement degree to a score in the range of 0 to 100. For example: If the robot successfully completes the grasping operation, it will score 100 points. If the task is partially completed (e.g., touching the object but not picking it up), the score is 50. If it fails completely, the score is 0.

[0245] This score serves as a feedback signal for subsequent dynamic adjustments to the maximum number of interactions per task (i.e., the maximum number of control steps allowed for the robot to execute a complete task).

[0246] In robot task training, a single task can be defined as the process by which a robot starts from an initial state (e.g., the starting position) and completes a full objective (e.g., "moving an object from point A to point B") through a series of actions (e.g., moving, grasping, placing, etc.). The maximum number of interactions in a single task refers to the upper limit of the number of control steps that the robot is allowed to execute during this process.

[0247] The dynamic interaction count adjustment module (which can be deployed in the controller of server 300 or robot 100) in this embodiment monitors the performance score obtained by agent 510 after each control step in real time. Based on the aforementioned scoring heuristic method, the system: Given the current maximum number of interactions, collect the initial K-step scores and calculate the baseline average score avg_score; Based on the interval (negative score, medium score, high score) of avg_score, the score increment score_increment is calculated using a three-segment nonlinear function, and a dynamic threshold threshold = avg_score + score_increment is generated. Monitor the achievement of scores in subsequent control steps. When the conditions of "complete mastery" (score exceeds the threshold for n consecutive steps) or "partial mastery" (achievement of the target in 1.5n steps within 2n steps) are met, increase the maximum number of interactions for the current single task by 2Q or Q respectively, thereby gradually increasing the complexity of the robot task.

[0248] For example, the initial maximum number of interactions is set to 10 steps (corresponding to simple tasks, such as "moving forward 1 meter"). When the robot can stably complete 10 consecutive tasks within this limit, the system will increase the limit to 13 or 16 steps, enabling it to tackle more complex tasks (such as "moving objects around obstacles").

[0249] Server 300 uses updated environmental interaction data (including corrected reward values) to train the agent 510 corresponding to the robot control network. Specifically, the policy update module (which can be deployed on server 300) collects trajectory data of robot 100 interacting with the environment and uses reinforcement learning algorithms (such as PPO, GRPO, etc.) to update the policy parameters of agent 510, enabling agent 510 to better adapt to the increasing task difficulty.

[0250] During training, the aforementioned dynamic adjustment mechanism and policy update work together: the dynamic interaction number adjustment module decides when to increase task complexity based on the agent's real-time performance, while the policy update module continuously optimizes the agent's decision-making ability. Together, they enable the robot to gradually learn from simple basic actions (such as "extending joints") to complex long-range operations (such as "completing the assembly line in sequence").

[0251] After training agent 510 is complete, the trained agent 510 is used to control the actions of the target robot (which can be the same robot 100 or another robot). At this point, the system can choose whether to continue enabling dynamic interaction adjustment based on actual task requirements. Fixed Depth Mode: For tasks with known stable complexity, the maximum number of interactions per task can be fixed to the upper limit reached during training, and the trained strategy can be used directly for control.

[0252] Adaptive Depth Mode: In real-world environments, if task complexity may change dynamically (e.g., different users have significantly different instructions), the dynamic adjustment module can still be enabled to adjust the interaction depth online based on real-time performance scores, balancing task success rate and response speed.

[0253] For example, in an industrial assembly scenario, a robot receives the task of "inserting a screw into a hole and tightening it." The agent gradually adjusts the pose of the end effector based on visual feedback. If it fails to complete the task within the initially allowed 10 steps, the system can determine whether to increase the number of steps (e.g., expand to 13 steps) based on the real-time score, thus giving the robot more opportunities to try and improving the assembly success rate.

[0254] The above application scenario embodiments are merely exemplary implementations of this disclosure in the field of robotics. Those skilled in the art should understand that the methods of this disclosure are also applicable to other related fields, such as autonomous vehicle control, industrial robotic arm operation, and drone navigation, all of which fall within the protection scope of this disclosure.

[0255] The intelligent agent control method provided in this disclosure can be applied to the field of intelligent question answering. Figure 7 A schematic diagram illustrating the application of the intelligent agent control method provided in this disclosure embodiment to the field of intelligent question answering is shown. Figure 7 As shown, the system includes: terminal 110, server 120, and terminal 130. Terminal 110 and server 120, and server 120 and terminal 130, communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0256] Terminal 110 is a computer device with a model training client installed, used to configure training parameters, monitor the training process, and display training results. Terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart TV, wearable device, or in-vehicle terminal, etc. Figure 7 The example only uses a desktop computer as terminal 110, but it is not a limitation.

[0257] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In this embodiment, server 120 deploys a question-answering agent (i.e., an agent to be trained or already trained), an environment module (simulating user interaction or interfacing with a real question-answering interface), a policy update module, and a dynamic interaction count adjustment module. Server 120 is responsible for executing the agent control method of this disclosure, performing multi-round interaction training on the question-answering agent, and dynamically adjusting the maximum number of interactions per task (i.e., the maximum number of rounds in a single question-answering dialogue) based on the agent's real-time performance.

[0258] Terminal 130 is a computer device with an application that has a question-and-answer function installed. This question-and-answer function can be a native application of Terminal 130 or a third-party application. Terminal 130 can be a smartphone, tablet, laptop, desktop computer, smart TV, wearable device, or in-vehicle terminal, etc. Figure 7 The example of terminal 130 being a desktop computer is used for illustration only, but it is not a limitation. After the question-answering agent is trained, terminal 130 is used to interact with the question-answering agent deployed on server 120, receive user-inputted question-answering requests, and display the answers returned by the agent.

[0259] During the training phase, the question-answering agent on server 120 interacts with the environment module (e.g., simulated users or standard question-answering datasets) in multiple rounds. In each round, the agent receives a user question (or a question generated by the environment), generates an answer, and the environment module calculates a performance score (ranging from 0 to 100) based on metrics such as the accuracy, completeness, and user satisfaction of the answer. This score is fed back to the dynamic interaction count adjustment module in real time. The dynamic interaction count adjustment module in server 120 dynamically determines whether to increase the upper limit based on the agent's score sequence under the current maximum number of interactions in a single task (i.e., the maximum number of question-answering rounds allowed in a single question-answering dialogue, for example, initially set to 5 rounds). The specific process is as follows: Benchmark calibration: At the current maximum number of interactions, collect scores for the initial K (e.g., 30) question-and-answer rounds and calculate the benchmark average score avg_score.

[0260] Threshold generation: Based on the numerical range (negative score, medium score, high score) of avg_score, a three-segment nonlinear function is used to calculate the score increment score_increment, and a dynamic threshold threshold = avg_score + score_increment is generated.

[0261] Performance assessment: Continue to collect scores from subsequent question-and-answer rounds to determine whether the conditions of "complete mastery" (scores exceeding the threshold for n consecutive steps) or "partial mastery" (achieving the standard in 1.5n steps within 2n steps) are met.

[0262] Deep adjustment: If complete mastery is achieved, increase the maximum number of interactions by 2Q (e.g., from 5 rounds to 7 rounds); if partial mastery is achieved, increase Q (e.g., from 5 rounds to 6 rounds); otherwise, remain unchanged.

[0263] Iterative loop: Repeat the above steps until the preset maximum number of interactions per task is reached (e.g., 15 rounds).

[0264] Through the aforementioned closed-loop feedback control, the question-answering agent can gradually learn from short dialogues (simple questions, such as "How is the weather?") to long dialogues (complex multi-turn questions, such as "reasoning and answering based on context"), avoiding training failure due to facing long dialogues too early, and also avoiding wasting too much computing resources on short dialogues.

[0265] During training, the policy update module in server 120 collects trajectory data of the agent's interaction with the environment (including user questions, agent answers, scores, reward signals, etc.) and uses reinforcement learning algorithms (such as PPO, GRPO, etc.) to update the policy parameters (i.e., the weights of the large language model) of the question-answering agent. The dynamic interaction number adjustment module works in conjunction with the policy update module: the former controls the rate of increase in task difficulty, while the latter improves the agent's basic capabilities. Together, they drive the agent's progressive learning from simple to complex.

[0266] After training the question-answering agent, server 120 exposes the inference interface of the trained agent to terminal 130. The user inputs questions through a question-answering application on terminal 130, and the question-answering agent on server 120 engages in multiple rounds of interaction based on the user's continuous questions. During this process, server 120 can select one of the following modes: Fixed Depth Mode: For scenarios with known stable complexity (such as FAQ), the maximum number of interactions per task is fixed to the upper limit reached during training (such as 10 rounds), and the answer is returned directly.

[0267] Adaptive Depth Mode: For open-ended dialogue scenarios, the dynamic interaction count adjustment module can be enabled to dynamically adjust the maximum number of rounds allowed in the current dialogue based on real-time user feedback (such as whether the user is satisfied or whether they ask follow-up questions). This avoids unresolved issues due to too few rounds or wasted computing resources and user time due to too many rounds.

[0268] For example, when a user asks "Please recommend a mobile phone" on terminal 130, the agent replies, "What is your budget?" If the user further replies "Around 5000 yuan," the agent then recommends a specific model. If the system's preset initial maximum number of interactions is 5 rounds, but 3 rounds are actually required, the system will not force an increase in the number of rounds. If the user asks multiple follow-up questions (such as "How is the photo quality?" or "What about the battery life?"), the system will determine whether to temporarily increase the allowed number of interaction rounds based on the real-time score to ensure a complete answer to the user's questions.

[0269] The above application scenario embodiments are merely exemplary implementations of this disclosure in the field of intelligent question answering. Those skilled in the art should understand that the methods of this disclosure are also applicable to other software and hardware combined scenarios, such as intelligent customer service systems, voice dialogue assistants, online education tutoring, etc., all of which fall within the protection scope of this disclosure.

[0270] The solution provided in this disclosure improves training efficiency in two ways: First, through a progressive learning path and dynamic threshold mechanism, the agent can evolve efficiently on the optimal learning curve, avoiding wasting time on tasks that are too difficult or too easy. Initially, the number of interactions is limited, reducing the policy search space and accelerating the learning of basic skills. Difficulty is increased only after full mastery, avoiding training oscillations. A dual-condition triggering mechanism ensures that fast learners can quickly progress to higher difficulty levels. Second, it saves computational resources: Interactions are allocated on demand, avoiding excessive interaction rounds on simple tasks or repeated failures on complex tasks due to insufficient rounds. Simple tasks can be completed with fewer interactions, saving computation. Complex tasks allow for higher interaction rounds only after the agent's capabilities improve, increasing sampling efficiency. A dynamic adjustment mechanism ensures resources are concentrated on tasks within the agent's current capabilities. Furthermore, the agent's adaptability to long-term tasks is enhanced: By progressively increasing the number of interactions, the agent can gradually build the ability to handle long-term complex tasks, breaking through the limitation of fixed interaction rounds. Starting from 10 steps, it gradually increases to 50 steps or even higher, ensuring that the agent fully masters each stage, establishing a solid skill foundation, and avoiding training collapse caused by directly facing extremely long interaction tasks. Meanwhile, training stability is improved: the dual-condition triggering mechanism and dynamic threshold setting ensure the stability of the training process, avoid strategy collapse caused by increasing difficulty too early, ensure that the conditions for complete mastery are met continuously and ensure skill solidification, allow certain fluctuations for partial mastery conditions to enhance fault tolerance, and eliminate evaluation bias across difficulty levels through dynamic benchmark calibration.

[0271] The solutions provided in this disclosure are tested on the ScienceWorld public dataset, a text-based virtual environment centered on completing tasks in a standardized basic science curriculum. The APIs provided by this dataset are designed to support scientific exploration through text-driven reasoning loops. These APIs enable agents to conduct experiments by interacting with various scientific instruments and performing operations such as measuring temperature, connecting circuits, and mixing chemicals.

[0272] Based on the ScienceWorld dataset, eight task subsets were selected from the original dataset, forming the following five sub-tasks: Find: Item search task; Measure: Temperature measurement task; Lifespan: Lifecycle tasks; Test-Cond: Electrical experiment task; Chem-Mix: Chemical Mixing Task.

[0273] The solution provided in this disclosure performs exceptionally well on the open-source dataset ScienceWorld, significantly outperforming other open-source models and achieving the highest total score of 64.64.

[0274] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0275] Optionally, the data involved in the embodiments of this disclosure may be stored in a computer device, or may be stored based on cloud storage technology or blockchain technology, without limitation.

[0276] See Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Figure 8 As shown, the computer device in this embodiment may include one or more processors 1701, a memory 1702, and an input / output interface 1703. The processor 1701, memory 1702, and input / output interface 1703 are connected via a bus 1704. The memory 1702 stores a computer program, which includes program instructions. The input / output interface 1703 receives and outputs data, such as for data interaction between the host machine and the computer device, or for data interaction between various virtual machines within the host machine. The processor 1701 executes the program instructions stored in the memory 1702.

[0277] The processor 1701 can perform the following operations: acquiring a first performance index sequence within a first training window when the agent interacts with the environment module through a communication interface in multiple rounds under the first single-task maximum number of interactions; determining a first benchmark performance index based on the first performance index sequence, and determining a first threshold based on the first benchmark performance index; acquiring a second performance index sequence between the agent and the environment module within a second training window under the first single-task maximum number of interactions; and adjusting the first single-task maximum number of interactions between the agent and the environment module to the second single-task maximum number of interactions based on the second performance index sequence and the first threshold.

[0278] In some feasible implementations, the processor 1701 may be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0279] The memory 1702 may include read-only memory and random access memory, and provides instructions and data to the processor 1701 and input / output interface 1703. A portion of the memory 1702 may also include non-volatile random access memory. For example, the memory 1702 may also store device type information.

[0280] In practice, the computer device can execute the implementation methods provided by each step in any of the above method embodiments through its built-in functional modules. For details, please refer to the implementation methods provided by each step in the figure shown in the above method embodiments, which will not be repeated here.

[0281] This disclosure provides a computer device including a processor, an input / output interface, and a memory. The processor retrieves a computer program from the memory and executes the steps of the method shown in any of the above embodiments.

[0282] This disclosure also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the intelligent agent control method provided in each step of any of the above embodiments. Specific implementations of each step in each of the above embodiments can be found therein and will not be repeated here. Furthermore, the beneficial effects of using the same method will not be repeated here either. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this disclosure, please refer to the description of the method embodiments of this disclosure. As an example, the computer program can be deployed to execute on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network.

[0283] The computer-readable storage medium can be the intelligent agent control device provided in any of the foregoing embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0284] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various alternative embodiments described above.

[0285] The above-disclosed embodiments are merely preferred embodiments of this disclosure and should not be construed as limiting the scope of this disclosure.

Claims

1. A method for controlling an intelligent agent, characterized in that, include: Get the sequence of first performance indicators within the first training window when the agent interacts with the environment module through the communication interface in multiple rounds under the maximum number of interactions in the first task. A first benchmark performance index is determined based on the first performance index sequence, and a first threshold is determined based on the first benchmark performance index. Obtain the second performance index sequence of the agent and the environment module within the second training window under the maximum number of interactions in the first single task; Based on the second performance index sequence and the first threshold, the first single-task maximum number of interactions between the agent and the environment module in multiple rounds is adjusted to the second single-task maximum number of interactions.

2. The method as described in claim 1, characterized in that, Determining the first threshold based on the first benchmark performance index includes: The first incremental index is obtained based on the first benchmark performance index; The first threshold is determined based on the first baseline performance index and the first incremental index.

3. The method as described in claim 2, characterized in that, The first incremental index is obtained based on the first benchmark performance index, including: The first benchmark performance index is input into an adaptive incremental generation model to determine candidate incremental indices. The adaptive incremental generation model is configured to: if the first benchmark performance index is determined to be lower than or equal to a first preset threshold, then determine the candidate incremental index based on the benchmark incremental ratio; if the first benchmark performance index is determined to be higher than the first preset threshold, then determine the candidate incremental index based on the first benchmark performance index and the benchmark incremental ratio. The first incremental indicator is determined based on the candidate incremental indicators.

4. The method as described in claim 2, characterized in that, The adaptive incremental generation model is further configured as follows: If it is determined that the first benchmark performance index is higher than the first preset threshold and less than or equal to the second preset threshold, then the difference between the preset value and the first benchmark performance index is obtained, and the candidate incremental index is determined based on the difference and the benchmark incremental ratio. If it is determined that the first benchmark performance index is greater than the second preset threshold, then the candidate incremental index is determined based on the difference, the benchmark incremental ratio, and the attenuation coefficient.

5. The method as described in claim 3 or 4, characterized in that, Determining the first incremental indicator based on the candidate incremental indicators includes: Obtain the maximum value of the candidate incremental index and the first preset parameter; The minimum value between the maximum value and the second preset parameter is obtained; The first incremental index is determined based on the minimum value.

6. The method as described in claim 1, characterized in that, The second training window includes a first consecutive step threshold; Specifically, adjusting the first maximum number of interactions per single task for the agent to interact with the environment module in multiple rounds, based on the second performance index sequence and the first threshold, to the second maximum number of interactions per single task includes: If, within the first consecutive step threshold, all performance indicators in the second performance indicator sequence are greater than the first threshold, then the first single-task maximum interaction count is increased by the first round increment to obtain the second single-task maximum interaction count.

7. The method as described in claim 6, characterized in that, The second training window also includes a second consecutive step threshold; The method of adjusting the maximum number of interactions in a single task between the agent and the environment module in multiple rounds, based on the second performance index sequence and the first threshold, to the maximum number of interactions in a single task, further includes: If, within the second consecutive step threshold, the proportion of performance indicators in the second performance indicator sequence that are greater than the first threshold reaches a predetermined proportion threshold, then the first maximum number of interactions is increased by the second round increment to obtain the second maximum number of interactions per task. Wherein, the increment of the first round is greater than the increment of the second round, and the threshold for the second consecutive number of steps is greater than the threshold for the first consecutive number of steps.

8. The method as described in claim 1, characterized in that, Also includes: Obtain the sequence of third performance indicators within the third training window when the agent interacts with the environment module through the communication interface in multiple rounds under the maximum number of interactions in the second single task; A second benchmark performance index is determined based on the third performance index sequence, and a second threshold is determined based on the second benchmark performance index; Obtain the sequence of fourth performance indicators of the agent and the environment module within the fourth training window under the maximum number of interactions in the second single task; Based on the fourth performance index sequence and the second threshold, the second maximum number of single-task interactions between the agent and the environment module is adjusted to the third maximum number of single-task interactions, until the preset maximum number of single-task interactions is reached.

9. An intelligent agent control device, characterized in that, include: The performance indicator acquisition unit is used to acquire the first performance indicator sequence within the first training window when the agent interacts with the environment module through the communication interface in multiple rounds under the maximum number of interactions in the first single task. A benchmark threshold determination unit is used to determine a first benchmark performance index based on the first performance index sequence, and to determine a first threshold based on the first benchmark performance index. The performance index acquisition unit is further configured to acquire the second performance index sequence of the agent and the environment module within the second training window under the maximum number of interactions in the first single task; The interaction count adjustment unit is used to adjust the first single-task maximum interaction count of the agent and the environment module to the second single-task maximum interaction count based on the second performance index sequence and the first threshold.

10. A computer device, characterized in that, Includes processor, memory, and input / output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive data and output data, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-8.

Images