Progressive thinking chain length calibration method for large model agent dialogue scene

Abstract

A progressive thought chain length calibration method for large-scale intelligent agent dialogue scenarios is proposed. For the input problem of large-scale intelligent agents, after estimating the global distribution of inference length and the length distribution of correct inference trajectories through multiple inference trajectories, the optimal inference length that maximizes the conditional probability of correct inference is calculated. For correct inference trajectories exceeding the optimal inference length, a length penalty is applied to guide excessively long correct inference trajectories to converge towards the optimal length. Simultaneously, a length reward coefficient is calculated based on the proportion of incorrect inference trajectories and the length penalty. Finally, based on a group relative policy optimization algorithm, the total reward and trajectory-level advantage value are calculated according to the length reward coefficient to update model parameters, achieving joint optimization of inference length compression and inference accuracy. This invention can significantly reduce the average response latency while ensuring response quality, achieving optimized computational allocation that is "deeper where necessary and shallower where necessary."

Description

Technical Field

[0001] This invention relates to a technology in the field of neural networks, specifically a progressive thought chain length calibration method for large-scale language model dialogue scenarios. Background Technology

[0002] Existing large-scale intelligent agent technology significantly improves performance in complex mathematical, code generation, and logical analysis tasks by extending the chain of thought and dynamically allocating computational resources for self-verification and logical correction. However, in dialogue applications, excessively long chain of thought directly translates into increased API call costs and longer user wait times, impacting the real-time interactive experience. Furthermore, when faced with simple queries, excessive self-verification may cause the model's logic to diverge, even deviating from the correct answer. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies that rely on manually labeled intermediate results or tree-structured intermediate inference steps, which cannot be applied to scenarios with only result feedback, and suffer from slow response times for simple queries and excessive thinking for complex problems in actual service deployments. It proposes a progressive thought chain length calibration method for large-scale intelligent agent dialogue scenarios. Through an optimal inference length estimation stage and dynamic reward optimization, the method dynamically adjusts the model's thinking steps based on the actual complexity of the query, significantly reducing average response latency while ensuring response quality. This achieves optimized computational allocation that is "deeper where necessary and shallower where necessary," providing high-precision and high-efficiency services for users of large-scale intelligent agent dialogue.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a progressive thought chain length calibration method for large-scale agent dialogue scenarios. Addressing the input problem of large-scale agents, it estimates the global distribution of inference length and the length distribution of correct inference trajectories using multiple inference trajectories, then calculates the optimal inference length that maximizes the conditional probability of correct inference. For correct inference trajectories exceeding the optimal length, a length penalty is applied to guide them towards convergence. Simultaneously, a length reward coefficient is calculated based on the proportion of incorrect inference trajectories and the length penalty. Finally, based on a group relative policy optimization algorithm, the total reward and trajectory-level advantage value are calculated using the length reward coefficient to update model parameters, achieving joint optimization of inference length compression and inference accuracy, providing users with an optimal experience in terms of efficiency and accuracy in agent dialogue.

[0006] The large model agent mentioned includes, but is not limited to, base models of different sizes, including deep inference models with parameter scales of 1.5B, 4B and 7B, and the number of training steps decreases as the model's capabilities increase.

[0007] The optimal inference length ,in: The optimal length for uncrunting is defined as follows: Its value may exceed the reasonable range of the thought chain length, so it needs to be truncated. , , respectively, are the mean and standard deviation of all trajectory lengths. , These are the mean and standard deviation of the correct trajectory length, respectively. It is a set of trajectory lengths.

[0008] Preferably, the optimal inference length is recalculated with each iteration of the training process. When the optimal inference length is greater than or equal to the maximum length of all correct trajectories, the length penalty is zero, and the method degenerates into classic group relative policy optimization training, with the training objective shifting to enhancing the model's inference ability rather than compressing the length.

[0009] The length penalty refers to one of the total rewards for a trajectory, which is usually negative and can be linearly superimposed on the result reward.

[0010] The length reward coefficient ,in: The proportion of erroneous trajectories. For the correctness reward list, For the length reward list, this coefficient ensures that the normalized advantage value of the correct trajectory is non-negative, mean() is the average value, and min() is the minimum value.

[0011] The length reward list is a matrix composed of the length rewards of all reasoning trajectories, the first of which is... The length of the trajectory is rewarded ,in: A reward is given for the correctness of the trajectory. The length of the trajectory is awarded as a reward. For the first The inference length of the trajectory, To estimate the optimal inference length, .

[0012] The first Total reward for each trajectory ,in: Rewards for accuracy, As a length bonus, For the correctness reward list, For the length reward list, This is the dynamic length reward coefficient.

[0013] The trajectory-level dominance value is calculated using a group normalization method: Where: G is the number of trajectories sampled in each group, For the first The total reward for each trajectory, where mean() is the average and std() is the variance.

[0014] This invention relates to a progressive thought chain length calibration system for large-scale intelligent agent dialogue scenarios that implements the above-mentioned method, comprising: a sampling unit, a length estimation unit, a reward calculation unit, and an update unit, wherein: the sampling unit performs trajectory sampling based on the input question to obtain several thought chains and inference results; the length estimation unit estimates the mean and variance of the lengths of all answers and correct answers based on the sampled trajectories to obtain the optimal length estimate; the reward calculation unit calculates the length penalty and correctness reward based on the optimal length and the accuracy of each trajectory to obtain the reward for each trajectory; and the update unit calculates the loss based on the reward for each trajectory and updates the model weights to obtain new model weights.

[0015] Technical effect

[0016] In the optimal inference length estimation stage of this invention, the optimal length that maximizes the probability of correct inference is dynamically estimated in each training iteration based on the trajectory group sampled for a given problem using the current strategy. In the dynamic reward optimization stage, an adaptive reward function is designed based on the estimated optimal length, and dynamic coefficients ensure the correct optimization direction of the correct trajectory. Compared with existing technologies, this invention achieves fine-grained control of trajectory rewards during training iterations, preventing the model from excessively reducing the length of the thought chain while improving the model's response accuracy on difficult problems. It can automatically adjust the compression strategy according to the relative difficulty of the problem—actively compressing for simple problems and retaining the necessary length for difficult problems. This avoids the "one-size-fits-all" problem of static design while achieving an average length compression of up to 52.5%, and improving accuracy, with an accuracy improvement of up to 16.6% on difficult benchmarks such as AIME25. It fundamentally avoids assigning a negative advantage to correct but long inference trajectories, protecting the model's exploration ability on complex problems without requiring additional value models or complex curriculum design. Only 150-50 training steps are needed to achieve results superior to hundreds of training steps of the baseline, and the efficiency further improves as the model size increases. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the invention;

[0018] Figure 2 This is a flowchart of an implementation example;

[0019] Figure 3 This is a schematic diagram illustrating the effect of an example. Detailed Implementation

[0020] like Figure 1 and Figure 2 As shown, this embodiment relates to a progressive thought chain length calibration method for dialogue scenarios of large-scale intelligent agents, including:

[0021] Step 1: Training Configuration and Data Preparation

[0022] Training data: The DeepScaleR-preview dataset was used, which contains 40,000 math question-answer pairs from AIME, AMC, Omni-MATH, and STILL, covering different difficulty levels, and no difficulty-based sampling strategy was adopted.

[0023] Hyperparameter settings: Batch size 64, group size G=8, minimum batch size 16, maximum response length 8000 tokens, learning rate 1×10⁻⁶. -6 To improve efficiency, the KL divergence loss is omitted.

[0024] Step 2, Optimal Inference Length Estimation Implementation, specifically includes:

[0025] 2.1 Given the current policy parameters and input issues The distribution of all response lengths l is as follows The conditional distribution of the correct response length is as follows: According to Bayes' theorem, the conditionally correct probability... By comparing the logarithms of the values ​​and taking the derivative, we obtain the result if and only if... At that time, there exists a unique maximum point: ;

[0026] 2.2 Utilizing the inherent group sampling characteristic of the GRPO algorithm, G inference trajectories are sampled for problem q in each training forward propagation. Record the length of each trajectory and correctness Estimating distribution parameters based on sample statistics: , ,in: , .

[0027] 2.3 Considering the stability of the estimation, the final calculated optimal length estimate is the result after truncation. .

[0028] In this embodiment: In each training iteration, 8 reasoning trajectories are sampled for each question in the training set of step 1, and the token count of each reasoning trajectory is recorded. Correctness is determined by answer matching. Then, the average length of all trajectories is calculated in real time. Standard deviation The mean of the correct trajectory (if it exists) and standard deviation ; final calculation And cut to Get within range .

[0029] Typically, for MATH 500 level problems, Approximately 2500-3500 tokens; for AIME level 25 problems, It can reach 5000-7000 tokens.

[0030] Step 3, Dynamic Reward Calculation and Policy Update: Penalize correct trajectories whose length exceeds the optimal estimate, guiding them to converge toward the optimal length while preserving the exploratory long inference required for difficult problems. Specifically, this includes:

[0031] Rewards are given for the accuracy of each trajectory calculation: If the answer is correct, then 0 is given.

[0032] Length calculation reward: only for correct and The trajectory In all other cases, the value is 0.

[0033] Calculate the dynamic coefficient: ,in: This represents the error rate within the group.

[0034] Total Rewards: .

[0035] Calculate the within-group mean and standard deviation to obtain the normalized odds. It is used for gradient updates in the GRPO policy.

[0036] Step 4, Performance Monitoring and Checkpoint Selection and Dialogue Inference Service Deployment: The accuracy of the inference model's responses is monitored in real time using automated model performance tracking technology. After the model training is completed, a high-throughput, low-latency dialogue service system is built to verify the effectiveness and stability of the dynamic length constraint strategy in a real production environment.

[0037] The aforementioned real-time monitoring preferably selects checkpoints near the plateau period of accuracy increase, or selects model checkpoints based on user satisfaction feedback through real-time online updates.

[0038] This embodiment uses vLLM as the service framework, combining multi-GPU parallel inference with a dynamic length control mechanism to achieve end-to-end optimization from training to deployment. To test service response efficiency and accuracy, dialogue simulations were performed using the MATH500, AIME25, and AMC23 datasets. The service API was set to temperature=0.6, top_p=1.0, and max_tokens=32768. Four responses were sampled for each question, and the average response length and response accuracy were calculated for each dataset.

[0039] The test results under different simulation model services and different dialogue question-and-answer tests are shown in the table.

[0040]

[0041] Compared with existing technologies, this invention can compress the length of the response thought chain by up to 52.5% on average, and improve the average accuracy of difficult problems by up to 16%.

[0042] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A progressive thought chain length calibration method for large model agent dialogue scenarios, characterized in that, To address the input problem of large-scale intelligent agents, this paper estimates the global distribution of inference length and the length distribution of correct inference trajectories by using multiple inference trajectories, and then calculates the optimal inference length that maximizes the conditional probability of correct inference. For correct inference trajectories exceeding the optimal length, a length penalty is applied to guide them towards the optimal length. Simultaneously, a length reward coefficient is calculated based on the proportion of incorrect inference trajectories and the length penalty. Finally, based on a group relative policy optimization algorithm, the total reward and trajectory-level advantage value are calculated according to the length reward coefficient to update model parameters, achieving joint optimization of inference length compression and inference accuracy.

2. The progressive thought chain length calibration method for large-scale intelligent agent dialogue scenarios according to claim 1, characterized in that, The optimal inference length where is , , are the mean and standard deviation of all trajectory lengths, respectively, , are the mean and standard deviation of correct trajectory lengths, respectively, is the set of trajectory lengths.

3. The method of claim 2, wherein the method is characterized by, The optimal inference length is recalculated with each iteration of the training process. When the optimal inference length is greater than or equal to the maximum length of all correct trajectories, the length penalty is zero, and the method degenerates into classic group relative policy optimization training. The training objective becomes enhancing the model's inference ability rather than compressing the length.

4. The progressive thought chain length calibration method for large-scale intelligent agent dialogue scenarios according to claim 1, characterized in that, The length reward coefficient ,in: The proportion of erroneous trajectories. For the correctness reward list, For the length reward list, this coefficient ensures that the normalized advantage value of the correct trajectory is non-negative, mean() is the average value, and min() is the minimum value.

5. The progressive thought chain length calibration method for large-scale intelligent agent dialogue scenarios according to claim 1, characterized in that, The length reward list is a matrix composed of the length rewards of all reasoning trajectories, the first of which is... The length of the trajectory is rewarded ,in: A reward is given for the correctness of the trajectory. The length of the trajectory is awarded as a reward. For the first The inference length of the trajectory, To estimate the optimal inference length, ; The first Total reward for each trajectory ,in: Rewards for accuracy, As a length bonus, For the correctness reward list, For the length reward list, This is the dynamic length reward coefficient.

6. The progressive thought chain length calibration method for large-scale intelligent agent dialogue scenarios according to claim 1, characterized in that, The trajectory-level dominance value is calculated using a group normalization method: Where: G is the number of trajectories sampled in each group, For the first The total reward for each trajectory, where mean() is the average and std() is the variance.

7. A progressive thought chain length calibration system for large-scale intelligent agent dialogue scenarios, implementing the method of any one of claims 1-6, characterized in that, include: The system comprises a sampling unit, a length estimation unit, a reward calculation unit, and an update unit. The sampling unit samples trajectories based on the input question, obtaining several thought chains and reasoning results. The length estimation unit estimates the mean and variance of the lengths of all answers and correct answers based on the sampled trajectories, obtaining the optimal length estimate. The reward calculation unit calculates length penalties and correctness rewards based on the optimal length and the accuracy of each trajectory, obtaining the reward for each trajectory. The update unit calculates the loss based on the reward for each trajectory and updates the model weights, obtaining new model weights.

Images