An adaptive thinking chain generation method, system and device based on dynamic punishment

Abstract

This invention discloses an adaptive thought chain generation method, system, and apparatus based on dynamic penalty, relating to the field of artificial intelligence technology, particularly to natural language processing and inference optimization of large-scale language models. The method includes: Step 1, fine-tuning and initializing the problem difficulty level based on the scaling law; Step 2, thought chain generation and dual-gating verification, verifying the generated thought chain through a dual-gating mechanism; Step 3, dynamic scoring using a multi-order asymmetric Gaussian curve, dynamically scoring the verified thought chain based on its length and the relationship between its length and a preset length range corresponding to the difficulty level using an asymmetric Gaussian reward curve; Step 4, dynamic anchor point update based on successful experience using EMA, forcing the model to precisely converge to the true optimal solution length for a specific problem; and Step 5, global annealing and boundary-oriented collapse, dynamically adjusting the length range boundary to approximate the absolute optimal solution for a specific problem.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence (AI) technology, and particularly to reasoning optimization of natural language processing (NLP) and large language model (LLM), especially to an adaptive thought chain generation method, system and device based on dynamic penalty. Background Technology

[0002] In recent years, large-scale language models have made significant progress in several complex tasks of natural language processing, such as automatic question answering, mathematical problem solving, code generation and assistance, and commonsense reasoning. To improve the performance of models in these tasks, Chain-of-Thought (CoT) technology has been widely adopted. This technology guides the model to generate a series of intermediate reasoning steps, simulating the human thought process. This enhances logical reasoning ability and the accuracy of the final answer. However, generating thought chains also introduces new technical challenges, the core of which lies in the inherent contradiction between the length and quality of the reasoning process and the cost of computing resources (such as computing power and latency). The ideal length of the thought chain should match the difficulty of the problem itself. Reasoning that is too short may lead to logical jumps and the loss of key information, while reasoning that is too long will introduce unnecessary computational overhead and may even produce "illusions" due to information redundancy or deviation from the topic, reducing the reliability of the answer.

[0003] Existing technologies have several shortcomings in controlling the length of reasoning chains. Some methods employ fixed or static upper limits on reasoning length, or use simple length penalty mechanisms. This "one-size-fits-all" strategy cannot dynamically and adaptively adjust the optimal length according to the inherent difficulty of the specific problem. For example, a simple problem may be forced into overthinking, or a complex problem may terminate reasoning prematurely due to length limitations. Furthermore, existing methods generally lack sufficient assessment of the logical consistency of the reasoning process, often relying on a single Outcome Reward Model (ORM) or Process Reward Model (PRM). The former cannot identify "coincidentally correct" erroneous logic, while the latter struggles to effectively assess the accuracy of the final answer; both have limitations when used alone. In terms of specific length control implementation, some dynamic windowing or dynamic truncation strategies have coarse-grained control and may inappropriately terminate the reasoning process when the model is performing effective self-correction (i.e., the model discovers errors during reasoning and backtracks to correct them), thus limiting the full potential of the model.

[0004] The limitations of these existing technologies directly lead to a series of engineering and application problems. First, the inference performance of the models exhibits significant instability and volatility across problems of varying difficulty and type. Second, the inability to perform fine-grained and adaptive length control of the inference process results in inefficient utilization of computational resources, especially in large-scale deployment scenarios where unnecessary inference length can significantly increase service costs. Furthermore, existing methods lack generalization ability across different problems and application scenarios, and a fixed length control strategy is difficult to apply universally.

[0005] Therefore, how to design a method, system, and device that can adaptively adjust the length of the thought chain according to the difficulty of the problem and effectively evaluate the logic of the reasoning process in order to balance reasoning quality, efficiency, and cost is a technical problem that urgently needs to be solved in the field of large-scale language models. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to solve the problems of static configuration of inference length and single reward signal in the process of generating thought chains in existing large-scale language models.

[0007] To achieve the above objectives, this invention provides an adaptive thought chain generation method based on dynamic penalty, which can achieve a balance between reasoning efficiency and answer accuracy. The method includes the following steps: Step 1: Fine-scale and baseline initialization of problem difficulty, determining the problem difficulty level based on the scaling law; Step 2: Mind chain generation and dual-gating verification. The generated mind chain is verified through a dual-gating mechanism, which includes an outcome reward model and a process reward model. Step 3: Dynamic scoring using multi-order asymmetric Gaussian curves. For the thought chain that has passed the verification, dynamic scoring is performed using the asymmetric Gaussian reward curve based on the relationship between its length and the preset length range of the corresponding difficulty level. Step 4: Update the EMA dynamic anchor point based on successful experience to force the model to converge precisely to the true optimal solution length for the specific problem; Step 5: Perform global annealing and boundary-oriented collapse on the plains to dynamically adjust the length interval boundaries in order to approximate the absolute optimal solution for a specific problem.

[0008] Further, step 1 includes the following sub-steps: Step 1.1: Construct a multi-scale model test matrix based on the scale law, and select language models with gradient differences in parameter scale; Step 1.2: Use each model in turn to solve the input problem, and use the minimum model parameter threshold that can stably output the correct answer as the basis for difficulty quantification; Step 1.3: Divide the problem into five difficulty levels from Level 1 to Level 5, and set a macro-optimal length benchmark for each difficulty level. .

[0009] Furthermore, the dual gating mechanism in step 2 is as follows: The result reward model is used to determine the correctness of the final answer; The process reward model adopts a two-stage evaluation: the first stage is that the large model automatically discovers logical doubts in the thought chain, and the second stage combines the original reasoning process with the logical doubts to make a comprehensive logical consistency judgment. If the result reward model determines the answer is correct and the process reward model's judgment logic is unbroken, the test is considered passed and the user is allowed to proceed to the next step; otherwise, the length reward for the current sample is cleared to zero, and the next step is terminated.

[0010] Furthermore, the asymmetric Gaussian reward curve in step 3 is based on the actual length of the currently generated thought chain. Implement differentiated rewards: when When falling into the left steep slope penalty zone, the shorter the length, the heavier the penalty, which is used to suppress logical jumps; when When a question falls into the perfect score zone, a full reward is given, which includes the length variance of different questions of the same difficulty. when When the model falls into the right-side gradual decay zone, the reward decreases slowly with increasing length, while protecting the model's self-correction ability. in, and This represents the boundary of the interval corresponding to the difficulty level.

[0011] Furthermore, the EMA dynamic anchor point update in step 4 specifically involves: The initial anchor point is set to the macroscopic optimal length reference μ0 corresponding to the difficulty level; Anchor point updates are triggered only when the thought chain passes the dual gating check; The updated formula is: in, This represents the actual length of thought required to arrive at the current solution. The dynamic anchor point at time t-1, Let t be the dynamic anchor point. At the very beginning of training, set the dynamic anchor point... The initial setting is the center point of the current difficulty macroscopic plain, i.e.: ; This is a hyperparameter.

[0012] Furthermore, the hyperparameters ∈[0.1, 0.3].

[0013] Furthermore, the plain global annealing in step 5 refers to the plain width. According to the formula, the global training steps are incremented by 100. It exhibits an exponential, smooth decay, among which, The attenuation coefficient is... This represents the initial width of the plain. The boundary directional collapse refers to re-anchoring the shrunken plain to the state updated in step 4. Then, dynamically calculate the legal boundaries for the next round: in, The right boundary Left boundary; By continuously narrowing the plain width, the model is forced to eventually converge to the optimal reasoning length, i.e., the absolute optimal solution for a specific problem, without compromising its self-correction ability.

[0014] Furthermore, the attenuation coefficient γ∈[0.99, 0.999].

[0015] The present invention also provides an adaptive thought chain generation system based on dynamic punishment, for implementing the method described in any one of the preceding claims, the system comprising: The problem difficulty scaling module is configured to perform multi-scale model testing based on the scaling law and a five-level difficulty classification. The dual-gated evaluation module includes an outcome reward model unit and a process reward model unit, used to verify logical consistency and answer correctness; The dynamic length reward calculation module stores the asymmetric Gaussian reward curves corresponding to each difficulty level. The successful experience anchor point update module uses the EMA mechanism to smoothly update the optimal length anchor points at the question level. The Plain Width Annealing and Boundary Update module is used to dynamically shrink full-score plains and adjust length intervals.

[0016] The present invention also provides an adaptive thought chain generation device based on dynamic punishment, wherein the device stores a computer program, and the computer program, when executed, implements the method described in any one of the preceding claims.

[0017] The adaptive thought chain generation method, system, and apparatus based on dynamic penalty provided by this invention have at least the following technical effects: 1. The technical solution provided by this invention can accurately match the difficulty of a problem and significantly improve the utilization rate of computing resources: By inversely applying the scaling law of large models, the difficulty is automatically calibrated using the solution performance of models with different parameter scales for the same problem. Based on this, the system extracts the average thinking length of representative successful samples at this difficulty level, and uses it as the macroscopic benchmark peak value for length control. This allows the model to adaptively match the most cost-effective target thinking length for problems of varying complexity, completely solving the problems of wasted computing power and efficiency imbalance caused by traditional static length configuration, which involves "overthinking simple problems and prematurely truncating complex problems."

[0018] 2. The technical solution provided by this invention can alleviate or even eliminate "pretended thinking," ensuring the high reliability of the length learning signal: It introduces a strict dual-gating mechanism of Outcome Reward (ORM) and Process Reward (PRM). Addressing the pain point of process evaluation easily generating illusions, PRM employs an innovative "two-stage logical evaluation," where a large model first "automatically finds" logical errors, and then makes a comprehensive judgment. A strict veto is enforced; the model is only allowed to receive a length reward and update its experience anchors if the final answer is completely correct and the reasoning steps do not involve any forced jumps or "guessing." This effectively prevents the model from using lengthy filler words to fake deep thinking, ensuring the absolute purity of the length learning signal.

[0019] 3. The technical solution provided by this invention can perform asymmetric Gaussian dynamic scoring, compatible with anti-cheating and self-correction mechanisms: by abandoning rigid static penalties, a customized asymmetric Gaussian reward curve is used to finely control the length. A steep penalty is set at the far left end of the curve to severely punish logical skipping and mechanical backboard cheating behaviors that fall below the lower limit of information content; a full-score plain area is set in the middle to fully reward the natural variance of the optimal length of different questions of the same difficulty; a gentle decay design is adopted on the right side, which suppresses redundancy by mildly deducting points, while greatly protecting the ability of large models to spend extra tokens for backtracking and self-correction when encountering dead ends.

[0020] 4. The technical solution provided by this invention can approach the individual optimum through a "dual-gear linkage" to achieve robust convergence with redundancy removal: To address the potential for the macro-level plain to become a breeding ground for "legitimate slacking" in the model, a dual-gear linkage mechanism of "finding anchor points" and "shrinking the plain" is innovatively introduced. First, the latest successful experience is smoothly absorbed through exponential moving average (EMA), ensuring that the dynamic anchor points steadily approach the true optimal solution length for the specific problem, resisting abnormal fluctuations. Second, as the number of training steps increases, the system gradually narrows the width of the full-score plain through an annealing algorithm, much like "boiling a frog in warm water." The boundary is dynamically updated. This strategy forces the model to eliminate excess information without compromising its error correction capabilities, ultimately converging robustly to the limit of the optimal inference length for a single question.

[0021] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0022] Figure 1 This is a flowchart of a preferred embodiment of the present invention. Detailed Implementation

[0023] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0024] Example 1 This invention relates to an adaptive thought chain generation method based on dynamic penalty, used to control the thought chain generation process of a large-scale language model when performing complex reasoning tasks.

[0025] This method aims to balance the model's inference efficiency and answer accuracy by introducing an adaptive mechanism that combines dynamic punishment and reward. Specific technical points include: automatically scaling the question difficulty based on the model's capabilities; employing dual gating (ORM+PRM) to evaluate the logical consistency of the reasoning process and the correctness of the final answer; implementing a dynamic reward strategy based on an asymmetric Gaussian function; and combining exponential moving average (EMA) anchoring with a plains width annealing mechanism that utilizes successful experience to achieve adaptive adjustment of the inference length. This invention can be applied to various scenarios requiring deep thinking, such as automatic question answering, mathematical problem solving, code generation and assisted programming, and logical reasoning.

[0026] Specifically, such as Figure 1 As shown in the figure, an adaptive thought chain generation method based on dynamic penalty provided by an embodiment of the present invention includes the following steps: Step 1: Fine-scale and baseline initialization of problem difficulty, determining the problem difficulty level based on the scaling law; Step 2: Mind chain generation and dual-gating verification. The generated mind chain is verified through a dual-gating mechanism, which includes an outcome reward model and a process reward model. Step 3: Dynamic scoring using multi-order asymmetric Gaussian curves. For the thought chains that pass the verification, dynamic scoring is performed using asymmetric Gaussian reward curves based on the relationship between their length and the preset length range of the corresponding difficulty level. Step 4: Update the EMA dynamic anchor point based on successful experience to force the model to converge precisely to the true optimal solution length for the specific problem; Step 5: Perform global annealing and boundary-oriented collapse on the plains to dynamically adjust the length interval boundaries in order to approximate the absolute optimal solution for a specific problem.

[0027] Example 2 Based on Example 1, step 1 includes the following sub-steps: Step 1.1: Construct a multi-scale model test matrix based on the scaling law, and select language models with gradient differences in parameter scale; Step 1.2: Use each model in turn to solve the input problem, and use the minimum model parameter threshold that can stably output the correct answer as the basis for difficulty quantification; Step 1.3: Divide the problem into five difficulty levels from Level 1 to Level 5, and set a macro-optimal length benchmark for each difficulty level. Among them, the macroscopic optimal length benchmark. It uses a strong ability model to solve representative problems of corresponding difficulty, and statistically analyzes the mean length of the thought process chain of the correct samples as the initial benchmark for the macroscopic peak of the asymmetric Gaussian curve specific to this difficulty level. .

[0028] In particular, The initial peak center corresponding to the difficulty level. The method for confirming this value is: extract a batch of representative problems at a specific difficulty level, and use the currently most powerful model (such as Gemini3) to... <think>After tagging the answers and filtering out all correctly answered samples, the average length of their Thinking Tokens is calculated.

[0029] Example 3 Based on Example 1 or 2, in step 2, the target model generates a model with specific features for the specific problem. <think>The thought chain (CoT) of the label and the final answer are then processed. The outcome reward model (ORM) first verifies the absolute correctness of the final answer. If the ORM verification passes, the process reward model (PRM) is triggered. The first stage involves a large model "automatically finding flaws" to uncover logical inconsistencies. The second stage combines these inconsistencies to comprehensively evaluate whether there are logical jumps or causal breaks in the reasoning process. Only when both ORM and PRM pass perfectly is the reasoning considered a "high-quality success," receiving a length reward / penalty signal and being allowed to proceed to subsequent length reward allocation and anchor point update stages to ensure that length learning is consistent with logic / correctness; otherwise, the length reward for the current sample is reset to zero, terminating subsequent steps.

[0030] Specifically, the dual-gating (ORM + PRM) mechanism is as follows: The outcome reward model is used to determine the correctness of the final answer; The process reward model adopts a two-stage evaluation: the first stage is that the big model automatically discovers logical doubts in the thought chain, and the second stage combines the original reasoning process with the logical doubts to make a comprehensive logical consistency judgment. If the result reward model determines the answer is correct and the process reward model's decision logic is unbroken, the test is considered passed and the user is allowed to proceed to the next step. Otherwise, the length reward for the current sample is reset to zero, and the next step is terminated.

[0031] Example 4 Based on Examples 1-3, assuming both gating mechanisms pass, step 3 automatically calls the specific asymmetric Gaussian reward curve assigned to the problem difficulty in step 1 and the actual length of the currently generated thought chain. Implement differentiated rewards: when When falling into the left steep slope penalty zone, the shorter the length, the heavier the penalty. This is used to suppress logical jumps and give severe penalties, accurately cracking down on logical jumps and mechanical backplate cheating behaviors. when When a score falls into the perfect score plateau, a full reward is given, which includes the length variance of different questions of the same difficulty and rewards efficient reasoning; when When the model falls into the right-side gradual decay zone, the reward decreases slowly with the length, and the points are deducted gently to punish redundancy while protecting the model's self-correction ability. in, and This represents the boundary of the interval corresponding to the difficulty level.

[0032] Example 5 Based on Examples 1-4, and after confirming that the solution passed the double gating and achieved full marks, the actual length of thought in this case was utilized. This updates the dynamic anchor point for the problem. To smooth the calculation, the anchor point is updated using the Exponential Moving Average (EMA) formula.

[0033] Specifically, the EMA dynamic anchor point update in step 4 is as follows: The initial anchor point is set to the macroscopic optimal length reference μ0 corresponding to the difficulty level; Anchor point updates are only triggered when the thought chain passes the double-gating check. The updated formula is: in, This represents the actual length of thought required to arrive at the current solution. The dynamic anchor point at time t-1, Let t be the dynamic anchor point. At the very beginning of training, set the dynamic anchor point... The initial setting is the center point of the current difficulty macroscopic plain, i.e.: ; This is a hyperparameter.

[0034] By setting hyperparameters It smoothly absorbs the latest successful experiences, enabling the anchor point to robustly move towards the true optimal solution length for that specific problem. To move closer together and resist abnormal fluctuations generated in a single instance.

[0035] In particular, hyperparameters ∈[0.1, 0.3]. Hyperparameters This is used to control the proportion of new and old experiences when updating the Exponential Moving Average (EMA). At each update, the latest successful experience accounts for... The weight of historical experience accounts for a significant portion. The weights are determined by this. This ensures that the dynamic anchor point can steadily approach the true optimal length without drastically fluctuating due to a single abnormal fluctuation in the solution.

[0036] Example 6 Based on Examples 1-5, in step 5, regardless of whether the model's single solution is correct, plain global annealing refers to the plain width. According to the formula, the global training steps are incremented by 100. It exhibits an exponential, smooth decay, among which, The attenuation coefficient is... This represents the initial width of the plain. This is the initial setting of the extremely wide plain width. It is recommended to cover the principal variance interval at this difficulty level to ensure that the model is provided with a wide plain for protection in the early stages of training, allowing it to explore freely.

[0037] Boundary directional collapse refers to re-anchoring the shrunken plain to the state updated in step 4. Then, dynamically calculate the legal boundaries for the next round: in, The right boundary Left boundary; By continuously narrowing the plain width, the model is forced to squeeze out excess "water" without compromising its self-correction ability, eventually converging to the optimal reasoning length, which is the absolute optimal solution for a specific problem.

[0038] Specifically, the decay coefficient γ ∈ [0.99, 0.999]. This parameter determines the rate at which the width of the perfect score plain shrinks in the later stages of reinforcement learning training. As the number of global training steps increases, the plain width will shrink exponentially and smoothly.

[0039] Example 7 This invention provides an adaptive thought chain generation system based on dynamic penalty, used to implement the method of any of the foregoing embodiments, including: The problem difficulty scaling module is configured to perform multi-scale model testing based on the scaling law and a five-level difficulty classification. The dual-gated evaluation module includes an outcome reward model unit and a process reward model unit, used to verify logical consistency and answer correctness; The dynamic length reward calculation module stores the asymmetric Gaussian reward curves corresponding to each difficulty level. The successful experience anchor point update module uses the EMA mechanism to smoothly update the optimal length anchor points at the question level. The Plain Width Annealing and Boundary Update module is used to dynamically shrink full-score plains and adjust length intervals.

[0040] By working together, the thought chain generation process of large-scale language models can be precisely controlled.

[0041] Specifically, regarding the scaling law in the problem difficulty scaling module: (1) The principle and calibration basis of the scaling law: In the field of large language models, the scaling law indicates that the comprehensive ability of the model (such as logical reasoning and mathematical calculation) usually increases systematically with the increase of parameter size. This system cleverly applies this mechanism in reverse to quantify the difficulty of the problem: the number of model parameters is used as a "ruler" to measure the complexity of the problem. For the same problem, the "minimum model parameter threshold" required to be stably solved is tested; the higher the threshold, the stronger the reasoning ability required for the problem, and the greater its inherent difficulty.

[0042] (2) Multi-scale model test matrix: Select a series of language models with different parameter scales (e.g., the Qwen series of models from small parameters to very large parameters) to construct a multi-scale test ladder, and test the same question in turn.

[0043] (3) Five-level difficulty classification criteria: Based on the minimum model threshold that can reliably output the correct answer in the test matrix, the questions are finely divided into five difficulty levels: Very low difficulty (Level 1): If a micro / edge model with a very small number of parameters can reliably give the correct answer, it means that the question only requires basic pattern matching and is judged as very low difficulty. Low difficulty (Level 2): ​​Micro models are prone to errors, but conventional small-scale models can reliably give the correct answer and are judged as low difficulty. Medium difficulty (Level 3): Small-scale models cannot reliably answer and mainstream medium-scale models are required to perform effective reasoning and obtain the correct answer and are judged as medium difficulty. High difficulty (Level 4): Medium-scale models struggle and must rely on models with a larger number of parameters to ensure the accuracy of the answer and are judged as high difficulty. Very high difficulty (Level 5): Only top-tier flagship models with a very large number of parameters can give the correct answer and are judged as very high difficulty. The difficulty results obtained after the above five-level calibration will serve as the core input for subsequent control strategies. They will be used to allocate the most cost-effective target thought chain length range for problems of different difficulty levels, and in this way determine the macroscopic peak position of the dynamic length reward curve (asymmetric Gaussian curve) specific to that difficulty level.

[0044] Specifically, the dual-gated evaluation module (ORM+PRM) aims to completely eliminate the potential for models to disguise "guessing" as deep thinking by "filling in the word count," ensuring that only truly high-quality reasoning processes receive the length reward. To this end, the system introduces a strict dual-gated mechanism: The first line of defense: The Outcome Reward Model (ORM) is mainly used to determine whether the final answer output by the model is absolutely correct.

[0045] The second line of defense: The Process Reward Model (PRM), as the core mechanism for evaluating logical consistency, will infiltrate... <think>Within the tags, each logical reasoning step of the model undergoes fine-grained scrutiny. Considering the serious "illusion" problem that can easily arise when directly using a large model to evaluate the logic of long texts, this system innovatively adopts a two-stage evaluation for PRM: Stage 1 (Automatic Error Finding): Utilizing the large model as an "error finder," it specifically uncovers potential logical errors or doubts in the current thought chain (CoT) reasoning steps. Stage 2 (Comprehensive Judgment): The complete original reasoning process, along with the "doubts" identified in Stage 1, are input into the evaluation model. With clear hints of doubts, the evaluation model can more accurately and effectively determine whether there are fatal logical jumps or self-contradictions in the reasoning process.

[0046] The "one-vote veto" and reward triggering logic system strictly enforces the one-vote veto system: even if the ORM determines that the final answer is 100% correct, as long as the PRM finds that any step in the middle is a "forced jump", "causal break" or "blind chance", the reward for the derivation process of that sample will be instantly cleared.

[0047] Gating: Samples will only pass through the gating process and enter the subsequent dynamic length reward / penalty calculation stage if the ORM determines the answer is correct and the PRM determines the entire reasoning logic is completely reliable. If the double verification fails, the system will not apply any length-related intervention signals, thereby completely cutting off the interference of "coincidentally correct" erroneous reasoning on the model's length learning mechanism.

[0048] Specifically, for the dynamic length reward calculation module (asymmetric Gaussian curve): it is triggered under the premise that the model passes dual gating (i.e., ensuring the correctness of the answer and the complete reliability of the reasoning logic), serving as the core mechanism for balancing computational cost and reasoning quality. For the five problem difficulty levels (Level 1 to Level 5) defined in the initial calibration, the system maintains a "curve family" at the underlying level containing five independent asymmetric Gaussian reward curves. Each difficulty level has its own exclusive macro-optimal length benchmark. (i.e., the initial peak center of the curve) and the dynamically calculated legal interval boundary ( and For any input question, the system will automatically invoke its corresponding difficulty level's exclusive reward curve, based on the current length of the thought process. Dynamic scoring is performed. The specific segmented control mechanism and design principle for a single curve are as follows: (1) Left-side far-end steep slope penalty area ( The operating mechanism is as follows: when the length of the generated thought chain... Less than the lower limit specific to this difficulty level At this point, the reward score exhibits a steep downward trend (i.e., a gradually descending steep slope). The shorter the length, the further it deviates from the legal lower limit, and the heavier the negative penalty imposed by the system. Design rationale: Specifically designed to defend against logical skips and mechanical backplanes in the model's "blind luck." When the length falls below the lower limit of the logical information required for that difficulty level, a logical gap is highly probable. Compared to a fixed cliff-like negative score, the steep slope design provides the model with a more nuanced and continuous gradient signal—a warning for slight deficiencies and a heavy penalty for severe deficiencies, thus guiding the model to smoothly and quickly "push" the inference length back to the rigorous legal range.

[0049] (2) Full marks plain area ( The operating mechanism is as follows: a constant full-score reward is maintained at an approximately constant level within this interval. The design rationale is twofold: firstly, it aims to reward truly "efficient geniuses," encouraging the model to complete tasks with minimal computational power while ensuring rigorous reasoning, thereby obtaining the highest cost-effectiveness reward; secondly, by setting a plateau rather than a single peak, it effectively accommodates the natural variance in the optimal solution length for different specific problems within the same difficulty level (e.g., both belonging to Level 4 high difficulty), providing the model with legitimate space for fine-tuning and exploration.

[0050] (3) The right-side gradual attenuation zone ( The operating mechanism is as follows: when the length exceeds the right boundary of the plain, the reward score decreases slowly in a Gaussian manner as the length increases. The design rationale is to mildly penalize lengthy and verbose ineffective reasoning, but its core purpose is to protect the self-correcting ability of large models. In complex reasoning, models often need to backtrack and try again or reconstruct their thinking. This mechanism allows the model to spend extra tokens to explore when encountering a dead end, rather than giving up halfway due to fear of severe length penalty; at the same time, it also maintains a certain degree of tolerance for semantically equivalent but lengthy reasonable thinking.

[0051] Specifically, the Experience Anchor Point Update (EMA) module is designed to address the core contradiction between macroscopic statistical regularities (Gaussian curves) and microscopic individual differences (specific problems). While the pre-defined "perfect score plateau" protects the length variance of different problems of equal difficulty, it also creates a breeding ground for "legal slacking off" in the model. To force the model to converge precisely to the true optimal solution length for a specific problem (…),… The system abandons blind collapse and instead introduces a "dynamic anchor point" mechanism based on correct historical experience.

[0052] The specific operating logic and parameter design are as follows: Initial state settings: At the very beginning of training (Epoch 0), the system has not yet mastered the optimal solution length for a specific problem, therefore dynamic anchors will be used. The initial setting is the center point of the current difficulty macro plain, i.e. .

[0053] The strict update triggering conditions (anti-pollution mechanism) operate as follows: Anchor point updates are subject to extremely strict gating restrictions. Anchor point updates are only triggered when the model provides the correct answer to the problem and successfully passes the aforementioned ORM and PRM dual verifications (i.e., achieves full marks). The rationale is that if the model makes a mistake, it means the length of its thought process is not relevant, and the anchor point remains unchanged. This completely prevents the system from being led astray by incorrect exploration directions.

[0054] The operating mechanism of the EMA smoothing update formula: assuming the model is in the... The first time I encountered this problem, I solved it perfectly, and the actual length of the Thinking Tokens consumed was [length missing]. The system will update the anchor point using an exponential moving average (EMA) according to the following formula: In the formula It is a smoothing hyperparameter (set to 0.2). This means that with each update, new successful experiences account for 20% of the weight, while historical experiences account for 80%. This design is because the generation of large models is random, and the role of EMA is to smoothly record the model's most recent successful experiences. This ensures that the anchor point can robustly approach the true optimal length of the problem, without drastic fluctuations due to an abnormally long or short solution.

[0055] Specifically, the plains width annealing and boundary update module is the "second gear" that meshes closely with the aforementioned "EMA dynamic anchor point." Its core logic is that the perfect plains cannot be seen as a safe haven where the model can "legally slack off" indefinitely. In the early stages of reinforcement learning (RL) training, the system needs broad plains to protect the model for sufficient exploration; however, in the later stages of training, the system needs to gradually collapse the plains into a peak, forcing the model to extract the absolute optimal solution for a specific problem.

[0056] The specific operational mechanism and mathematical reconstruction are as follows: Plain width ( The global annealing mechanism works as follows: the width of the plain gradually decreases with the total number of global steps in the entire RL training. This is a "boiling frog" process, calculated using the following formula: .in, It is the initially set extremely wide plain width. This is the decay factor (e.g., 0.99). As the number of training steps increases, It will get smaller and smaller.

[0057] Left and right boundaries ( and The dynamic directional movement mechanism of ) is as follows: when a center point (anchor point) is established... ) and the current plain width Then we can calculate the left and right boundaries: half of the plain is to the left of the anchor point, and half is to the right of the anchor point.

[0058] Calculation formula: Movement of the maximum point (right boundary): .

[0059] Movement of the minimum point (left boundary): .

[0060] Example 8 This invention provides an adaptive thought chain generation device based on dynamic punishment. The device stores a computer program, which, when executed, implements any of the methods described in the foregoing embodiments.

[0061] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.< / think> < / think> < / think>

Claims

1. A method for adaptive thought chain generation based on dynamic penalty, characterized in that, The method includes the following steps: Step 1: Fine-scale and baseline initialization of problem difficulty, determining the problem difficulty level based on the scaling law; Step 2: Mind chain generation and dual-gating verification. The generated mind chain is verified through a dual-gating mechanism, which includes an outcome reward model and a process reward model. Step 3: Dynamic scoring using multi-order asymmetric Gaussian curves. For the thought chain that has passed the verification, dynamic scoring is performed using the asymmetric Gaussian reward curve based on the relationship between its length and the preset length range of the corresponding difficulty level. Step 4: Update the EMA dynamic anchor point based on successful experience to force the model to converge precisely to the true optimal solution length for the specific problem; Step 5: Perform global annealing and boundary-oriented collapse on the plains to dynamically adjust the length interval boundaries in order to approximate the absolute optimal solution for a specific problem.

2. The dynamic penalty based adaptive thought chain generation method of claim 1, wherein, Step 1 includes the following sub-steps: Step 1.1: Construct a multi-scale model test matrix based on the scale law, and select a language model with gradient differences in parameter scale; Step 1.2: Use each model in turn to solve the input problem, and use the minimum model parameter threshold that can stably output the correct answer as the basis for difficulty quantification; Step 1.3, the questions are divided into five levels of difficulty from Level 1 to Level 5, and macro-optimal length benchmarks are set for each level of difficulty .

3. The dynamic penalty based adaptive thought chain generation method of claim 1, wherein, The dual gating mechanism in step 2 is as follows: The result reward model is used to determine the correctness of the final answer; The process reward model adopts a two-stage evaluation: the first stage is that the large model automatically discovers logical doubts in the thought chain, and the second stage combines the original reasoning process with the logical doubts to make a comprehensive logical consistency judgment. If the result reward model determines the answer is correct and the process reward model's judgment logic is unbroken, the test is considered passed and the user is allowed to proceed to the next step; otherwise, the length reward for the current sample is cleared to zero, and the next step is terminated.

4. The adaptive thought chain generation method based on dynamic punishment as described in claim 2, characterized in that, the asymmetric gaussian reward curve in the step 3 is according to the current generated thought chain actual length , a differentiated reward is made: When a left-side steep slope penalty area is fallen into, the shorter the length is, the heavier the penalty is, for inhibiting logical jump; When the full score of the plain area, give the full amount of reward, including different lengths of the variance under the same difficulty of the subject; when When the model falls into the right-side gradual decay zone, the reward decreases slowly with increasing length, while protecting the model's self-correction ability. in, and This represents the boundary of the interval corresponding to the difficulty level.

5. The adaptive thought chain generation method based on dynamic punishment as described in claim 4, characterized in that, The EMA dynamic anchor point update in step 4 specifically involves: The initial anchor point is set to the macroscopic optimal length reference μ0 corresponding to the difficulty level; Anchor point updates are triggered only when the thought chain passes the dual gating check; The updated formula is: in, This represents the actual length of thought required to arrive at the current solution. The dynamic anchor point at time t-1, Let t be the dynamic anchor point. At the very beginning of training, set the dynamic anchor point... The initial setting is the center point of the current difficulty macroscopic plain, i.e.: ; This is a hyperparameter.

6. The adaptive thought chain generation method based on dynamic punishment as described in claim 5, characterized in that, The hyperparameters ∈[0.1, 0.3].

7. The adaptive thought chain generation method based on dynamic punishment as described in claim 2, characterized in that, The plain global annealing mentioned in step 5 refers to the plain width. According to the formula, the global training steps are incremented by 100. It exhibits an exponential, smooth decay, among which, The attenuation coefficient is... This represents the initial width of the plain. The boundary directional collapse refers to re-anchoring the shrunken plain to the state updated in step 4. Then, dynamically calculate the legal boundaries for the next round: in, The right boundary Left boundary; By continuously narrowing the plain width, the model is forced to eventually converge to the optimal reasoning length, i.e., the absolute optimal solution for a specific problem, without compromising its self-correction ability.

8. The adaptive thought chain generation method based on dynamic punishment as described in claim 7, characterized in that, The attenuation coefficient γ∈[0.99, 0.999].

9. An adaptive thought chain generation system based on dynamic penalty, used to implement the method described in any one of claims 1-8, characterized in that, The system includes: The problem difficulty scaling module is configured to perform multi-scale model testing based on the scaling law and a five-level difficulty classification. The dual-gated evaluation module includes an outcome reward model unit and a process reward model unit, used to verify logical consistency and answer correctness; The dynamic length reward calculation module stores the asymmetric Gaussian reward curves corresponding to each difficulty level. The successful experience anchor point update module uses the EMA mechanism to smoothly update the optimal length anchor points at the question level. The Plain Width Annealing and Boundary Update module is used to dynamically shrink full-score plains and adjust length intervals.

10. An adaptive thought chain generation device based on dynamic punishment, characterized in that, The device stores a computer program that, when executed, implements the method as described in any one of claims 1-8.

Images