Dynamic reference stable grouping relative strategy optimization method

Abstract

The invention provides a grouping relative strategy optimization method with stable dynamic reference, which comprises the following steps of: performing multiple independent sampling on the same input prompt by a current strategy model, generating a candidate answer set, calculating an absolute quality score of each candidate answer through a preset determinacy evaluation function, and calculating a quality score of each candidate answer; calculating an average score of all candidate answers in the current group; dynamically updating a global historical reference value according to the score of the current batch of candidate answers through a dynamic global historical reference line maintenance module; through a composite advantage calculation module, in-group relative comparison and global historical comparison are fused, and a composite advantage value of each candidate answer is calculated; and through a strategy gradient updating module, the logarithmic probability gradient of the strategy model is calculated and the parameters of the strategy model are updated by taking the composite advantage value as a weighted weight, so that the generation probability of answers with high advantage values is enhanced, and the generation probability of answers with low advantage values is inhibited. The method has the beneficial effect that the answer which is absolutely progressive compared with the historical performance of the model can be accurately identified.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a dynamic benchmark-stabilized grouping relative strategy optimization method. Background Technology

[0002] In recent years, reinforcement learning based on human feedback (RLHF) and its variants have become a core technological paradigm for aligning large language models with human intent. Its standard process typically involves three steps: supervised fine-tuning, reward-based model training, and reinforcement learning-based policy optimization. However, RLHF's reliance on high-quality, large-scale human preference data and its complex training process (involving multiple independent models) limits its applicability and efficiency.

[0003] To overcome the aforementioned challenges, an unsupervised or self-supervised optimization method called Group Relative Policy Optimization (GRPO) has been proposed. The core innovation of GRPO lies in abandoning the independent reward model. Its optimization signal comes directly from the relative quality comparison of multiple output answers (i.e., a "group") sampled from the policy model itself under the same input cues.

[0004] The basic working principle is as follows: For a given prompt, the current policy model generates a set of candidate answers through multiple samplings. Then, an absolute score is calculated for each answer using a fixed, training-free reward function (e.g., rule-based task completion score, code execution accuracy, or similarity to the reference answer). The key step of GRPO is that it does not directly maximize these absolute scores, but rather constructs the policy gradient by calculating the advantage of each answer's score relative to the **average score within its group**.

[0005] Answers with positive advantage values ​​are considered "winners," and their generation probability is enhanced in subsequent optimizations; while answers with negative advantage values ​​are considered "laggards," and their generation probability is suppressed. This method aims to implicitly learn better generation strategies through relative comparisons of the model's own outputs within the same context.

[0006] While GRPO simplifies the training process and reduces data requirements, it has an inherent structural flaw that affects training stability and final performance: the baseline of the advantage function is dynamically and instantaneously determined by the quality distribution of randomly sampled answers in the current batch. This leads to training instability and policy degradation, as the magnitude and sign of the advantage value are highly dependent on the random quality of other answers in the same group. Even if an answer makes a significant improvement in absolute quality, reaching the model's historical best, it will still be given a negative advantage if a higher-scoring answer happens to appear in the same batch, thus being incorrectly suppressed during optimization. Conversely, a mediocre answer may gain a positive advantage simply because other answers in the same group are of lower quality, causing the policy to update towards a suboptimal direction, affecting the stable improvement and evolution of model performance. Summary of the Invention

[0007] To address the problems in the prior art, this invention provides a dynamic benchmark-stabilized grouping relative strategy optimization method.

[0008] This invention provides a dynamic benchmark-stabilized grouping relative strategy optimization method, comprising the following steps: Step S1: The current strategy model performs multiple independent samplings on the same input prompt to generate a set of candidate answers. The absolute quality score of each candidate answer is calculated using a preset deterministic evaluation function, and the average score of all candidate answers in the current group is calculated. Step S2: The global historical baseline value is dynamically updated based on the scores of the current batch of candidate answers through the dynamic global historical baseline maintenance module. Step S3: Using the composite advantage calculation module, the composite advantage value of each candidate answer is calculated by integrating intra-group relative comparisons and global historical comparisons. The formula for calculating the composite advantage value is as follows: in, For the first The absolute quality score of each answer. The average score of the answers within the current group. This is a dynamically updated global historical baseline value. To meet Adjustable balance coefficient; Step S4: Through the policy gradient update module, the log probability gradient of the policy model is calculated and the policy model parameters are updated using the composite advantage value as the weight. This increases the probability of generating answers with high advantage values ​​and suppresses the probability of generating answers with low advantage values.

[0009] The present invention is further improved in that, in step S2, the initial value of the global historical benchmark is determined at the initial training time. The value is set as the median or expected value of the task score distribution based on prior knowledge of the task.

[0010] In a further improvement to this invention, the method for dynamically updating the global historical baseline value in step S2 is as follows: Step S21: Obtain the average score of the current batch. and highest score ; Step S22: Update the mean baseline using the exponential moving average method. The update formula is as follows: in, Let t be the baseline for the mean of round t. This serves as the baseline for the average value of the (t-1)th round. Step S23: Update the peak baseline using a monotonically non-decreasing strategy. The update formula is as follows: in, Let t be the peak baseline. This serves as the baseline for the peak value in the (t-1)th round. Step S24: Obtain the global historical baseline value through weighted fusion. The fusion formula is as follows: in, For the combined weight parameters, .

[0011] The present invention is further improved in step S22, The adaptive smoothing factor is calculated using the following formula: in, Based on the smoothing factor, The amplitude coefficient of the adaptive adjustment is controlled. The standard deviation of the current batch score. This is the reference standard deviation.

[0012] The present invention is further improved in that, in step S2, after updating the global historical baseline value, anomaly suppression processing is also included: Calculate the relative rate of change of the global historical benchmark. ,in It is a very small constant; like The final global historical baseline value is Otherwise, the final global historical baseline value will remain at [value]. ,in As an inhibitor, This is the threshold for the maximum allowable rate of change.

[0013] The present invention is further improved in that, in step S2, boundary constraints are also included after the anomaly suppression processing: in, and These are the minimum and maximum scores for the task, respectively.

[0014] In a further improvement to this invention, in step S3, the absolute quality score is standardized before calculating the composite advantage value: ,in and The mean and standard deviation of the training set scores are given; after standardization, the within-group mean score is updated to... The global historical baseline value has been updated to The formula for calculating the composite advantage value has been updated accordingly: .

[0015] This invention is further improved with a balance coefficient. The parameters are dynamically adjusted as the training process progresses, and the adjustment formula is: ,in As the initial weights, The decay time constant, This represents the current training iteration number.

[0016] In a further improvement to this invention, in step S3, the calculation of the composite advantage value also incorporates intra-group score ranking information, specifically: calculating the... Ranking of candidate answers ,in for Ranked in descending order within the group The size of the candidate answer set; The relative advantage component within the group has been updated to , This is a ranking enhancement factor. The formula for calculating the composite dominance value has been updated accordingly: .

[0017] The present invention is further improved by repeating steps S1 to S4 to form an iterative optimization loop. By improving the quality of the answer, the global historical baseline is gradually moved upward, and the model is driven to evolve towards the global optimum.

[0018] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a dynamic benchmark-stable grouping relative policy optimization method, which can effectively solve the problems of training instability and policy degradation in existing grouping relative policy optimization methods. By controlling the policy model to generate candidate answers and calculating the absolute quality score and the average score within the group, combined with the dynamic update of the dynamic global historical benchmark, and then integrating the comparison between the group and the global history through the composite advantage value formula, it can accurately identify the answer that has made absolute progress compared with the model's historical performance. Even if the answer is not optimal within the current group, a positive signal can still be obtained through the composite advantage value. This fundamentally avoids the problem in existing technologies where high-quality strategies are wrongly suppressed and suboptimal strategies are wrongly rewarded due to simply relying on the comparison of random samples within the group, thus ensuring the correctness of the policy update direction. Attached Figure Description

[0019] To more clearly illustrate the solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a dynamic benchmark-stabilized grouping relative strategy optimization method according to the present invention. Detailed Implementation

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having” and any variations thereof in the specification, claims and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0024] This invention provides a dynamic benchmark-stabilized grouping relative strategy optimization method, such as... Figure 1 As shown, the detailed implementation method of using dual optical distance sensors in law enforcement recorders in this example is as follows: 1. The current strategy model performs multiple independent samplings of the same input prompt to generate a set of candidate answers. It calculates the absolute quality score of each candidate answer using a preset deterministic evaluation function and calculates the average score of all candidate answers in the current group.

[0025] Given input prompts Based on the current strategy model (Parameters are) )conduct Sub-independent sampling generates a set of candidate answers. The specific sampling process can be represented as follows: To measure the quality of each generated answer, a pre-defined deterministic evaluation function is used. The function scores the answers. It does not depend on model parameters, but is usually based on task-related evaluation metrics (such as task completion, matching degree, executability, etc.) for any answer. Return scalar quality score For each candidate answer Calculate its absolute quality score: This leads to the score set of the current group's answers. .

[0026] To support subsequent advantage calculations, we further calculate the statistics of the within-group scores, i.e., the group average score: And the standard deviation of the scores within the group (optional, used for normalization, etc.): At the same time, in preparation for updating the global historical benchmark, the highest score of the current batch is recorded:

[0027] And the average score of the current batch (consistent with the group average score, explicitly used here for baseline updates): 2. The global historical baseline maintenance module dynamically updates the global historical baseline value based on the scores of the candidate answers in the current batch.

[0028] The initial moment of training (denoted as moment) Therefore, it is necessary to set reasonable initial values ​​for the baseline system. Considering that the distribution range of scores for different tasks may vary, the initial value of the global historical baseline should be... Based on prior knowledge of the task, the mean is set as the median or expected value of the task score distribution, providing a reasonable starting reference point for subsequent training. and peak baseline This value is also used for synchronization initialization.

[0029] The method for dynamically updating the global historical baseline value is as follows: Step S21: Obtain the average score of the current batch. and highest score ; Batch average score : Reflects the average performance level of the model in the current batch; highest score in the batch : Represents the model's best performance in the current batch; Batch score standard deviation : Measures the stability of the model's performance in the current batch; Step S22: The mean baseline aims to capture the long-term trend of model performance. It is updated using an exponential moving average method, assigning appropriate weights to recent data while maintaining the continuity of historical information. The update formula is: in, Let t be the baseline for the mean of round t. This serves as the baseline for the average value of the (t-1)th round. This update method ensures that the mean baseline can smoothly reflect the evolution of the model's capabilities, avoiding drastic fluctuations caused by a single batch of outlier data.

[0030] Step S23: Unlike the mean baseline, the peak baseline focuses on recording the model's historical best performance. A monotonically non-decreasing strategy is used to update the peak baseline, ensuring it always maintains an upward trend. The update formula is: in, Let t be the peak baseline. This serves as the baseline for the peak value in the (t-1)th round. This design makes the peak baseline a target that the model must continuously surpass, providing a clear direction for training progress and a relentless driving force.

[0031] Step S24: A single type of baseline often struggles to simultaneously meet the requirements of stability and challenge. A weighted fusion method is used to obtain a global historical baseline value. Combining the robustness of the mean baseline with the aggressiveness of the peak baseline, the fusion formula is as follows: in, For the combined weight parameters, In the early stages of training, a larger [set] can be used. The value can be adjusted to obtain a more stable optimization direction; as training progresses, it can be appropriately reduced. To increase the challenge and drive the model to continuously break through its own limits.

[0032] The smoothness of the baseline update needs to be adaptively adjusted based on the stability of the current model performance. When the model performance is stable (small score standard deviation), we want the baseline to reflect the latest trend more quickly; when performance fluctuates greatly, a more conservative update is needed to smooth out noise. Therefore, an adaptive smoothing factor is designed. The calculation formula is as follows: in, Based on the smoothing factor, The amplitude coefficient of the adaptive adjustment is controlled. The standard deviation of the current batch score. The reference standard deviation is used. This design allows the smoothing factor to... Based on this, it is dynamically adjusted according to the changes in batch score stability.

[0033] To promptly detect anomalies during baseline updates, continuous monitoring of the baseline's rate of change is necessary. The calculation of the relative rate of change takes into account the current size of the baseline, and anomaly suppression processing is also included after updating the global historical baseline value. Calculate the relative rate of change of the global historical benchmark. ,in This is a very small constant used to prevent the denominator from being zero. Monitoring the rate of change provides a quantitative basis for subsequent anomaly handling.

[0034] When an abnormal change in the baseline is detected, an suppression mechanism needs to be activated to prevent interference with the training process. The final global historical baseline value is Otherwise, the final global historical baseline value will remain at [value]. ,in As an inhibitor, This mechanism, with its maximum permissible rate of change threshold, allows for normal baseline adjustments while effectively mitigating the impact of abnormal fluctuations.

[0035] To ensure that the baseline always remains within the reasonable range defined by the task, boundary constraints need to be applied to the updated baseline. After anomaly suppression processing, boundary constraints are also included. in, and These are the minimum and maximum scores for the task, respectively. This constraint ensures the mathematical rationality of the baseline system.

[0036] The trend of the baseline can also provide valuable information for monitoring the training process. By analyzing the average change of the baseline within the sliding window, it is possible to help determine whether the model has reached convergence. in To adjust the sliding window size, This serves as a threshold for determining convergence. This determination can provide data support for training strategies such as early stopping.

[0037] 3. Through the composite advantage calculation module, which integrates intra-group relative comparisons and global historical comparisons, the composite advantage value of each candidate answer is calculated. The formula for calculating the composite advantage value is as follows: in, For the first The absolute quality score of each answer. The average score of the answers within the current group. This is a dynamically updated global historical baseline value. To meet The adjustable balance coefficient.

[0038] To ensure the comparability of scores with different dimensions and distribution ranges, the raw scores are first standardized. Considering that the score distribution may shift during training, we use training set statistics as a normalization benchmark, and standardize the absolute quality score before calculating the composite advantage value: ,in and This processing, which uses the mean and standard deviation of the training set scores, not only eliminates the influence of dimensions but also maps the scores to a relatively stable numerical range. After standardization, the within-group mean score is updated to The global historical baseline value has been updated to The formula for calculating the composite advantage value has been updated accordingly: .

[0039] Balance coefficient The weights are not fixed but dynamically adjusted according to the training process. In the early stages of training, more reliance should be placed on intra-group comparisons to encourage exploration, while in the later stages, global historical comparisons should be emphasized to promote convergence. To this end, we designed an exponentially decaying weight adjustment strategy: a balancing coefficient... The parameters are dynamically adjusted as the training process progresses, and the adjustment formula is: in As the initial weights, The decay time constant, This represents the current training iteration number.

[0040] To improve the discrimination of intra-group comparisons, in addition to considering the absolute score difference, the calculation of the composite advantage value also incorporates intra-group score ranking information. This ranking information better reflects the relative position of the answer within the group. Specifically, it involves calculating the... Ranking of candidate answers ,in for Rank the groups in descending order (the best value is 1). Size of the candidate answer set; ranking ratio It varies between 0 and 1, with smaller values ​​indicating a higher ranking.

[0041] Based on standardized scores, mean score differences, and ranking information, an enhanced within-group relative advantage component is calculated. The introduction of ranking information allows high-quality answers to receive appropriate enhancement even when the score difference is not significant: the within-group relative advantage component is updated as follows: in, This is the ranking enhancement factor (usually set to 0.2-0.5), which degenerates into the base version when ω=0.

[0042] The formula for calculating the composite dominance value has been updated accordingly: .

[0043] Global historical comparisons consider not only the absolute gap with the current benchmark but also the progress relative to the previous benchmark. This design rewards performance that surpasses historical levels and encourages continuous improvement. in and These are the standardized global baseline values ​​for the current and previous time steps, respectively. For the progress reward coefficient, The function ensures that the reward range is bounded.

[0044] Considering that the reliability of scores for different answers may vary, a confidence weight is also introduced when calculating the composite advantage value: in The standard deviation of the within-group scores is the standardized score. As a sensitivity parameter, it controls the strength of the uncertainty penalty. The more outlier the score is in the within-group distribution (the larger the standardized distance from the mean), the lower its reliability tends to be. The formula for calculating the composite dominance value has been updated accordingly: .

[0045] The basic composite dominance value is calculated by organically combining dynamic weights, within-group dominance components, and global dominance components. Confidence weights act on both components simultaneously here, ensuring that the overall influence of unreliable scores is suppressed. To prevent excessive scale differences in the dominance value across different batches from causing training instability, the process after obtaining the base composite dominance value also includes: batch-level normalization of the base composite dominance value. , and These are the basic composite advantage values ​​for the current batch. The mean and standard deviation, As a numerically stable term, this operation ensures that the distribution of the advantage value remains relatively stable during training.

[0046] To further enhance training stability, a soft-pruning operation is performed on the normalized dominance values. ,in, The pruning threshold is usually set to 2.0-3.0. This method can prevent extreme values ​​from interfering with the optimization process and avoid information loss that may be caused by hard pruning.

[0047] After the complete processing flow, the final advantage value is generated through a gentle nonlinear transformation: ,in This is a reference value (usually set to 2.0). hour, When the term is close to 1, the transformation is approximately linear; when When the value is large, this term decays rapidly, effectively suppressing extreme values. This transformation further suppresses the influence of outliers while maintaining linearity within the normal range.

[0048] After the above ten sub-steps of progressive calculation, the final composite advantage value is obtained. It fully integrates information from both intra-group competition and global surpassing, while possessing good numerical stability and training adaptability, providing high-quality optimization signals for subsequent policy gradient updates.

[0049] By using the calculated final composite advantage value as a guiding signal for policy updates, a complete gradient calculation and parameter update mechanism is constructed to ensure that the model can systematically enhance its ability to generate high-quality answers while effectively suppressing the generation of low-quality answers.

[0050] 4. Through the policy gradient update module, the log probability gradient of the policy model is calculated and the policy model parameters are updated using the composite advantage value as the weight. This increases the probability of generating answers with high advantage values ​​and suppresses the probability of generating answers with low advantage values.

[0051] Before updating the policy model parameters, the following is also included: Before calculating the policy gradient, the composite advantage value is first calibrated to ensure its numerical characteristics are suitable for gradient optimization. Considering the potential distribution bias in the advantage value, we center it by subtracting the average of the composite advantage values ​​of all candidate answers within the group. ,in To calculate the final composite advantage value, The number of answers within the group. It is an index variable: representing the sequence number of the candidate answer within the current group (and the set of candidate answers). The corresponding subscript, The value range is 1 to ), It is the first The composite advantage value (and) of each candidate answer The definitions are the same, only the subscripts are used. (By distinguishing different candidate answers), centralization can eliminate the overall bias between batches, making the mean of the advantage value within a batch 0, avoiding the overall high / low advantage value from misleading the model update, and making the advantage value better reflect the relative merits of the answers.

[0052] The core idea of ​​policy gradient descent is to adjust the parameters of the policy model so that the probability of generating high-dominance answers increases and the probability of generating low-dominance answers decreases. This process is achieved by calculating the weighted log-probability gradient.

[0053] Suppose the policy model is given a prompt Generate answer below The probability is Then its logarithmic probability gradient can be expressed as: This formula reflects the core role of the advantage value as a weighting factor: the answer corresponding to a positive advantage value will receive a positive gradient, encouraging the model to increase its generation probability; while a negative advantage value will generate a negative gradient, prompting the model to reduce the probability of the corresponding generation path.

[0054] To obtain a stable update direction and improve computational efficiency, the log probability gradients of all candidate answers within the group are aggregated. The aggregation process involves not only simple summation but also gradient magnitude normalization to prevent a single sample from dominating the update direction. This is the batch aggregated gradient ultimately used for model parameter updates, and it is the output of this formula; The number of candidate answers in the current group (and the set of candidate answers) (The sample size is consistent). It is the first The original policy gradients corresponding to each candidate answer are as described above. Calculated , It is the first The original gradient The reciprocal of the L2 norm (the L2 norm is the square root of the sum of the squares of the elements of the gradient vector; taking the reciprocal is for "scaling suppression" of large gradients). It is the average of the inverse of the L2 norm of all original gradients within the group, used to globally normalize the gradient's "scaling factor" and avoid scale differences between different batches.

[0055] This adaptive weighting method first scales the original gradient of each sample according to its own norm, then takes the average within the group, and finally obtains a more robust batch gradient (which preserves the gradient direction of each sample and balances the gradient intensity of different samples), ensuring that the contribution of gradients of different magnitudes to the final update vector is more balanced and enhancing the stability of training.

[0056] Before feeding the aggregated gradients into the optimizer, gradient clipping is performed to further ensure the numerical stability of the training process. Gradient clipping is achieved by limiting the maximum norm of the gradient vector: in This is a preset maximum gradient norm threshold. The clipped gradient retains the original direction information while avoiding training instability caused by gradient explosion.

[0057] Based on the pruned gradients, the policy model parameters are finally updated using optimization algorithms (such as Adam, SGD, etc.) based on the processed gradients. Taking the Adam optimizer as an example, its update rule can be formally expressed as: in For the first Model parameters at the next iteration For learning rate, and These are the bias correction terms for the first-order moment estimate and the second-order moment estimate of the gradient, respectively. This is the numerical stability constant. This update process systematically adjusts the parameters of the policy model along the optimization direction indicated by the composite advantage value.

[0058] 5. Repeat steps S1 to S4 to form an iterative optimization loop. By improving the quality of the answers, the global historical baseline is gradually moved upward, driving the model to evolve towards the global optimum.

[0059] By continuously iterating through steps S1 to S4, a dynamic optimization mechanism with self-reinforcing characteristics is established. Each iteration follows a standard process of "generation-evaluation-calibration-update": the policy model generates diverse candidate answers based on current parameters, the evaluation module objectively scores the quality of these answers, the baseline system dynamically adjusts historical reference standards based on the latest performance, the advantage calculation module comprehensively analyzes the relative and absolute advantages of the answers, and finally, the policy gradient module precisely adjusts the model parameters accordingly.

[0060] This cycle is not a simple repetitive operation, but an intelligent process with a clear evolutionary direction. As the quality of the model's answers gradually improves, the global historical baseline is adjusted upwards accordingly, forming a self-driven development cycle of "performance improvement driving higher standards, and higher standards forcing performance breakthroughs." This mechanism ensures that the model never remains in the comfort zone of local optima, but continuously faces new challenges and pushes its own capability boundaries. The systematic upward shift of the baseline injects continuous evolutionary pressure into training, forcing the model to explore better policy spaces; at the same time, the stability of the optimization process is guaranteed by multiple calibration mechanisms, avoiding training oscillations caused by excessively rapid increases in standards. Ideally, this self-driven cycle will achieve a dynamic balance between model performance, baseline level, and optimization difficulty, propelling the model steadily towards the global optimum along a smooth evolutionary trajectory, ultimately achieving a qualitative leap from "passive learning" to "active evolution," laying a solid algorithmic foundation for the continuous optimization of complex tasks.

[0061] This invention can accurately achieve the stability and continuous evolution of the policy optimization process. Whether it is preventing policy degradation, breaking through local optima, or improving training efficiency, it can significantly enhance the model's ability evolution trajectory.

[0062] Compared with the prior art, the present invention has the following innovative features: (1) The present invention can introduce a dynamically updated global historical baseline. By comparing the score of each answer with the historical baseline, the limitations of simple intra-group comparison are made up. The long-term performance reference provided by the historical baseline not only helps the model identify answers that have made absolute progress compared to "their past", but also effectively avoids the incorrect suppression of high-quality strategies due to random fluctuations within the group by rewarding absolute progress, thus fundamentally ensuring the correctness and monotonicity of the optimization direction. (2) This invention can provide a clear and continuous direction for strategy evolution by constructing the global historical baseline as a dynamically upgradable catch-up target; the model must continuously generate answers that can refresh the historical record to obtain the maximum reward, thereby being actively guided to escape the comfort zone of local optima; this mechanism deeply strengthens the driving force of the model to continuously break through itself, and promotes the strategy to explore and approach the global better solution space in a stable and continuous manner, realizing the evolutionary paradigm shift from "relatively good" to "absolutely excellent"; (3) The present invention can use a smoothly updated global baseline as a stable reference and significantly reduce the random noise of the advantage signal, making the estimation of the policy gradient more reliable. This improvement makes the training process smoother and the convergence speed faster, while reducing the sensitivity to harmful updates. It achieves higher data utilization efficiency and overall training robustness at the same sampling cost, and finally achieves efficient and stable policy optimization.

[0063] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A dynamic benchmark-stabilized grouping relative strategy optimization method, characterized in that, Includes the following steps: Step S1: The current strategy model performs multiple independent samplings on the same input prompt to generate a set of candidate answers. The absolute quality score of each candidate answer is calculated using a preset deterministic evaluation function, and the average score of all candidate answers in the current group is calculated. Step S2: The global historical baseline value is dynamically updated based on the scores of the current batch of candidate answers through the dynamic global historical baseline maintenance module. Step S3: Using the composite advantage calculation module, the composite advantage value of each candidate answer is calculated by integrating intra-group relative comparisons and global historical comparisons. The formula for calculating the composite advantage value is as follows: in, For the first The absolute quality score of each answer. The average score of the answers within the current group. This is a dynamically updated global historical baseline value. To meet Adjustable balance coefficient; Step S4: Using the policy gradient update module, the log probability gradient of the policy model is calculated and the policy model parameters are updated with the composite advantage value as the weighting weight, thereby increasing the probability of generating answers with high advantage value and suppressing the probability of generating answers with low advantage value.

2. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 1, characterized in that: In step S2, the initial value of the global historical benchmark is set at the initial training time. The value is set as the median or expected value of the task score distribution based on prior knowledge of the task.

3. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 2, characterized in that: In step S2, the method for dynamically updating the global historical baseline value is as follows: Step S21: Obtain the average score of the current batch. and highest score ; Step S22: Update the mean baseline using the exponential moving average method. The update formula is as follows: in, Let t be the baseline for the mean of round t. This serves as the baseline for the average value of the (t-1)th round. Step S23: Update the peak baseline using a monotonically non-decreasing strategy. The update formula is as follows: in, Let t be the peak baseline. This serves as the baseline for the peak value in the (t-1)th round. Step S24: Obtain the global historical baseline value through weighted fusion. The fusion formula is as follows: in, For the combined weight parameters, .

4. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 3, characterized in that: In step S22 The adaptive smoothing factor is calculated using the following formula: in, Based on the smoothing factor, The amplitude coefficient of the adaptive adjustment is controlled. The standard deviation of the current batch score. This is the reference standard deviation.

5. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 3, characterized in that: In step S2, after updating the global historical baseline value, anomaly suppression processing is also included: Calculate the relative rate of change of the global historical benchmark. ,in It is a very small constant; like The final global historical baseline value is ; Otherwise, the final global historical baseline value will remain at [value]. ,in As an inhibitor, This is the threshold for the maximum allowable rate of change.

6. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 5, characterized in that: In step S2, after anomaly suppression processing, boundary constraints are also included: in, and These are the minimum and maximum scores for the task, respectively.

7. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 1, characterized in that: In step S3, the absolute quality score is standardized before calculating the composite advantage value: ,in and The mean and standard deviation of the training set scores are given; after standardization, the within-group mean score is updated to... The global historical baseline value has been updated to The formula for calculating the composite advantage value has been updated accordingly: .

8. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 7, characterized in that: The balance coefficient The parameters are dynamically adjusted as the training process progresses, and the adjustment formula is: ,in As the initial weights, The decay time constant, This represents the current training iteration number.

9. The dynamic benchmark-stabilized grouping relative strategy optimization method according to claim 7, characterized in that: In step S3, the calculation of the composite advantage value also incorporates intra-group score ranking information, specifically: calculating the first... Ranking of candidate answers ,in for Ranked in descending order within the group The size of the candidate answer set; The relative advantage component within the group has been updated to , This is a ranking enhancement factor. The formula for calculating the composite dominance value has been updated accordingly: .

10. The dynamic benchmark-stabilized grouping relative strategy optimization method according to any one of claims 1-9, characterized in that: Repeat steps S1 to S4 to form an iterative optimization loop. By improving the quality of the answers, the global historical baseline is gradually moved upward, driving the model to evolve towards the global optimum and stability.

Images