A double-flow concurrent evolutionary trigger method and system based on overfitting signal perception

Abstract

A dual-stream concurrent evolution triggering method and system based on overfitting signal perception continuously senses the overfitting degree of the current criteria through a three-signal fusion overfitting index strategy and compares this index with a hysteresis threshold. The judgment result directly drives the switching of three scheduling states between the monitoring computation stream and the main training computation stream on the graphics processor: when not triggered, the two streams execute concurrently, causing monitoring and training to overlap; when triggered, a synchronization event is inserted, causing the main stream to pause and wait for evolution to complete; after the hysteresis is reversed, concurrency is resumed. Simultaneously, monitoring statistics are performed directly on GPU memory using a parallel reduction kernel function; bi-subset normalization reuses the batch normalization kernel and is accelerated using shared memory; changes in the criterion dimension caused by evolution are managed through GPU memory pre-allocation and effective masking. This invention enables the hardware computation stream scheduling strategy to be completely dynamically adapted to the algorithm criteria, achieving deep collaboration and mutual adaptation between algorithm characteristics, the graphics processor computation stream scheduler, and the parallel resources of the stream processor.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and computer architecture, and in particular to a dual-stream concurrent evolution triggering method and system based on overfitting signal sensing. Background Technology

[0002] In reinforcement learning that trains large language models using the "criterion as reward" approach, a structural defect chain emerges if the criterion set is hardcoded before training and remains unchanged during training (i.e., static criterions). As the policy model progresses, the failure patterns described by the original criterions are gradually corrected, and the quality gap between the policy response and the reference response narrows. At the same time, after each criterion is "learned," it gradually becomes saturated—almost all responses can be satisfied, the satisfaction rate tends to reach its upper limit, and the discriminative power decreases, leading to a weakening of the reward signal, a reduction in the within-group reward standard deviation, and training stagnation. More seriously, the policy model may also learn to satisfy only the surface wording of the criterion rather than its true objective, resulting in dynamic reward manipulation.

[0003] To address these issues, the industry has proposed a self-evolutionary principle: allowing the criterion set to be updated concurrently with the policy model during training, ensuring the evaluation criteria maintain their discriminative power and blocking newly emerging scoring paths. However, all self-evolutionary methods must answer a crucial question: when and how frequently should the criteria evolve? – This is the problem of evolution scheduling. Too frequent evolution can cause severe fluctuations in the reward signal and unstable training; too sparse evolution can degenerate into near-static behavior, failing to block new scoring paths in time. Existing self-evolutionary methods focus heavily on "how to generate better criteria," but almost all of them hastily address "when to evolve" with a manually fixed rule, avoiding the core issue of evolution scheduling.

[0004] In existing self-evolutionary methods, fixed-frequency evolution schemes (such as DR-Tulu / RLER, Online Rubrics, etc.) perform criterion updates every fixed number of training steps. The timing and frequency of the updates are pre-set constants that do not change with the training state, resulting in three drawbacks: the evolution timing is out of sync with the policy state—frequent evolution when the policy is not yet saturated is unnecessary jitter, while when the policy is severely overfitted, it is too late to block the scoring path; the only addition without deletion leads to the criterion set constantly expanding and becoming redundant during training; and the evolution frequency is a hyperparameter that requires repeated trial and error, and the trade-off between frequency and training stability lacks a systematic characterization. Another type of conditional triggering scheme (such as SibylSense's memory convergence triggering and RLCER's correlation screening) introduces update conditions based on a certain signal, but its triggering signal does not target "overfitting / scoring of the policy to the current criterion" - SibylSense looks at the convergence of the memory itself, and RLCER looks at the correlation between the criterion and correctness. Neither directly monitors whether the policy model is scoring the current criterion or whether it has saturated. Therefore, it cannot evolve in time when overfitting just occurs.

[0005] More importantly, existing self-evolutionary schemes, in their engineering implementation, only treat the graphics processing unit (GPU) as a passive platform for algorithm execution, without considering the coordination between the algorithmic characteristics of evolution scheduling and hardware resource scheduling. Regardless of whether a fixed frequency or conditional triggering is used, existing methods execute the evolution decision and training processes serially on the same computational stream—monitoring statistics, trigger decision, evolution execution, and policy gradient updates are all crammed into the same GPU computational stream. This serial architecture leads to two problems: First, the monitoring statistics required for trigger decision (such as the standard deviation of intra-group rewards and criterion satisfaction rate) must be performed in every training step. If calculated serially on the CPU, this introduces host-side blocking at each step; if it shares a stream with training on the GPU, it competes for resources with training computation. Second, evolution is an occasional but significant event—once triggered, it requires calling a large language model to compare and generate new criteria, which takes much longer than a normal training step. When executed serially, the evolution process completely blocks the entire training pipeline, causing the GPU stream processor to be idle during this period, resulting in a serious waste of computing resources. Existing methods fail to recognize that the algorithmic characteristic of "evolution being an occasional heavyweight event" has the potential to drive hardware computation flow scheduling—the vast majority of training steps do not require evolution, and in this case, monitoring and training should be parallel; only a few steps that trigger evolution require synchronous waiting, and existing serial architectures cannot implement this mechanism of dynamically switching hardware scheduling strategies based on the evolution trigger state.

[0006] In summary, existing self-evolutionary methods generally suffer from a disconnect between the "evolutionary scheduling algorithm" and "hardware execution": the disconnect between evolution timing and policy state leads to unnecessary jitter or delayed responses; frequency hyperparameters rely on trial and error without systematic characterization; and trigger signals are not targeted at policy overfitting. In engineering implementation, monitoring and statistics, trigger determination, evolution execution, and training updates are sequentially crowding the same computational flow, without differentiated configuration of hardware computational flow scheduling based on the "occasional heavyweight" characteristics of the evolutionary algorithm. This disconnect causes GPU computing power to be consumed by resource contention for monitoring and training in most non-triggered training steps, and wasted by pipeline blockage during evolution in occasionally triggered steps, resulting in limited overall system training efficiency. Therefore, there is an urgent need for a method that can use the algorithmic characteristics of evolutionary scheduling in reinforcement learning training—especially the structural difference of "overfitting signal monitoring must be done at every step, and evolutionary triggering is sporadic and heavyweight"—as a hardware scheduling-driven perceptual signal, so as to enable adaptive concurrency and synchronous scheduling of monitoring computation, trigger determination and main training process on different computational flows of the graphics processor, so as to achieve deep collaboration and mutual adaptation between algorithmic features and the internal structure of the computer system (including GPU computational flow scheduler and parallel resources of stream processor).

[0007] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a dual-stream concurrent evolution triggering method and system based on overfitted signal sensing.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A dual-stream concurrent evolution triggering method based on overfitting signal perception is applied to a reinforcement learning training system running on a computing device equipped with a graphics processor. The reinforcement learning training system uses a set of scoring criteria to evaluate the generated responses of the policy model. The method includes the following steps: S1. In each training step, obtain a set of responses sampled by the policy model for the same prompt. The evaluation model scores each response according to the current criterion set and weights them to synthesize a reward. At the same time, the standard deviation of the reward within the group, the satisfaction rate of each criterion, and the quality gap between the policy response and the reference answer are calculated. S2. The raw data used to calculate the within-group reward standard deviation, the satisfaction rate of each criterion, and the quality gap reside in the video memory of the graphics processor. On the first computing stream of the graphics processor, based on the decay magnitude of the within-group reward standard deviation relative to its baseline, the proportion of criteria whose satisfaction rate exceeds a preset saturation threshold, and the shrinkage magnitude of the quality gap relative to its baseline, three overfitting signals are calculated and weighted and fused into an overfitting index. S3. Compare the overfitting index with a preset trigger threshold. When the overfitting index reaches the trigger threshold, trigger the criterion evolution; otherwise, continue normal training and keep the first computing stream and the second computing stream carrying the main training process running concurrently. S4. In response to the trigger, a synchronization event is inserted between the first computing stream and the second computing stream to pause the main training process on the second computing stream and perform criterion evolution: analyze the new failure mode and saturated criteria from the answers sampled by the current policy, add and delete positive and negative criteria in both directions, and output the old criterion subset that is retained and the new criterion subset added this time. S5. Calculate the reward for each group of answers using the old criterion subset and the new criterion subset respectively. After independently standardizing the rewards within each subset, two sets of advantages are obtained. The two sets of advantages are convexly combined according to the preset merging weight to obtain the final advantage. S6. Feed the final advantage back into the reinforcement learning algorithm to update the policy model parameters, remove the synchronization event, restore the main training process on the second computation flow, and reset the trigger state.

[0010] A dual-stream concurrent evolution triggering system based on overfitting signal sensing, applied to reinforcement learning training, includes: A graphics processing unit (GPU) includes multiple stream processors, the GPU being configured to run a monitoring computation stream and a main training computation stream, the monitoring computation stream and the main training computation stream being executed concurrently; The host memory, coupled to the graphics processor, stores computer program instructions; When the graphics processor or a processor coupled to the graphics processor executes the computer program instructions, it implements the dual-stream concurrent evolution triggering method based on overfitting signal sensing.

[0011] The present invention has the following beneficial effects: This invention relates to a dual-stream concurrent evolution triggering method and system based on overfitting signal perception. It utilizes the structural difference in evolution scheduling algorithms during reinforcement learning training—namely, the requirement for overfitting signal monitoring at every step, and the sporadic yet heavyweight nature of evolution triggering—as the basis for GPU computational flow scheduling. This achieves deep collaboration and mutual adaptation between algorithmic features and hardware resources. Specifically, this invention executes the monitoring computational flow and the computational flow carrying the main training process concurrently on the GPU, ensuring that the parallel reduction statistics of the monitoring module and the training computation overlap in most training steps without blocking each other. When the overfitting index obtained from the fusion of monitoring signals reaches a trigger threshold, a synchronization event is inserted between the two computational flows in response to the triggering of this index, causing the main training flow to pause and wait for evolution to complete. When the overfitting index falls below the lower threshold, the synchronization event is removed, and the concurrent execution of the two computational flows resumes. Therefore, the determination result of "whether the overfitting index has reached the trigger threshold" in the algorithm of this invention directly drives the switching of the three scheduling states of "concurrency / synchronous waiting / resuming concurrency" between the two computing streams of the graphics processor. When not triggered, pure concurrency is used to hide the monitoring overhead. When triggered, synchronization is inserted to ensure that the main training continues after the evolution is completed. After falling back, concurrency is resumed to minimize pipeline blockage. This makes the hardware scheduling strategy fully adapt to the algorithm criteria dynamically, avoiding the dual efficiency loss of monitoring and training competing for resources and evolution blocking the entire pipeline in the existing serial architecture.

[0012] Furthermore, the hardware collaboration mechanism of this invention is also reflected in the optimization of monitoring statistics, stabilization reduction, and GPU memory management. In the monitoring stage, this invention stores a set of answer reward vectors, satisfaction matrices, and reference vectors in the GPU memory. Multiple stream processors simultaneously perform block summation and sum-of-squares reduction using a parallel reduction kernel function. A single kernel function call can obtain all the statistics required for the three overfitting signals, eliminating the need to transfer intermediate results back to host memory and avoiding data transfer between host and device at each step. In the stabilization stage, this invention implements the z-score normalization of the old and new criterion subsets as a GPU kernel function using a batch normalization mode. Shared memory is used to complete the mean and sum-of-squares reduction within a thread block, keeping the overhead of this step to the microsecond level. It can also be integrated with advantage calculation to further eliminate the need for intermediate tensor GPU memory read / write operations. In managing the dynamic changes of the criterion set, this invention pre-allocates video memory space once according to the preset maximum number of criteria. It uses an effective mask to mark the currently effective criteria to cope with changes in the number of criteria caused by evolution, avoiding fragmentation and overhead caused by repeated allocation and release of video memory during runtime. The reference answer and fixed evaluation model reside in page-locked memory or video memory, and the entire training process is reused, further compressing the I / O latency of each step. Discrete operations such as adding and deleting criterion text and semantic management are performed on the CPU side, while numerically intensive satisfaction scoring, signal reduction and standardization are performed on the GPU side. When evolution is triggered, only a small amount of metadata, namely the updated criterion index and mask, is synchronized to the GPU, which clarifies the heterogeneous division of labor. The above mechanism implements the "monitoring-evolution-stabilization" algorithm characteristics of this invention one by one into the specific configuration and scheduling of the graphics processor's parallel reduction, multi-computation stream concurrency, shared memory reduction and video memory layout. This enables the graphics processor's stream processor parallel resources, computation stream scheduler and shared memory bandwidth to be optimized under the drive of algorithm characteristics, forming a closed-loop collaboration from algorithm state perception to hardware scheduling decision-making, effectively overcoming the technical defect in existing solutions where the algorithm can calculate but the hardware cannot keep up.

[0013] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0014] Figure 1 This is a flowchart of the overall process of the dual-stream concurrent evolution triggering method based on overfitted signal sensing in an embodiment of the present invention.

[0015] Figure 2 This is a diagram illustrating the overall architecture of the adaptive triggering evolution scheme in the reinforcement learning closed loop according to an embodiment of the present invention.

[0016] Figure 3 This is a flowchart of the overfitting signal monitoring and adaptive triggering criterion in an embodiment of the present invention.

[0017] Figure 4This is a flowchart illustrating the stabilization process of independent normalization of bi-subsets before and after evolution in an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0019] like Figure 1 As shown, this embodiment of the invention provides a dual-stream concurrent evolution triggering method based on overfitting signal perception, applied to a reinforcement learning training system running on a computing device equipped with a graphics processor. The reinforcement learning training system uses a set of scoring criteria to evaluate the generated responses of the policy model. (See reference...) Figure 2 The overall architecture of the adaptive triggered evolution method shown is as follows: The method includes the following steps: Step S1: In each training step, obtain a set of responses sampled by the policy model for the same prompt. The evaluation model scores each response according to the current criterion set and weights them to synthesize a reward. At the same time, the standard deviation of the reward within the group, the satisfaction rate of each criterion, and the quality gap between the policy response and the reference answer are calculated. Step S2: The raw data used to calculate the within-group reward standard deviation, the satisfaction rate of each criterion, and the quality gap reside in the video memory of the graphics processor. On the first computing stream of the graphics processor, based on the decay magnitude of the within-group reward standard deviation relative to its baseline, the proportion of criteria whose satisfaction rate exceeds a preset saturation threshold, and the shrinkage magnitude of the quality gap relative to its baseline, three overfitting signals are calculated and weighted and fused into an overfitting index. Reference Figure 3 The overfitting signal monitoring and adaptive triggering criterion process shown in the illustration includes, in some embodiments, the three overfitting signals in step S2 as follows: a first signal, which is the percentage decrease of the within-group reward standard deviation relative to its baseline, used to characterize the degree of overfitting of the policy model to the current criterion; a second signal, which is the ratio of the number of criteria whose satisfaction rate exceeds a preset saturation threshold to the total number of criteria, used to characterize the overall saturation degree of the current criterion set; and a third signal, which is the percentage reduction of the quality gap relative to its baseline, used to characterize the degree of convergence of the gap between the policy answer and the reference answer.

[0020] In some embodiments, the weighted fusion in step S2 includes: multiplying the three overfitted signals by their corresponding fusion weights and summing the results to obtain the overfitting index. The overfitting index has a value between 0 and 1, and each fusion weight is non-negative and its sum is 1.

[0021] In some embodiments, the statistical calculations of the within-group reward standard deviation, criterion satisfaction rate, and quality gap in step S2 are performed on the graphics processor by a parallel reduction kernel function: a set of answer reward vectors, satisfaction matrix, and reference vectors reside in the graphics processor's video memory, and multiple stream processors of the graphics processor perform block summation and sum-of-squares reduction in parallel. A single kernel function call simultaneously obtains the within-group reward standard deviation, criterion satisfaction rate, and quality gap without having to transfer intermediate results back to host memory.

[0022] Step S3: Compare the overfitting index with a preset trigger threshold. When the overfitting index reaches the trigger threshold, trigger the criterion evolution; otherwise, continue normal training and maintain the concurrent execution of the first computational flow and the second computational flow carrying the main training process. In some embodiments, the preset trigger threshold in step S3 includes a preset upper threshold and a preset lower threshold, wherein the preset upper threshold is greater than the preset lower threshold; the condition for the evolution of the trigger criterion is: currently in a triggerable state and the overfitting index is greater than or equal to the preset upper threshold; after triggering, it enters an untriggerable state until the overfitting index falls back to less than or equal to the preset lower threshold and then it returns to a triggerable state.

[0023] Step S4: In response to the trigger, a synchronization event is inserted between the first computing stream and the second computing stream to pause the main training process on the second computing stream and perform criterion evolution: analyze the new failure modes and saturated criteria from the answers sampled by the current policy, add and delete positive and negative criteria in both directions, and output the old criterion subset that is retained and the new criterion subset added this time. In some embodiments, the positive and negative bidirectional addition and deletion in step S4 includes: adding criteria for characterizing new failure patterns exposed in the current strategy response, deleting criteria whose satisfaction rate exceeds a preset saturation threshold and whose contribution to the variance of the in-group reward is close to zero, and merging criteria with semantic repetition.

[0024] In some embodiments, the first computational flow is a monitoring computational flow, and the second computational flow is a main training computational flow. The monitoring computational flow and the main training computational flow are executed concurrently on the graphics processor. When the overfitting index reaches the trigger threshold, in response to the triggering of the overfitting index, a synchronization event is inserted between the monitoring computational flow and the main training computational flow, causing the main training computational flow to pause and wait for the criterion evolution to complete before continuing. When the overfitting index falls below the preset lower threshold, the monitoring computational flow resumes concurrent execution with the main training computational flow, and no further synchronization events are inserted.

[0025] In some embodiments, the change in the number of criteria caused by the evolution of the criterion set is managed by pre-allocation of video memory: the video memory space of the satisfaction matrix and weight tensor is allocated at once according to the preset maximum number of criteria, an effective mask vector with a length equal to the preset maximum number of criteria is maintained, the corresponding mask bit is set when a new criterion is added, the corresponding mask bit is cleared when a criterion is deleted, and the criteria that are masked are skipped during the operation.

[0026] Step S5: Calculate the reward for each group of answers using the old criterion subset and the new criterion subset respectively. After independently standardizing the rewards within each subset, two sets of advantages are obtained. The two sets of advantages are convexly combined according to the preset merging weight to obtain the final advantage. Combination Figure 4 The evolution-before-and-after bi-subset independent normalization process is shown in some embodiments. In step S5, the independent normalization includes: calculating the mean reward and standard deviation of the old criterion subset and the new criterion subset respectively; subtracting the mean of the subset from each reward within each subset and then dividing by the standard deviation of the subset to obtain the advantages of the two sets of independent normalization; the preset merging weight is determined according to the proportion of the number of criteria in the new criterion subset to the total number of criteria in the old and new criterion subsets.

[0027] Step S6: Feed the final advantage back into the reinforcement learning algorithm to update the policy model parameters, remove the synchronization event, restore the main training process on the second computation flow, and reset the trigger state.

[0028] See also Figure 1 and Figure 2 This invention also provides a dual-stream concurrent evolution triggering system based on overfitting signal sensing, applied to reinforcement learning training. The system includes a graphics processor and a host memory coupled to the graphics processor.

[0029] The graphics processing unit (GPU) includes multiple stream processors configured to run a monitoring computation stream and a main training computation stream, which execute concurrently. The host memory stores computer program instructions, and the GPU, or a processor coupled to the GPU, executes these computer program instructions to perform the following operations: First, at each training step, a set of responses sampled by the policy model for the same prompt is obtained. The evaluation model scores each response according to the current criterion set and weights them to synthesize a reward. At the same time, the standard deviation of the reward within the group, the satisfaction rate of each criterion, and the quality gap between the policy response and the reference answer are calculated.

[0030] Subsequently, the raw data used to calculate the within-group reward standard deviation, the satisfaction rate of each criterion, and the quality gap reside in the video memory of the graphics processor. On the monitoring computation stream, based on the decay magnitude of the within-group reward standard deviation relative to its baseline, the proportion of criteria whose satisfaction rate exceeds a preset saturation threshold, and the shrinkage magnitude of the quality gap relative to its baseline, three overfitting signals are calculated and weighted and fused into an overfitting index.

[0031] Next, the overfitting index is compared with a preset trigger threshold. When the overfitting index reaches the trigger threshold, the criterion evolution is triggered; otherwise, normal training continues, maintaining the concurrent execution of the monitoring computation stream and the main training computation stream.

[0032] In response to the trigger, a synchronization event is inserted between the monitoring computation stream and the main training computation stream, causing the main training computation stream to pause and execute criterion evolution: analyze new failure modes and saturated criteria from the responses sampled from the current policy, perform bidirectional addition and deletion of the criterion set, and output the retained subset of old criteria and the newly added subset of criteria.

[0033] Then, the rewards for each group of answers are calculated using the old criterion subset and the new criterion subset respectively. After independently standardizing the rewards within each subset, two sets of advantages are obtained. The two sets of advantages are then combined convexly according to a preset merging weight to obtain the final advantage.

[0034] Finally, the final advantage is fed back into the reinforcement learning algorithm to update the policy model parameters, the synchronization event is removed, the main training computation flow is restored, and the trigger state is reset.

[0035] In this embodiment, the above operations are executed by the graphics processor and the host memory in cooperation, making full use of the graphics processor's multi-stream concurrency capability and parallel computing resources, and realizing the collaborative optimization of overfitting signal perception and dual-stream concurrent evolution triggering.

[0036] The dual-stream concurrent evolution triggering method and system proposed in this invention, based on overfitting signal perception, uses the algorithmic characteristics of evolution scheduling in reinforcement learning training—namely, the structural difference of "overfitting signal monitoring being mandatory at every step, and evolution triggering being sporadic yet heavyweight"—as the basis for perceiving the GPU computation flow scheduling. This achieves mutual adaptation and cooperation between algorithmic features and hardware resources at the technical implementation level. Specifically, the algorithm continuously outputs a status signal indicating "whether evolution has been triggered" through overfitting index and hysteresis threshold determination using a three-signal fusion method. This signal directly drives the dynamic switching of three scheduling strategies between the GPU's monitoring computation flow and the main training computation flow: when not triggered, the two streams execute concurrently, with the monitoring parallel reduction statistics overlapping with the training computation, hiding the monitoring overhead at each step; when triggered, a synchronization event is inserted, the main training flow pauses and waits for evolution to complete, ensuring that the criterion set takes effect promptly after evolution; after evolution is completed, the synchronization event is removed, and the two streams resume concurrency, allowing the hardware scheduling strategy to be fully dynamically adapted to the algorithm criteria. The switching between the three scheduling states is entirely determined by whether the overfitting metric calculated by the algorithm exceeds the upper threshold or falls back to the lower threshold, forming a direct and clear causal relationship between the hardware scheduling logic and the algorithm criteria. Correspondingly, the statistical calculations of the monitoring module are directly completed on GPU memory using a parallel reduction kernel function, without needing to be sent back to host memory; the bi-subset normalization reuses the batch normalization kernel and accelerates it with shared memory, avoiding global GPU memory read / write operations of intermediate tensors; changes in the criterion dimension caused by evolution are eliminated by GPU memory pre-allocation and effective mask management, eliminating runtime reallocation overhead. Therefore, the overfitting signal calculated by the algorithm not only determines "when to evolve," but also directly determines whether the GPU computational streams run concurrently or wait synchronously, enabling the GPU computational stream scheduling strategy to dynamically adapt to the algorithm criteria, achieving a closed-loop collaboration from "algorithm state awareness" to "hardware scheduling decision-making."

[0037] The features, working principle and advantages of specific embodiments of the present invention are further described below.

[0038] Overall Architecture and System Composition

[0039] This embodiment provides a dual-stream concurrent evolution triggering method and system based on overfitting signal perception. The system inserts two core modules (running on a GPU-equipped server) into the "criterion is reward" reinforcement learning loop. The overall architecture is as follows: Figure 2 As shown. Figure 2 Each part shown completes overfitting monitoring, evolution triggering, and stabilization at the algorithm logic level, while its operating characteristics correspond to the hardware execution mechanism of the GPU. The batch statistics of the monitoring module are implemented through parallel reduction, the evolution triggering judgment and the main training process are executed concurrently on independent computational flows, and the bi-subset reduction of the stabilization module is accelerated by shared memory. This makes the algorithm design of the present invention compatible with the scheduling strategy of GPU hardware resources at the architecture level.

[0040] Overfitting Signal Monitoring Module: At each training step, three signals reflecting "overfitting / saturation of the policy with the current criterion" are statistically analyzed and weighted into an overfitting index. Exceeding a threshold triggers evolution. The statistical computation of this module is implemented using a GPU parallel reduction kernel function. The reward vector, satisfaction matrix, and reference vector are stored in GPU memory, and summation and square reduction are performed in parallel by multiple stream processors, avoiding data transfer between the host and device at each step.

[0041] Criterion Evolution Module: When triggered, it samples responses from the current policy and updates the criterion set based on the new failure modes it exposes (both positive and negative additions and deletions). Evolution is an occasional but significant event; once triggered, it requires calling a large language model to generate new criteria. In this embodiment, the monitoring module and the criterion evolution module are placed on independent GPU computing streams and executed concurrently with the computing stream carrying the main training process. When not triggered, the two streams run in parallel to hide the monitoring overhead. When triggered, a synchronization event is inserted between the streams to make the main training stream wait for the evolution to complete, allowing the scheduling strategy of the hardware computing stream to dynamically adapt to the evolution trigger state.

[0042] The evolution stabilization module independently normalizes the advantage signals of two subsets: the "old criteria before evolution" and the "new criteria after evolution," before merging them to ensure that abrupt changes in the reward distribution do not disrupt training stability. This module's bi-component z-score normalization reuses the batch normalization kernel function, using GPU shared memory to perform mean and sum-of-squares reduction within thread blocks. It can also be integrated with advantage calculation to eliminate intermediate tensor memory reads and writes, keeping the additional overhead of the stabilization step to the microsecond level.

[0043] The remaining components are standard: a policy model (the trained large language model), a judge model, a reference answer, and a GRPO optimizer. The judge model is a fixed (not involved in training, and its parameters are not updated) large language model. It receives "the current criterion set + one answer," and determines the satisfaction level of each answer for each criterion (values ​​[0,1]). These satisfaction levels are weighted and summed according to the weights of each criterion (in this embodiment, each criterion is equally weighted or its weight is inherent in the criterion set), thus obtaining the scalar reward for that answer, which is then fed back into the GRPO. The criterion set is directly provided to the judge model as the basis for its scoring: once the criterion set evolves and changes, the criteria used by the judge model for scoring also change, thereby updating the reward signal accordingly. (Note: The term "judge model" here is different from the reference answer mentioned below.) gap The "fixed independent evaluation model" used in the calculation consists of two different components—the former scores each criterion in the criterion set and generates training rewards, while the latter only assigns an external reference score to the overall quality of the answer and does not use the criterion set.

[0044] Combination Figure 2The overall architecture and data flow are as follows, with the data flow for each training step as follows: First, the policy model samples a set of responses for each batch of prompts (this is the original GRPO step); this set of responses is fed into the evaluation model, which scores each response according to the current criterion set and weights them to obtain the reward for each response. Simultaneously, it calculates the intra-group information (intra-group reward standard deviation, criterion satisfaction rate, and reference gap calculated using a fixed independent evaluation model) – the reward generation is the original GRPO, and the intra-group statistics are calculated incidentally in this step and provided to the monitoring module (the monitoring module does not additionally call the policy model for sampling, but only reuses existing data in this step to reduce overhead and coupling). Next, the overfitting signal monitoring module reads the above intra-group statistics, calculates three signals and overfitting indicators, and determines whether to trigger evolution based on the hysteresis threshold. The statistical calculations of this module are executed on the monitoring computation flow of the GPU, running concurrently with the computation flow carrying the main training process, so that the monitoring overhead overlaps with the training computation in most training steps. If no overfitting criterion is triggered, the data directly enters the stabilization module. If triggered, the criterion evolution module is called—in response to the overfitting criterion, a synchronization event is inserted between the monitoring stream and the mainstream, causing the mainstream to pause and wait for the evolution to complete. The evolution module analyzes new failure modes and saturated criteria from the currently sampled responses (this analysis is done by calling a large language model, which can be from the same source as the evaluation model or a different one), and adds or removes criteria in both positive and negative directions accordingly, outputting "old criterion subset + new criterion subset". Regardless of whether evolution is triggered, the data eventually enters the evolution stabilization module: it calculates the advantage for the old and new criterion subsets separately, standardizes them independently, and then convexly combines them into the final advantage. This final advantage replaces the advantage originally calculated by the unified reward in GRPO and is sent to the GRPO optimizer to update the policy model; the rest of the GRPO process (gradient calculation, parameter update) remains unchanged. After evolution is complete, the synchronization event between the streams is removed, restoring pure concurrent execution of the two computation streams; the monitoring stream continues to provide real-time awareness of the overfitting criterion, allowing the hardware scheduling strategy to continuously and dynamically adapt to the algorithm criteria. Therefore, the three modules of "monitoring-evolution-stabilization" are embedded into GRPO through the interface of "reading intragroup statistics and outputting the advantages after replacement", without changing the original sampling, reward weighting and gradient update of GRPO.

[0045] Overfitting signal monitoring module

[0046] In this embodiment, the overfitting signal monitoring module calculates the following three signals at each (or every several) training steps. All three signals are normalized to [0,1] (0 indicates no overfitting, 1 indicates strong overfitting), and each normalization is referenced to a baseline (signal one uses a base...). std Baseline, signal, and three bases gap(Using this as a baseline, the two are independent and calculated separately). The statistical calculations of this module are performed on the monitoring computation flow of the graphics processor, running concurrently with the computation flow carrying the main training process, so that the acquisition of overfitting metrics does not block the main training pipeline.

[0047] Baseline initialization and update: Where B represents the number of training steps used to establish the baseline (default B=50). To avoid baseline distortion in the initial training steps due to poor policy performance, low response quality, and high variance, the baseline is not directly taken as a simple average of the first B steps, but rather uses an exponential moving average (EMA) for smooth initialization: The training is first run for B steps to initially stabilize the policy, during which... std reward , ref gap Each maintains an EMA value (base ← β·base + (1-β)·current value, with the smoothing coefficient β defaulting to 0.9), using this EMA value as the initial baseline. The baseline is continuously and slowly updated in subsequent training, but only on "training steps that do not trigger evolution" (to avoid polluting the baseline by abrupt changes in the reward distribution caused by evolution). The update also uses the above EMA form but takes a smaller update rate (e.g., 1-β=0.01), that is, maintaining a long-term moving average as the baseline; in this way, the baseline slowly follows the overall improvement of the policy's capabilities, without the early fixed values ​​becoming meaningless in the later stages, and is not disturbed by the evolutionary events themselves.

[0048] The specific deterministic formulas for normalizing each signal are as follows: Signal 1: Within-group reward standard deviation decay g1. Calculate the standard deviation of the rewards for each response within the same group for the same prompt. std reward (Multiple hints to take the mean). Overfitting leads to convergent responses within groups. std reward Decrease. Normalized to the percentage decrease relative to the baseline and truncated to [0,1]: g1 = clip( (base std - std reward ) / (base std + ε), 0, 1) (Equation 1) Signal 2: Criterion satisfaction rate saturation g2. Calculate the average satisfaction rate of each criterion, and take the value when the satisfaction rate exceeds the high-order threshold. ρ sat The percentage of criteria number (default 0.9) is used as the saturation level: g2 = (satisfaction rate> ρ sat (Number of criteria) / (Total number of criteria m) (Naturally in [0,1]) (Equation 2) Signal 3: Reference gap narrowing g3. The quality gap between the statistical strategy response and the reference response, ref_gap (its deterministic calculation method is explained in "Calculation of Reference Gap ref_gap" below). Normalized to the reduction ratio relative to the baseline and truncated to [0,1]: g3 = clip( (base gap - ref gap ) / (base gap + ε), 0, 1) (Equation 3) In the above formula, g1 , g2 , g3 These are three overfitted signals (all taking values ​​[0,1]); std reward The standard deviation of the rewards for each group of responses to the same prompt (take the mean for multiple prompts); base std for std reward The baseline (obtained using the aforementioned EMA method); ρ sat The threshold for the high-order satisfaction rate of the saturation criterion (default 0.9). m This represents the number of criteria in the current criterion set. ref gap The quality gap between the strategy-based answer and the reference answer; base gap for ref gap The baseline; clip(·, 0, 1) This means that the value will be truncated to the interval [0,1]. ε To prevent extremely small positive numbers with a denominator of zero (such as 1e-8).

[0049] The weighted fusion of three signals is used as an overfitting metric (since g1, g2, g3 ∈ [0,1] and the sum of the weights is 1, Φ ∈ [0,1], and the threshold can be consistently interpreted across the training process): Φ = α1·g1 + α2·g2 + α3·g3 (α1+α2+α3 = 1, default α1=α2=α3=1 / 3) (Formula 4) In the formula, Φ is the overfit index after fusion (with a value of [0,1], the larger the value, the more severe the overfitting of the strategy to the current criterion); α1, α2, and α3 are the fusion weights of the three signals (all are non-negative, and the sum of the three is 1, with each taking 1 / 3 by default).

[0050] When the overfitting metric reaches the trigger threshold, a criterion evolution is triggered; otherwise, no evolution occurs, and training continues (wherein...). Φthreshold The overfitting threshold that triggers evolution is a decision threshold with a value of [0,1]. To prevent the evolution from being repeatedly triggered (jittering) due to fluctuations in Φ around the threshold, this embodiment adopts a hysteresis threshold (dual threshold) mechanism: Let the upper threshold be... Φ hi (Default 0.6) and lower threshold Φ lo (Default 0.4) Φ hi >Φ lo It maintains a boolean state `armed` (indicating whether it is in a triggerable state, initially true). Its debouncing logic is as follows: armed = True (Initially in a triggerable state) Each training step: Calculate Φ If armed and Φ ≥ Φ hi : (Triggers only when "armed" and above the upper threshold)

[0051] Triggering a criterion evolution

[0052] armed = False (Disarms upon triggering, temporarily disables re-triggering)

[0053] If (not armed) and Φ ≤ Φ lo : (Re-arming will only begin when Φ falls below the lower threshold)

[0054] armed = True (Equation 5)

[0055] That is: after an evolution is triggered, the system enters an "unarmed" state and temporarily disables monitoring triggers, and must wait for Φ to fall back to the lower threshold. Φ lo Only then will the next trigger be allowed, thus allowing a hysteresis interval. Φ lo , Φ hi Isolation thresholds are isolated from fluctuations. Therefore, the timing of evolution is adaptively determined by the policy's true overfitting state, rather than at a fixed frequency.

[0056] The above ρ sat B, β, α, Φ hi / Φ lo All of these provide default values ​​that can be used directly, which allow those skilled in the art to reproduce a runnable instance without parameter scanning; parameter tuning beyond these default values ​​is an optional optimization step rather than a requirement.

[0057] Calculation of the reference gap: The reference gap (ref_gap) used in Signal 3 measures the "quality gap between the strategy answer and the reference answer," and its calculation is as follows. First, the "judgment" used is performed by a pre-trained, independent evaluation model that remains unchanged throughout the training process. This evaluation model is independent of the evaluation model that evolves with training in this system and does not use the currently evolving criterion set for scoring; instead, it assigns a scalar score of [0,1] to the "overall quality of the answer." The reason for choosing a fixed, independent evaluation model is to provide... ref gap Provide an external stable reference that does not drift with the evolution of criteria. Secondly, the calculation process is as follows: For each prompt, take the reference answer (a high-quality reference answer, which can be manually written or generated and fixed by a strong model) and the current policy answer, and assign an overall quality score to each using the aforementioned fixed evaluation model. The difference between the two is the gap for that prompt; then, average all prompts within a batch to obtain the score for this step. ref gap (That is, "first calculate the difference between the reference score and the policy score for each prompt, and then take the average across prompts", which is a single scalar, not a criterion-by-criterion score). Third, to control overhead, the reference answer is pre-generated and cached, and the entire training process is reused without the need to regenerate it at each step; the evaluation model call can share a forward pass with the statistics of signal three, with the overhead being scoring each batch of prompts once at each step. In some embodiments, the reference answer and the weights of the fixed evaluation model can reside in page-locked memory or GPU memory, and the entire training process can be reused to avoid reloading or transferring reference data at each step.

[0058] In some embodiments, the statistical calculation of the above three signals can be implemented on a graphics processor using a parallel reduction kernel function: a set of answer reward vectors, satisfaction matrices, and reference subvectors are stored in video memory, and multiple stream processors perform block summation and sum-of-squares reduction in parallel, obtaining the results simultaneously in a single kernel function call. std reward Satisfaction rate of each criterion ref gap There is no need to send intermediate results back to host memory. This parallel reduction is performed on the GPU's monitoring computation stream and runs concurrently with the main training computation stream, so that the additional computation of monitoring statistics is covered by the computation of training itself and does not become a bottleneck in the training pipeline.

[0059] Figure 3 The complete process of the above-mentioned overfitting signal monitoring and adaptive triggering criteria is shown.

[0060] Criterion Evolution Module

[0061] Once triggered, a criterion set update will be performed: Sample several answers from the current strategy; Analyze the new failure patterns and saturation criteria revealed by these responses; Add and delete in both positive and negative directions: Add criteria that can characterize the current new failure mode (negative: block new scoring paths), add criteria that are still lacking in the current strategy (positive), and delete / merge criteria that are saturated or redundant. Output the updated set of criteria, distinguishing between the "retained subset of old criteria" and the "new subset of criteria added this time", for use by the stabilization module.

[0062] The specific steps of each sub-step in this module are as follows: (1) Sampling: directly reuse the group of answers that have been sampled in the current GRPO step, without additional sampling; (2) Failure mode analysis and new criterion generation: take the answers with high scores but still judged as poor by a fixed strong evaluation model as negative examples, and the answers with low scores but judged as good as positive examples, and use a large language model for comparison generation, so that the model can compare the positive and negative examples and summarize new criteria that can distinguish between the two. This comparison generation paradigm can refer to existing training-free methods such as RLCER or OpenRubrics; (3) Deletion / merging: delete / merge the answers with a satisfaction rate exceeding ρ sat Furthermore, criteria whose contribution to the current group reward variance is close to zero are judged to be saturated and deleted, and criteria whose text semantics are nearly identical with other criteria are merged; (4) Marking: Output the updated criterion set and mark "retained old criterion subset" and "new criterion subset added this time" for use by the stabilization module.

[0063] The improvement of this module does not lie in the generation method of a single criterion (which can reuse existing comparative generation paradigms), but in the evolution triggered by the overfitting criterion mentioned above, and the bidirectional addition and deletion of positive and negative subsets and the labeling of new and old subsets. In some embodiments, the generation of new criterion text and semantic deduplication are performed on the CPU side. After the evolution is completed, only the updated criterion index and effective mask are synchronized to the GPU, without the need to transfer large amounts of text data, thereby reducing the data transfer overhead in heterogeneous computing environments.

[0064] Evolution Stabilization Module

[0065] Evolution can cause a sudden change in the reward distribution at this step (the addition / deletion of criteria alters the reward composition). Directly standardizing the mutated rewards as advantages would lead to abrupt changes in the advantage signal scale and training oscillations. In this embodiment, drawing on the idea of ​​decoupling normalization, the advantages of the old criterion subset (criters that existed before evolution and are retained this time) and the new criterion subset (newly added criteria) are calculated separately and standardized independently, then merged according to a determined convex combination: Each subset is standardized using its own mean and standard deviation of rewards for within-group z-score standardization: A old = ( Rold - mean(R old ) ) / ( std(R old ) + ε ) (Old criteria subset: Only rewards calculated using the old criteria are standardized) A new = ( R new - mean(R new ) ) / ( std(R new ) + ε ) (New criteria subset: Only rewards calculated using the new criteria are standardized) Convex combinations based on the criterion ratio of the two subsets: λ = m new / (m old + m new ) A = (1-λ)·A old + λ·A new (Equation 6) in, R old , R new The rewards are calculated using only the old criteria subset and only the new criteria subset, respectively. mean(·) , std(·) The mean and standard deviation are given. A old , A new The advantages of standardization within each subset; m old , m new λ represents the number of criteria in the old and new criterion subsets, respectively; λ is the merging weight; A is the final advantage used for policy update; and ε is a very small positive number to prevent division by zero. This standardization process can be implemented in-block reduction using shared memory on the graphics processor: the batch standardization kernel function is reused for z-score calculation, and the mean and sum of squares reduction within the thread block is completed using shared memory, keeping the overhead of this step to the microsecond level. Furthermore, it can be integrated with subsequent advantage calculations to further reduce intermediate tensor memory reads and writes.

[0066] The merging weight λ is set based on the following: In this embodiment, the default value is... λ = m new / (m old +m new) This means allocating weights according to the proportion of new and old criteria. The rationale is that, in the absence of prior knowledge, each criterion contributes approximately equally to the advantage, and weighting by quantity is an unbiased approximation of the overall contribution of each subset, and is parameterless, analytical, and stable. When it is necessary to consider differences in criterion importance or discriminative power, λ can be replaced by weighting based on the variance of subset rewards (such as by...). std(R new ) and std(R old ) Alternative approaches include proportional allocation (giving greater weight to subsets with strong discriminative power and high variance), or introducing a learnable scaling factor that is automatically determined by training.

[0067] Degradation and stability handling: when no new criteria are introduced in this step ( m new When λ = 0 (i.e., evolution was not triggered or only deletion occurred), λ = 0, degenerating to standardization using only the old subset (equivalent to the regular GRPO advantage). When the sample size of a subset is too small—that is, the number of responses from that subset participating in the statistics is less than a lower limit—λ degenerates into standardization using only the old subset (equivalent to the regular GRPO advantage). n min (default n min When the standard deviation is λ = 8, its estimation is unreliable. Therefore, a lower bound is imposed on the standard deviation of this subset: the larger of the standard deviation of this subset and 10% of the standard deviation of all global responses is taken to avoid amplifying noise by using a near-zero standard deviation. Furthermore, to prevent a sudden change from λ→1 due to the addition of a large number of criteria in a single evolution, an upper bound is set on λ. λ max (Default 0.5), that is λ ← min(λ, λ max ) This ensures that regardless of the number of new criteria, the contribution of a subset of old criteria to the advantage is no less than 1- λ max This avoids the advantage being dominated solely by a subset of the new criteria.

[0068] Since the two subsets are standardized within themselves, the difference in reward scale introduced by the new criterion is absorbed. A new Within the standardization framework, it will not cause disruption. A old The scale of the dominant signal remains smooth before and after evolution, ensuring stable training.

[0069] The difference between this implementation and the existing decoupling normalization method (PAPO) is that PAPO's decoupling normalization is used for decoupling different signal sources under static criteria; while this implementation extends it to two dynamically generated subsets during training: the "old criterion subset" and the "new criterion subset," with the subset division itself dynamically determined by the evolutionary actions. This dynamic partitioning method allows the shared memory reduction on the hardware side to be flexibly scheduled according to the actual effective size of the criterion subset, further improving the computational efficiency of the stabilization module.

[0070] Figure 4 The stabilization process of independent normalization of bi subsets before and after the above evolution is shown.

[0071] Implementation Guidelines

[0072] Typical platform: GRPO, policy model can be Qwen3-8B in scale; evaluation tasks can use public benchmarks such as HealthBench, AlpacaEval, GPQA, ArenaHard; a single machine with multiple GPUs can complete a round of training in 1-2 days, with low engineering overhead.

[0073] The baseline for the three signals is taken as an exponential moving average of the early training steps; the threshold... Φ hi / Φ lo Both the fusion weight α and the fusion weight α have default values ​​that can be used directly, so the program can run without parameter scanning. Further parameter scanning is an optional optimization.

[0074] The monitoring module performs statistical calculations, which have a much lower overhead than a single forward pass of the model; the stabilization module only changes the dominant standardization method and does not increase sampling overhead.

[0075] Example

[0076] To enable those skilled in the art to implement this method, an implementation example is given below, where all parameters are specific values. The values ​​used in this example are exemplary preferred values ​​and are not intended to limit the invention.

[0077] (a) Specific configuration (values ​​for each component)

[0078] Policy model: Qwen3-8B (a large language model that has been trained).

[0079] Evaluation model: A fixed small to medium-sized large language model (such as Qwen2.5-7B-Instruct) is not used in training. Soft satisfaction is calculated one by one according to the current criterion set.

[0080] Fixed independent evaluation model (for) ref gap ): A powerful and consistent large language model (such as GPT-4 level) scores the overall quality of the response in the range of [0,1].

[0081] Criterion generation model (used during evolution): A large language model (which may be of the same origin as the evaluation model) generates new criteria according to the aforementioned comparative generation method.

[0082] Reinforcement learning algorithm: GRPO; group size G=8; batch size 8; learning rate 1e-5.

[0083] Baseline parameters: Baseline establishment steps B=50; EMA initial smoothing coefficient β=0.9; Baseline long-term update rate 1-β=0.01 (updated only in steps where evolution is not triggered).

[0084] Signal and triggering parameters: saturation threshold ρ sat =0.9; Fusion weights α1=α2=α3=1 / 3; Hysteresis upper threshold Φ hi =0.6, lower threshold Φ lo =0.4.

[0085] Stabilization parameter: Combined weight λ = m new / (m old +m new And the upper limit λ max =0.5; Subset sample lower limit n min =8; the lower limit of std is 10% of the global std.

[0086] Operating environment: Single-machine 8-GPU server; Example evaluation tasks: HealthBench, AlpacaEval, GPQA, ArenaHard.

[0087] The execution flow for each training step is as follows: policy samples a set of answers → the evaluation model scores and awards rewards according to the current criterion set, and the std is calculated. reward / satisfaction rate / ref gap → Monitoring module calculates Φ → Hysteresis determination to see if triggered → (If triggered) Evolution module compares and generates addition / deletion criteria, marks new and old subsets → Stabilization module calculates A old / A new The A→A combination obtained by λ convex combination is then fed into the GRPO update strategy. In some embodiments, the above monitoring statistics are executed on the monitoring computation stream of the graphics processor and run concurrently with the computation stream carrying the main training process, so that the monitoring overhead overlaps with the training computation; when evolution is triggered, a synchronization event is inserted between the streams to make the main training stream wait for the evolution to complete, and after the evolution is completed, the concurrent execution of the two streams is resumed.

[0088] (II) Experimental Verification

[0089] This embodiment employs the following verification experiment to test the aforementioned beneficial effects: The comparison group settings were as follows: (a) static criterion baseline (the criterion set remained unchanged throughout the training process); (b) fixed-frequency evolution baseline (evolved once every fixed number of steps); (c) this embodiment (adaptive triggering of overfitting signals + bisubset stabilization). All three groups used the same strategy model, the same initial criterion set, and the same computing power.

[0090] Observation indicators: ① Training curve – Records the changes in rewards and within-group reward standard deviations for each group as training steps. It can evolve in a timely manner when saturation occurs, keeping the reward standard deviation within the effective range and alleviating training stagnation at the static baseline; ② Changes in criterion set size – Records the changes in the number of criteria as training steps. Due to the addition and deletion of positive and negative criteria, the number of criteria tends to stabilize, while the number of criteria in the fixed-frequency "only adds, never deletes" baseline continues to expand; ③ Anti-score manipulation – Uses a fixed independent evaluation model to measure the degree of deviation (score manipulation rate) of "the judged score increases but the actual quality does not increase". The score manipulation rate in this embodiment is lower than that of the static and fixed-frequency baselines; ④ Triggering timing – Records Φ and evolution trigger points. Triggers are concentrated in the stage of saturation / score manipulation signal increase, verifying that adaptive triggering is indeed aligned with the strategy state.

[0091] The above experiments verified the effectiveness of the overfitting signal monitoring and adaptive triggering mechanism in this embodiment. In specific engineering implementations, the above steps are further combined with the hardware resource scheduling of the graphics processor to fully utilize the hardware acceleration capabilities.

[0092] As an alternative, regarding the triggering signal, any one of the three signals mentioned above (reward standard deviation decay, satisfaction rate saturation, and reference gap reduction) or a combination of two can be used. Other overfitting proxy signals can also be added, such as a decrease in the correlation between reward and true quality, the output of a score-boosting detector, or the KL drift rate. Besides weighted summation, the fusion method can also be maximum value, logical OR (triggering if any signal exceeds the threshold), or a learned classifier. However, it should be noted that when using the above alternative combinations, it is recommended to retain at least the in-group reward standard deviation decay signal as part of the triggering criterion to ensure a direct correlation between the triggering signal and the policy overfitting state.

[0093] Regarding triggering criteria, fixed thresholds can be replaced by adaptive thresholds, hysteresis thresholds with upper and lower limit stabilization, or statistical criteria based on change point detection.

[0094] Regarding the evolution of content, the bidirectional addition and deletion of positive and negative values ​​can be replaced by simply adding new values, adjusting weights, or refreshing the memory bank; the generation of new criteria can be achieved through comparative generation, failure induction, strong model distillation, and other methods.

[0095] Regarding stabilization mechanisms, independent normalization of new and old subsets can be replaced by unified normalization with smoothing, pre-warming the new criteria with weights, or pruning the advantages, applying KL constraints, and other stabilization methods.

[0096] Regarding reinforcement learning algorithms, this embodiment is not limited to GRPO and can be replaced by other policy optimization algorithms such as PPO, RLOO, and REINFORCE.

[0097] In terms of monitoring granularity, monitoring and triggering can be performed at each step, each window, or by KL drift amount.

[0098] In some embodiments, the above alternative implementations can be flexibly combined and adopted according to specific application scenarios and hardware conditions. For example, when hardware resources are limited, evolution can be triggered by only a single overfitted signal, and a unified normalization and smoothing stabilization method can be adopted to reduce computational overhead; when the criterion size is large, a cluster-based redundancy removal step can be added. All the above alternative solutions are equivalent or obvious variations of this embodiment and should be considered to fall within the scope of this invention.

[0099] The operational characteristics of the three modules in this embodiment of the invention—monitoring every step and performing batch reduction, evolution occurring sporadically but as heavyweight events, and stabilization into small and frequent intra-block reduction—correspond to three hardware mechanisms: parallel reduction of the graphics processing unit (GPU), concurrent multi-computation streams, and shared memory acceleration. In the monitoring phase, a set of response reward vectors, satisfaction matrices, and reference vectors reside in the GPU's video memory. Multiple stream processors simultaneously perform block summation and sum-of-squares reduction using a parallel reduction kernel function. A single kernel function call yields the intra-group reward standard deviation, the satisfaction rate of each criterion, and the reference gap, without needing to transfer intermediate results back to host memory. This parallel reduction is executed on the GPU's monitoring computation stream, running concurrently with the computation stream carrying the main training process. This ensures that the additional computations for monitoring and statistics overlap with the training computations in most training steps, preventing them from becoming a bottleneck in the training pipeline.

[0100] In the evolution trigger determination stage, this embodiment of the invention fully utilizes the algorithmic characteristic that "criterion evolution is an occasional heavyweight event" to differentiate hardware scheduling. Normally, when evolution is not triggered, the monitoring stream and the main training stream execute in parallel without blocking each other. When the overfitting index exceeds a preset upper threshold and triggers evolution, a synchronization event is inserted between the two streams—the trigger determination is calculated by the monitoring stream on the GPU side, and the trigger signal is sent back to the CPU side. The CPU side executes the criterion evolution module, calling the large language model to generate new criterion text and add / delete criterion sets. After evolution is complete, the CPU synchronizes the updated criterion set index and effective mask to the GPU memory, allowing the main training stream to continue. This synchronization wait introduces overhead only once in a few steps where evolution is actually triggered. In the vast majority of steps where evolution is not triggered, monitoring and training are completely concurrent, and the hardware computing stream scheduling strategy dynamically adapts to the evolution trigger state. When the overfitting index falls below a preset lower threshold, the synchronization event is removed, and the pure concurrent execution of the two computing streams resumes. Therefore, in this embodiment, the algorithm criterion of "whether to trigger evolution" directly determines the switching between the three scheduling states of "concurrency / synchronous waiting / resumption of concurrency" between the two computing streams of the graphics processor.

[0101] In the stabilization phase, the z-score normalization of the old and new criterion subsets reuses the batch normalization kernel function. Mean and sum-of-squares reduction within thread blocks are completed using the shared memory of the graphics processor (GPU). This can be combined with advantage computation to further eliminate the need for intermediate tensor read / write operations on GPU memory. For dynamic management of criterion set changes, GPU memory is pre-allocated once according to a preset maximum number of criteria. Effective masks are used to mark currently active criteria to address changes in the number of criteria caused by evolution, avoiding fragmentation and overhead caused by repeated allocation and release of GPU memory during runtime. The reference answer and fixed evaluation model reside in page-locked memory or GPU memory, and the entire training process is reused, further compressing the input / output latency at each step. Discrete operations such as adding, deleting, and semantic deduplication of criterion text are performed on the CPU side, while numerically intensive satisfaction scoring, signal reduction, and normalization are performed on the GPU side. When evolution is triggered, only a small amount of metadata—the updated criterion index and mask—is synchronized to the GPU, clearly defining heterogeneous division of labor.

[0102] Through the hardware-aware implementation described above, the overfitting index output by the algorithm in this embodiment not only determines "when to trigger evolution," but also directly determines whether the graphics processor's computational streams run concurrently or wait synchronously. This enables the graphics processor's stream processor parallel resources, computational stream scheduler, and shared memory bandwidth to be optimally configured under the drive of the algorithm's characteristics, forming a closed-loop collaborative link from "algorithm state awareness" to "hardware scheduling decision." Among these, dual-stream concurrent scheduling is directly driven by the triggering determination of the overfitting index and has the strongest correlation with the hardware structure, making it the preferred implementation. Parallel reduction, shared memory acceleration, video memory pre-allocation, resident caching, and heterogeneous division of labor are optional auxiliary implementations, which can be selected or combined according to hardware conditions.

[0103] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0104] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0105] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0106] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0107] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0108] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0109] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0110] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0112] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0113] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0114] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0115] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A dual-stream concurrent evolution triggering method based on overfitting signal perception, applied to a reinforcement learning training system running on a computing device equipped with a graphics processor, wherein the reinforcement learning training system uses a set of scoring criteria to evaluate the generated responses of the policy model, characterized in that, Includes the following steps: S1. In each training step, obtain a set of responses sampled by the policy model for the same prompt. The evaluation model scores each response according to the current criterion set and weights them to synthesize a reward. At the same time, the standard deviation of the reward within the group, the satisfaction rate of each criterion, and the quality gap between the policy response and the reference answer are calculated. S2. The raw data used to calculate the within-group reward standard deviation, the satisfaction rate of each criterion, and the quality gap reside in the video memory of the graphics processor. On the first computing stream of the graphics processor, based on the decay magnitude of the within-group reward standard deviation relative to its baseline, the proportion of criteria whose satisfaction rate exceeds a preset saturation threshold, and the shrinkage magnitude of the quality gap relative to its baseline, three overfitting signals are calculated and weighted and fused into an overfitting index. S3. Compare the overfitting index with a preset trigger threshold. When the overfitting index reaches the trigger threshold, trigger the criterion evolution; otherwise, continue normal training and keep the first computing stream and the second computing stream carrying the main training process running concurrently. S4. In response to the trigger, a synchronization event is inserted between the first computing stream and the second computing stream to pause the main training process on the second computing stream and perform criterion evolution: analyze the new failure mode and saturated criteria from the answers sampled by the current policy, add and delete positive and negative criteria in both directions, and output the old criterion subset that is retained and the new criterion subset added this time. S5. Calculate the reward for each group of answers using the old criterion subset and the new criterion subset respectively. After independently standardizing the rewards within each subset, two sets of advantages are obtained. The two sets of advantages are convexly combined according to the preset merging weight to obtain the final advantage. S6. Feed the final advantage back into the reinforcement learning algorithm to update the policy model parameters, remove the synchronization event, restore the main training process on the second computation flow, and reset the trigger state.

2. The method according to claim 1, characterized in that, The three overfitting signals mentioned in step S2 include: a first signal, which is the percentage decrease of the within-group reward standard deviation relative to its baseline, used to characterize the degree of overfitting of the policy model to the current criteria; a second signal, which is the ratio of the number of criteria whose satisfaction rate exceeds a preset saturation threshold to the total number of criteria, used to characterize the overall saturation degree of the current criterion set; and a third signal, which is the percentage decrease of the quality gap relative to its baseline, used to characterize the degree of convergence of the gap between the policy answer and the reference answer.

3. The method according to claim 1, characterized in that, The weighted fusion in step S2 includes: multiplying the three overfitted signals by their corresponding fusion weights and summing them to obtain the overfitting index. The overfitting index has a value between 0 and 1, and each fusion weight is non-negative and its sum is 1.

4. The method according to claim 1, characterized in that, In step S2, the statistical calculations of the within-group reward standard deviation, the satisfaction rate of each criterion, and the quality gap are performed on the graphics processor through a parallel reduction kernel function: the reward vector, satisfaction matrix, and reference vector of a set of answers reside in the video memory of the graphics processor, and the block summation and sum of squares reduction are performed in parallel by multiple stream processors of the graphics processor. The within-group reward standard deviation, the satisfaction rate of each criterion, and the quality gap are obtained simultaneously in one kernel function call, without the need to transfer intermediate results back to the host memory.

5. The method according to claim 1, characterized in that, The preset trigger threshold in step S3 includes a preset upper threshold and a preset lower threshold, wherein the preset upper threshold is greater than the preset lower threshold; the condition for the evolution of the trigger criterion is: the current state is triggerable and the overfitting index is greater than or equal to the preset upper threshold; After being triggered, it enters an untriggerable state until the overfitting index falls back to less than or equal to the preset lower threshold, at which point it returns to a triggerable state.

6. The method according to claim 1, characterized in that, The positive and negative bidirectional addition and deletion in step S4 includes: adding criteria to characterize new failure patterns exposed in the current strategy response, deleting criteria whose satisfaction rate exceeds a preset saturation threshold and whose contribution to the variance of the in-group reward is close to zero, and merging criteria with semantic repetition.

7. The method according to claim 1, characterized in that, The first computing stream is a monitoring computing stream, and the second computing stream is a main training computing stream. The monitoring computing stream and the main training computing stream are executed concurrently on the graphics processor. When the overfitting index reaches the trigger threshold, in response to the triggering of the overfitting index, a synchronization event is inserted between the monitoring computation stream and the main training computation stream, causing the main training computation stream to pause and wait for the criterion evolution to complete before continuing. When the overfitting index falls below the preset lower threshold, the monitoring computation stream resumes concurrent execution with the main training computation stream, and no more synchronization events are inserted.

8. The method according to claim 1, characterized in that, The change in the number of criteria caused by the evolution of the criterion set is managed through pre-allocation of video memory: the video memory space of the satisfaction matrix and weight tensor is allocated at once according to the preset maximum number of criteria, and an effective mask vector with a length equal to the preset maximum number of criteria is maintained. When a new criterion is added, the corresponding mask bit is set, and when a criterion is deleted, the corresponding mask bit is cleared. During the operation, the criteria that are masked are skipped.

9. The method according to claim 1, characterized in that, The independent standardization in step S5 includes: calculating the mean reward and standard deviation of the reward for the old criterion subset and the new criterion subset respectively; subtracting the mean of the subset from each reward within each subset and then dividing by the standard deviation of the subset to obtain the advantages of the two sets of independent standardization; the preset merging weight is determined according to the proportion of the number of criteria in the new criterion subset to the total number of criteria in the old and new criterion subsets.

10. A dual-stream concurrent evolution triggering system based on overfitting signal sensing, applied to reinforcement learning training, characterized in that, include: A graphics processing unit (GPU) includes multiple stream processors, the GPU being configured to run a monitoring computation stream and a main training computation stream, the monitoring computation stream and the main training computation stream being executed concurrently; The host memory, coupled to the graphics processor, stores computer program instructions; When the graphics processor or a processor coupled to the graphics processor executes the computer program instructions, it implements the dual-stream concurrent evolution triggering method based on overfit signal sensing as described in any one of claims 1 to 9.

Images