A method and system for adaptive switching of computing mode based on weight sparsity perception

Abstract

A weight sparsity-aware adaptive switching method and system for computation modes addresses the problem that the sparse weight structure output by existing reinforcement learning algorithms fails to drive corresponding changes in hardware computation modes and resource scheduling. This invention uses the sparse structure generated after water-filled allocation of criterion weights below a threshold and pushing them to a lower limit as a sensing signal for hardware scheduling. The sparsity rate is calculated by counting the number of invalid weights pushed to the lower limit. When the sparsity rate reaches a preset condition, in response to the triggering of the sparsity rate, the reward synthesis operation is adaptively switched from dense matrix multiplication mode to sparse matrix multiplication mode and executed by the GPU sparse computation acceleration unit. Simultaneously, with memory-aware dynamic window configuration and validity mask-driven cross-card gradient selective synchronization, deep collaboration and mutual adaptation between algorithm features and GPU memory hierarchy, tensor cores, and interconnect bandwidth are achieved.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and computer architecture, and in particular to a method and system for adaptive switching of computing modes based on weight sparsity awareness. Background Technology

[0002] In recent years, generative artificial intelligence, represented by Large Language Models (LLM), has rapidly improved its capabilities on open-ended tasks, and its training increasingly relies on reinforcement learning (RL) to align with human preferences. For open-ended, unverifiable tasks (i.e., tasks without a single standard answer and that cannot be automatically judged by a program), how to provide reliable reward signals for reinforcement learning is a core challenge.

[0003] The "rubric-as-reward" approach is one of the current mainstream solutions: First, a set of natural language scoring criteria is written by a human or a model for the task. During training, a judge model evaluates the satisfaction of each criterion with the responses generated by the policy model. The reward signal is then obtained by weighting and summing the criteria according to their respective weights, and used for reinforcement learning optimization. The effectiveness of this paradigm largely depends on how the weights of each criterion are determined—different criteria contribute unequally to distinguishing good from bad responses, and the rationality of the weights themselves changes dynamically during training (criters gradually "saturate" as the policy improves, and their discriminative power disappears). Therefore, the weights of the scoring criteria should not be calculated and frozen all at once before training, but should be dynamically adjusted online as the policy model's capabilities drift.

[0004] Existing methods for determining criterion weights, such as static weighting schemes (e.g., discriminative filtering in OpenRubrics, whitening-uniform weighting in RRD), determine the weights once before training begins and freeze them throughout the training process, failing to adapt to criterion saturation and policy drift that occur during training. While existing online dynamic weighting schemes achieve weight updates during training at the algorithm level, their weight calculation process lacks coordination with the underlying hardware execution, resulting in the following shortcomings: First, existing solutions only focus on the accuracy of weight calculation at the algorithm level, neglecting the impact of weight distribution on hardware computing efficiency. Dynamic weighting methods, such as water-filling allocation, compress the weights of many low-information criteria to their lower limits, resulting in a naturally highly sparse weight vector. However, existing solutions still perform dense matrix multiplication (GEMM) operations on all criteria in the subsequent reward synthesis stage—regardless of whether the weights are valid, the graphics processing unit (GPU) must multiply all criteria one by one and then sum them. This causes the GPU's streaming multiprocessors (SM) and tensor cores to waste computing power on a large number of ineffective multiply-accumulate operations, and many zero or near-zero values ​​in the weights are not utilized at the instruction set level, resulting in the actual effective floating-point throughput being far lower than the hardware's peak computing power. Existing methods fail to recognize that the sparse structure of the weights output by the algorithm has the potential to drive hardware computing mode switching.

[0005] Secondly, existing solutions only treat the GPU as a passive platform for the algorithm in engineering implementation, without considering the data transfer bottleneck between GPU memory and host memory during reinforcement learning training. Current dynamic weighting schemes typically use a fixed-capacity sliding window to cache sample data, lacking a dynamic adaptation mechanism between this fixed capacity and the available GPU memory. When the fixed window capacity exceeds the current available GPU memory, the system triggers an out-of-memory error, causing training interruption, or a page fault between the GPU and host memory is triggered due to memory overload. This forces some data to be transferred to host memory via the PCIe bus, whose bandwidth is much lower than that of GPU memory (HBM), and then transferred back when needed. This repeated data transfer between GPU memory and host memory introduces unpredictable long-tail latency into the training process, significantly reducing GPU computing resource utilization and limiting the scalability of reinforcement learning training on large-scale GPU clusters.

[0006] Third, in multi-GPU training environments, existing solutions treat the gradient components of all criteria equally for cross-GPU synchronization, failing to establish an information-driven communication scheduling mechanism. When using multiple GPUs for data-parallel training, each GPU needs to synchronize gradients via high-speed interconnects (such as PCIe or NVLink) after each iteration. Existing solutions do not consider the differences in information content among criteria, synchronizing all regardless of their validity, making communication overhead proportional to the total number of criteria. Since PCIe / NVLink bandwidth is much lower than the GPU's internal memory bandwidth, and multi-GPU synchronization often becomes a critical bottleneck in the training pipeline, this approach becomes particularly prominent as the scale of criteria increases, with a large amount of bandwidth being consumed by the gradient synchronization of invalid criteria.

[0007] Fourth, existing solutions do not consider the differentiated value of criteria in computational resource scheduling. Different criteria have significantly different judgment noise rates; high-noise criteria have low scoring reliability, and their corresponding forward / backward computations contribute little to model updates. However, existing solutions treat all criterion computation tasks equally, queuing them uniformly for execution. This causes high-noise, low-value computation tasks to compete with low-noise, high-value computation tasks for the same streaming multiprocessor resources in the GPU's CUDA Stream. High-noise computations may block the execution of high-value computations, leading to a decrease in GPU pipeline efficiency and slowing down the overall training progress.

[0008] In summary, existing dynamic weight adjustment schemes generally suffer from a disconnect between algorithm design and hardware execution: while the algorithm can calculate unevenly distributed weights, the reward synthesis calculation mode, memory management strategy, cross-card communication scheduling, and computation flow priority allocation are not dynamically adjusted according to the weight distribution or criterion characteristics, and hardware resources are not differentiated based on the structural information generated by the algorithm. This disconnect prevents the GPU computing power, memory bandwidth, and interconnect bandwidth from achieving optimized scheduling driven by algorithm characteristics, resulting in limited overall system training efficiency. Therefore, there is an urgent need for a method that can use the sparse weight structure generated by the dynamic weight allocation in reinforcement learning training as a perceptual signal to drive hardware scheduling, enabling reward synthesis operations, memory management, cross-card communication, and computation flow scheduling to adaptively adjust according to the weight distribution output by the algorithm. This would achieve deep collaboration and mutual adaptation between algorithm features and the internal structure of the computer system (including GPU memory hierarchy, tensor computation cores, interconnect bandwidth, and computation flow scheduler).

[0009] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0010] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a method and system for adaptive switching of computing modes based on weight sparsity awareness.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: An adaptive switching method for computational modes based on weight sparsity awareness is applied to a reinforcement learning training system running on a computing device equipped with a graphics processing unit. The reinforcement learning training system uses a set of scoring criteria to evaluate the generated responses of the policy model. The method includes the following steps: S1. At the start of training, query the current available video memory capacity of the graphics processor, and dynamically determine the sample pool capacity of the sliding window based on the available video memory capacity, the number of scoring criteria, and the number of storage bytes required for a single satisfaction data point, so that the sample pool resides entirely in the video memory of the graphics processor to avoid page swapping; obtain the sample pool data for multiple training steps within the sliding window according to the sample pool capacity, wherein the sample pool data includes the satisfaction matrix of each answer for each scoring criterion, the true quality signal of each answer, and the judgment noise rate of each scoring criterion; S2. For each scoring criterion, calculate the correlation measure between its satisfaction vector and the true quality signal, and convert the correlation measure into the corresponding information content estimate. S3. For each scoring criterion, calculate the average correlation between its satisfaction vector and the satisfaction vectors of the other scoring criteria, and discount the information content estimate based on the average correlation to obtain the information content after redundancy removal. S4. Based on the noise rate of each scoring criterion, the redundancy-free information is monotonically attenuated to obtain the effective information after reliability correction. S5. The effective information is allocated in a water-filling manner, and the criterion weights corresponding to the information below the preset threshold are reduced to the lower limit value to obtain the weight vector of each scoring criterion. S6. Count the number of invalid weights in the weight vector that are compressed to the lower limit value, and calculate the ratio of the number of invalid weights to the total number of scoring criteria as the sparsity rate. When the sparsity rate reaches a preset condition, in response to the triggering of the sparsity rate, switch the reward synthesis operation based on the weight vector from the dense matrix multiplication mode that performs multiplication and addition operations on all scoring criteria to the sparse matrix multiplication mode that performs multiplication and addition operations only on the criteria corresponding to the valid weights, and call the sparse computing acceleration unit of the graphics processor to execute the reward synthesis operation.

[0012] A weight sparsity-aware adaptive switching system for computational patterns, applied to reinforcement learning training, includes: A graphics processor includes video memory and a sparse computation acceleration unit, wherein the video memory is used to store sliding window sample pool data during reinforcement learning training, and the sparse computation acceleration unit is used to perform sparse matrix multiplication operations. The host memory, coupled to the graphics processor, stores computer program instructions; When the graphics processor or a processor coupled to the graphics processor executes the computer program instructions, it performs the following operations: (1) When training starts, query the current available video memory capacity of the graphics processor, and dynamically determine the sample pool capacity of the sliding window based on the available video memory capacity, the number of scoring criteria, and the number of storage bytes required for a single satisfaction data, so that the sample pool is completely resident in the video memory to avoid page swapping; obtain the sample pool data for multiple training steps within the window according to the sample pool capacity, the sample pool data including the satisfaction matrix of each answer to each scoring criterion, the true quality signal of each answer, and the judgment noise rate of each scoring criterion; (2) For each scoring criterion, calculate the correlation measure between its satisfaction vector and the real quality signal, and convert the correlation measure into the corresponding information content estimate; (3) For each scoring criterion, calculate the average correlation between its satisfaction vector and the satisfaction vectors of the other scoring criteria, and discount the information content estimate based on the average correlation to obtain the information content after redundancy removal. (4) Based on the judgment noise rate of each scoring criterion, the redundancy-free information is monotonically attenuated to obtain the effective information after reliability correction. (5) The effective information is allocated in a water-filling manner, and the criterion weights corresponding to the information below the preset threshold are reduced to the lower limit value to obtain the weight vector of each scoring criterion; (6) Count the number of invalid weights in the weight vector that are compressed to the lower limit value, calculate the ratio of the number of invalid weights to the total number of scoring criteria as the sparsity rate, and when the sparsity rate reaches the preset condition, in response to the triggering of the sparsity rate, switch the reward synthesis operation based on the weight vector from the dense matrix multiplication mode that performs multiplication and addition operations on all scoring criteria to the sparse matrix multiplication mode that performs multiplication and addition operations only on the criteria corresponding to the valid weights, and call the sparse computing acceleration unit to perform the reward synthesis operation.

[0013] The present invention has the following beneficial effects: The most prominent technical effect of this invention lies in achieving deep collaboration and mutual adaptation between algorithm features and graphics processor hardware resources by using the sparse structure of weights generated by dynamic weight allocation during reinforcement learning training as a perceptual signal driven by hardware scheduling. Specifically, the online weight calculation module of this invention uses real quality signals as anchors and calculates the effective information content of each scoring criterion through three steps: correlation measurement, symmetric redundancy discount, and judgment noise rate attenuation. Then, through a water-filling allocation, the weights of low-information-content criteria are compressed to the lower limit, so that the weight vector presents a sparse distribution dynamically determined by the algorithm. On this basis, this invention calculates the sparsity rate, which characterizes the sparsity of the weights, according to the sparse distribution. When the sparsity rate reaches a preset condition, in response to the triggering of the sparsity rate, the reward synthesis operation is adaptively switched from dense matrix multiplication mode to sparse matrix multiplication mode. Only the multiplication and addition operations are performed on the criteria corresponding to the effective weights, and the sparse computing acceleration unit of the graphics processor is called to perform the operation. Therefore, the weight distribution output by the algorithm directly drives the selection of the hardware computation path—the graphics processor switches from "performing dense multiply-accumulate operations on all criteria equally" to "performing sparse multiply-accumulate operations only on effective criteria," allowing the computing power of the GPU streaming multiprocessor and tensor cores to focus on effective computation, avoiding the waste of hardware resources caused by a large number of invalid multiply-accumulate operations. Compared to the existing approach where weight calculation and hardware execution are decoupled, this invention significantly improves the effective floating-point throughput of the graphics processor and reduces the computational latency of a single reward synthesis.

[0014] Furthermore, the hardware collaboration mechanism of this invention is also reflected in the optimization of video memory management and data transmission. At the start of training, this invention queries the current available video memory capacity of the graphics processor (GPU). Based on the available video memory capacity, the number of scoring criteria, and the number of bytes required to store a single satisfaction score, it dynamically determines the sample pool capacity of the sliding window, ensuring that the entire satisfaction matrix remains entirely in the GPU's video memory during training. This memory-aware dynamic configuration mechanism ensures the adaptation between the sliding window capacity and the available GPU video memory, avoiding page swapping triggered by a fixed-capacity window exceeding the video memory threshold—that is, avoiding the long-tail latency and training pauses caused by repeated data transfer between GPU video memory and host memory via the PCIe bus, whose bandwidth is much lower than that of video memory. This improves the determinism of the training process and the utilization rate of GPU computing resources. In multi-GPU parallel training scenarios, this invention also generates a validity mask based on the effective information content of each criterion. Only the gradient components corresponding to valid criteria participate in cross-GPU reduction synchronization, while the gradient components corresponding to invalid criteria are not synchronized. This reduces the data communication bandwidth occupied on PCIe or NVLink interconnects between multiple GPUs, alleviating the key bottleneck in the training pipeline of multi-GPU synchronization. At the same time, this invention allocates GPU computing flows with different priorities based on the judgment noise rate of each criterion, so that low-noise, high-value computing tasks are executed first, avoiding high-noise, low-value computing tasks from blocking the computing pipeline, thereby improving the overall utilization efficiency of GPU computing resources.

[0015] At the algorithmic level, the online dynamic weighting mechanism of this invention also brings significant technical benefits. This invention uses the real quality signal as an anchor and measures the correlation between each criterion and the real quality through Gaussian-Cauchy mutual information proxy, enabling the distinction between truly discriminative criteria and noise criteria that are constant or irrelevant to quality. Symmetrical redundancy discounting reduces the overall weight of related criterion clusters and distributes it within the cluster, avoiding the distortion of the reward signal caused by repeated scoring of the same meaning. Monotonic decay of the judgment noise rate reduces the contribution of more unstable criteria to the reward signal, weakening the pollution of training by unreliable criteria. Simultaneously, setting a weight lower limit allows criteria that are saturated but still represent a certain ability to retain a small positive weight, preventing catastrophic forgetting caused by their ability being completely cleared. The effective information calculated by the above algorithm modules not only improves training quality but, more importantly, provides a driving signal for sparse sensing and adaptive switching at the hardware level—the more accurate the algorithm's calculations, the more reliable the basis for hardware scheduling, thus forming a positive cycle of "computation-sensing-scheduling" between the algorithm and hardware.

[0016] In summary, this invention extends the dynamic weight allocation in reinforcement learning training from the algorithm level to the hardware execution level, enabling the reward synthesis calculation mode, memory management strategy, cross-card communication scheduling, and computation flow priority allocation to adaptively adjust with the weight distribution of the algorithm output. This achieves deep collaboration between algorithm features and the internal structure of the graphics processor (including memory hierarchy, tensor calculation core, interconnect bandwidth, and computation flow scheduler), effectively overcoming the technical defect of existing solutions where "the algorithm can calculate, but the hardware cannot keep up."

[0017] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a system architecture for adaptive switching of computation modes based on weight sparsity awareness, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the information content measurement, redundancy removal, reliability decay, and water-filled weight allocation process in the online weight calculation module of this invention. Figure 3 This is a flowchart of the online dynamic weighting and reward synthesis adaptive switching training closed loop based on weight sparsity awareness, according to an embodiment of the present invention. Detailed Implementation

[0019] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0020] See Figures 1 to 3 This invention provides a method for adaptive switching of computation modes based on weight sparsity awareness, applied to a reinforcement learning training system running on a computing device equipped with a graphics processor. The reinforcement learning training system uses a set of scoring criteria to evaluate the generated responses of the policy model, and includes the following steps: Step S1: At the start of training, query the current available video memory capacity of the graphics processor, and dynamically determine the sample pool capacity of the sliding window based on the available video memory capacity, the number of scoring criteria, and the number of storage bytes required for a single satisfaction data point, so that the sample pool resides entirely in the video memory of the graphics processor to avoid page swapping; obtain the sample pool data for multiple training steps within the sliding window according to the sample pool capacity, the sample pool data includes the satisfaction matrix of each answer for each scoring criterion, the true quality signal of each answer, and the judgment noise rate of each scoring criterion; In some embodiments, the step S1 of dynamically determining the sample pool capacity of the sliding window includes: dividing the available video memory capacity by the product of the number of scoring criteria and the number of bytes of a single satisfaction data storage and rounding down to obtain the maximum number of samples that can be accommodated, and determining the sample pool capacity without exceeding the maximum number of samples that can be accommodated, so that the entire satisfaction matrix resides in the video memory during the training process, avoiding data exchange between the graphics processor and the host memory due to insufficient video memory.

[0021] Step S2: For each scoring criterion, calculate the correlation measure between its satisfaction vector and the true quality signal, and convert the correlation measure into the corresponding information content estimate. In some embodiments, the correlation measure in step S2 is the Pearson correlation coefficient, and the information content estimate is obtained by taking the absolute value of the Pearson correlation coefficient and then taking the logarithmic transformation. When the variance of the satisfaction vector of a certain rating criterion approaches zero, the Pearson correlation coefficient approaches zero, and the information content estimate corresponding to the criterion automatically returns to zero.

[0022] Step S3: For each scoring criterion, calculate the average correlation between its satisfaction vector and the satisfaction vectors of the other scoring criteria, and discount the information content estimate based on the average correlation to obtain the information content after redundancy removal. In some embodiments, the average correlation degree in step S3 is: for each rating criterion, calculate the absolute value of the correlation coefficient between its satisfaction vector and the satisfaction vectors of the other rating criteria, and then take the arithmetic mean of all the absolute values ​​of the correlation coefficients; the discount coefficient of the discounting process is equal to 1 minus the average correlation degree, and the information content after redundancy removal is equal to the information content estimate multiplied by the discount coefficient.

[0023] Step S4: Based on the judgment noise rate of each scoring criterion, the redundancy-removed information is monotonically attenuated to obtain the effective information after reliability correction. In some embodiments, the monotonic attenuation processing in step S4 includes: deconstructing the redundancy-free information quantity into an equivalent information intensity scalar, multiplying the equivalent information intensity scalar by an attenuation factor that monotonically decreases with the judgment noise rate and takes the value 1 when the judgment noise rate is 0 and 0 when the judgment noise rate is 1, and then converting it back to the information quantity form to obtain the effective information quantity after reliability correction.

[0024] Step S5: Perform water-filling distribution on the effective information amount, and reduce the criterion weights corresponding to the information amount below the preset threshold to the lower limit value to obtain the weight vector of each scoring criterion; In some embodiments, the water-filling allocation in step S5 includes: first, assigning an initial weight equal to the lower limit value to each scoring criterion, the sum of which is the product of the total number of criteria and the lower limit value; then, taking a preset quantile as a threshold for the effective information vector, subtracting the threshold from each effective information amount and taking a non-negative value as the additional weight base, normalizing the additional weight base, and multiplying it by the remaining allocable weights to obtain the additional weight of each criterion; the initial weight and the additional weight are superimposed to obtain the final weight, and the sum of all weights is strictly 1; when all effective information amounts are lower than the threshold, each scoring criterion is allocated with equal weight.

[0025] Step S6: Count the number of invalid weights in the weight vector that are compressed to the lower limit value, and calculate the ratio of the number of invalid weights to the total number of scoring criteria as the sparsity rate. When the sparsity rate reaches a preset condition, in response to the triggering of the sparsity rate, switch the reward synthesis operation based on the weight vector from the dense matrix multiplication mode that performs multiplication and addition operations on all scoring criteria to the sparse matrix multiplication mode that performs multiplication and addition operations only on the criteria corresponding to the valid weights, and call the sparse computing acceleration unit of the graphics processor to execute the reward synthesis operation.

[0026] In some embodiments, the preset condition for the sparsity rate in step S6 is that the sparsity rate is greater than or equal to a preset ratio threshold; when the sparsity rate meets the preset condition, the reward synthesis operation only performs multiplication and addition operations on the criterion row vector corresponding to the effective weight in the sparse matrix multiplication mode, skipping the criteria that are compressed to the lower limit value; when the sparsity rate does not meet the preset condition, the reward synthesis operation reverts to the dense matrix multiplication mode.

[0027] In some embodiments, when the reinforcement learning training system uses multiple graphics processors for parallel data training, it generates an effectiveness mask vector based on the effective information content of each scoring criterion. In the mask vector, the bits corresponding to criteria with effective information content higher than the threshold are 1, and the bits corresponding to criteria with effective information content lower than or equal to the threshold are 0. During cross-card gradient synchronization, only the gradient components corresponding to the bits with 1 in the mask vector participate in cross-card reduction synchronization, while the gradient components corresponding to the bits with 0 in the mask vector are not synchronized, so as to compress the data communication bandwidth between multiple graphics processors.

[0028] In some embodiments, the computational flow priority is assigned to the computations related to the scoring criteria running on the graphics processor based on the judgment noise rate of each scoring criterion: the graphics processor driver interface is called to create computational flows with different priority parameters, the computational tasks corresponding to criteria with a judgment noise rate lower than a preset noise threshold are assigned to high-priority computational flows, and the computational tasks corresponding to criteria with a judgment noise rate higher than or equal to the preset noise threshold are assigned to low-priority computational flows, so that the graphics processor scheduler prioritizes scheduling kernel functions in the high-priority computational flows when performing resource contention.

[0029] See again Figure 1 This invention also provides a computation mode adaptive switching system based on weight sparsity awareness, which is applied to reinforcement learning training scenarios. The system includes a graphics processor and a host memory coupled to the graphics processor.

[0030] The graphics processing unit (GPU) includes video memory and a sparse computation acceleration unit. The video memory stores sliding window sample pool data during reinforcement learning training, and the sparse computation acceleration unit performs sparse matrix multiplication operations. The host memory stores computer program instructions, and the GPU, or a processor coupled to the GPU, executes these computer program instructions to perform the following operations: First, at the start of training, the current available video memory capacity of the graphics processor is queried. Based on the available video memory capacity, the number of scoring criteria, and the number of storage bytes required for a single satisfaction data point, the sample pool capacity of the sliding window is dynamically determined, so that the sample pool resides entirely in the video memory to avoid page swapping. The sample pool data for multiple training steps within the window is obtained according to the sample pool capacity. The sample pool data includes the satisfaction matrix of each answer for each scoring criterion, the true quality signal of each answer, and the judgment noise rate of each scoring criterion.

[0031] Subsequently, for each scoring criterion, the correlation metric between its satisfaction vector and the true quality signal is calculated, and the correlation metric is converted into a corresponding information content estimate. Then, for each scoring criterion, the average correlation between its satisfaction vector and the satisfaction vectors of the other scoring criteria is calculated, and the information content estimate is discounted based on the average correlation to obtain the information content after redundancy removal. Next, based on the judgment noise rate of each scoring criterion, the information content after redundancy removal is monotonically attenuated to obtain the effective information content after reliability correction.

[0032] Then, the effective information is allocated in a water-filling manner, and the criterion weights corresponding to information below the preset threshold are reduced to the lower limit value to obtain the weight vector of each scoring criterion.

[0033] Finally, the number of invalid weights in the weight vector that are compressed to the lower limit is counted, and the ratio of the number of invalid weights to the total number of scoring criteria is calculated as the sparsity rate. When the sparsity rate reaches a preset condition, in response to the triggering of the sparsity rate, the reward synthesis operation based on the weight vector is switched from a dense matrix multiplication mode that performs multiplication and addition operations on all scoring criteria to a sparse matrix multiplication mode that performs multiplication and addition operations only on the criteria corresponding to the valid weights, and the sparse computing acceleration unit is called to execute the reward synthesis operation.

[0034] In this embodiment, the above operations are executed by the graphics processor, which makes full use of the parallel computing capabilities and sparse computing acceleration unit of the graphics processor, and realizes the collaborative optimization of weight sparsity awareness and adaptive switching of computing mode.

[0035] The main technical advantages of the method and system proposed in this invention are: using the weight sparse structure generated by the dynamic weight allocation in reinforcement learning training as a perceptual signal to drive the hardware resource scheduling of the graphics processor, thereby realizing the mutual adaptation and cooperation between algorithm features and hardware execution at the technical implementation level. Specifically, the algorithm outputs a weight distribution with a naturally sparse structure through online information content measurement and water-filled allocation—the weights of low-information criteria are compressed to the lower limit, while the weights of effective criteria are highlighted—this sparse structure directly triggers three collaborative responses at the hardware layer: (i) the reward synthesis operation adaptively switches between dense matrix multiplication (GEMM) and sparse matrix multiplication (SpMM) according to the sparsity rate, so that the computing power of the GPU tensor core is focused on the criteria corresponding to the effective weights, avoiding the waste of hardware resources by a large number of invalid multiplication and addition operations, and significantly improving the throughput of effective floating-point operations; (ii) the window capacity is dynamically configured according to the available video memory, so that the data resides completely in the GPU video memory, avoiding the PCIe data transfer delay caused by page swapping, and stabilizing the training pipeline; (iii) in the multi-card training scenario, only the gradient components of the effective criteria participate in cross-card synchronization, while the invalid criteria are not synchronized, compressing the interconnect bandwidth occupation. Therefore, the weight distribution calculated by the algorithm not only serves the training accuracy, but also directly determines the computing mode, memory management and communication scheduling at the hardware level. This enables the GPU's memory hierarchy, tensor cores and interconnect bandwidth to be optimized under the drive of the algorithm's characteristics, forming a closed-loop collaboration between the algorithm and the hardware in "computation-perception-scheduling".

[0036] The features, working principle and advantages of specific embodiments of the present invention are further described below.

[0037] Overall Architecture and System Composition

[0038] Figure 1 is a schematic diagram of the overall architecture of InfoRubric in the "criterion as reward" reinforcement learning closed loop. This invention provides an online dynamic information allocation method and system for the weights of scoring criteria in reinforcement learning training, named InfoRubric (Information Criterion). After calculating the weight distribution of each scoring criterion at the algorithm level, this method and system use this weight distribution as a perceptual signal to drive the hardware resource scheduling of the graphics processing unit (GPU), thereby achieving mutual adaptation and cooperation between algorithm features and hardware execution at the technical implementation level. The system is inserted as an online weight calculation module in the "criterion as reward" reinforcement learning closed loop, and its overall architecture is shown in Figure 1, consisting of the following components (all of which can run on a general-purpose computer / server equipped with a GPU): Policy model: A trained large language model that samples a group of responses for each input cue x. {y 1 , ..., y G } .

[0039] Judge model: Outputs soft satisfaction for each (response, criterion). s ij ∈ [0,1] (i is the criterion number, j is the response number).

[0040] True quality estimation module: Estimates the true quality signal for each answer j. q j (Verifiable tasks are evaluated based on correctness; unverifiable tasks are evaluated based on scores from one or more independent evaluators.)

[0041] Sliding-window buffer: A sliding-window buffer is maintained in the GPU's memory to cache the (S, q, η) triples within the most recent W training steps, forming a sample pool. The pool size N can reach an exemplary empirical threshold (based on synthetic data pre-experiments, the correlation ranking estimation tends to stabilize when N is around 160; this threshold varies with the number of criteria m and the task, and in practice, the confidence interval width of bootstrap (a method of estimating the stability of statistics by repeatedly sampling with replacement) can be used as a general criterion for determining the threshold). The window size is dynamically configured based on the available GPU memory at the start of training to ensure that the sample pool remains completely in the GPU memory.

[0042] The online weight calculation module calculates the weights w for each criterion based on the satisfaction matrix S, the true quality vector q, and the judgment noise rate vector η within the window, using the InfoRubric online information content measurement and water-filling weight allocation method described later. The output weight vector, after water-filling allocation, exhibits a sparse structure dynamically determined by the algorithm. This sparse structure is quantified by the sparsity rate in step S6 and serves as the driving signal for triggering hardware calculation mode switching. See details... Figure 2 .

[0043] Reward synthesis module: Synthesizes rewards based on the current weight w. R(y j ) = Σ i w i · s ij The reward synthesis operation adaptively switches between dense matrix multiplication mode and sparse matrix multiplication mode according to the sparsity of the weight vector, so as to call the corresponding computing unit of the graphics processor to perform the operation and feed back reinforcement learning algorithm (such as GRPO, Group Relative Policy Optimization, a reinforcement learning algorithm that samples a group of answers for the same prompt and uses the relative advantage within the group as the optimization signal) to update the policy model.

[0044] Module Responsibility Boundary Description (Distinguishing Between the Evaluation Model and the True Quality Estimation Module): In this invention, the "evaluation model" and the "true quality estimation module" are two components with distinct responsibilities and should not be confused. The evaluation model is only responsible for answering "the degree to which a certain answer satisfies a certain criterion s". ij The output "how much" represents the fine-grained satisfaction level of each criterion, serving as the raw material for reward synthesis; in this invention, it can be handled by a small to medium-sized LLM. The true quality estimation module is responsible for providing a reference quantity q for the overall quality of the response. j , is an anchor used to "calibrate the weights of each criterion" and does not participate in the scoring of individual satisfaction. This module has three mutually exclusive value sources (see QE1 / QE2 / QE3 in Figure 1): verifiable tasks are directly judged for correctness by rules or procedures; unverifiable tasks are given an overall score by an independent evaluator, which uses different prompts from the evaluation model, or can use different models or different model instances to reduce the risk of them sharing the same bias; human annotations can also be integrated for periodic calibration when necessary. Section 3.2.2 further provides the deterministic definitions and acquisition process of q and η.

[0045] Symbols and settings

[0046] The input to this module is an input prompt x (i.e., a question for the policy model to answer) and its corresponding criterion set. {r 1 , ..., r m } Where m is the number of criteria, r i Let represent the i-th scoring criterion (i = 1, 2, …, m), i.e., a natural language evaluation standard. The policy model samples responses to this prompt, and denoted by G, is the number of responses sampled for the same prompt within a group; a total of N responses are pooled within a sliding window (N is formed by pooling G responses in each training step within the window). y j This represents the j-th answer (j = 1, …, N). The sample pool capacity N is dynamically determined based on the available video memory capacity of the graphics processor at the start of training, so that the entire sliding window sample pool can reside in the graphics processor's video memory for high-speed access.

[0047] The evaluation model outputs a soft satisfaction score for each (response, criterion), denoted as _____. s ij Let represent "the degree of satisfaction of the j-th answer with the i-th criterion", taking values ​​[0,1]; when it is emphasized that it is a function of the answer y, it is denoted as s i (y) The two are synonymous. Arrange all satisfaction levels into a matrix. S ∈ [0,1] m×N The element in its i-th row and j-th column is... s ij The satisfaction matrix S is stored in the graphics processor's video memory and is directly read by the graphics processor's computing unit when calculating statistics such as correlation coefficients, without needing to be transferred through the host memory. The true quality estimation module provides a true quality signal for each answer, denoted as... q j Let be the true quality signal of the j-th answer, with a value of [0,1] (its deterministic definition and acquisition process are described in the defined calculation method of q and evaluator configuration process), and record it as . q For all q j The true mass vector formed by this.

[0048] In weight calculation, ρ i The Pearson correlation coefficient represents the satisfaction level of the i-th criterion (i.e., the i-th row of matrix S) and the true quality vector q. Î Indicates the amount of information. Î(s i; q) That is, the amount of effective information provided by the i-th criterion for the true quality, which is estimated by the correlation coefficient (called "Gaussian-copula mutual information proxy", where "proxy" means that it is a reliable estimate of mutual information, and the calculation is shown in Equation 1). η i The value represents the decision noise rate of the i-th criterion, which is [0,1] (i.e., the hard decision majority flip rate of repeated decisions in the evaluation model, see the flip rate calculation formula of η, resampling estimation scheme and cross-window smoothing strategy). η For all η i The vector formed by the vector. τ This represents the threshold for water-filled allocation (a certain quantile of effective information, defaulting to the 25th quantile).

[0049] The output of this module is a weight vector. w = (w 1 , ..., w m ) ,in w i For the i-th criterion r i The weights are the target quantities that this invention dynamically calculates online. w min This is a preset lower limit for the weights (a tiny positive number, default 0.02) to ensure that the weights of any criterion are not reset to zero and to prevent catastrophic forgetting. The weights satisfy the constraints. w i ≥ w min > 0 and Σ i w i = 1 (where the summation sign Σ) i (This represents the summation over all criteria). The water-filled allocation compresses the weights of criteria below a threshold to a lower limit, resulting in a naturally highly sparse weight vector. This sparse structure is quantified by the sparsity rate in subsequent steps, directly driving the reward synthesis operation to adaptively switch between dense and sparse matrix multiplication modes, thus focusing the graphics processor's tensor core computing power on the criteria corresponding to the effective weights. The final reward for each synthesized answer is... R(y j ) = Σ i w i · s ij In addition, W represents the sliding window length, which is the most recent training step counted in the statistics.

[0050] (I) Deterministic definition and acquisition process of the true quality signal q

[0051] True quality signal q j It is a scalar falling within the interval [0,1], used to characterize the overall quality of the j-th answer, and its value is determined in two categories according to the task type.

[0052] For verifiable tasks (such as math problems, coding problems, and objective problems with a single correct answer), q j The determination is made directly by rules or procedures: in its simplest form, it takes the value 0 / 1 (1 for correct, 0 for incorrect). When there are multiple decomposable sub-items, the "proportion of the number of passed sub-items to the total number of sub-items" can be taken as a continuous score of [0,1] (e.g., the proportion of unit tests that pass the code). This value is objective, programmatically reproducible, and serves as the clean anchor (upper bound) of this invention.

[0053] For unverifiable tasks (such as open writing and idea generation), q j The overall score is given by one or more independent evaluators and normalized to [0,1]. This invention provides the following deterministic process for the evaluators and their aggregation: (i) Each evaluator provides a score for the answer y. j Output a raw global score (e.g., a Likert scale of 1–10), linearly normalized to [0,1]; (ii) when using K evaluators, q j Take the arithmetic mean of the K normalized scores (default), or take the median to resist outliers; (iii) The evaluator can be an LLM, a rule system or human annotation. When the evaluator is an LLM, it is required to use different prompt templates and different models or different model instances as much as possible to avoid sharing the same systematic bias with the evaluation model and generating circular dependencies.

[0054] To address the risk that "using a model as an evaluator may amplify bias," this invention establishes two mitigation mechanisms: First, the correlation between the evaluator's score and the gold standard (manually annotated) is periodically measured on a small subset as a monitoring indicator of the evaluator's reliability. When the correlation significantly decreases, the evaluator is replaced or recalibrated. Second, q j It is only used for "calibration criterion weights" and not directly included in the policy gradient as a reward (the reward is still determined by the satisfaction degree s of each criterion). ij (Weighted synthesis), so the bias of the evaluator only indirectly affects the training through the weighted channels to a limited extent, and will not be directly maximized.

[0055] Regarding the timeliness and amortization of q: q is amortized within a sliding window, meaning that the evaluator is not called for every set of answers. Instead, within each window, the evaluator is called on a randomly selected subset (e.g., 20%–30% of the answers within the window) to obtain q, and the remaining samples are not involved in the statistics related to q. To ensure timeliness, the refresh of q is linked to policy drift—when the policy's KL divergence (Kullback-Leibler divergence, relative entropy, used to measure the difference between two probability distributions, here measuring the degree of change before and after the policy update) increases by more than a threshold (e.g., KL > 0.05), the current window's q subset is forcibly refreshed, thus ensuring that q used for weight calculation does not lag significantly behind the current policy. The aforementioned evaluator call and resampling estimation can be executed asynchronously on a different computational flow than the main training, overlapping with the forward / backward computation of the policy model to hide the delay in evaluation and statistics.

[0056] (ii) Deterministic definition, estimation scheme and cost of noise rate η

[0057] Determine the noise rate η i ∈ [0,1] The instability of the evaluation model's decision on the i-th criterion is measured. This invention explicitly adopts a hard-decision-based flip rate definition, and provides a calculable mathematical expression as follows: For a certain (answer j, criterion i) pair, let the evaluation model independently and repeatedly sample T times, each time taking the soft satisfaction degree... s ij Binarization with a threshold of 0.5 is used for hard decision-making. b ij (t) ∈{0,1} (the superscript (t) indicates the t-th sampling, t = 1, …, T; i.e., s≥0.5 is denoted as satisfying); let the proportion of the majority class in T samplings be denoted as p ij = max(number of hits of 1, number of hits of 0) / T Then the flip rate of the pair is 1 - p ij η is obtained by averaging several (response, criterion) pairs of a criterion i over the estimation subset. i This definition is the majority flip rate of hard-determined values, consistent with the attenuation motivation in M3 that "the greater the noise, the weaker the effective signal" (distinct from the "variance" definition to avoid motivation mismatch).

[0058] To control costs, this invention does not resample all (answer, criterion) pairs. Instead, it only randomly selects a small subset of (answer, criterion) pairs within each window (e.g., 8–16 answers per criterion) and performs T resampling operations (e.g., T=5) to estimate their η. iThen, pooling / extrapolation is applied to all samples of the criterion within this window; and a cross-window exponentially weighted moving average (EWMA, a smoothing method that exponentially decays the weight of historical values ​​over time) η estimate can be maintained for each criterion, avoiding re-estimation from zero for each window and reducing variance. Therefore, the additional evaluation call for the η estimate is approximately "number of criteria m × number of samples per criterion × T", which is a small proportion (on the order of a single-digit percentage) relative to the basic cost of scoring all m × N pairs in each step, thus ensuring feasibility; when further reducing T to minimize costs, the aforementioned cross-window EWMA smoothing should be used to suppress the variance of the η estimate.

[0059] The noise rate ηi is not only used for M3 attenuation mapping to correct the effective information content, but also serves as the basis for allocating the graphics processor computing flow priority to computing tasks with different criteria in subsequent steps. The computing tasks corresponding to the criteria of noise rate being lower than the threshold are allocated to the high-priority computing flow, while the high noise criteria are allocated to the low-priority computing flow, so that the graphics processor scheduler prioritizes scheduling high-value computing tasks when performing resource contention.

[0060] (III) Selection Criteria for M3 Attenuation Mapping √(1-η)

[0061] M3 adopts √(1-η) i The attenuation factor is based on the "attenuation due to measurement error" in classical measurement error theory: when a variable is observed with noise, its observable correlation with the target is suppressed by the square root of the reliability coefficient. The reliability can be approximated by 1-η (consistent decision ratio), so the attenuation factor is √(1-η). This form satisfies the reasonable boundary conditions that there is no attenuation when η=0, zero when η=1, and monotonically decreasing with η. It should be emphasized that this invention does not consider this specific functional form as the only limitation: at the claim level, the attenuation of M3 is described as "an attenuation mapping that monotonically decreases with the decision noise rate, and takes the value 1 when η=0 and 0 when η=1", √(1-η) is only a preferred example, and mappings such as (1-η), exp(-k·η), etc., that also satisfy the above boundary conditions are all included in this invention (for other examples of attenuation mappings that satisfy the same boundary conditions, see the alternative implementations below). The effective information output by this decay mapping not only serves the weight calculation, but more importantly, it provides a driving signal for the sparse sensing and adaptive switching of the subsequent hardware layer. The more accurate the information, the more reliable the basis for hardware scheduling, thus forming a closed-loop collaboration between the algorithm and the hardware in "computation-sensing-scheduling".

[0062] The core algorithm of the online weight calculation module (see...) Figure 2 )

[0063] This module sequentially performs three metrics (M1 Correlation / M2 Redundancy Removal / M3 Reliability) and a one-step water-filling allocation, all based on reliable correlation statistics rather than unreliable non-parametric information estimates. The weight vector calculated by this module, after water-filling allocation, exhibits a sparse structure dynamically determined by the algorithm. This sparse structure is quantified by the sparsity rate in subsequent steps, serving as a driving signal to trigger the adaptive switching of reward synthesis operations between dense and sparse matrix multiplication modes.

[0064] (M1) Correlation – Gaussian-copula Mutual Information Agent

[0065] For each criterion i, compute its satisfaction row vector on the window sample pool. S[i] With the true mass vector q Pearson correlation coefficient ρ i = corr(S[i], q) (For columns with zero variance, robust processing is performed by setting them to 0), and then converted to Gaussian-copula mutual information: Î(s i ; q) = -½ · log(1 - ρ i 2 ) (Equation 1) In the formula Î(s i ; q) This represents the amount of effective information provided by the i-th criterion for the true quality ("proxy" refers to it as a reliable estimate of mutual information in information theory). ρ i The aforementioned Pearson correlation coefficient, where log is the natural logarithm. This quantity is monotonic with respect to... |ρ i | (Right now ρ i The absolute value of the criterion can be reliably estimated with small samples (with a bootstrap confidence interval), uniformly characterizing the "importance / discrimination" of a criterion—both are essentially related to the criterion satisfaction and the true quality. When a criterion saturates with training (within-group satisfaction variance approaches zero), ρ i → 0、 Î → 0, the relevance information of the criterion is automatically reduced to zero, which allows the water-filled allocation to compress the weights of a large number of low-information criteria to the lower limit, so that the weight vector presents a naturally high sparse structure, providing a perception basis for subsequent hardware computing mode switching.

[0066] (M2) Redundancy Removal – Symmetric Redundancy Discount

[0067] To avoid double-counting of a set of highly correlated criteria, the relevance information of each criterion is symmetrically discounted based on its average relevance to all other criteria: redundancy i = mean j≠i |corr(S[i], S[j])| (Values ​​range from [0,1], with larger values ​​indicating redundancy) discount i = 1 - redundancy i (Values ​​[0,1]) cmi i = Î(s i ; q) · discount i (Equation 2) In the formula redundancy i The redundancy of the i-th criterion is equal to the absolute value of its correlation coefficient with the satisfaction of each of the other criterions j. |corr(S[i], S[j])| The mean (mean means taking the average) j≠i This indicates traversing all criteria except i), with values ​​ranging from 0 to 1; the larger the value, the more redundant the criteria. discount i The corresponding discount factor (equal to 1 minus redundancy); cmi i The effective information content after redundancy removal for the i-th criterion (i.e., the information content of M1) Î(s i ; q) Multiply by the discount factor.

[0068] Special Note (Key Improvement of This Invention Compared to a Naïve Implementation): An early naïve approach used "greedy partial correlation"—retaining all information of a cluster representing a criterion and pushing its copies one by one to the floor. This approach is asymmetric, concentrating weights on a single criterion, making that dimension more susceptible to score manipulation. This invention uses symmetric discounting: the total weight of the correlated clusters is reduced as a whole and distributed within the cluster (similar to the RRD approach but an online version), rather than collapsing onto a single criterion, thus balancing redundancy removal and resistance to score manipulation.

[0069] Degenerate implementation (non-preferred): M2 can also be implemented using a greedy, biased, closed-loop Gaussian conditional mutual information formula. Î(s i ; q | s <i ) = -½·log(1-ρ i·q|<i 2 ) To achieve this, the vertical bar "|" in the formula indicates "under the given / controlled conditions of the subsequent variables". s <i This represents the set of criteria that have been selected up to the i-th criterion. Î(s i ; q | s <i ) That is, "in the set of selected criteria already deducted" s <i "After the influence, the conditional information content between the i-th criterion and the true quality", ρ i·q|<i For the corresponding partial correlation coefficient (i.e., control) s <i After s i The correlation coefficient with q). Its calculation method is as follows: first, use least squares... S[i] and q Regression to eliminate the selected criteria set s <i The influence of the two factors is then considered, and the correlation between their residuals is calculated. However, as mentioned earlier, greedy partial correlation is asymmetric—it retains all the information of the cluster representation criterion, pushes the copies to the floor, concentrates the weights on a single criterion, and makes that dimension more susceptible to score manipulation. Therefore, this invention uses symmetric discount as the core preferred implementation, only mentioning greedy partial correlation as a degenerate control branch, and does not claim that it has the same anti-score manipulation effect as symmetric discount.

[0070] This invention employs symmetric discounting rather than greedy partial correlation to ensure that the total weight of the correlation criterion cluster is reduced as a whole and distributed within the cluster, avoiding weight collapse and concentration on a single criterion, thereby eliminating redundancy while also ensuring anti-score manipulation capabilities.

[0071] (M3) Reliability – Assessing Noise Attenuation

[0072] Determine the noise rate η i The deterministic definition, low-cost estimation scheme, and selection criteria for the attenuation factor have been given in the defined calculation and estimation method of the reversal rate for judging the noise rate, as well as the selection criteria for the attenuation map discussed (η). i Take the majority reversal rate determined by hard criteria; decay factor √(1-η) i Based on the principle of correlation attenuation caused by classical measurement errors, the calculation process for M3 is as follows: The effective information quantity after redundancy removal via M2... cmi i First, an equivalent information strength scalar is obtained by inverse solving, and then... √(1-η i )The information content is then converted back after factor attenuation. This attenuation operator satisfies the boundary conditions of "monotonically decreasing with noise rate, no attenuation when η=0, and zero when η=1". √(1-η i ) For a preferred example (see alternative implementations below for other examples of attenuation mappings that satisfy the same boundary conditions): v i = sqrt(1 - exp(-2·cmi) i )) (The equivalent information strength scalar is derived from the information content after redundancy removal, and its value is between 0 and 1.) v effi = v i · sqrt(1 - η i ) (Modularly decays according to noise rate) Î effi = -½ · log(1 - v effi 2 ) (Returning valid information) (Equation 3) In the formula v i The amount of information after redundancy removal by the i-th criterion cmi i The equivalent information strength scalar obtained from the inverse solution (with a value between 0 and 1). η i Let be the judgment noise rate for the i-th criterion. v effi The effective information strength after noise attenuation. Î effi Let be the final effective information content of the i-th criterion (sqrt represents the square root, exp represents the natural exponent, and log represents the natural logarithm).

[0073] Reliability is incorporated into the information content only through this monotonically decaying mapping, without the addition of independent penalty terms, thus avoiding repeated counting of "unreliable" cases. This effective information content, corrected for reliability, is output by M3 and then enters a water-filling allocation process—criters with higher effective information content are assigned higher weights, while criteria with effective information content below a threshold have their weights reduced to the lower limit, resulting in a sparse distribution of the weight vector. This sparse distribution provides a direct trigger for switching between dense and sparse computation modes at the hardware level, achieving mutual adaptation and cooperation between algorithm characteristics and hardware execution at the technical implementation level.

[0074] Equivalent / substitutable implementation: The attenuation of M3 can also be applied to the original correlation ρ after M1 and before M2. i (i.e., first attenuate the true correlation and then remove redundancy), or use other mappings that monotonically decrease with η and do not attenuate when η=0, and return to zero when η=1 (such as...). (1-η i ) , exp(-λη i ) (where λ is a positive attenuation rate hyperparameter; the larger λ is, the stronger the penalty for noise.) (Alternative) √ (1-η i ) All of these fall within the scope of this invention.

[0075] Water-filled weight allocation, including weight lower limit

[0076] The effective information vector after redundancy removal by M2 and attenuation by M3 Î eff = (Î eff1 , ..., Î effm ) Perform a water-filling allocation, ensuring that the weight of each criterion is not lower than a preset lower limit. w min And the sum of all weights is 1. Specifically, each scoring criterion is first assigned an amount equal to 1. w min The initial weights, the sum of which is m × w min Then from the remaining weight 1 - m× w min The allocation is based on the amount of effective information: a certain quantile of the effective information is taken as the threshold τ, and the non-negative value after subtracting the threshold from each effective information is taken as the additional weight raw. i = max(0, Î effi -τ), for raw i After normalization, multiply by the remaining weights to obtain the additional weights for each criterion. Adding these to the initial weights yields the final weight w. i = w min + (1 - m × w min )×raw i / Σ raw. When all valid information amounts are below a threshold, each scoring criterion is assigned equal weight. This process ensures that the weight of each criterion is no less than [a certain threshold]. w min And the sum of the weights is strictly 1.

[0077] Weight lower bound w min Ensure that any criterion (even if it is saturated and Î_eff→0) retains a small positive weight to prevent its capacity from being completely cleared and causing catastrophic forgetting.

[0078] Compatibility Constraints and Coordination: To prevent the lower limit from overshadowing the concentrated effect of water injection, the product of the total number of criteria and the lower limit value must be less than 1 when setting the weight lower limit. When the number of criteria is large, the weight lower limit should be reduced accordingly (for example, taking a weight lower limit no greater than twice the reciprocal of the number of criteria), so that the two objectives of "ensuring a minimum lower limit" and "concentrating weight on high-information criteria through water injection" are quantitatively compatible rather than mutually canceling. If satisfied... m·w min < 1 (As default) w min =0.02 , m≤40 (When it holds true); when the criterion number m is very large, it should be reduced accordingly. w min (If you take) w min ≤ 1 / (2m) ).

[0079] Figure 2 shows the flowchart of the three-dimensional weighting and water-filling allocation process of the online weight calculation module (InfoRubric core).

[0080] After the aforementioned water-filling distribution, the criterion weights below the threshold are compressed to the lower limit, resulting in a highly sparse structure in the weight vector dynamically determined by the algorithm. This sparse structure is quantified by the sparsity rate in subsequent steps, serving as the direct basis for triggering the graphics processor to switch the reward synthesis operation from dense matrix multiplication mode to sparse matrix multiplication mode. The sparser the weight vector, the fewer criteria participate in the effective computation, the higher the proportion of invalid multiplication and addition operations skipped by the graphics processor's tensor cores, and the more significant the improvement in hardware computing power utilization efficiency—thus forming a complete driving link from "algorithm-calculated sparse distribution of weights" to "adaptive switching of hardware execution computation mode."

[0081] The linear and training closed-loop workflow (see Figure 3)

[0082] The "linearity" of this module is reflected in the following: the aforementioned statistics such as ρi are estimated on a sliding window (a sample pool formed by the group responses of the most recent W steps, with a pool size reaching the aforementioned threshold); as the policy model drifts during training, the correlation structure between criterion satisfaction and true quality changes accordingly, and the weights w are recalculated in each window. Simultaneously, the sample pool size of the sliding window is dynamically configured based on the available GPU memory at the start of training, ensuring that the data within the window remains completely resident in GPU memory; each time the weights within a window are recalculated, the output sparse weight structure drives the reward synthesis operation of the current and subsequent windows to adaptively switch between dense and sparse computation modes.

[0083] The implementation method and parameter setting basis of sliding window pooling: The so-called "sliding window pooling" refers to not estimating the correlation coefficient on a single group (only G answers, G is usually 8-16), but merging the most recent W training steps, with G answers per step, into a sample pool of approximately N = W·G, and uniformly calculating ρ on this pool. i The system employs equal statistical measures; the window moves forward with each training step (discarding the oldest step and incorporating the latest step with each forward step), thus ensuring estimation accuracy while utilizing sufficient samples to reflect only the behavior of recent strategies. The parameter settings are based on preliminary experiments using synthetic data from the applicant (see the preliminary experimental data in the specific implementation section below): directly estimating the absolute information content of a single criterion with a small single-group sample (G=8–16) is unreliable (the estimated standard deviation is on the same order of magnitude as the signal); however, when the sample pooling reaches approximately 160 or more, the correlation ranking estimation of the criteria tends to stabilize (the correlation coefficient estimation reaches an accuracy of 0.142±0.027 when pooling 320 samples). Therefore, the window length W is set with the lower limit of "making N = W·G reach approximately 160," and the specific N can be determined during implementation by whether the bootstrap confidence interval width meets the standard. The window capacity N must also not exceed the maximum number of samples that the graphics processor's available video memory can hold, in order to ensure that the entire sample pool resides in video memory during training and avoid PCIe data transfer delays caused by page swapping.

[0084] The overall training closed-loop process is as follows: Sampling: The strategy model samples a set of responses for each batch of prompts; Evaluation: The evaluation model assigns a soft satisfaction rating to each (response, criterion). s ij The noise rate is determined by resampling the subset T times. η i ; Quality estimation: The true quality estimation module provides the following for each answer. q j (Amortization within the window, no need to adjust the evaluator for each group); Window entry: Write (S, q, η) into the sliding window buffer; Weighting: When the number of samples in the window reaches the threshold, the online weight calculation module (Figure 2) is called to obtain the dynamic weight w. The weight vector is sparsely distributed after water-filling allocation and is characterized by sparsity rate. Reward composition: Based on the sparsity rate, select either dense matrix multiplication mode or sparse matrix multiplication mode to perform the reward composition operation. R(y j ) = Σ i w i · s ij —When the sparsity reaches the preset condition, the sparse computation acceleration unit of the graphics processor is invoked to execute in sparse matrix multiplication mode; otherwise, it is executed in dense matrix multiplication mode. Update strategy: Feedback is used to update the policy model using reinforcement learning algorithms such as GRPO; Slide: The window moves forward, returning to step 1. The weights are continuously updated as the policy drifts. The sparse structure generated by the weight vectors in each window continuously drives the adaptive switching of the calculation mode of the subsequent reward synthesis operation.

[0085] Figure 3 This demonstrates the closed-loop training process of InfoRubric's online dynamically weighted reinforcement learning.

[0086] Through the closed-loop process described above, the weight calculation in each training step not only serves the synthesis of the reward signal, but also continuously provides the graphics processor with a sparsity rate sensing signal of the current weight distribution, so that the switching of hardware computing mode and the dynamic changes of algorithm weights are kept synchronized, realizing continuous collaboration between algorithm and hardware.

[0087] Cost and feasibility verification

[0088] The additional overhead of this invention mainly comes from three sources, all of which are designed and controlled within an acceptable range. First, the acquisition of the true quality signal q is amortized within the window. The evaluator call budget is only "scoring a random subset (e.g., 20%–30% of samples) per window", without needing to call the evaluator for every set of answers, and adaptively determining the refresh frequency according to strategy drift (e.g., KL divergence growth exceeding the threshold) (see the calculation method of q and the evaluator configuration process defined above for details). Second, the estimation of the judgment noise rate η only involves sampling a small number of (answer, criterion) pairs for each criterion and performing T times (e.g., T=5) of resampling, repooling, and cross-window EWMA smoothing. Its additional evaluation call amount is approximately "number of criteria m × number of samples per criterion × T", which is a small proportion relative to the basic overhead of scoring all m×N (answer, criterion) pairs once per step (typically on the order of single-digit percentages, see the calculation method and estimation method of η flip rate defined above for details). Third, the three metrics M1 / M2 / M3 and the water-filling allocation are all closed-loop calculations involving correlation, partial correlation, and element-wise operations, with complexity far lower than that of a single-pass forward pass model, and can be ignored. In summary, the additional computational overhead of this invention compared to the original reinforcement learning algorithm is very small; and the GPU computing power saved by using the sparse weight structure output by the algorithm as a hardware scheduling-driven perception signal to trigger the switching between dense and sparse computation modes far outweighs the aforementioned statistical overhead. The solution demonstrates good engineering feasibility and hardware acceleration benefits in a single-machine multi-GPU environment.

[0089] Experimental Example

[0090] The following is a specific experimental example, listing the specific values ​​of each component and a set of synthetic data pre-experimental results. The parameter values ​​in this example are exemplary preferred values ​​and are not intended to limit the invention.

[0091] (a) Specific configuration (values ​​for each component)

[0092] Policy model: Qwen2.5-3B-Instruct (the large language model being trained).

[0093] Criterion generation model: Qwen2.5-7B-Instruct (used to generate a set of criteria for prompts; this step is not innovative in this invention).

[0094] Evaluation model: gpt-4o-mini (outputs soft satisfaction score s for each (response, criterion)) ij ).

[0095] The true quality estimation module: For verifiable tasks (such as mathematics), the correctness of the program judgment is taken as 0 / 1; for unverifiable tasks (such as open writing, instruction compliance), the overall score of 1–10 from an independent evaluator LLM is normalized to [0,1]. This evaluator uses a different prompt template than the judgment model.

[0096] Reinforcement learning algorithm: GRPO; group size G = 8; number of reinforcement learning steps per round 50; batch size 8; learning rate 1e-5.

[0097] Criterion size: Each hint has a set of approximately m = 5–10 criteria.

[0098] Sliding window: The window length W is set such that the pool capacity N = W·G reaches approximately 160 or more (e.g., N = 160 when W = 20 and G = 8); the window capacity is dynamically determined based on the available video memory of the graphics processor at the start of training to ensure that the sample pool remains completely in video memory.

[0099] Hyperparameter for weight calculation: Lower limit of weight w min = 0.02; The water injection threshold is taken as the 25th percentile (τ) of the effective information content. quantile = 0.25); Enable M2 redundancy removal and M3 reliability degradation.

[0100] To determine the noise rate estimation: 8–16 responses are sampled for each criterion, and η is estimated after T = 5 resampling iterations. i It is smoothed by a cross-window exponentially weighted moving average.

[0101] Operating environment: Single-machine 8-GPU server.

[0102] Evaluation tasks (examples): Verifiable anchors use a math problem (q = correctness) as an upper bound; unverifiable anchors use open-ended writing and instruction-following tasks.

[0103] With the above configuration, each training step executes the process described in section 3.2.4: policy sampling → evaluation and scoring to obtain S and η → estimation of q → windowing → when the pooling capacity reaches 160, proceed as per section 3.2.4. Figure 2 The weight w is calculated, and the weight vector exhibits a sparse structure after being distributed using a water-sprinkling method. This sparseness is then quantified and represented by the sparsity rate, leading to the synthesis of the reward R(y). j )=Σ i w i ·s ij →GRPO Update→ Window Moved Forward.

[0104] (II) Results of Preliminary Experiments with Synthetic Data (Verifying the Feasibility of the Estimator)

[0105] To verify the feasibility of the core estimator (the correlation estimate upon which M1 / M2 depends) before large-scale training, the applicant conducted preliminary experiments on synthetic data: constructing synthetic samples (n) with satisfaction s ∈ [0,1] and true quality q, and a potential correlation coefficient ρ between the two. trials = 300), examining the estimation performance under different sample sizes G. The main results are as follows.

[0106] First, under a small sample size, directly estimating the absolute information content of a single criterion is unreliable: when ρ = 0.5 (true information content approximately 0.1438) and G = 16, the Gaussian-copula estimate is 0.173 ± 0.137, and the estimated standard deviation is on the same order of magnitude as the signal itself, making it unusable. This confirms the design choice of this invention to abandon "estimating absolute information content group by group" and instead "estimating correlation on a pooling window".

[0107] Secondly, distinguishing (ranking) two criteria is feasible with a small sample size—while the weighting method of this invention only requires correctly ranking the criteria and does not require absolute information content. In the test of "distinguishing between criterion A with ρ=0.6 and criterion B with ρ=0.4", the probability of correct ranking increases with the sample size as shown in Table 1: Table 1

[0108] It is evident that the Gaussian-copula related agent outperforms binning mutual information and KSG nearest neighbor estimation (KSG stands for the initials of the three authors Kraskov, Stögbauer, and Grassberger, and is a nonparametric mutual information estimation method based on nearest neighbor distance) at various sample sizes. Therefore, this invention selects it as the estimator for M1 / M2.

[0109] Third, the pooling / amortization estimation recovers accuracy: with ρ = 0.5 (true value 0.1438), when pooling the samples to 320, the Gaussian-copula estimation achieves an accuracy of 0.142 ± 0.027, indicating that stepping sliding window pooling is feasible, and providing a basis for determining the method of N defined above and the parameter setting principle of sliding window pooling.

[0110] The above are preliminary experiments using synthetic data to verify the rationality of the estimator selection and window parameters. The degree to which dynamic weight adjustment improves the final training effect (such as convergence speed and anti-score-boosting performance) needs to be further measured in real training with the aforementioned specific configuration before a quantitative statement can be made. No unverified numerical commitments are made here. The above preliminary experiments verify the feasibility of the information content estimation algorithm. The weight sparse structure generated by the effective information content output by the estimation algorithm after water-filled allocation is the signal source driving the adaptive switching of hardware computing modes. The more accurate the algorithm estimation, the more reliable the basis for hardware scheduling. In this embodiment, the two work together and cooperate with each other.

[0111] In a specific implementation of the present invention, the above-mentioned online weight calculation and reward synthesis steps are combined with the hardware resource scheduling of the graphics processor, which can make full use of the hardware acceleration capabilities.

[0112] Specifically, in the sliding window configuration stage, the GPU driver interface's memory query function can be called at the start of training to obtain the number of currently available GPU memory bytes. The maximum number of samples that can be accommodated can be calculated by combining the number of scoring criteria and the number of bytes stored for a single satisfaction data. The window capacity is determined with the constraint of not exceeding the maximum number of samples that can be accommodated, so that the entire satisfaction matrix can remain completely in GPU memory during the training process. This avoids the data transfer overhead caused by page swapping triggered by the fixed window capacity exceeding the GPU memory threshold.

[0113] In the reward synthesis stage, after the water-filled allocation is completed, the number of effective weights is counted and the sparsity rate is calculated. When the sparsity rate reaches the preset condition, the reward synthesis operation is switched from dense matrix multiplication mode to sparse matrix multiplication mode by calling the relevant interface of the graphics processor sparse computing library. This allows the computing power of the tensor core to focus on the criteria corresponding to the effective weights, skipping the multiplication and addition operations on the invalid criteria that have been pushed to the lower limit. The triggering condition for this switch is directly determined by the weight distribution output by the algorithm.

[0114] In multi-GPU data-parallel training scenarios, a validity mask is generated based on the effective information content of each criterion. Only the gradient components corresponding to valid criteria participate in cross-GPU reduction synchronization, while the gradient components corresponding to invalid criteria are not synchronized, thereby reducing the bandwidth occupied by the interconnection between multiple GPUs. At the same time, computational flow priorities are allocated to computational tasks based on the judgment noise rate of each criterion: computational tasks corresponding to low-noise criteria are assigned to high-priority computational flows, and those corresponding to high-noise criteria are assigned to low-priority computational flows, so that the GPU scheduler prioritizes the execution of high-value computational tasks.

[0115] Furthermore, regarding memory optimization, the satisfaction matrix can be stored in half-precision floating-point or fixed-point format, and only converted to single-precision floating-point format as needed when calculating statistics. The sliding window can utilize a circular buffer to reuse storage space in-place within the memory, avoiding fragmentation and overhead caused by repeated allocation and release. Evaluation requests from the evaluation model can be batched and operator fused to reduce the number of kernel function startups. Evaluator calls and resampling estimations can be allocated to a different computational flow than the main training process and executed asynchronously, overlapping with the forward / backward computation of the policy model. Simultaneously, a central processing unit-side thread prefetches the data required for the next window, forming an asynchronous pipeline to hide the latency of evaluation and statistics. When a single GPU's memory is insufficient to accommodate the target window, the sample pool can be sharded across multiple GPUs according to the response dimension. Intermediate statistics are accumulated locally on each GPU and then merged into global statistics after a lightweight reduction, thereby expanding the effective window capacity across multiple GPUs and controlling communication overhead.

[0116] Through the aforementioned hardware-aware implementation method, the sparse structure of the weights output by the algorithm, after being characterized by sparsity rate quantization, directly drives the adaptive switching between dense matrix multiplication and sparse matrix multiplication in the reward synthesis operation. The window capacity is dynamically configured according to the available video memory, the computational load of cross-card gradient synchronization is dynamically reduced with the number of effective criteria, and the computational flow priority is allocated differentially according to the criterion noise rate. Thus, the weight distribution calculated by the algorithm not only serves to improve the quality of the reward signal, but also directly determines the selection of the computing mode, video memory management strategy, cross-card communication scheduling, and computing resource allocation at the hardware level. This enables the video memory hierarchy, tensor cores, and interconnect bandwidth of the graphics processor to be optimized under the drive of the algorithm characteristics, forming an adaptive collaborative link from algorithm computation to hardware execution.

[0117] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A computational mode adaptive switching method based on weight sparsity awareness, applied to a reinforcement learning training system running on a computing device equipped with a graphics processor, wherein the reinforcement learning training system uses a set of scoring criteria to evaluate the generated responses of the policy model, characterized in that, Includes the following steps: S1. When training starts, query the current available video memory capacity of the graphics processor, and dynamically determine the sample pool capacity of the sliding window based on the available video memory capacity, the number of scoring criteria, and the number of storage bytes required for a single satisfaction data, so that the sample pool resides completely in the video memory of the graphics processor to avoid page swapping. The sample pool data for multiple training steps within the sliding window is obtained according to the sample pool capacity. The sample pool data includes the satisfaction matrix of each answer to each scoring criterion, the true quality signal of each answer, and the judgment noise rate of each scoring criterion. S2. For each scoring criterion, calculate the correlation measure between its satisfaction vector and the true quality signal, and convert the correlation measure into the corresponding information content estimate. S3. For each scoring criterion, calculate the average correlation between its satisfaction vector and the satisfaction vectors of the other scoring criteria, and discount the information content estimate based on the average correlation to obtain the information content after redundancy removal. S4. Based on the noise rate of each scoring criterion, the redundancy-free information is monotonically attenuated to obtain the effective information after reliability correction. S5. The effective information is allocated in a water-filling manner, and the criterion weights corresponding to the information below the preset threshold are reduced to the lower limit value to obtain the weight vector of each scoring criterion. S6. Count the number of invalid weights in the weight vector that are compressed to the lower limit value, and calculate the ratio of the number of invalid weights to the total number of scoring criteria as the sparsity rate. When the sparsity rate reaches a preset condition, in response to the triggering of the sparsity rate, switch the reward synthesis operation based on the weight vector from the dense matrix multiplication mode that performs multiplication and addition operations on all scoring criteria to the sparse matrix multiplication mode that performs multiplication and addition operations only on the criteria corresponding to the valid weights, and call the sparse computing acceleration unit of the graphics processor to execute the reward synthesis operation.

2. The method according to claim 1, characterized in that, The step S1 of dynamically determining the sample pool capacity of the sliding window includes: dividing the available video memory capacity by the product of the number of scoring criteria and the number of bytes of a single satisfaction data storage and rounding down to obtain the maximum number of samples that can be accommodated; determining the sample pool capacity without exceeding the maximum number of samples that can be accommodated; and ensuring that the entire satisfaction matrix remains resident in the video memory during the training process to avoid triggering data exchange between the graphics processor and the host memory due to insufficient video memory.

3. The method according to claim 1, characterized in that, The correlation measure mentioned in step S2 uses the Pearson correlation coefficient. The information content estimate is obtained by taking the absolute value of the Pearson correlation coefficient and then taking the logarithmic transformation. When the variance of the satisfaction vector of a certain scoring criterion approaches zero, the Pearson correlation coefficient approaches zero, and the information content estimate corresponding to the criterion automatically returns to zero.

4. The method according to claim 1, characterized in that, The average correlation in step S3 is as follows: for each rating criterion, calculate the absolute value of the correlation coefficient between its satisfaction vector and the satisfaction vectors of the other rating criteria, and then take the arithmetic mean of all the absolute values ​​of the correlation coefficients; the discount coefficient of the discounting process is equal to 1 minus the average correlation, and the information content after redundancy removal is equal to the information content estimate multiplied by the discount coefficient.

5. The method according to claim 1, characterized in that, The monotonic attenuation processing in step S4 includes: deconstructing the redundancy-free information quantity into an equivalent information intensity scalar, multiplying the equivalent information intensity scalar by an attenuation factor that monotonically decreases with the judgment noise rate and takes the value 1 when the judgment noise rate is 0 and 0 when the judgment noise rate is 1, and then converting it back to the information quantity form to obtain the effective information quantity after reliability correction.

6. The method according to claim 1, characterized in that, The water-filling allocation in step S5 includes: first, assigning an initial weight equal to the lower limit value to each scoring criterion, the sum of which is the product of the total number of criteria and the lower limit value; then, taking a preset quantile as a threshold for the effective information vector, subtracting the threshold from each effective information amount and taking a non-negative value as the additional weight base, normalizing the additional weight base, and multiplying it by the remaining allocable weights to obtain the additional weight of each criterion; the initial weight and the additional weight are superimposed to obtain the final weight, and the sum of all weights is strictly 1; when all effective information amounts are lower than the threshold, each scoring criterion is allocated equally.

7. The method according to claim 1, characterized in that, The preset condition for the sparsity rate in step S6 is that the sparsity rate is greater than or equal to a preset ratio threshold. When the sparsity rate meets the preset condition, the reward synthesis operation only performs multiplication and addition operations on the criterion row vector corresponding to the effective weight in the sparse matrix multiplication mode, skipping the criteria that are compressed to the lower limit value. When the sparsity rate does not meet the preset condition, the reward synthesis operation reverts to the dense matrix multiplication mode.

8. The method according to claim 1, characterized in that, When the reinforcement learning training system uses multiple graphics processors for parallel data training, it generates an effectiveness mask vector based on the effective information content of each scoring criterion. In the mask vector, the bits corresponding to criteria with effective information content higher than the threshold are 1, and the bits corresponding to criteria with effective information content lower than or equal to the threshold are 0. During cross-card gradient synchronization, only the gradient components corresponding to the bits with 1 in the mask vector participate in cross-card reduction synchronization, while the gradient components corresponding to the bits with 0 in the mask vector are not synchronized, so as to compress the data communication bandwidth between multiple graphics processors.

9. The method according to claim 1, characterized in that, Based on the noise rate of each scoring criterion, the computational flow priority is allocated to the scoring criterion-related calculations running on the graphics processor: the graphics processor driver interface is called to create computational flows with different priority parameters. The computational tasks corresponding to criteria with a noise rate lower than a preset noise threshold are allocated to high-priority computational flows, and the computational tasks corresponding to criteria with a noise rate higher than or equal to the preset noise threshold are allocated to low-priority computational flows. This allows the graphics processor scheduler to prioritize scheduling kernel functions in the high-priority computational flows when performing resource contention.

10. A computational mode adaptive switching system based on weight sparsity awareness, applied to reinforcement learning training, characterized in that, include: A graphics processor includes video memory and a sparse computation acceleration unit, wherein the video memory is used to store sliding window sample pool data during reinforcement learning training, and the sparse computation acceleration unit is used to perform sparse matrix multiplication operations. The host memory, coupled to the graphics processor, stores computer program instructions; When the graphics processor or a processor coupled to the graphics processor executes the computer program instructions, it performs the following operations: (1) When training starts, query the current available video memory capacity of the graphics processor, and dynamically determine the sample pool capacity of the sliding window based on the available video memory capacity, the number of scoring criteria, and the number of storage bytes required for a single satisfaction data, so that the sample pool is completely resident in the video memory to avoid page swapping; obtain the sample pool data for multiple training steps within the window according to the sample pool capacity, the sample pool data including the satisfaction matrix of each answer to each scoring criterion, the true quality signal of each answer, and the judgment noise rate of each scoring criterion; (2) For each scoring criterion, calculate the correlation measure between its satisfaction vector and the real quality signal, and convert the correlation measure into the corresponding information content estimate; (3) For each scoring criterion, calculate the average correlation between its satisfaction vector and the satisfaction vectors of the other scoring criteria, and discount the information content estimate based on the average correlation to obtain the information content after redundancy removal. (4) Based on the judgment noise rate of each scoring criterion, the redundancy-free information is monotonically attenuated to obtain the effective information after reliability correction. (5) The effective information is allocated in a water-filling manner, and the criterion weights corresponding to the information below the preset threshold are reduced to the lower limit value to obtain the weight vector of each scoring criterion; (6) Count the number of invalid weights in the weight vector that are compressed to the lower limit value, calculate the ratio of the number of invalid weights to the total number of scoring criteria as the sparsity rate, and when the sparsity rate reaches the preset condition, in response to the triggering of the sparsity rate, switch the reward synthesis operation based on the weight vector from the dense matrix multiplication mode that performs multiplication and addition operations on all scoring criteria to the sparse matrix multiplication mode that performs multiplication and addition operations only on the criteria corresponding to the valid weights, and call the sparse computing acceleration unit to perform the reward synthesis operation.

Images