Reinforcement learning data selection method and device based on off-line strategy influence estimation

Abstract

The invention provides a reinforcement learning data selection method and device based on off-line strategy influence estimation. According to the method, firstly, offline approximation of a target strategy gradient is realized through importance sampling and KL divergence constraint; carrying out gradient dimensionality reduction by applying sparse random projection to optimize storage calculation; calculating gradient inner product similarity based on the current strategy check point and the verification set to generate an influence score; and finally, iteratively selecting a high-influence data subset according to the score, and optimizing strategy parameters under a course learning framework. According to the method, the data utilization efficiency is remarkably improved, the calculation cost is reduced, and the high-dimensional gradient processing performance is effectively optimized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a reinforcement learning data selection method and apparatus based on offline policy influence estimation. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in complex tasks such as mathematical reasoning, Reinforcement Learning with Verifiable Rewards (RLVR) has become a core technology for improving model reasoning capabilities. Among related technologies, a data-driven model optimization system has been constructed through the collaborative operation of influence functions, offline policy gradient estimation, and sparse random projection. Specifically, this technology covers the entire process from initial policy evaluation to multi-stage data selection, including key steps such as gradient calculation, dimensionality compression, and cross-task data fusion. Based on this, existing RLVR frameworks typically employ supervised learning pre-training strategies combined with online policy trajectory generation (rollout) for model iteration, where policy gradient estimation and data selection mechanisms constitute the core optimization path. However, while traditional data selection methods achieve initial screening through heuristic indicators such as difficulty and uncertainty, they suffer from fundamental flaws such as insufficient theoretical support and poor cross-task generalization ability. Furthermore, the computational cost of online rollout increases exponentially with model size, making it difficult to meet industrial-grade training efficiency requirements.

[0003] Existing RLVR data selection methods directly employ heuristic metrics (such as difficulty thresholds or confidence levels) for data screening, without establishing a theoretical quantitative model of data contribution. This can lead to a deviation between data selection and policy optimization objectives. Specifically, rollout-based online gradient estimation requires generating a large number of policy trajectories. For LLMs with 1.5B-7B parameters, a single-stage training requires 5.2 hours of computational resources, significantly limiting training efficiency. Furthermore, the complexity of storing and computing high-dimensional gradients further exacerbates the resource bottleneck. While existing random projection methods can reduce dimensionality, they do not consider the interference of numerical noise on inner product similarity. These technical shortcomings make it difficult for existing methods to maintain the effectiveness of data selection during dynamic policy iteration, causing performance stagnation in the later stages of training and limited cross-task generalization ability, thus affecting the practical deployment of large-scale language models in complex inference scenarios. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a reinforcement learning data selection method based on offline policy influence estimation.

[0006] The second objective of this invention is to propose a reinforcement learning data selection device based on offline policy influence estimation.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a reinforcement learning data selection method based on offline policy influence estimation, comprising: S1, acquiring pre-collected offline trajectory data, and calculating the gradient estimate of the target policy based on the trajectory generated by the behavior policy, and realizing offline approximation of the policy gradient through importance sampling and KL divergence constraints; S2, applying sparse random projection technology to the gradient estimate, randomly selecting a subset of the gradient dimension for projection dimensionality reduction, and reducing storage and computational overhead and suppressing numerical noise by setting a sparsity ratio; S3, calculating the gradient inner product similarity between training data points and validation data points based on the current checkpoint of the target policy and the validation set of the target policy, and generating a data influence score; S4, iteratively selecting the most influential training data subset under the current policy according to the data influence score, and gradually optimizing the policy model parameters using a course learning framework.

[0008] In one embodiment of the present invention, S1 further includes: S11, the weight of importance sampling is calculated using the logarithmic probability ratio of the trajectory, specifically: based on the difference in trajectory generation probability between the current policy πθ and the behavior policy β; S12, the KL divergence constraint is implemented by introducing a KL loss term, specifically: during policy update, the KL divergence between the current policy πθ and the behavior policy β is calculated, and a threshold is set to limit the corresponding maximum value to ensure the stability of policy estimation.

[0009] In one embodiment of the present invention, S2 further includes: S21, the method for constructing the sparse random projection matrix includes: randomly selecting a subset S of the gradient dimensions, wherein the size of the subset S is determined by a preset sparsity ratio α; S22, a random dropout operation is performed before projection, specifically: each dimension of the gradient vector g is independently set to zero with probability 1-α, and the retained dimensions are used to generate non-zero columns of the projection matrix through a standard normal distribution N(0,1).

[0010] In one embodiment of the present invention, S3 further includes: S31, the gradient features are normalized before the gradient inner product similarity calculation, specifically: the gradient vector g of each training data point is divided by its L2 norm to obtain a unit vector before calculating the inner product; S32, the calculation of the data influence score is combined with semantic similarity, specifically: the semantic similarity between the training data points and the validation data points is calculated through a pre-trained semantic embedding model, and then weighted and fused with the gradient inner product similarity, with the weight coefficient dynamically adjusted according to the task type.

[0011] In one embodiment of the present invention, S4 further includes: S41, the selection of training data subsets in the iterative phase of the course learning framework adopts the Reciprocal Rank Fusion (RRF) method, specifically: calculating the POPI-R ranking rj(so) of the training data points for each validation set Dval,j, and merging the rankings of all validation sets; S42, in the optimization step of the policy model parameters, the learning rate is dynamically adjusted according to the online pass rate of the current stage, specifically: if the online pass rate increases beyond a preset threshold, the learning rate is reduced to stabilize training, and vice versa to accelerate convergence.

[0012] In one embodiment of the present invention, the method further includes: S5, adjusting the composition of the validation set according to the dynamic requirements of the model training stage, specifically: prioritizing the use of the target validation set in the initial stage, and gradually introducing non-target validation sets as the training progresses, in order to improve the model's cross-task generalization ability; S6, dynamically adjusting the sparsity ratio of the projected gradient features, specifically: at the beginning of each training stage, automatically calculating the sparsity ratio α based on the gradient noise level of the current strategy; if the noise level is higher than the threshold, increasing α to enhance the noise reduction effect, and vice versa, decreasing α to retain more gradient information.

[0013] To achieve the above objectives, a second aspect of the present invention proposes a reinforcement learning data selection device based on offline policy influence estimation, comprising: a policy gradient estimation module, used to acquire pre-collected offline trajectory data and calculate the gradient estimate of the target policy based on the trajectory generated by the behavioral policy, and to achieve offline approximation of the policy gradient through importance sampling and KL divergence constraints; a sparse projection dimensionality reduction module, used to apply sparse random projection technology to the gradient estimate, randomly select a subset of the gradient dimension for projection dimensionality reduction, and reduce storage and computational overhead and suppress numerical noise by setting a sparsity ratio; a gradient similarity analysis module, used to calculate the gradient inner product similarity between training data points and validation data points based on the current checkpoint and validation set of the target policy, and generate a data influence score; and a curriculum learning optimization module, used to iteratively select the most influential subset of training data under the current policy according to the data influence score, and gradually optimize the policy model parameters using a curriculum learning framework.

[0014] In one embodiment of the present invention, the policy gradient estimation module is further configured to: use the log probability ratio of the trajectory when calculating the importance sampling weight, specifically based on the difference in trajectory generation probability between the current policy πθ and the behavior policy β; calculate the KL divergence between the current policy πθ and the behavior policy β when updating the policy, and set a threshold to limit the corresponding maximum value.

[0015] In one embodiment of the present invention, the sparse projection dimensionality reduction module is further configured to: randomly select a subset S of the gradient dimension when constructing the sparse random projection matrix, wherein the size of the subset S is determined by a preset sparsity ratio α; perform a random dropout operation on the gradient vector g before projection, specifically by independently setting each dimension to zero with probability 1-α, and the retained dimensions are used to generate non-zero columns of the projection matrix through a standard normal distribution N(0,1).

[0016] In one embodiment of the present invention, the gradient similarity analysis module is further configured to: normalize the gradient features before calculating the gradient inner product similarity, specifically by dividing the gradient vector g of each training data point by its L2 norm to obtain a unit vector before calculating the inner product; and combine semantic similarity when calculating the data influence score, specifically by calculating the semantic similarity between the training data points and the validation data points through a pre-trained semantic embedding model, and then weighting and fusing it with the gradient inner product similarity, with the weight coefficient dynamically adjusted according to the task type.

[0017] In one embodiment of the present invention, the course learning optimization module is further configured to: employ the Reciprocal Rank Fusion (RRF) method in the selection of training data subsets during the iteration phase, specifically calculating the POPI-R ranking rj(so) of training data points for each validation set Dval,j, and fusing the rankings of all validation sets; and dynamically adjust the learning rate according to the online pass rate at the current stage in the policy model parameter optimization step, specifically reducing the learning rate to stabilize training if the online pass rate increases beyond a preset threshold, and increasing the learning rate to accelerate convergence if the online pass rate decreases.

[0018] In one embodiment of the present invention, the invention further includes: a validation set adjustment module, used to adjust the composition of the validation set according to the dynamic needs of the model training phase, specifically, prioritizing the use of the target validation set in the initial stage, and gradually introducing non-target validation sets as the training progresses, in order to improve the model's cross-task generalization ability; and a dynamic sparsity ratio adjustment module, used to dynamically adjust the sparsity ratio of the projected gradient features, specifically, automatically calculating the sparsity ratio α based on the gradient noise level of the current strategy at the beginning of each training phase, increasing α if the noise level is higher than a threshold to enhance the noise reduction effect, and decreasing α otherwise to retain more gradient information.

[0019] The methods and apparatus of this invention significantly improve the data utilization efficiency of large-scale language models in reinforcement learning training, reduce the computational cost of policy trajectory generation, and effectively reduce the storage and computational overhead of high-dimensional gradients through sparse random projection, thereby accelerating the training convergence speed and improving the model inference performance.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0022] Figure 1 This is a flowchart of a reinforcement learning data selection method based on offline policy influence estimation according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of a reinforcement learning data selection device based on offline policy influence estimation according to an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] The following description, with reference to the accompanying drawings, illustrates a reinforcement learning data selection method and apparatus based on offline policy influence estimation, according to embodiments of the present invention.

[0027] Example 1

[0028] Figure 1 This is a flowchart of a reinforcement learning data selection method based on offline policy influence estimation according to an embodiment of the present invention, such as... Figure 1 As shown, it includes:

[0029] S1: Acquire pre-collected offline trajectory data and calculate the gradient estimate of the target policy based on the trajectory generated by the behavior policy. The offline approximation of the policy gradient is achieved through importance sampling and KL divergence constraints.

[0030] Specifically, the core technical objective of this step is to compute the policy gradient estimate of the target policy based on trajectories generated from offline trajectory data and behavioral policies, and to achieve an offline approximation of the policy gradient through importance sampling and KL divergence constraints. This method aims to reduce the computational overhead required for online policy gradient estimation while maintaining the accuracy of the gradient estimation, thereby improving training efficiency within the reinforcement learning and verifiable reward (RLVR) framework.

[0031] Specifically, pre-collected offline trajectory data is first acquired. These trajectories are generated by the behavior policy β, typically the initial policy π0. The gradient estimation of the target policy πθ is approximated by importance sampling, that is, by using the trajectory weights of the behavior policy to perform a weighted summation of the gradient of the target policy.

[0032] Furthermore, the coefficient of the KL divergence constraint is typically set to 0.001 to balance the magnitude and stability of policy updates. During importance sampling, the calculation of trajectory weights depends on the likelihood ratio of the policy, and its numerical precision is typically in float16 format to reduce storage and computational overhead. In addition, to improve the robustness of gradient estimation, gradient dimensions can optionally be randomly dropped before projection, with a sparse ratio set to 0.1 to reduce noise influence and preserve the inner product relationship between gradients.

[0033] Specifically, this method replaces online rollout with offline trajectory data, significantly reducing the computational cost of policy gradient estimation, while using KL constraints to ensure the stability of policy updates. In practical applications, this method is suitable for RLVR training of large-scale language models (such as 1.5B and 7B parameter sizes), especially in multi-stage data selection, where it can efficiently filter samples that have the greatest impact on the current policy, thereby accelerating training and improving performance. Experiments show that this step can achieve a 2.66x training speedup on a 1.5B model while using only 10% of the training data, validating its advantages in data efficiency and computational cost.

[0034] Furthermore, S1 includes:

[0035] S11, the weight of importance sampling is calculated using the logarithmic probability ratio of the trajectory, specifically: based on the difference in trajectory generation probability between the current policy πθ and the behavior policy β.

[0036] Specifically, in this step, the importance sampling weights are calculated using the logarithmic probability ratio of the trajectories. The core principle is based on the policy gradient method and importance sampling theory in reinforcement learning. Specifically, the weight s0 represents the probability difference between the current policy π_θ and the behavior policy β in generating the trajectory T under the initial state s0. This weight is used to measure the importance of the trajectory under the current policy, thereby adjusting its contribution to policy updates in offline policy influence estimation.

[0037] Furthermore, the trajectory T is pre-generated by the behavior policy β and stored in an offline trajectory library. The current policy π_θ is used to calculate its log probability with respect to trajectory T via a policy gradient estimator (such as the improved policy gradient in TRPO or PPO). Since π_θ and β are typically constrained on the KL divergence (e.g., the KL coefficient is set to 0.001 in GRPO), the weight estimation exhibits low variance and high stability. In addition, to improve computational efficiency, weight calculations are typically performed at floating-point 16 precision to reduce memory usage and computational overhead.

[0038] Specifically, by sampling weights based on the importance of offline trajectories, CROPI can efficiently evaluate the potential impact of each training sample on the current policy without real-time sampling, thereby achieving phased data filtering and policy optimization. Experiments show that this method achieves a 2.66x training speedup on a 1.5B model using only 10% of the data, significantly improving the data utilization efficiency of RLVR.

[0039] S12, KL divergence constraint is implemented by introducing KL loss term, specifically: when updating the policy, calculate the KL divergence between the current policy πθ and the behavior policy β, and set a threshold to limit the corresponding maximum value.

[0040] Specifically, during the policy update process, introducing a KL loss term to constrain the KL divergence between the current policy πθ and the behavioral policy β is a key technique in the CROPI framework to ensure the stability of policy estimation. Specifically, during each policy update, the system calculates the KL divergence between the current policy πθ and the behavioral policy β, and sets a threshold (usually 0.1) to limit its maximum value from exceeding this threshold. This operation is based on the idea of ​​Trust Region Policy Optimization (TRPO), controlling the magnitude of policy updates to prevent policy estimation instability or training divergence caused by excessively large policy updates.

[0041] Specifically, the KL divergence is typically calculated using the expected form of the log-probability ratio. In CROPI, the behavioral policy β is set to the initial policy πθ0 to ensure continuity during policy updates. During policy optimization, the KL divergence is added as a regularization term to the objective function.

[0042] At the parameter level, the KL divergence is typically calculated using the expected form of the log-probability ratio, i.e., KL(πθ||β)=E_{s~β}[KL(πθ(·|s)||β(·|s))]. In CROPI, the behavioral policy β is set to the initial policy πθ0 to ensure continuity during policy updates. During policy optimization, the KL divergence is added as a regularization term to the objective function, taking the form KL(πθ||β)×λ, where λ is the KL coefficient, typically set to 0.001, to adjust the strength of the KL constraint.

[0043] Furthermore, this step is primarily used in multi-stage RLVR training, particularly during the policy iteration optimization phase. By limiting the KL divergence, CROPI can maintain the stability of policy updates while using only 10% of the training data in each stage, thereby achieving efficient data selection and training acceleration.

[0044] Specifically, by introducing KL constraints, the problem of training instability caused by excessive changes in policy distribution during policy updates is effectively alleviated, while the robustness and efficiency of data selection are improved, providing theoretical support and practical guarantee for the training of large-scale language models in RLVR.

[0045] S2, apply sparse random projection technique to the gradient estimation, randomly select a subset of the gradient dimension for projection dimensionality reduction, reduce storage and computation overhead and suppress numerical noise by preset sparsity ratio.

[0046] Specifically, this step involves applying sparse random projection techniques to the gradient estimation to achieve dimensionality reduction, decrease storage and computational overhead, and suppress numerical noise. The technique combines gradient vector sparsification with random projection methods, and the specific operations are as follows:

[0047] Specifically, in some implementations, gradient estimation is often a high-dimensional vector. To reduce computational complexity, a subset of the gradient dimension is first randomly selected. Its size is determined by the preset sparsity ratio r s Control. For example, if the model parameter dimension is d = 10^6 and the sparsity ratio is set to 0.1, then only 10^5 dimensions are retained for subsequent processing. This process can be regarded as performing a random "dropout" operation on the gradient vector, that is, masking some dimensions before projection.

[0048] Furthermore, a sparse random projection matrix is ​​constructed, and the projection operation can be represented as g. proj =P sparse ·g[S], where g[S] is the subvector of the gradient vector indexed by S. This method significantly reduces dimensionality while preserving the inner product structure of the gradient vectors, thereby reducing storage requirements and computational resource consumption.

[0049] Specifically, the sparsity ratio r s Typically set to 0.1, the projection target dimension k can be adjusted according to the model size and hardware conditions. For example, in a 1.5B model, k can be set to 10^4. In addition, to ensure the fidelity of the inner product of the gradient after projection, the Johnson-Lindenstrauss lemma can be used as a theoretical basis to guarantee the approximate accuracy of the inner product under high probability.

[0050] Specifically, this step is primarily used in the reinforcement learning (RL) training process of large-scale language models (LLM), particularly within the Curriculum RL with Off-Policy Influence Guidance (CROPI) framework, to efficiently evaluate the influence of data points on the current policy.

[0051] Specifically, this step effectively alleviates the storage and computation bottleneck caused by high-dimensional gradients, while reducing the impact of numerical noise on gradient estimation through sparsification, thereby improving the robustness and efficiency of data selection.

[0052] Furthermore, S2 includes:

[0053] S21, the method for constructing the sparse random projection matrix includes: randomly selecting a subset S of the gradient dimension, wherein the size of the subset S is determined by a preset sparsity ratio α.

[0054] Specifically, the method for constructing sparse random projection matrices aims to reduce the dimensionality of policy gradients in large-scale language models (LLMs), thereby improving computational and storage efficiency. Specifically, this method first randomly selects a subset S from the complete set of dimensions of the policy gradient, the size of which is determined by a preset sparsity ratio α. In some implementations, α ranges from 0.05 to 0.2, indicating that only α×d dimensions of the gradient are retained (where d is the total dimension of the gradient).

[0055] The sparsity ratio α is a key parameter controlling the sparsity of the projection, and its value directly affects the proportion of gradient information retained and the computational cost. In experiments, α is usually set to 0.1, meaning it retains 10% of the dimension in the gradient. The dimension k of the projected gradient is usually much smaller than the original dimension d; for example, in a 1.5B model, d can reach millions of times, while k is usually set to 10. 4 ~10 5 Furthermore, the preprocessing steps before gradient normalization (such as cosine similarity calculation) also need to consider numerical accuracy. The float16 format is usually used to save memory, but attention should be paid to the numerical errors it may introduce.

[0056] This method is widely used in policy gradient estimation based on influence functions, particularly within the Curriculum RL with Off-Policy Influence Guidance (CROPI) framework. In each training phase of CROPI, the policy gradient needs to be estimated via offline trajectories, followed by sparse random projection to compress the gradient dimension, accelerating subsequent similarity calculations and data selection. For example, on the GSM8K and MATH validation sets, sparse projection can reduce gradient storage requirements by over 90% while maintaining the sorting accuracy of the gradient inner product.

[0057] Specifically, sparse random projection not only significantly reduces the storage and computational overhead of gradients, but also effectively suppresses numerical noise in gradients by randomly discarding some dimensions, thereby improving the robustness of gradient similarity. Experiments show that when α = 0.1, the top-10% ranking accuracy (precision@10%) of the projected gradients can reach over 85%, approaching the ranking effect of full gradients. This step provides CROPI with an efficient data selection mechanism, enabling the model to achieve a 2.66x training speedup even when trained using only 10% of the data, demonstrating its practical value in large-scale reinforcement learning training.

[0058] S22, the random dropout operation is performed before projection. Specifically, each dimension of the gradient vector g is independently set to zero with probability 1-α, and the remaining dimensions are used to generate non-zero columns of the projection matrix through the standard normal distribution N(0,1).

[0059] Specifically, in this step, the "random dropout operation performed before projection" is a sparsity processing mechanism for gradient feature compression. Its core lies in reducing the computational complexity of subsequent random projection and improving the representation quality of gradient features in low-dimensional space by randomly zeroing a portion of the gradient vector g. Specifically, this operation independently retains each dimension of the gradient vector g with probability 1-α and zeros it with probability α, where α is a preset sparsity ratio, typically ranging from 0.1 to 0.5, used to control the proportion of gradient dimensions ultimately retained. The retained dimensions are then used to generate non-zero columns of the projection matrix through a standard normal distribution N(0,1), thereby achieving sparse random projection.

[0060] Specifically, this step first performs dimensional random sampling on the gradient vector g, that is, for each dimension i∈{1,...,d}, it is determined with probability α whether to set it to zero. This process can be regarded as a gradient masking operation, the purpose of which is to reduce the interference of redundant information and numerical noise on the projection result. Subsequently, random projection is performed only on the retained dimensions. The projection matrix Psparse∈Rk×d is constructed as follows: for the set of retained dimensions S, the corresponding columns of Psparse are sampled from N(0,1), and the remaining columns are set to zero vectors. This method has a significant advantage in computational efficiency because it only needs to process about (1-α)d dimensions, rather than all d dimensions.

[0061] Furthermore, the value of α directly affects the sparsity of the gradient features and the ability to preserve similarity after projection. Experiments show that when α = 0.1, the projected gradient features perform best in terms of inner product preservation, achieving a high level of precision at 10%. In addition, the projection dimension k is usually set much smaller than the original gradient dimension d, for example, k = 512, d = 10240, to achieve dual optimization of storage and computation.

[0062] Specifically, this step is primarily used in reinforcement learning (RLVR) training of large-scale language models (LLMs) to reduce the overhead of gradient storage and computation. By introducing random dropout before projection, the CROPI framework can efficiently select the most influential training samples in multi-stage training, thereby improving training efficiency and model performance. The technical value of this step lies in its effective mitigation of the numerical instability problem of high-dimensional gradients during projection, while significantly reducing computational resource consumption through sparsification. Combined with subsequent random projection and cosine similarity calculation, this step provides an efficient and accurate gradient feature representation mechanism for data selection based on influence functions, and is one of the key technologies for achieving efficient RLVR training.

[0063] S3 calculates the gradient inner product similarity between training data points and validation data points based on the current checkpoint of the target policy and the validation set of the target policy, and generates a data influence score.

[0064] Specifically, the core objective of this step is to generate a data influence score based on the current checkpoint and validation set of the target policy, by calculating the gradient inner product similarity between training data points and validation data points. This score measures the potential contribution of each training sample to the performance improvement of the current policy on the validation set, thus providing a basis for data selection for subsequent curriculum learning.

[0065] Specifically, at the technical implementation level, this step first utilizes offline trajectories to calculate the policy gradient of the training data points. Specifically, for each training sample s0, several trajectories are generated using the behavior policy β. The policy gradient s0 is then calculated based on these trajectories. Since the policy gradient dimension is high (e.g., the gradient dimension of a 1.5B model can reach millions), a sparse random projection technique is employed to reduce storage and computational overhead. This method randomly selects a subset of the gradient dimensions. Constructing a sparse projection matrix P sparse ∈\mathbbR k×d , where k << d, only the gradient information of the dimension in S is retained, and the other dimensions are set to zero. The sparseratio is usually set to 0.1, that is, 10% of the gradient dimension is retained.

[0066] Furthermore, the gradient inner product similarity calculation adopts the cosine similarity form, s_0, where s_0 represents the validation set samples. To avoid the influence of gradient norm bias on the similarity calculation, the gradient is normalized before projection. In addition, to improve computational efficiency, the gradient calculation uses the float16 format and is optimized using parallel computing.

[0067] Specifically, this step is embedded in each phase of the CROPI framework to dynamically select the most influential training samples under the current strategy. By fusing the influence scores of multiple validation sets using the RRF (Reciprocal Rank Fusion) method, the final selected samples have higher semantic relevance and learning potential, thus significantly improving training efficiency and model performance. Experiments show that on the 1.5B model, this step achieves a training speedup of 2.66 times while using only 10% of the training data.

[0068] Specifically, the technical value of this step lies in the fact that, through gradient inner product similarity evaluation supported by theory, it enables fine-grained screening of training data, overcomes the limitations of traditional heuristic methods, and provides a scalable data selection mechanism for efficient reinforcement learning of large-scale language models.

[0069] Furthermore, S3 includes:

[0070] S31, before calculating the gradient inner product similarity, the gradient features are normalized. Specifically, the gradient vector g of each training data point is divided by its L2 norm to obtain a unit vector before calculating the inner product.

[0071] Specifically, in the CROPI method, normalizing the gradient features before calculating the gradient inner product similarity is a key step in improving data selection accuracy and computational efficiency. This step is technically implemented based on L2 norm normalization of the gradient vector, that is, the gradient vector g∈\mathbbR for each training data point. d Divide it by its L2 norm ||g||2, thus mapping it to a unit vector. The mathematical essence of this operation is to project the gradient vector onto the unit sphere, eliminating the interference of the gradient magnitude on the inner product calculation, so that the similarity between gradients is determined only by the direction, rather than the magnitude.

[0072] Specifically, at the implementation level, the normalization process is typically performed after gradient calculation and before inner product similarity evaluation. Specifically, for each training sample s... o The gradient *g* is calculated using the GRPO algorithm based on pre-collected offline trajectories. Subsequently, a numerically stable L2 normalization method (such as the `F.normalize` function in PyTorch) is used to process the gradient vector, ensuring that it satisfies ||g|² = 1. In some implementations, to avoid division by zero errors caused by zero vectors in the gradient vector, a small smoothing term ε can be introduced, i.e. Where e -8 .

[0073] Furthermore, the normalized gradient vector is used to calculate cosine similarity. In application scenarios, this normalization step is widely used in multi-stage data selection frameworks to measure the potential impact of training samples on the current strategy. By eliminating gradient magnitude differences, the model can more accurately identify the training samples most similar to the validation samples in the gradient direction, thus achieving efficient data filtering. Experiments show that on the 1.5B model, this normalization step combined with sparse random projection can achieve an accuracy of over 90% at 10% for inner product similarity calculation, significantly outperforming the unnormalized direct projection method.

[0074] Specifically, this step effectively alleviates the gradient-scale issue and improves the representation ability of gradient features in low-dimensional space. At the same time, by standardizing the gradient direction, the comparability of influence between different samples is enhanced, providing more reliable input for subsequent Rank Fusion (such as RRF), thereby improving the overall training efficiency and generalization ability of the CROPI framework.

[0075] S32, the calculation of the data influence score combines semantic similarity, specifically: the semantic similarity between training data points and validation data points is calculated through a pre-trained semantic embedding model, and then weighted and fused with gradient inner product similarity, with the weight coefficient dynamically adjusted according to the task type.

[0076] Specifically, this step proposes a data influence scoring method that combines semantic similarity and gradient inner product similarity, aiming to improve the accuracy and efficiency of data selection during reinforcement learning and verifiable reward (RLVR) training. The technique is implemented based on a pre-trained semantic embedding model and policy gradient estimation, achieving dynamic evaluation of training data points through multi-dimensional feature fusion.

[0077] Specifically, firstly, a pre-trained semantic embedding model (such as BGE-large-en-V1.5) is used to represent the training data point *so* and the validation data point *s%* as vectors, and their semantic similarity is calculated. This similarity is typically quantified using cosine similarity, calculated as: *cos_sim(so,s%) = (so·s%) / (|so||s%|). Subsequently, the gradient inner product similarity of the training data points with respect to the current policy πθ is calculated using an offline policy gradient estimation method. This gradient estimation is based on offline trajectories generated by the behavior policy β and employs importance sampling techniques to avoid the high computational overhead of online policy sampling.

[0078] Furthermore, semantic similarity and gradient inner product similarity are weighted and fused using an adjustable weight coefficient α. The final influence score is expressed as: Inf(so; s%) = α·cos_sim(so, s%) + (1-α)·grad_sim(so, s%). Here, α ranges from [0, 1] and is dynamically adjusted according to the task type (e.g., mathematical reasoning, language understanding) to balance semantic relevance and policy optimization requirements. In experiments, α is typically set to 0.5, but can be optimized to the range of 0.3–0.7 for specific tasks to adapt to the training characteristics of different models.

[0079] Specifically, this method is integrated into the CROPI (Curriculum RL with Off-Policy Influence Guidance) framework for data selection in multi-stage training. Through joint analysis of offline trajectories and semantic embeddings, CROPI achieves significant performance improvements by using only 10% of the training data in each stage, particularly demonstrating a 2.66x training speedup in mathematical reasoning tasks. By introducing semantic similarity as a supplement to gradient similarity, it effectively alleviates the over-reliance of traditional data selection methods on policy gradients, improving the robustness and generalization ability of data selection. Simultaneously, the dynamic weight adjustment mechanism enhances the method's adaptability, enabling it to maintain efficient data selection capabilities across different tasks and model sizes.

[0080] S4. Based on the data influence score, iteratively select the most influential subset of training data under the current strategy, and gradually optimize the strategy model parameters using the course learning framework.

[0081] Specifically, the step of "iteratively selecting the most influential subset of training data under the current policy based on the data influence score and gradually optimizing the policy model parameters using the curriculum learning framework" is the core optimization mechanism in the CROPI (Curriculum RL with Off-Policy Influence guidance) framework. Its technical implementation is based on the combination of influence function and curriculum learning, aiming to improve the training efficiency and inference ability of large-scale language models in RLVR (Reinforcement Learning with Verifiable Rewards) tasks.

[0082] Specifically, this step first uses offline policy influence estimation to calculate the gradient influence of each training sample on the current policy model using pre-collected trajectory data. Specifically, for each training cue so, the policy gradient is estimated through importance sampling based on the behavioral policy β (usually set as the initial policy πθ0), avoiding the high-cost rollouts required for online policies.

[0083] Specifically, CROPI uses a selection ratio α = 0.1, meaning that only 10% of the training data is selected for optimization in each stage. The number of training stages M is set to 4–6, and the number of training steps E in each stage is set to 200–300. POPI (Policy Influence) scores from multiple validation sets are fused using RRF (Reciprocal Rank Fusion) to improve the robustness and generalization ability of data selection.

[0084] Furthermore, this step is applicable to LLM training scenarios requiring reinforcement learning optimization, such as mathematical reasoning and multimodal reasoning. By dynamically selecting the most influential samples, the model can achieve more efficient policy updates with limited data, thereby accelerating convergence and improving final performance. Experiments show that on the 1.5B model, CROPI improves training speed by 2.66 times when using 10% of the data, and outperforms full-data training and traditional data selection baselines on multiple validation sets.

[0085] Specifically, through the theoretically supported influence function and course learning mechanism, data-driven strategy optimization was achieved, effectively solving key problems in RLVR such as high cost of online rollouts and large gradient dimensionality, and providing a scalable solution for efficient training of large-scale models.

[0086] Furthermore, S4 includes:

[0087] S41, the selection of training data subsets in the iterative phase of the course learning framework adopts the Reciprocal Rank Fusion (RRF) method, specifically: for each validation set Dval,j, calculate the POPI-R rank rj(so) of the training data points, and fuse the ranks of all validation sets.

[0088] Specifically, in the CROPI (Curriculum RL with Off-Policy Influence guidance) framework, the selection of training data subsets during the iterative phase employs the Reciprocal Rank Fusion (RRF) method. Its core lies in fusing data through the POPI-R (Policy Gradient-based Influence Rank) rankings of multiple validation sets to improve the robustness and generalization ability of data selection. Specifically, for each validation set D... val,j The system first calculates the training data points s o POPI-R rankings j s o This ranking reflects s o For the current policy π_θ on the validation set D val,j The potential contribution to performance improvement. Then, through the formula... The rankings of all validation sets are merged, where k is the merging parameter, usually set to 20 or 30, to smooth the ranking values ​​and reduce the impact of extreme rankings on the final result.

[0089] Specifically, the RRF fusion process first relies on the computation of POPI-R, which is based on policy gradient estimation of offline trajectories and uses sparse random projection for gradient compression to reduce computational and storage overhead. In each iteration, the system scores all training samples according to the current policy using POPI-R and calculates their rankings on multiple validation sets. These rankings are then weighted and fused using the RRF formula, ultimately selecting the top D samples with the highest RRF scores. train A subset of samples is used as the training set for the next stage, where α is the selection ratio, which is usually set to 0.1 (i.e., 10%).

[0090] Furthermore, by fusing rankings from multiple validation sets, the system can more comprehensively evaluate the combined impact of training samples on different task objectives, thereby avoiding the bias that may arise from a single validation set. In addition, the RRF method has been proven in the field of information retrieval to effectively improve the accuracy of ranking fusion (Cormack et al., 2009), and its introduction into RLVR further enhances the dynamic adaptability and task relevance of data selection. Experiments show (see Figure 3) that on the 1.5B model, this method achieves a 2.66x training speedup using only 10% of the training data, significantly outperforming full-data training and traditional heuristic methods.

[0091] In step S42, the learning rate is dynamically adjusted based on the online pass rate at the current stage. Specifically, if the online pass rate increases beyond a preset threshold, the learning rate is reduced to stabilize training; otherwise, the learning rate is increased to accelerate convergence.

[0092] Specifically, in the optimization step of the policy model parameters, dynamically adjusting the learning rate based on the online pass rate of the current stage is an adaptive optimization mechanism based on model training state feedback. Its technical implementation principle is based on the control effect of the learning rate on the model update magnitude in the gradient descent optimization algorithm. The specific operation is as follows: After each training stage, the system first evaluates the online pass rate of the current policy model on the validation set, that is, the proportion of correct responses of the model to validation samples under the current policy. If the pass rate increases by more than a preset threshold (e.g., 0.05 or 5%) compared to the previous stage, it indicates that the current policy has entered a relatively stable optimization range. At this time, the learning rate should be reduced to decrease the magnitude of policy updates, thereby avoiding policy oscillations or overfitting caused by an excessively high learning rate. Conversely, if the pass rate does not reach the threshold or decreases, it indicates that the model has not yet fully adapted to the current data distribution, and the learning rate needs to be increased to accelerate the policy update speed and improve convergence efficiency.

[0093] Specifically, learning rate adjustment typically employs exponential decay or linear adjustment strategies. For example, the learning rate is multiplied by a decay factor (e.g., 0.95) when the pass rate increases, and by a growth factor (e.g., 1.05) when the pass rate decreases. Furthermore, this mechanism needs to be combined with gradient clipping and KL divergence constraints to ensure the stability of policy updates. In the CROPI framework, this step works synergistically with the POPI-based multi-stage data selection mechanism to dynamically adjust the learning rate, enabling the model to achieve more efficient policy updates and performance improvements on training data at different stages, thereby significantly accelerating the RLVR training process.

[0094] S5. Adjust the composition of the validation set according to the dynamic needs of the model training phase. Specifically, in the initial stage, the target validation set is used first, and non-target validation sets are gradually introduced as the training progresses to improve the model's cross-task generalization ability.

[0095] Specifically, this step proposes a method to dynamically adjust the composition of the validation set based on the needs of the training phase, aiming to improve the model's cross-task generalization ability. Its core technical principle lies in using an influence function to quantify the contribution of training samples to the current policy, and achieving efficient data filtering and evaluation through off-policy gradient estimation and sparse random projection.

[0096] In the specific implementation, the target validation set, such as the GSM8K or MATH dataset, is used first in the early stages of model training to ensure the model's performance improvement on key tasks. As the training progresses, untargeted validation sets, such as AMC23 and AIME24, are gradually introduced to evaluate the model's generalization performance on different tasks. This process is implemented through a Rank-based POPI (Policy Gradient Influence) scoring mechanism. In each stage m, the POPI scores of the training sample so against multiple validation sets Dval,j are calculated based on the current policy πθ(m). The ranking results of different validation sets are then fused using the Reciprocal Rank Fusion (RRF) method. Finally, the sample with the highest score α is selected for training, where α is usually set to 0.1 (i.e., only 10% of the training data is used in each stage).

[0097] Furthermore, POPI score calculation relies on the cosine similarity of gradient features. The gradient is normalized before projection to eliminate biases caused by differences in length and pass rate. The sparse ratio of sparse random projection is typically set to 0.1, meaning only 10% of the gradient dimension is retained, thus significantly reducing computational and storage overhead while preserving gradient information.

[0098] Specifically, this step functions as Curriculum Learning within the CROPI framework. Through a phased data selection mechanism, it gradually focuses the model on the most influential samples, thereby improving performance on the target task while enhancing cross-task generalization ability. Experiments show that this method achieves a 2.66x training speedup on the 1.5B model and demonstrates significant performance improvements on non-target tasks, validating its practical value in large-scale language model reinforcement learning.

[0099] S6. Dynamically adjust the sparsity ratio of the projected gradient features. Specifically, at the beginning of each training stage, automatically calculate the sparsity ratio α based on the gradient noise level of the current strategy. If the noise level is higher than the threshold, increase α to enhance the noise reduction effect, and vice versa to retain more gradient information.

[0100] Specifically, this step involves dynamically adjusting the sparsity ratio of the projected gradient features. Its core lies in automatically calculating the sparsity ratio α based on the gradient noise level of the current policy. In practice, the gradient features are first compressed into a low-dimensional space through sparse random projection to reduce storage and computational overhead. Subsequently, at the start of each training phase, the system evaluates the stability of the gradient features by statistically analyzing the gradient noise level of the current policy (such as gradient variance or the dispersion of the gradient distribution). If the noise level exceeds a preset threshold (such as the variance threshold σ), the system will adjust the sparsity ratio accordingly. 2 If the noise level is low (e.g., α = 0.05), then increase α (e.g., from 0.1 to 0.2) to enhance the noise reduction effect; conversely, if the noise level is low, decrease α (e.g., from 0.1 to 0.05) to retain more gradient information, thereby improving the model's sensitivity to key features. This process is usually combined with a sliding window mechanism or exponential moving average (EMA) to smooth noise estimation and avoid misjudgments caused by single fluctuations.

[0101] Furthermore, the sparsity ratio α is adjusted based on the quantization metrics of the gradient noise, such as the mean squared error (MSE), variance, or entropy of the gradient distribution. In the CROPI framework, the threshold for gradient noise is typically set to 0.05, while the initial value of α is 0.1, with a maximum value not exceeding 0.3 and a minimum value not lower than 0.03. Additionally, the projection dimension k of the gradient features is typically set to 1 / 10 to 1 / 5 of the original gradient dimension d (e.g., k = 128 or 256) to ensure sufficient information is retained during dimensionality reduction. The projection matrix Psparse is constructed using a combination of random index selection and Gaussian distribution sampling, where non-zero elements follow an N(0,1) distribution, and the remainder are zero vectors.

[0102] Specifically, this step is applicable to the multi-stage training process of large-scale language models (LLMs) in RLVR tasks. Before the start of each training stage, the system dynamically adjusts the sparsity ratio α based on the gradient noise level of the current policy to optimize the accuracy of subsequent data selection. For example, in the early stages of training, when the policy has not yet converged and the gradient noise is high, the system increases α to filter out noise; while in the later stages of training, when the policy tends to stabilize and the noise decreases, the system decreases α to retain more gradient details, thereby improving the model's adaptability to complex tasks. By dynamically adjusting the sparsity ratio α, this step effectively balances the contradiction between gradient denoising and information preservation, improving the discriminative ability of gradient features in data selection. Experiments show that this method achieves a training speedup of 2.66× on a 1.5B model while using only 10% of the data volume. Furthermore, this step enhances the robustness and adaptability of the CROPI framework, enabling it to maintain efficient data selection capabilities across different training stages and model sizes, thus significantly improving the overall training efficiency and model performance of RLVR.

[0103] The reinforcement learning data selection method based on offline policy influence estimation in this invention improves the generalization ability and policy update stability of the RLVR method across different tasks by dynamically adjusting the composition of the validation set and the gradient feature sparsity ratio during training, thereby achieving more efficient and robust model convergence with limited data.

[0104] Example 2

[0105] To implement the above embodiments, embodiments of the present invention provide a reinforcement learning data selection device 10 based on offline policy influence estimation, such as... Figure 2 As shown, it includes:

[0106] The policy gradient estimation module 100 is used to acquire pre-collected offline trajectory data and calculate the gradient estimate of the target policy based on the trajectory generated by the behavior policy. The offline approximation of the policy gradient is achieved through importance sampling and KL divergence constraints.

[0107] Specifically, when calculating the importance sampling weight, the log probability ratio of the trajectory is used, which is based on the difference in trajectory generation probability between the current policy πθ and the behavior policy β. When updating the policy, the KL divergence between the current policy πθ and the behavior policy β is calculated, and a threshold is set to limit its maximum value.

[0108] The sparse projection dimensionality reduction module 200 is used to apply sparse random projection technology to the gradient estimation, randomly select a subset of the gradient dimension for projection dimensionality reduction, and reduce storage and computation overhead and suppress numerical noise by setting a sparsity ratio.

[0109] Specifically, when constructing the sparse random projection matrix, a subset S of the gradient dimension is randomly selected, where the size of the subset S is determined by a preset sparsity ratio α. Before projection, a random dropout operation is performed on the gradient vector g, specifically by setting each dimension to zero independently with probability 1-α. The retained dimensions are used to generate non-zero columns of the projection matrix through a standard normal distribution N(0,1).

[0110] The gradient similarity analysis module 300 is used to calculate the gradient inner product similarity between training data points and validation data points based on the current checkpoint and validation set of the target policy, and generate a data influence score.

[0111] Specifically, before calculating the gradient inner product similarity, the gradient features are normalized by dividing the gradient vector g of each training data point by its L2 norm to obtain a unit vector before calculating the inner product. When calculating the data influence score, semantic similarity is combined. Specifically, the semantic similarity between the training data points and the validation data points is calculated through a pre-trained semantic embedding model, and then weighted and fused with the gradient inner product similarity. The weight coefficients are dynamically adjusted according to the task type.

[0112] The course learning optimization module 400 is used to iteratively select the most influential subset of training data under the current strategy based on the data influence score, and gradually optimize the strategy model parameters using the course learning framework.

[0113] Specifically, the Reciprocal Rank Fusion (RRF) method is used in the selection of training data subsets during the iteration phase. Specifically, the POPI-R rank rj(so) of the training data points is calculated for each validation set Dval,j, and the ranks of all validation sets are merged. In the policy model parameter optimization step, the learning rate is dynamically adjusted according to the online pass rate at the current stage. Specifically, if the online pass rate increases beyond a preset threshold, the learning rate is reduced to stabilize training, and vice versa to accelerate convergence.

[0114] The reinforcement learning data selection device based on offline policy influence estimation in this invention effectively controls the stability of policy updates by introducing KL divergence constraints and a dynamic threshold mechanism; it employs an adjustable sparsity ratio random projection technique, significantly reducing computational complexity while maintaining gradient feature representation capabilities; it improves the accuracy of data selection by combining gradient inner product and semantic similarity multi-dimensional influence evaluation; and it achieves smooth convergence of the training process by combining an RRF-based ranking course learning strategy and a dynamic learning rate adjustment mechanism. This device demonstrates excellent data utilization efficiency and policy optimization effects in multiple task scenarios, significantly improving the overall performance of reinforcement learning training.

[0115] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0116] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A reinforcement learning data selection method based on offline policy influence estimation, characterized in that, include: S1. Acquire pre-collected offline trajectory data and calculate the gradient estimate of the target policy based on the trajectory generated by the behavior policy. The offline approximation of the policy gradient is achieved through importance sampling and KL divergence constraint. S2, apply sparse random projection technique to the gradient estimation, randomly select a subset of the gradient dimension for projection dimensionality reduction, reduce storage and computation overhead and suppress numerical noise by preset sparsity ratio; S3, based on the current checkpoint of the target policy and the validation set of the target policy, calculates the gradient inner product similarity between the training data points and the validation data points, and generates a data influence score; S4. Based on the data influence score, iteratively select the most influential subset of training data under the current strategy, and gradually optimize the strategy model parameters using the course learning framework.

2. The method as described in claim 1, characterized in that, S1 further includes: S11, The weight of importance sampling is calculated using the log probability ratio of the trajectory, specifically: based on the difference in trajectory generation probability between the current policy πθ and the behavior policy β; S12, KL divergence constraint is implemented by introducing KL loss term, specifically: when updating the policy, calculate the KL divergence between the current policy πθ and the behavior policy β, and set a threshold to limit the corresponding maximum value.

3. The method as described in claim 1, characterized in that, S2 further includes: S21, The method for constructing the sparse random projection matrix includes: randomly selecting a subset S of the gradient dimension, wherein the size of the subset S is determined by a preset sparsity ratio α; S22, the random dropout operation is performed before projection. Specifically, each dimension of the gradient vector g is independently set to zero with probability 1-α, and the remaining dimensions are used to generate non-zero columns of the projection matrix through the standard normal distribution N(0,1).

4. The method as described in claim 1, characterized in that, S3 further includes: S31, before calculating the gradient inner product similarity, the gradient features are normalized, specifically: the gradient vector g of each training data point is divided by its L2 norm to obtain a unit vector before calculating the inner product. S32, the calculation of the data influence score combines semantic similarity, specifically: the semantic similarity between training data points and validation data points is calculated through a pre-trained semantic embedding model, and then weighted and fused with gradient inner product similarity, with the weight coefficient dynamically adjusted according to the task type.

5. The method as described in claim 1, characterized in that, S4 further includes: S41, the selection of training data subsets in the iterative phase of the course learning framework adopts the Reciprocal Rank Fusion (RRF) method, specifically: for each validation set Dval,j, calculate the POPI-R rank rj(so) of the training data points, and fuse the ranks of all validation sets; In step S42, the learning rate is dynamically adjusted based on the online pass rate at the current stage. Specifically, if the online pass rate increases beyond a preset threshold, the learning rate is reduced to stabilize training; otherwise, the learning rate is increased to accelerate convergence.

6. The method as described in claim 1, characterized in that, Also includes: S5. Adjust the composition of the validation set according to the dynamic needs of the model training phase. Specifically, in the initial stage, the target validation set is used first, and non-target validation sets are gradually introduced as the training progresses. S6. Dynamically adjust the sparsity ratio of the projected gradient features. Specifically, at the beginning of each training stage, automatically calculate the sparsity ratio α based on the gradient noise level of the current strategy. If the noise level is higher than the threshold, increase α to enhance the noise reduction effect, and vice versa to retain more gradient information.

7. A reinforcement learning data selection device based on offline policy influence estimation, characterized in that, include: The policy gradient estimation module is used to acquire pre-collected offline trajectory data and calculate the gradient estimate of the target policy based on the trajectory generated by the behavior policy. It achieves offline approximation of the policy gradient through importance sampling and KL divergence constraints. The sparse projection dimensionality reduction module is used to apply sparse random projection technology to the gradient estimate, randomly select a subset of the gradient dimension for projection dimensionality reduction, and reduce storage and computation overhead and suppress numerical noise by setting a sparsity ratio. The gradient similarity analysis module is used to calculate the gradient inner product similarity between training data points and validation data points based on the current checkpoint and validation set of the target policy, and generate a data influence score. The course learning optimization module is used to iteratively select the most influential subset of training data under the current strategy based on the data influence score, and gradually optimize the strategy model parameters using the course learning framework.

8. The apparatus as claimed in claim 7, characterized in that, The policy gradient estimation module is also used for: The importance sampling weights are calculated using the log probability ratio of the trajectories, specifically based on the difference in trajectory generation probabilities between the current policy πθ and the behavioral policy β. When updating the policy, calculate the KL divergence between the current policy πθ and the behavior policy β, and set a threshold to limit its maximum value.

9. The apparatus as claimed in claim 7, characterized in that, The sparse projection dimensionality reduction module is also used for: When constructing the sparse random projection matrix, a subset S of the gradient dimension is randomly selected, where the size of the subset S is determined by a preset sparsity ratio α. Before projection, a random dropout operation is performed on the gradient vector g. Specifically, each dimension is independently set to zero with probability 1-α. The remaining dimensions are used to generate non-zero columns of the projection matrix through a standard normal distribution N(0,1).

10. The apparatus as claimed in claim 7, characterized in that, The gradient similarity analysis module is also used for: Before calculating the gradient inner product similarity, the gradient features are normalized. Specifically, the gradient vector g of each training data point is divided by its L2 norm to obtain a unit vector before calculating the inner product. When calculating the data influence score, semantic similarity is combined. Specifically, the semantic similarity between training data points and validation data points is calculated through a pre-trained semantic embedding model, and then weighted and fused with gradient inner product similarity. The weight coefficients are dynamically adjusted according to the task type.

Images