Reinforcement learning training method and device based on data dynamic selection, and electronic equipment

Abstract

The invention discloses a reinforcement learning model training method and device based on dynamic data selection and electronic equipment, and the method comprises the steps: carrying out the screening from an original mathematical problem data pool according to preset data quality, diversity and difficulty evaluation indexes, and obtaining a candidate training set; calculating a data validity score of each training sample in the candidate training set; difficulty filtering is carried out on the candidate training set, and an adaptive training course is constructed; further filtering the candidate training set according to the solution reasoning process and the reward score of the candidate training set; dynamically selecting a data subset for the current training period by setting a temperature parameter attenuated along with the training process and a probability threshold based on the further filtered candidate training set and the data validity score; the selected data subsets are put into a reinforcement learning model for training, and the reinforcement learning model is trained in an optimized mode according to the correctness reward and the format reward. The model training period can be remarkably shortened, and the performance equivalent to or even better than that of full data training is achieved.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning for large language models, specifically to an efficient training method, apparatus, and electronic device for reinforcement learning models based on dynamic data selection. Background Technology

[0002] In recent years, reinforcement learning techniques based on human feedback, with large-scale language models at their core, have demonstrated unprecedented performance in handling complex reasoning tasks, especially in the field of mathematical reasoning. These techniques, through dynamic, reward-based training, enable models to learn autonomously and develop complex reasoning abilities.

[0003] In mathematical reasoning scenarios, current mainstream reinforcement learning training paradigms typically rely on large-scale static datasets. To enable models to acquire mathematical reasoning capabilities, researchers often use datasets containing hundreds of thousands of question-answer pairs for long-term training. However, this training paradigm based on maximizing data size not only consumes enormous computational resources but is also inefficient.

[0004] More importantly, the quality and difficulty of training samples in mathematical reasoning scenarios have a crucial impact on the model's learning trajectory: in the early stages of training, the model struggles to learn effective information from excessively difficult samples; while as the model's capabilities improve, samples with excessively low difficulty fail to provide meaningful gradients for policy optimization, thus reducing sample utilization and training stability. Therefore, simply increasing the amount of training data has an inherent bottleneck in its effectiveness.

[0005] To address this issue, data-centric methods have emerged in existing technologies. The core idea is to achieve, or even surpass, the performance of training on the entire dataset by training on a carefully selected, information-intensive small subset. However, existing data selection methods have a fundamental flaw: they typically require a complete reinforcement learning pre-training on the entire dataset before evaluating and selecting effective samples based on reward signals. This process consumes significant computational resources and time, contradicting the goal of improving training efficiency. Furthermore, their evaluation criteria are relatively simplistic, often neglecting key dimensions such as problem difficulty and solution complexity, leading to suboptimal data selection results.

[0006] In the process of developing this invention, the inventors discovered that the prior art has at least the following problems when training mathematical reasoning models: First, existing methods generally employ static, full-scale mathematical problem datasets for training, leading to a significant waste of computational resources and lengthy training cycles. Second, these methods overlook a crucial fact: the "teaching" value of the same mathematical problem dynamically changes for models at different learning stages. Furthermore, while existing data selection techniques acknowledge these issues, their high upfront pre-training costs render them impractical in real-world applications. Therefore, there is an urgent need to develop an efficient reinforcement learning training method that requires no costly pre-training and can dynamically and adaptively select data based on the model's current mathematical capabilities. Summary of the Invention

[0007] To overcome the above technical problems, this application implements and provides an efficient training method, apparatus, and electronic device for reinforcement learning models based on dynamic data selection, ensuring improved training efficiency and effectiveness.

[0008] According to a first aspect of the embodiments of this application, a reinforcement learning training method based on dynamic data selection is provided, comprising: S1: Based on the preset data quality, diversity and difficulty evaluation indicators, candidate training sets are obtained by screening from the original mathematical problem data pool; S2: For each training sample in the candidate training set, calculate its data validity score; S3: Filter the candidate training set by difficulty and construct an adaptive training course; S4: Further filter the candidate training set based on the reasoning process and reward score of the candidate training set; S5: Based on the further filtered candidate training set and data validity score, dynamically select a subset of data for the current training cycle by setting temperature parameters and probability thresholds that decay with the training process; S6: Input the selected subset of data into the reinforcement learning model for training, and optimize the training of the reinforcement learning model based on the correctness reward and the format reward.

[0009] According to a second aspect of the embodiments of this application, a data-selective reinforcement learning model training apparatus is provided, comprising: The dataset construction module is used to select candidate training sets from the original mathematical problem data pool based on preset data quality, diversity and difficulty evaluation indicators; The validity evaluation module is used to calculate the data validity score for each training sample in the candidate training set; The adaptive curriculum module is used to filter the candidate training set by difficulty and construct an adaptive training curriculum. A further filtering module is used to further filter the candidate training set based on the solution reasoning process and historical reward scores. The data selection module is used to dynamically select a subset of data for the current training cycle based on the further filtered candidate training set and data validity score, by setting a temperature parameter and probability threshold that decay with the training process; The model training module is used to feed the selected subset of data into the reinforcement learning model for training, and optimizes the training of the reinforcement learning model based on the correctness reward and the format reward.

[0010] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0011] The technical solutions provided by the embodiments of this application may include the following beneficial effects: This application calculates a data validity score for each training sample in the candidate training set. Through validity scoring in a dynamic mathematical reasoning scenario, it can quantify the value of mathematical problems to the model at a specific learning stage in real time and at low cost. The candidate training set is then filtered by difficulty to construct an adaptive training curriculum. The candidate training set is further filtered based on the solution reasoning process and reward scores. Based on the further filtered candidate training set and data validity scores, a data subset is dynamically selected for the current training cycle by setting a temperature parameter and probability threshold that decays with the training process. By combining the above scoring mechanism with a sampling strategy that sets a temperature parameter and probability threshold that decays with the training process, this solution can adaptively adjust the distribution of training problems according to the model's mathematical ability learning trajectory. In the early stages of training, the focus is on providing basic arithmetic and algebra problems; as the model's ability improves, the proportion of more complex problems such as geometry and calculus is automatically increased, achieving efficient dynamic curriculum learning.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0014] Figure 1 This is an overall framework diagram of an efficient training method for reinforcement learning models based on dynamic data selection, provided in an embodiment of this application.

[0015] Figure 2This is a structural block diagram of an efficient training device for a reinforcement learning model based on dynamic data selection, provided in an embodiment of this application. Detailed Implementation

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0017] Existing reinforcement learning models, especially large language models, generally rely on large-scale static datasets for training. While direct, this approach leads to extremely high computational costs and lengthy training cycles, and suffers from serious sample efficiency issues. Specifically, the value of data samples changes dynamically at different learning stages: overly simple samples fail to provide effective gradients as the model's capabilities improve, while overly difficult samples are incomprehensible to the model in the early stages of training, both resulting in wasted and redundant training resources. Furthermore, some existing data selection methods attempt to address this problem, but they themselves require a full training run on the complete dataset to obtain evaluation metrics, a process that consumes significant upfront computational resources, contradicting the initial goal of efficiency optimization.

[0018] To overcome the above technical problems, embodiments of this application provide an efficient training method, apparatus, and electronic device for reinforcement learning models based on dynamic data selection, in order to improve the fairness, accuracy, and efficiency of predictions.

[0019] The efficient training method for reinforcement learning models provided in this application is mainly applicable to training scenarios of mathematical reasoning questions in large-scale language models, and provides a good solution for how to efficiently utilize training data.

[0020] Figure 1 This application provides an overall framework diagram of an efficient training method for reinforcement learning models based on dynamic data selection, according to an exemplary implementation example. Figure 1 As shown, the method may include the following steps: S1: Based on preset data quality, diversity, and difficulty evaluation indicators, candidate training sets are obtained by filtering from the original mathematical problem data pool; this step may include the following sub-steps: S11: Based on the preset data quality, gather raw mathematical problem data from multiple mathematical competitions and forums, and filter data whose data quality is lower than the set threshold.

[0021] Specifically, the process begins with a screening process using a raw pool of question-answer pairs. This pool includes data collected from multiple authoritative sources, such as the American Invitational Mathematics Competition (AIME), the American Mathematical Competitions (AMC), the International Mathematical Olympiad, and online mathematics forums. Based on this, data quality filtering is performed, systematically identifying and removing samples with formatting errors, logical breaks, missing key fields, or vague question descriptions or incorrect mathematical expressions. Only samples with clear questions, rigorous notation, and complete solutions are retained as the basis for subsequent processing.

[0022] S12: The original mathematical problem data is filtered and selected according to the classification criteria of mathematical problem domains to ensure diversity.

[0023] Specifically, to ensure the model can learn comprehensive mathematical concepts and avoid ability bias, all quality-filtered problem samples are classified by domain. Based on the mathematical attributes of the problems, they are categorized into different branches of mathematics, such as geometry, algebra, number theory, and combinatorics. During the screening process, the system monitors and maintains a balance in the number of different problem types to ensure that the final candidate training set comprehensively and evenly covers diverse mathematical knowledge points.

[0024] S13: Set difficulty assessment indicators, use baseline model inference assessment, classify the difficulty of the selected original mathematical problem data, and obtain candidate training sets.

[0025] Specifically, this step aims to construct a dataset whose difficulty matches the model's learning curve. We employ a model inference-based difficulty assessment method, which uses a baseline model with a small number of parameters to solve all problems. During the assessment, problems that can be easily and stably solved by the baseline model are removed because they are considered too rudimentary. Simultaneously, to ensure computational efficiency during training, problems with inference trajectories exceeding 8K tokens are also excluded. Through this method, a candidate training set with appropriate difficulty is ultimately obtained.

[0026] S2: For each training sample in the candidate training set, calculate its data validity score; the data validity score is a weighted combination of the problem difficulty coefficient, the length of the problem-solving thought chain, and the reward fitness. The weight coefficients of the weighted combination are dynamically adjusted by the Softmax parameterization method to adaptively control the relative importance of each evaluation component according to the model training state.

[0027] Specifically, to quantify the potential value of each training sample for model updates, this method introduces an innovative evaluation metric—the data validity score. This score is a composite function calculated by weighted summation of three key evaluation components: a difficulty coefficient D(x) measuring the inherent challenge of the problem; a thought chain complexity C(x) evaluating the sophistication of the solution path; and a reward adaptability R(x) evaluating the ability of the sample to provide effective learning signals. Its formal expression is as follows: In this formula, As candidate training samples, , , These are the sample's difficulty coefficient, mental chain complexity, and reward adaptability scores, respectively. These are the weight coefficients corresponding to the three components, which control the relative importance of each component in the final DES score, enabling dynamic parameterized adjustment of the weight coefficients.

[0028] In order for DES to adapt to the model's learning state, the weight coefficients The weights are dynamically adjusted during training to ensure that they always form an effective probability distribution (i.e., satisfy...). This invention uses the Softmax parameterization method to derive the weights. The specific calculation formula is as follows: In this formula: It is with three components Corresponding real-valued parameters. These parameters can be updated based on feedback during training, thereby dynamically adjusting the relative importance of each component, enabling DES to more intelligently reflect the data features most valuable to the model at a specific training stage.

[0029] S3: Filter the difficulty of the candidate training set using Gaussian distribution sampling to construct an adaptive training curriculum; this step may include the following sub-steps: S31: Deploy large language models of different sizes on the base to perform multiple inferences on the candidate training set, and divide the static initial difficulty level for different accuracy intervals.

[0030] Specifically, to assign an objective initial difficulty benchmark to each mathematical problem, this method first employs a static evaluation method based on model performance. The core of this method is to utilize a powerful base model (such as DeepSeek-R1) as an "evaluator" to perform multiple independent sampling inferences on each problem in the candidate set. By statistically analyzing the model's solution accuracy across multiple attempts, the initial difficulty of the problem can be categorized into different static levels.

[0031] S32: Employ a Gaussian distribution-based function to dynamically track the optimal difficulty level; based on the initial difficulty level, implement an adaptive difficulty adjustment mechanism, design time-dependent weights that evolve with the training process, and construct an adaptive training course.

[0032] Specifically, a dynamic difficulty coefficient evaluation function is introduced based on S31. By dynamically tracking the optimal difficulty level at different learning stages of the model, adaptive difficulty learning can be achieved. The difficulty score for each sample is recalculated using Gaussian sampling: in, This represents the static difficulty score of the current sample, i.e., the initial difficulty level obtained through step S31. The average difficulty score represents all samples in the entire candidate dataset and serves as the global difficulty benchmark. This represents the standard deviation of the difficulty scores for all samples in the dataset, and is used to control the dispersion of the difficulty distribution. These are time-dependent weights, whose values ​​decay linearly from 1 to 0 as training progresses. In the early stages of training... When the value approaches 1, the target difficulty of the formula is close to... Therefore, they tend to choose problems of moderate difficulty. As training progresses, As the target approaches zero, the difficulty gradually shifts towards a more challenging level. This offset guides the model to solve more difficult problems. This is a positive coefficient representing the adaptive difficulty adjustment rate, which controls the maximum deviation of the target difficulty from the average difficulty in the later stages of training. In this way, the present invention constructs a smoothly transitioning adaptive difficulty curriculum without manual intervention, ensuring that the model is exposed to the most beneficial training samples for improving its capabilities at each development stage.

[0033] S4: Based on Gaussian distribution sampling, filter the solution reasoning process and historical reward scores of the candidate training set to remove overly simple or lengthy samples; this step may include the following sub-steps: S41: Define the complexity of the thought chain, sample the problem-solving reasoning process of different complexity through Gaussian distribution, and adaptively adjust it through time-dependent weights.

[0034] Specifically, the complexity of the thought chain The aim is to quantitatively evaluate the sophistication and verbosity of the reasoning process generated by the model for solving a problem using a Gaussian distribution-based function. This differs from directly assessing the difficulty of the problem. This approach focuses on the characteristics of the solution itself. Its evaluation method involves sampling the problem multiple times using a benchmark model (such as DeepSeek-R1) and analyzing the average length of the generated solutions. Its formal expression is as follows: in, This represents the length of the solution for the current sample (e.g., the number of tokens). This represents the average length of all solutions in the entire candidate dataset. It is a variance parameter of the control function's sensitivity to length deviation. and It shares the same adaptive adjustment mechanism as the difficulty coefficient function in S3. Time-dependent weights. Similarly, it will evolve with the training process, allowing the target solution length to be dynamically adjusted, gradually favoring more complex and refined reasoning chains in the later stages of training.

[0035] S42: Calculate rewards based on correctness and format, and capture the dynamic changes in reward values ​​of candidate training sets during the training process.

[0036] Specifically, to subsequently evaluate the reward adaptability of a sample, an immediate reward value needs to be calculated for the answer generated by the model after each reasoning attempt. This invention employs a rule-based structured reward system that can simultaneously evaluate the correctness and format compliance of the answer. This reward mechanism is designed to guide the model not only to arrive at correct mathematical results but also to learn to present its reasoning process and final answer in a clear, standardized, and machine-readable manner.

[0037] The reward function is specifically defined as a three-level scoring system: (1) Highest reward: When the final answer generated by the model is verified to be completely correct, the system will give a positive reward (e.g., a value of 1).

[0038] (2) Moderate penalty: When the model generates an incorrect answer, but its entire output (including the reasoning process and the presentation of the final answer) follows the preset format specifications, the system will give a small negative reward or penalty (e.g., a value of -0.5).

[0039] (3) Maximum penalty: When the model generates an answer that is not only wrong, but also has problems with its output format (e.g., the final answer is not wrapped with the specified "boxed{}" label), the system will give a more severe negative reward or penalty (e.g., a value of -1).

[0040] To achieve this automated reward calculation, this invention requires all model outputs to follow a structured format; for example, the final answer must be encapsulated within specific symbols or labels. This design not only ensures accurate reward calculation but also helps promote model learning and the generation of systematic and well-organized problem-solving strategies.

[0041] S43: Based on the reward score, further filter out samples in the candidate training set that are all correct or all wrong.

[0042] Specifically, this invention introduces a third core component of the Data Validity Score (DES)—reward fitness. This component evaluates the effectiveness of the reward signals generated by training samples in optimizing the model's policy. This design aims to address the gradient saturation (or gradient diminishing) problem prevalent in reinforcement learning training. Specifically, if a training sample is too easy for the model at the current stage (resulting in consistently saturated positive rewards, such as +1) or too difficult (resulting in consistently saturated negative rewards, such as -1), then the policy gradient generated by that sample will approach zero, thus failing to provide effective guidance for updating the model's parameters. Therefore, a reward fitness function is constructed to quantify this effect. Its core mechanism is to assign lower scores to "invalid" samples that generate extreme (saturated) reward values, thereby reducing their influence and sampling weights during training. Its formal expression is as follows: in, This represents the average reward value of the current sample x over the most recent training periods. This represents the average reward value across all samples. It is also a parameter of the adaptive adjustment mechanism.

[0043] The core mechanism of this formula lies in the fact that when a sample's recent average reward is -1 or 1, its reward fitness score is directly set to 0, thus excluding it from subsequent sampling. For other samples, a score is calculated based on how close their reward value is to a dynamically changing target reward value. In this way, the invention ensures that the training process always focuses on problems that are appropriately challenging for the model and can generate meaningful gradient signals, greatly promoting the stability and efficiency of learning.

[0044] S5: Based on the data validity score and adaptive training curriculum, exploratory sampling is performed by dynamically adjusting the temperature coefficient to dynamically select a subset of data for the current training cycle; this step may include the following sub-steps: S51: Based on the data validity score and the temperature parameter that decays with the training process, calculate the sampling probability of each sample in the further filtered candidate training set.

[0045] Specifically, after calculating the Data Validity Score (DES) for each sample in the candidate dataset, this invention employs a hybrid sampling strategy combining temperature scheduling and threshold filtering to select training data. At the beginning of each training cycle, a temperature-controlled probability distribution is first calculated based on the DES scores of all candidate samples in the current batch. This probability distribution is generated using a Softmax function to ensure that the sum of the sampling probabilities of all samples is 1.

[0046] in, Let represent the probability that sample x is selected at the t-th training epoch. DES(x) is the data validity score of sample x. is the temperature parameter at the t-th training epoch; it acts as a regulator, controlling the degree to which the DES score affects the final probability distribution. B represents the batch of candidate samples currently being evaluated.

[0047] S52: Dynamically select samples with a sampling probability greater than the preset probability threshold to form a subset of data for the current training period.

[0048] Specifically, in order to balance the exploration of new samples and the utilization of known high-value samples during training, the temperature parameter... It is not a fixed value, but follows a preset decay schedule. This decay strategy ensures that in the early stages of training, when the model's capabilities are weak, the system can explore more diverse data; while in the later stages of training, when the model's capabilities are strong, the system can focus more on the high-scoring samples that are most conducive to improving its capabilities. The specific decay formula is as follows: In this formula, `t` is the temperature value at the t-th training cycle. `t` is the preset initial temperature, usually a relatively high value to encourage exploration. `t` is the current training cycle number. `t` is the preset maximum total number of training cycles. In the early stages of training (when `t` is small)... A higher value will result in a higher probability distribution calculated in S51. The relatively smooth surface increases the chance of selecting samples with lower DES scores, thus enabling broader exploration. This is achieved as training progresses (t increases). As the DES score decreases, the probability distribution becomes steeper, meaning that only samples with the highest DES scores have a significant probability of being selected, thus enabling effective utilization of high-value samples.

[0049] The sampling probability of each sample is calculated through temperature scheduling. Subsequently, this invention introduces a final step of threshold filtering to determine the final training subset. The system presets a probability threshold. Only those sampling probabilities Only samples that are greater than or equal to this threshold will be ultimately selected. The formal expression of this selection rule is as follows: in, This is the final set of samples selected for this round of training. It is the set of candidate samples for the current batch. It is a preset probability threshold. This hybrid approach, by combining temperature scheduling and threshold filtering, provides flexible control over the quality and quantity of samples throughout the training process.

[0050] S6: Input the selected subset of data into the reinforcement learning model for training, and optimize the training of the reinforcement learning model based on the correctness reward and the format reward.

[0051] Specifically, the Proximal Policy Optimization (PPO) algorithm is employed to train the reinforcement learning model using the aforementioned data subset. By limiting the difference between the new and old policies in each update step, training crashes caused by excessive policy updates are avoided. The training objective of the PPO algorithm is as follows: Among them, It is the probability ratio between the old and new strategies. It is the advantage function at time step t. Clipping cuts the probability ratio to a preset interval. This limits the magnitude of single-step policy updates and ensures training stability.

[0052] Ultimately, the training reward function is defined as a multi-level structure, which assigns positive rewards, slight penalties, or severe penalties based on the correctness and format compliance of the model's output answer. This invention employs a rule-based, multi-level reward system that evaluates not only the final correctness of the model's generated answer but also its format compliance. This design aims to guide the model to learn both accurate and standardized problem-solving abilities. Reward Function Specifically defined as: To ensure that the reward function is calculated automatically, consistently, and accurately, this invention requires the model's output to follow a predefined structured format. This requirement is crucial for achieving automated evaluation and training loop closure. For example, the system stipulates that the final answer generated by the model must be encapsulated within specific tags, such as "boxed{}". Simultaneously, the complete reasoning process (i.e., the thought process chain) should also be wrapped in corresponding tags. This structured output method not only greatly facilitates the development of automated evaluation programs, enabling error-free reward calculation, but also helps promote model learning and the generation of clearer, more systematic, and logically sound problem-solving strategies.

[0053] It also includes: repeating steps S4 to S6, and before each iteration, re-executing S2 based on the latest capability level of the reinforcement learning model to update the data validity scores of all samples in the candidate training set until the model performance converges.

[0054] Specifically, a closed-loop iterative process is established throughout the training process. After the model is trained using a batch of data, its mathematical reasoning ability changes. At the start of the next iteration, the system recalculates the validity scores of all candidate samples based on the model's latest ability level, achieving automatic and progressive course learning.

[0055] The efficient training method for reinforcement learning models based on dynamic data selection of this invention is implemented and run on a server equipped with eight NVIDIA A800 GPUs. To verify the performance of this embodiment (hereinafter referred to as ScalingRL), we tested it on several publicly available mathematical inference benchmarks (such as MATH 500, AMC 2023, and AIME 2024). For model performance metrics, the PASS@1 metric is used for evaluation. It assesses the model's single-pass capability by generating k answers for each question and calculating their average accuracy. A higher PASS@1 value indicates stronger inference performance. The specific calculation formula is as follows: Here, k represents the total number of answers generated for each question, which is usually set to 16 in experiments. This represents a Boolean value indicating the accuracy of the answer to the i-th question. If the i-th answer is correct, then... =1, otherwise =0.

[0056] The performance of the ScalingRL method proposed in this invention was analyzed through experimental testing, and the results are shown in Tables 1 and 2. It can be seen that ScalingRL achieves comparable or even superior performance on multiple benchmarks compared to existing baseline models trained on large-scale static data (such as PRIME-Zero-7B) and other data-efficient methods (such as LIMR). For example, on the MATH 500 benchmark, this method achieved 84.2% performance using only 1.5K samples, surpassing the PRIME-Zero-7B model trained with 230K samples (83.8%). This is mainly due to the adaptive curriculum learning mechanism constructed by this invention through dynamic data selection, which can match the most suitable training data according to the current learning stage of the model, avoiding the overfitting and low training efficiency problems caused by static datasets. Thus, it achieves better generalization ability while significantly reducing the amount of data and training time.

[0057] Table 1: Comparison of algorithm performance on MATH 500, AIME2024, and AMC2023 benchmarks Table 2: Performance Comparison of Different Data Selection Algorithms on MATH 500, AIME2024, and AMC2023 Benchmarks Corresponding to the aforementioned embodiments of the efficient training method for reinforcement learning models based on dynamic data selection, this application also provides embodiments of an efficient training apparatus for reinforcement learning models based on dynamic data selection.

[0058] Figure 2 This is a block diagram of the efficient training device for the reinforcement learning model based on dynamic data selection according to this application. (Refer to...) Figure 2 The device includes: Dataset construction module 1 is used to filter from the original mathematical problem data pool based on preset data quality, diversity and difficulty evaluation indicators to obtain candidate training sets; The validity evaluation module 2 is used to calculate the data validity score for each training sample in the candidate training set; Adaptive Course Module 3 is used to filter the candidate training set by difficulty and construct an adaptive training course. Further filtering module 4 is used to further filter the candidate training set based on the solution reasoning process and historical reward scores of the candidate training set; The data selection module 5 is used to dynamically select a subset of data for the current training cycle based on the further filtered candidate training set and data validity score, by setting a temperature parameter and probability threshold that decay with the training process; Model training module 6 is used to feed the selected data subset into the reinforcement learning model training, and optimize the training of the reinforcement learning model based on the correctness reward and the format reward.

[0059] It also includes: a model iteration module, which repeatedly executes the steps of the further filtering module 4 to the model training module 6, and before each iteration, re-executes the validity evaluation module 2 based on the latest capability level of the reinforcement learning model to update the data validity scores of all samples in the candidate training set until the model performance converges.

[0060] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0061] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0062] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the efficient training method for reinforcement learning models based on dynamic data selection as described above.

[0063] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described efficient training method for reinforcement learning models based on dynamic data selection.

[0064] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0065] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A reinforcement learning model training method based on dynamic data selection, characterized in that, include: S1: Based on the preset data quality, diversity and difficulty evaluation indicators, candidate training sets are obtained by screening from the original mathematical problem data pool; S2: For each training sample in the candidate training set, calculate its data validity score; S3: Filter the candidate training set by difficulty and construct an adaptive training course; S4: Further filter the candidate training set based on the reasoning process and reward score of the candidate training set; S5: Based on the further filtered candidate training set and data validity score, dynamically select a subset of data for the current training cycle by setting temperature parameters and probability thresholds that decay with the training process; S6: Input the selected subset of data into the reinforcement learning model for training, and optimize the training of the reinforcement learning model based on the correctness reward and the format reward.

2. The method according to claim 1, characterized in that, Based on preset data quality, diversity, and difficulty evaluation indicators, candidate training sets are obtained by screening from the original mathematical problem data, including: Based on the preset data quality, raw mathematical problem data is collected from multiple mathematical competitions and forums, and data with quality below the set threshold is filtered out. The original mathematical problem data was filtered and selected according to the classification criteria of mathematical problem domains to ensure diversity; Difficulty assessment indicators are set, and baseline model inference is used for assessment. The original mathematical problem data after screening is classified into difficulty levels to obtain candidate training sets.

3. The method according to claim 1, characterized in that, The data validity score is a weighted combination of the problem difficulty coefficient, the length of the problem-solving thought chain, and the reward fitness. The weight coefficients of the weighted combination are dynamically adjusted using the Softmax parameterization method to adaptively control the relative importance of each evaluation component according to the model training state.

4. The method according to claim 1, characterized in that, The candidate training set is filtered for difficulty, and an adaptive training course is constructed, including: Different sized base large language models are deployed to perform multiple inferences on the candidate training set, and static initial difficulty levels are divided for different accuracy ranges; A Gaussian distribution-based function is used to dynamically track the optimal difficulty level; based on the initial difficulty level, an adaptive difficulty adjustment mechanism is implemented, and time-dependent weights that evolve with the training process are designed to construct an adaptive training course.

5. The method according to claim 1, characterized in that, The candidate training set is further filtered based on the reasoning process and reward score of the solutions in the candidate training set, including: Define the complexity of the thought chain, sample problem-solving reasoning processes of different complexity using a Gaussian distribution, and adaptively adjust them using time-dependent weights; Rewards are calculated based on correctness and format, capturing the dynamic changes in reward values ​​of candidate training sets during the training process; Based on the reward score, samples that are all correct or all wrong in the candidate training set are further filtered out.

6. The method according to claim 1, characterized in that, Based on the further filtered candidate training set and data validity score, a subset of data is dynamically selected for the current training cycle by setting a temperature parameter and probability threshold that decay as the training progresses. This subset includes: Based on the data validity score and the temperature parameter that decays with the training process, the sampling probability of each sample in the further filtered candidate training set is calculated. Based on a preset probability threshold, samples with a sampling probability greater than the threshold are dynamically selected to form a subset of the data for the current training period.

7. The method according to claim 1, characterized in that: The training reward function is defined as a multi-level structure, which gives positive rewards, slight penalties, or severe penalties based on the correctness and format compliance of the model's output answer.

8. The method according to claim 1, characterized in that, Also includes: Repeat steps S4 through S6, and before each iteration, re-execute S2 based on the latest capability level of the reinforcement learning model to update the data validity scores of all samples in the candidate training set, until the model performance converges.

9. A reinforcement learning model training device based on dynamic data selection, characterized in that, include: The dataset construction module is used to select candidate training sets from the original mathematical problem data pool based on preset data quality, diversity and difficulty evaluation indicators; The validity evaluation module is used to calculate the data validity score for each training sample in the candidate training set; The adaptive curriculum module is used to filter the candidate training set by difficulty and construct an adaptive training curriculum. A further filtering module is used to further filter the candidate training set based on the solution reasoning process and historical reward scores. The data selection module is used to dynamically select a subset of data for the current training cycle based on the further filtered candidate training set and data validity score, by setting a temperature parameter and probability threshold that decay with the training process; The model training module is used to feed the selected subset of data into the reinforcement learning model for training, and optimizes the training of the reinforcement learning model based on the correctness reward and the format reward.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

Images