Mathematical reasoning-oriented large language model training method and device based on difficulty grouping

Abstract

The application belongs to the cross field of artificial intelligence, machine learning and large language model, and discloses a mathematical reasoning-oriented large language model training method and device based on difficulty grouping, which comprises the following steps: obtaining a plurality of mathematical problems and dividing them into a plurality of difficulty subsets according to the difficulty of the problems; obtaining the average accuracy of the to-be-trained model on each difficulty subset; and using an improved GRPO method to train the to-be-trained model according to the plurality of mathematical problems and the average accuracy, wherein the improved GRPO method is obtained by replacing the advantage value of the GRPO method with a grouped advantage estimation value, and the grouped advantage estimation value is obtained according to the reward value of the answer of the to-be-trained model based on the mathematical problem and the average accuracy corresponding to the difficulty subset to which the mathematical problem belongs. The application can reduce the estimation error of the advantage function under the condition of limited sampling, improve the stability of reinforcement learning training and the efficiency of sample utilization, and thus improve the training effect and reasoning performance by using the difficulty information of the mathematical problem to group and estimate the advantage function.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence, machine learning, and large language models, and relates to a method and apparatus for training large language models for mathematical reasoning based on difficulty grouping. Background Technology

[0002] In recent years, large language models have made significant progress in areas such as mathematical problem solving, mathematical proof generation, formula derivation, scientific computing assistance, and intelligent education. Particularly in mathematical reasoning tasks, large language models, by generating multi-step reasoning processes to arrive at the final answer, have become an important direction in current artificial intelligence research and industrial applications. To further enhance the model's reasoning ability, supervised fine-tuning alone is often insufficient to fully unlock the model's potential. Therefore, reinforcement learning training has gradually become an important technique in the post-training stage of large language models for mathematical reasoning. By automatically validating the model's generated results and providing reward signals, the model can be guided to learn more effective reasoning strategies, thereby improving the accuracy of solving mathematical problems.

[0003] Currently, Group Relative Policy Optimization (GRPO) is a widely adopted training method in reinforcement learning training of mathematical reasoning models. This method generates multiple candidate solutions to the same mathematical problem and constructs an advantage function using reward statistics among these solutions, thereby updating the policy. Because it does not require additional training of a value network, GRPO is simple to implement and stable in training, making it widely used in training large language models for mathematical reasoning. However, in real-world mathematical reasoning tasks, there are often significant differences in difficulty between different mathematical problems. For example, the model typically has a high accuracy rate for basic algebraic operations, while its accuracy rate drops significantly for complex proofs, multi-step combinatorial reasoning problems, and competitive mathematical problems. Therefore, the true reward probabilities for different problems are usually distributed in different intervals. Existing GRPO methods typically use a uniform advantage function estimation method for all problems during training, failing to fully utilize the information on the difficulty differences between problems. When the sample size for each problem is small, the intra-group reward statistics are easily affected by random sampling noise, leading to increased advantage function estimation errors. Especially in mathematical reasoning training, the above problems are more prominent because the model generation process is costly and actual training often uses a small number of samples.

[0004] Therefore, how to utilize the difficulty characteristics of mathematical problems to construct a more accurate advantage function estimation method, reduce the advantage estimation error under finite sampling conditions, and improve the training efficiency and final performance of reinforcement learning for large language models for mathematical reasoning has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and apparatus for training a large language model for mathematical reasoning based on difficulty grouping.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, this invention provides a method for training a large language model for mathematical reasoning based on difficulty grouping. The method uses a large language model for mathematical reasoning to be trained as the training model, and includes: acquiring several mathematical problems and dividing them into several difficulty subsets based on problem difficulty; acquiring the average accuracy of the training model on each difficulty subset; and training the training model using an improved GRPO method based on the several mathematical problems and the average accuracy, thereby obtaining a trained large language model for mathematical reasoning. The improved GRPO method is obtained by replacing the advantage value of the GRPO method with a grouped advantage estimate, wherein the grouped advantage estimate is obtained based on the reward value of the mathematical problem's answer based on the training model and the average accuracy corresponding to the difficulty subset to which the mathematical problem belongs.

[0007] Optionally, obtaining several mathematical problems and dividing them into several difficulty subsets according to the difficulty of the problems includes: using a model to be trained to obtain several answers to each mathematical problem and calculating the average accuracy rate, obtaining the average accuracy rate of each mathematical problem based on the model to be trained and using it as a difficulty index; dividing the several mathematical problems into several difficulty subsets according to several preset difficulty index intervals; wherein, the several difficulty index intervals are continuous and each difficulty index interval is of equal length.

[0008] Optionally, obtaining the average accuracy of the model to be trained on each difficulty subset includes: obtaining the average accuracy of the model to be trained on each difficulty subset using the following formula:

[0009] in, Let be the average accuracy of the model to be trained on the i-th difficulty subset. To form a validation set consisting of several identically distributed mathematical problems independently sampled from the i-th difficulty subset, for The number of mathematical problems in For the j-th mathematical problem, Let be the average accuracy of the j-th mathematical problem based on the model to be trained.

[0010] Optionally, the number of mathematical problems in the identically distributed validation set. Determined by the following formula:

[0011] Where n is a hyperparameter representing the upper bound of the number of mathematical problems in the identically distributed validation set. The total number of math problems. It is the set of all mathematical problems.

[0012] Optionally, the grouping advantage estimate is obtained by the following formula:

[0013] in, For the grouping advantage estimate of the m-th answer to the j-th math problem, The reward value for the m-th answer to the j-th math problem. Let be the average accuracy rate corresponding to the i-th difficulty subset to which the j-th math problem belongs.

[0014] Optionally, the reward value is obtained by: obtaining the comparison result of the answer to the mathematical problem with the standard solution to the mathematical problem; when the comparison result is correct, the reward value of the answer to the mathematical problem is a first preset value; when the comparison result is incorrect, the reward value of the answer to the mathematical problem is a second preset value; wherein, the first preset value is greater than the second preset value.

[0015] Optionally, the step of training the model to be trained using the improved GRPO method based on several mathematical problems and each average accuracy to obtain the trained large language model for mathematical reasoning includes: when the number of training iterations of the model to be trained using the improved GRPO method reaches the preset number of periodic iterations, obtaining the average accuracy of the current model to be trained on each difficulty subset; and updating each average accuracy based on the average accuracy of the current model to be trained on each difficulty subset.

[0016] In a second aspect, this invention provides a training system for a large language model for mathematical reasoning based on difficulty grouping. The system uses a large language model for mathematical reasoning to be trained as the training model and includes: a grouping module for acquiring several mathematical problems and dividing them into several difficulty subsets based on problem difficulty; a calculation module for acquiring the average accuracy of the training model on each difficulty subset; and a training module for training the training model using an improved GRPO method based on the several mathematical problems and the average accuracy, to obtain a trained large language model for mathematical reasoning. The improved GRPO method is obtained by replacing the advantage value of the GRPO method with a grouped advantage estimate, where the grouped advantage estimate is obtained based on the reward value of the mathematical problem's answer based on the training model and the average accuracy corresponding to the difficulty subset to which the mathematical problem belongs.

[0017] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for training a large language model for mathematical reasoning based on difficulty grouping.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for training a large language model for mathematical reasoning based on difficulty grouping.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention presents a training method for a large language model for mathematical reasoning based on difficulty grouping. It divides problems into several difficulty subsets according to their difficulty, grouping mathematical problems of similar difficulty together, thus laying a data foundation for subsequent refined advantage function estimation. By obtaining the average accuracy of the model under test on each difficulty subset, and utilizing the statistical information of multiple mathematical problems within the same difficulty subset, a more stable and reliable reward probability estimate is obtained. This effectively alleviates the problem of large fluctuations in reward statistics for individual mathematical problems under finite sampling conditions, improving the stability of advantage estimation. The model is trained using an improved GRPO method based on several mathematical problems and their average accuracies. The improved GRPO method replaces the advantage value of the GRPO method with... The grouped advantage estimate is obtained by comparing the reward value of the model's response to a mathematical problem with the average accuracy of the subset of difficulty to which the mathematical problem belongs. This method fully utilizes the difficulty correlation information between mathematical problems to construct a more accurate advantage function, which more accurately reflects the reward characteristics of mathematical problems of different difficulties. It reduces the advantage function estimation error without increasing the number of samples per problem, improving the stability of reinforcement learning training and sample utilization efficiency. This enhances the training effect and final inference performance of the mathematical inference model. It achieves better training results without increasing the number of responses per mathematical problem, reducing inference overhead and computational resource consumption during the training phase, making it suitable for resource-constrained training environments. Furthermore, this training method is compatible with existing GRPO training methods and other policy gradient-based reinforcement learning training methods, allowing for deployment without modifying the training framework. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method for training a large language model for mathematical reasoning based on difficulty grouping, according to an embodiment of the present invention.

[0021] Figure 2 This is a comparison chart of the estimation errors of the method of the present invention and GRPO in an embodiment of the present invention.

[0022] Figure 3 This is a graph showing the contribution of each difficulty subset of the method of the present invention to the overall error in an embodiment of the present invention.

[0023] Figure 4 This is a comparison chart of experimental results on different difficulty subsets according to an embodiment of the present invention.

[0024] Figure 5 The test curves of the method of the present invention and GRPO in the embodiments of the present invention are shown on Minerva, Math500, and GSM8K.

[0025] Figure 6 This is a comparison chart of the reward curves of the method of the present invention and GRPO training in an embodiment of the present invention.

[0026] Figure 7 This is a block diagram of the training system for a large language model for mathematical reasoning based on difficulty grouping, according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1In one embodiment of the present invention, a training method for a large language model for mathematical reasoning based on difficulty grouping is provided to solve the problem that in the GRPO reinforcement learning training process of a large language model for mathematical reasoning, due to the obvious differences in difficulty between mathematical problems but the difficulty information of the problems being difficult to fully utilize, the estimation error of the dominance function under limited sampling conditions is large, which affects the training efficiency and inference performance of the model.

[0030] Specifically, the method for training a large language model for mathematical reasoning based on difficulty grouping in this invention includes: S1: Obtain several mathematical problems and divide them into several difficulty subsets based on their difficulty.

[0031] S2: Obtain the average accuracy of the model to be trained on each difficulty subset.

[0032] S3: Based on several mathematical problems and average accuracy rates, the improved GRPO method is used to train the model to be trained, resulting in a trained large language model for mathematical reasoning.

[0033] The improved GRPO method is obtained by replacing the advantage value of the GRPO method with a grouped advantage estimate, which is obtained based on the reward value of the mathematical problem based on the answer of the model to be trained and the average accuracy corresponding to the difficulty subset to which the mathematical problem belongs.

[0034] Specifically, step S1 preprocesses the mathematical problems in the training data by grouping them according to their relative difficulty, based on an assessment of the relative difficulty of each problem. In mathematical reasoning tasks, different mathematical problems can have vastly different difficulty levels for the same model. For example, basic arithmetic problems and high-level competitive mathematical proofs have completely different probabilities of being correctly solved by the model being trained. By grouping problems of similar difficulty into the same subset, problems within the same subset can have similar true reward probability distributions, thus providing a data foundation for subsequent advantage function estimation using within-group statistics. Here, "several mathematical problems" refers to all the training data used for reinforcement learning training. Each mathematical problem contains its corresponding standard answer, which is used in subsequent steps to verify the correctness of the model's generated response, thereby generating a reward value. A difficulty subset refers to multiple disjoint sets of mathematical problems formed by dividing all mathematical problems. This division ensures that all mathematical problems are categorized into difficulty subsets corresponding to their respective difficulty levels.

[0035] Specifically, step S2 calculates the average accuracy of all questions within each difficulty subset obtained in step S1. Since questions within the same difficulty subset have similar true reward probabilities, the average accuracy of that subset can serve as a robust estimate of the true reward probability of questions at that difficulty level. Compared to relying solely on a small sample of a single question to estimate its reward probability, utilizing statistical information from multiple questions within the same subset yields a more stable and accurate estimate.

[0036] Specifically, step S3 is the core of the entire training process. It improves the traditional GRPO method during the reinforcement learning training phase by utilizing the difficulty subset information from step S1 and the average accuracy of each subset calculated in step S2. Specifically, for each mathematical problem and each generated answer, instead of using the within-group statistics based on a small number of sampled answers to the same problem to calculate the dominance value in the original GRPO method, it constructs a grouped dominance estimate using the global average accuracy corresponding to the difficulty subset to which the problem belongs. Because this grouped dominance estimate utilizes the statistical information of all problems within the same subset, it has a lower estimation error under small sampling conditions and can provide a more accurate gradient direction for policy updates.

[0037] Explained, in step S3, the improved GRPO method refers to replacing its core advantage function estimation module with the grouping statistics-based estimation method proposed in this invention, while maintaining the overall training framework and policy update formula of the original GRPO method. Therefore, this method is highly compatible with the existing GRPO training framework and can be deployed without modifying the model structure. The grouping advantage estimate is a new advantage estimate proposed in this invention, which quantifies the superiority or inferiority of an answer to a mathematical problem relative to the average performance of that difficulty level based on the deviation between the reward value of the answer and the average accuracy of the subset of difficulty to which the problem belongs.

[0038] In summary, the training method of this invention, through the synergistic effect of steps S1 to S3, divides problems into several difficulty subsets based on their difficulty, grouping mathematical problems of similar difficulty into the same group, thus laying a data foundation for subsequent refined advantage function estimation. By obtaining the average accuracy of the model to be trained on each difficulty subset, and utilizing the statistical information of multiple mathematical problems within the same difficulty subset, a more stable and reliable reward probability estimate is obtained, effectively alleviating the problem of large fluctuations in reward statistics for a single mathematical problem under finite sampling conditions, and improving the stability of advantage estimation. The improved GRPO method is used to train the model to be trained based on several mathematical problems and their average accuracies, wherein the improved GRPO method replaces the advantage value of the GRPO method with... The grouped advantage estimate is obtained by comparing the reward value of the model's response to a mathematical problem with the average accuracy of the subset of difficulty to which the mathematical problem belongs. This method fully utilizes the difficulty correlation information between mathematical problems to construct a more accurate advantage function, which more accurately reflects the reward characteristics of mathematical problems of different difficulties. It reduces the advantage function estimation error without increasing the number of samples per problem, improving the stability of reinforcement learning training and sample utilization efficiency. This enhances the training effect and final inference performance of the mathematical inference model. It achieves better training results without increasing the number of responses per mathematical problem, reducing inference overhead and computational resource consumption during the training phase, making it suitable for resource-constrained training environments. Furthermore, this training method is compatible with existing GRPO training methods and other policy gradient-based reinforcement learning training methods, allowing for deployment without modifying the training framework.

[0039] In one possible implementation, the step of acquiring several mathematical problems and dividing them into several difficulty subsets according to the difficulty of the problems includes: using a model to be trained to acquire several answers to each mathematical problem and calculating the average accuracy rate, obtaining the average accuracy rate of each mathematical problem based on the model to be trained and using it as a difficulty index; dividing the several mathematical problems into several difficulty subsets according to several preset difficulty index intervals; wherein, the several difficulty index intervals are continuous and each difficulty index interval is of equal length.

[0040] Specifically, for each math problem in the training dataset, the model to be trained generates a certain number of answers. Each answer is then compared to the standard answer, and the proportion of correct answers is calculated. This proportion is used as the average accuracy of the problem under the current model. This average accuracy ranges between 0 and 1; a lower value indicates a more difficult problem, and a higher value indicates a less difficult problem. Then, the (0, 1) interval is divided into a predetermined number of k consecutive difficulty index intervals of equal length. Each interval corresponds to a difficulty subset, and math problems whose average accuracy falls within the same interval are assigned to the corresponding difficulty subset.

[0041] Specifically, as shown in the following formula, the i-th difficulty subset The mathematical definition of is as follows:

[0042] in, Let j represent the j-th mathematical problem. Let the average accuracy of the model being trained be the value for the j-th mathematical problem, i.e., the difficulty index. and The boundary values ​​of the preset difficulty index range satisfy... k is the preset total number of difficulty subsets.

[0043] Explanatoryly, in this embodiment, by using equal-length intervals, all mathematical problems can be evenly distributed across various difficulty levels in the simplest way, facilitating subsequent statistical calculations and performance comparisons. However, it should be noted that equal-length intervals are a preferred embodiment and not a limitation of the invention. In practical applications, non-equal-length intervals can also be used depending on the characteristics of the data distribution.

[0044] In one possible implementation, obtaining the average accuracy of the model to be trained on each difficulty subset includes: obtaining the average accuracy of the model to be trained on each difficulty subset using the following formula:

[0045] in, Let be the average accuracy of the model to be trained on the i-th difficulty subset. To form a validation set consisting of several identically distributed mathematical problems independently sampled from the i-th difficulty subset, for The number of mathematical problems in For the j-th mathematical problem, Let be the average accuracy of the j-th mathematical problem based on the model to be trained.

[0046] Specifically, after obtaining k difficulty subsets in step S1, a validation subset is randomly and independently sampled from each difficulty subset to obtain a validation subset for calculating the average accuracy. The number of mathematical problems in this verification subset is Then, calculate the average accuracy of all questions in that validation subset. The subset of difficulty is obtained by taking the arithmetic mean. Corresponding average accuracy .

[0047] Explanatoryly, in this embodiment, using an independently sampled validation set instead of directly using the entire training subset to calculate the average accuracy avoids overfitting at group boundaries and effectively avoids statistical bias caused by accidental uneven distribution of training data at group boundaries, resulting in more accurate calculations. It can more robustly and accurately represent the true average accuracy rate of this difficulty range.

[0048] In one possible implementation, the number of mathematical problems in the identically distributed validation set Determined by the following formula:

[0049] Where n is a hyperparameter representing the upper bound of the number of mathematical problems in the identically distributed validation set. The total number of math problems. It is the set of all mathematical problems.

[0050] Specifically, the formula means: first, calculate 20% of the total number of math problems, ensuring it is at least 1; then compare this value with a preset hyperparameter n, and take the smaller of the two as the final validation set size N. Explainingly, in this implementation, the above formula is used to strike a balance between validation set size and computational efficiency. When the training dataset is large, using all the data to calculate the average accuracy would incur unnecessary computational overhead. Setting an upper bound n controls the size of the validation set, thereby reducing computational costs. When the training dataset is small, setting a lower bound of 1 ensures that the validation set has a sufficient sample size, thus guaranteeing the statistical reliability of the calculated average accuracy.

[0051] In one possible implementation, the grouping advantage estimate is obtained by the following formula:

[0052] in, For the grouping advantage estimate of the m-th answer to the j-th math problem, The reward value for the m-th answer to the j-th math problem. Let be the average accuracy rate corresponding to the i-th difficulty subset to which the j-th math problem belongs.

[0053] Specifically, during reinforcement learning training, for the i-th difficulty subset... any mathematical problem The model to be trained generates several answers to the question, each answer corresponding to a reward value. Then, the difficulty subset of the mathematical problem is used. average accuracy As a benchmark, the reward value for this answer is calculated according to the above formula. After standardization, the grouping advantage estimate of the answer is obtained. .

[0054] Interpretively, this grouped advantage estimate measures how well the reward of the current answer is performed relative to the global average performance at the difficulty level of the math problem. A positive advantage estimate indicates an answer that is better than average if the reward is higher than the average accuracy of the subset of difficulty; a negative advantage estimate indicates an answer that is worse than average if the reward is lower than the average accuracy of the subset of difficulty. This is achieved by dividing by... Normalization is performed to make the advantage estimates comparable across different difficulty levels.

[0055] Explanatoryly, in this embodiment, the grouped advantage estimate constructed using the above formula replaces the within-group normalization-based advantage estimate in the original GRPO method. The original GRPO method samples several answers separately for each question and normalizes using only the statistics of these few answers; when the sample size is small, these statistics fluctuate greatly. In contrast, this invention utilizes statistical information from all questions within a subset of similar difficulty to obtain the advantage estimate. This estimator is based on a larger sample size, and is therefore more stable and accurate. This implementation provides an analytical expression for the estimation error of the dominance function, demonstrating that the estimation error continuously decreases as the number of difficulty groups increases, providing a theoretical basis for parameter selection and system deployment.

[0056] In one possible implementation, the reward value is obtained by: obtaining a comparison result between the answer to the mathematical problem and the standard solution to the mathematical problem; when the comparison result is correct, the reward value for the answer to the mathematical problem is a first preset value; when the comparison result is incorrect, the reward value for the answer to the mathematical problem is a second preset value; wherein the first preset value is greater than the second preset value.

[0057] Specifically, reward values ​​are obtained by comparing the answer with the standard answer. For each mathematical problem in the training set, the standard answer is known in advance. The answer generated by the model to be trained is compared with the standard answer to determine whether the answer is correct. The comparison method can be exact string matching, semantic equivalence judgment, or, for mathematical reasoning tasks, symbolic expression equivalence judgment, etc. When the answer is consistent with the standard answer or mathematically equivalent, the comparison result is correct, and the reward value of the answer is set to the first preset value. When the answer is inconsistent with the standard answer or mathematically inequivalent, the comparison result is incorrect, and the reward value of the answer is set to the second preset value.

[0058] For example, the first preset value can be 1, and the second preset value can be 0.

[0059] Explanatoryly, in this embodiment, by comparing the answer with the standard answer to obtain the reward value, the quality of each answer can be clearly evaluated, providing accurate reward input for subsequent advantage value calculation.

[0060] In one possible implementation, the step of training the model to be trained using the improved GRPO method based on several mathematical problems and each average accuracy to obtain the trained large language model for mathematical reasoning includes: when the number of training iterations of the model to be trained using the improved GRPO method reaches a preset number of periodic iterations, obtaining the average accuracy of the current model to be trained on each difficulty subset; and updating each average accuracy based on the average accuracy of the current model to be trained on each difficulty subset.

[0061] Specifically, as reinforcement learning progresses, the reasoning ability of the model under training continuously improves, thus increasing the probability of correctly answering the same mathematical problem. This means the true reward probability for each mathematical problem changes. To ensure that the average accuracy of each difficulty subset consistently reflects the actual performance of the current model, the average accuracy of each difficulty subset needs to be recalculated every preset number of iterations (T steps), and the new calculation results should replace the original values. This ensures the timeliness and accuracy of the grouped advantage estimation.

[0062] Explained in this embodiment, by introducing a dynamic update mechanism, the method of the present invention can adaptively follow changes in model performance, continuously provide an accurate advantage estimation benchmark, and thus maintain stable training results throughout the training process.

[0063] In one possible implementation, the present invention's method for training a large language model for mathematical reasoning based on difficulty grouping includes the following detailed steps: Step 1: Prepare the training dataset ,in For the question, For each question, a standard answer is generated. 10 answers and calculate their accuracy. .

[0064] Step 2: Divide all problems into k difficulty subsets based on their difficulty characteristics. Each difficulty subset Corresponding to a continuous range of difficulty ,Right now: in .

[0065] Step 3: Sampling independently from each difficulty subset The mathematical problems constitute an identically distributed verification set. .

[0066] Step 4: For each difficulty subset ( ), calculate the average accuracy of the subset. : ;in, Using an independently sampled validation set instead of a training set here avoids overfitting at group boundaries, thus... It is more representative of the true average accuracy rate for that difficulty range.

[0067] Step 5: In the subsequent GRPO training phase, for the subset of training data Any question in Each answer generated for this question ( G represents the number of samples for each problem, defining the grouped advantage estimation. as follows: ;in, This is the reward for the answer. This estimate directly replaces the advantage function in the original GRPO and is used for subsequent policy updates.

[0068] Step 6: Use the grouping advantage estimate obtained in Step 5 The policy network parameters of the language model are updated according to the policy gradient update formula of GRPO. Optimize accordingly. A typical update method is:

[0069] in, For policy networks in the problem Generate answer below The probability. In actual training, common techniques such as PPO pruning and KL penalty can be combined.

[0070] Step 7: As training progresses and the policy network performance improves, the true reward probability for the same mathematical problem will change. To ensure the timeliness of the estimation, a dynamic update mechanism is implemented: every training... After the first step, the empirical accuracy is recalculated for each question in the validation set, and steps 5 and 6 are performed for training.

[0071] The method of this invention will now be explained from the perspective of mathematical principles.

[0072] Specifically, the ideal advantage signal is defined as:

[0073] The mean square estimation error of the method proposed in this invention under a given number of groups k is:

[0074] in, Let f(p) be the difficulty interval of the i-th difficulty subset, and let f(p) be the probability density function of p. This represents the weight of the i-th difficulty subset. When the length of the difficulty interval approaches 0 (i.e., k is sufficiently large), this error can become arbitrarily small. Similarly, the mean squared estimation error of GRPO can be derived as:

[0075] See Figure 2 When a uniformly distributed F and equally partitioned intervals are used, the error of the present invention decreases monotonically as k increases, and the rate of decrease is significantly better than that of the original GRPO method under a finite G (number of samples).

[0076] The relevant derivation and proof are as follows.

[0077] Lemma 1 For a given p, let Record within the group and within-group mean Within-group standard deviation The advantages of GRPO standardization are:

[0078] The advantages of true standardization are:

[0079] The conditional mean square error can then be expressed as:

[0080] Its exact analytical expression is:

[0081] Proof: Due to symmetry, we only need to consider Expand the sum of squares and utilize Bernoulli's property. , can be obtained therefore .

[0082] when or At that time, it was agreed Substituting the advantages of GRPO and simplifying, we get:

[0083] Given p, ,and Press MSE by S and break down: .

[0084] in,

[0085] :but .

[0086] :but .

[0087] :like : .

[0088] therefore: .

[0089] like .

[0090] therefore: .

[0091] Substituting the above results, we get:

[0092] remember ,but:

[0093] make ,so:

[0094] It can be proven through algebraic operations that:

[0095] therefore:

[0096] Q.E.D.

[0097] Lemma 2 set up :

[0098]

[0099] For each difficulty subset

[0100] When the size of the validation set within the difficulty subset At that time, the global advantage of the group is estimated as follows:

[0101] relative to real advantages The conditional mean square error satisfies:

[0102] right After integrating over the subset of difficulty, the overall grouped MSE is:

[0103] Proof: For a dataset, through each question The difficulty is divided into k subsets, namely:

[0104] Consider a subset of data sets, from which N validation sets are randomly sampled. When the size of the validation sets... At that time, global estimation And the variance is Therefore, its MSE converges to a constant:

[0105] For a fixed p, let , ,but:

[0106]

[0107] because:

[0108] Therefore:

[0109] because so: .

[0110] The covariance is: .

[0111] The calculation yields: .

[0112] and: .so: .

[0113] therefore: .

[0114] Overall MSE:

[0115] Therefore, considering k subsets of the dataset, the error can be expressed as:

[0116]

[0117] Q.E.D.

[0118] In one possible implementation, to verify the effectiveness of the present invention’s method for training large language models for mathematical reasoning based on difficulty grouping, a comparative experiment was conducted on a mathematical reasoning task.

[0119] The error analysis formula shows that when the dataset is a subset of the original dataset with similar difficulty, the error is relatively smaller. With fixed parameters... Under these conditions, the contribution of each difficulty subset to the overall error follows a U-shaped curve, specifically as follows: Figure 3 As shown (take) This pattern indicates that training on difficulty subsets can achieve lower estimation errors, and the method of dividing difficulty into subsets and training on these subsets is easy to implement in practice. Therefore, this experiment uses difficulty subset training for comparative analysis. It can also be observed that the estimation error is lowest for difficulty subsets with intermediate accuracy, while the estimation error is relatively larger for relatively high or low difficulty subsets.

[0120] Let the original dataset be D, and divide it into k subsets according to their difficulty. , Select three representative subsets from them. The following comparative experiment was conducted, corresponding to three difficulty levels: hard, medium, and easy. 1. Training data was taken from... (Difficulty level: Hard) Compare the results of the GRPO method with the validation set estimation; 2. Training data is taken from... (Difficulty level: Medium) Compare the results of the GRPO method with the validation set estimation; 3. Training data is taken from... (Difficulty level: Easy) Compare the results of the GRPO method with the validation set estimation. In the experiment, N data points were randomly sampled from the training difficulty subset to form the validation set. Since random sampling is used, the validation set can be regarded as a subset with the same distribution as the difficulty subset. Therefore, the average accuracy of the model on the validation set can be approximated as its average accuracy on the corresponding difficulty subset.

[0121] The specific parameters for the experiment are set as follows: reward function hyperparameters Number of difficult subset partitions The training set uses the MATH dataset, and the test set includes AMC-23, Minerva, Math500, GSM8K, AIME24, and AIME25. The maximum number of tokens in the model is set to 512, and the batch size is 24. The experimental reporting metric is avg@32.

[0122] See Figure 4 As can be seen, the method of this invention outperforms GRPO on all subsets of the dataset, with the best performance on the medium-difficulty subset. This aligns with the U-shaped curve of the estimation error, meaning that the estimation error is lowest on datasets with moderate difficulty. See also Figure 5 The test curves are shown on Minerva, Math500, and GSM8K. See also... Figure 6 The training reward curve is shown. The above experiments effectively demonstrate the superiority of the method of this invention compared with GRPO under small sampling conditions, and can be generalized to any number of samples.

[0123] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0124] See Figure 7 In another embodiment of the present invention, a training system for a large language model for mathematical reasoning based on difficulty grouping is provided, which can be used to implement the above-mentioned training method for a large language model for mathematical reasoning based on difficulty grouping. Specifically, the training system for a large language model for mathematical reasoning based on difficulty grouping includes a grouping module, a calculation module, and a training module.

[0125] The system comprises the following modules: a grouping module for acquiring several mathematical problems and dividing them into several difficulty subsets; a calculation module for obtaining the average accuracy of the model to be trained on each difficulty subset; and a training module for training the model to be trained using an improved GRPO method based on the mathematical problems and the average accuracy, to obtain a trained large language model for mathematical reasoning. The improved GRPO method is obtained by replacing the dominance value of the GRPO method with a grouped dominance estimate, which is obtained based on the reward value of the mathematical problem's answer based on the model to be trained and the average accuracy corresponding to the difficulty subset to which the mathematical problem belongs.

[0126] All relevant content of each step involved in the aforementioned embodiment of the method for training large language models for mathematical reasoning based on difficulty grouping can be referenced to the functional description of the corresponding functional module of the system for training large language models for mathematical reasoning based on difficulty grouping in the embodiments of the present invention, and will not be repeated here.

[0127] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0128] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or function. The processor described in this embodiment of the present invention can be used for the operation of a large language model training method for mathematical reasoning based on difficulty grouping.

[0129] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for training a large language model for mathematical reasoning based on difficulty grouping in the above embodiments.

[0130] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for training a large language model for mathematical reasoning based on difficulty grouping, characterized in that, The large language model for mathematical reasoning to be trained is used as the training model, including: Obtain a number of math problems and divide them into several difficulty subsets based on their difficulty. Obtain the average accuracy of the model to be trained on each difficulty subset; Based on several mathematical problems and average accuracy rates, an improved GRPO method is used to train the model to be trained, resulting in a trained large language model for mathematical reasoning. The improved GRPO method is obtained by replacing the advantage value of the GRPO method with a grouped advantage estimate, which is obtained based on the reward value of the mathematical problem based on the answer of the model to be trained and the average accuracy corresponding to the difficulty subset to which the mathematical problem belongs.

2. The method for training a large language model for mathematical reasoning based on difficulty grouping according to claim 1, characterized in that, The process of obtaining several mathematical problems and dividing them into several difficulty subsets based on their difficulty includes: The training model is used to obtain several answers to each mathematical problem and calculate the average accuracy. The average accuracy of each mathematical problem based on the training model is then used as a difficulty indicator. Based on several preset difficulty index ranges, several mathematical problems are divided into several difficulty subsets; Among them, several difficulty index intervals are continuous, and each difficulty index interval is of equal length.

3. The method for training a large language model for mathematical reasoning based on difficulty grouping according to claim 1, characterized in that, The process of obtaining the average accuracy of the model to be trained on each difficulty subset includes: The average accuracy of the model to be trained on each difficulty subset is obtained by the following formula: in, Let be the average accuracy of the model to be trained on the i-th difficulty subset. To form a validation set consisting of several identically distributed mathematical problems independently sampled from the i-th difficulty subset, for The number of mathematical problems in For the j-th mathematical problem, Let be the average accuracy of the j-th mathematical problem based on the model to be trained.

4. The method for training a large language model for mathematical reasoning based on difficulty grouping according to claim 3, characterized in that, The number of mathematical problems in the same-distributed validation set Determined by the following formula: Where n is a hyperparameter representing the upper bound of the number of mathematical problems in the identically distributed validation set. The total number of math problems. It is the set of all mathematical problems.

5. The method for training a large language model for mathematical reasoning based on difficulty grouping according to claim 1, characterized in that, The grouping advantage estimate is obtained by the following formula: in, For the grouping advantage estimate of the m-th answer to the j-th math problem, The reward value for the m-th answer to the j-th math problem. Let be the average accuracy rate corresponding to the i-th difficulty subset to which the j-th math problem belongs.

6. The method for training a large language model for mathematical reasoning based on difficulty grouping according to claim 1, characterized in that, The reward value is obtained in the following manner: Obtain the comparison results between the answer to a math problem and the standard solution to the math problem; When the comparison result is correct, the reward value for answering the math question is a first preset value; when the comparison result is incorrect, the reward value for answering the math question is a second preset value. The first preset value is greater than the second preset value.

7. The method for training a large language model for mathematical reasoning based on difficulty grouping according to claim 1, characterized in that, The improved GRPO method is used to train the model based on several mathematical problems and average accuracy rates, resulting in a trained large language model for mathematical reasoning, including: When the number of training iterations for training the model to be trained using the improved GRPO method reaches the preset number of periodic iterations, the average accuracy of the current model to be trained on each difficulty subset is obtained. Update the average accuracy based on the average accuracy of the current model to be trained on each difficulty subset.

8. A training system for a large language model for mathematical reasoning based on difficulty grouping, wherein the large language model for mathematical reasoning to be trained is used as the training model, characterized in that, include: The grouping module is used to obtain several mathematical problems and divide them into several difficulty subsets based on the difficulty of the problems; The calculation module is used to obtain the average accuracy of the model to be trained on each difficulty subset; The training module is used to train the model to be trained using an improved GRPO method based on several mathematical problems and average accuracy rates, so as to obtain a trained large language model for mathematical reasoning. The improved GRPO method is obtained by replacing the advantage value of the GRPO method with a grouped advantage estimate, which is obtained based on the reward value of the mathematical problem based on the answer of the model to be trained and the average accuracy corresponding to the difficulty subset to which the mathematical problem belongs.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for training a large language model for mathematical reasoning based on difficulty grouping as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for training a large language model for mathematical reasoning based on difficulty grouping as described in any one of claims 1 to 7.

Images