Training optimization acceleration method of inference model, storage medium and program product
Through the advantages-based answer pruning and graphics card scheduling strategies, the problems of large computing overhead and low efficiency when training inference models are solved, and the training efficiency is significantly improved and the graphics card utilization rate is improved.
Patent Information
- Application Number
- CN202510452417.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-11
- Publication Date
- 2025-08-05
AI Technical Summary
The existing reinforcement learning methods have problems such as high computational overhead and low efficiency when training inference models. Especially in the group relative strategy optimization algorithm, generating a large number of answers and calculating the predicted probability ratio leads to high training costs, limiting its efficiency and scalability.
Advantage-based answer pruning strategy is adopted, and pruning unimportant answers by calculating the relative advantage value of the answer, and combining the answer scheduling strategy, the graphics card utilization rate is optimized and training efficiency is improved.
The training efficiency has been improved, the graphics card utilization rate has been improved, and the training has been accelerated by 1.76 times. At the same time, the model performance has not been reduced, and the rationality of the pruning strategy and the effectiveness of the graphics card utilization strategy have been verified.
Smart Images
Figure CN120430401A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of training acceleration of neural network models, and in particular to a training optimization acceleration method for an inference model. Background Art
[0002] Recent years have seen significant progress in the development of reasoning models, with the emergence of a series of advanced models, such as OpenAI-o1, Deepseek-R1, and Kimi-1.5. These models, through their sophisticated reasoning capabilities, have demonstrated exceptional performance in fields such as mathematics, coding, and scientific reasoning. At their core, they rely on step-by-step reasoning and reflection mechanisms to solve complex problems.
[0003] Reasoning models have been widely used in various fields, particularly in tasks requiring logical reasoning and complex decision-making. For example, they can solve complex mathematical problems, write code, and perform scientific reasoning. By generating high-quality solutions, these models provide powerful support for fields such as education, scientific research, and automated systems.
[0004] Existing reinforcement learning methods suffer from significant computational overhead when training inference models. For example, the Group Relative Policy Optimization (GRPO) algorithm requires generating a large number of completions (e.g., 16) for each problem and estimating the advantage of each completion through group scoring, resulting in high training costs. Furthermore, GRPO requires calculating the ratio of predicted probabilities between the policy model, the reference model, and the old policy model, further increasing the training burden and severely limiting its efficiency and scalability. Summary of the Invention
[0005] In response to the problems existing in the prior art, the present invention provides a training optimization and acceleration method for an inference model to improve training efficiency.
[0006] To achieve the above object, the technical solution adopted by the present invention is as follows: A method for optimizing and accelerating training of an inference model, comprising the following steps: Step 1: For each question prompt in the training dataset, let the old strategy model generate a set of G answers; Step 2: Use the reward calculation system to calculate the reward value for each answer; Step 3: Calculate the relative advantage of a set of G responses; Step 4: Calculate the pruning importance index, that is, the absolute value of the relative advantage value; Step 5: Prune the answers based on the pruning importance index and pruning rate, giving priority to the answers with smaller absolute values. The pruning rate is the set value, which defaults to 0.5. Step 6: Input the remaining answers after pruning into the policy model, reference model, and old policy model to calculate the predicted probabilities of the three models for the answers, calculate the gradient, guide the training of the policy model, and train to obtain the inference model.
[0007] In step 2, the reward value of each answer consists of two parts: one is the format reward value of the answer, and the other is the accuracy reward value. If the format is correct, the reward is increased by 1, and if the answer matches the answer, the reward value is increased by 1, otherwise the reward value is 0.
[0008] In step 3, the relative advantage value of each answer is: the reward value minus the mean reward value of G answers, and then divided by the variance of the reward values of the G answers.
[0009] In step 6, the gradient is calculated as follows: obtain the first part of the value, that is, the Kullback-Leibler divergence of the reference model and the policy model for the predicted probability of the answer, obtain the second part of the value, that is, the ratio of the predicted probability of the answer by the policy model and the old policy model, and then subtract the first part from the second part to obtain the final gradient.
[0010] The training optimization acceleration method also includes answering the scheduling strategy, which is as follows: After step 5, count the number of answers on each graphics card. If the number of answers on the graphics card is less than the set value, resample the answers to new questions from the training set. The sampling process is carried out according to steps 1-5, and then assigned to the graphics card; until the number of answers on the graphics card is greater than the set value, then proceed to step 6.
[0011] A computer-readable storage medium stores a computer program / instruction thereon, which, when executed by a processor, implements the steps of the training optimization acceleration method of an inference model as described above. A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the training optimization acceleration method of an inference model as described above.
[0012] After adopting the above scheme, the present invention adopts an advantage-based answer pruning strategy during the training process. The traditional group relative strategy optimization process requires generating a large number of answers, and the training overhead is proportional to the number of answers, which makes the traditional group relative strategy optimization efficiency low. Therefore, the advantage-based answer pruning strategy of the present invention can prune unimportant answers, thereby improving training efficiency.
[0013] In addition, the present invention also adds an answer scheduling strategy during the training process. Since the number of answers on different graphics cards will be reduced after adopting the advantage-based answer pruning strategy, in order to fully utilize the parallel computing capabilities of the graphics cards, the answer scheduling strategy is used to assign new answers to each graphics card, thereby improving the utilization rate of the graphics card and thus improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION
[0015] like Figure 1 As shown, the present invention discloses a training optimization acceleration method for an inference model, which specifically includes the following steps: Step 1: For each question prompt in the training dataset, let the old strategy model generate a set of G answers; Step 2: Use the reward calculation system to calculate the reward value for each answer.
[0016] Specifically, the reward value for each answer consists of two parts: one is the format reward value of the answer, and the other is the accuracy reward value. If the format is correct, the reward is plus 1, and if the answer and the answer match, the reward value is plus 1. Otherwise, the reward value is 0. Then, the reward value of each answer has three cases: if the format is incorrect and the answer and the answer do not match, the reward value is 0; if only the format is correct or only the answer and the answer match, the reward value is 1; if the format is correct and the answer and the answer match, the reward value is 2.
[0017] Step 3. Calculate the relative advantage of a set of G answers. The relative advantage of each answer is: use the reward value minus the mean reward value of the G answers, and then divide it by the variance of the reward values of the G answers.
[0018] Step 4: Calculate the pruning importance index, that is, the absolute value of the relative advantage value.
[0019] Step 5: Prune according to the pruning importance index and the pruning rate, giving priority to pruning the answers with smaller absolute values. The pruning rate is a set value. In this embodiment, the pruning rate is set to 0.5 by default.
[0020] Step 6: Input the remaining answers after pruning into the policy model, the reference model, and the old policy model to calculate the predicted probabilities of the answers from these three models. This calculation then calculates the gradients to guide the training of the policy model, resulting in the inference model. This portion of guiding the training of the policy model can be implemented using existing techniques, typically using backpropagation to calculate the gradients of each model parameter and then update the policy model parameters.
[0021] In the present invention, the gradient is calculated as follows: obtain the first part of the value, that is, the Kullback-Leibler divergence of the reference model and the policy model for the predicted probability of the answer, obtain the second part of the value, that is, the ratio of the predicted probability of the answer by the policy model and the old policy model, and then subtract the first part from the second part to obtain the final gradient.
[0022] In the present invention, the strategy model, the old strategy model and the reference model are initialized to the same model. During the training process, the reference model does not update the parameters. The old strategy model is synchronized with the strategy model every k (such as 2) training steps, and the strategy model is updated every training step.
[0023] In order to improve the utilization rate of the graphics card, the present invention also adds a response scheduling strategy during the training optimization acceleration process. The specific steps are as follows: After step 5, the number of answers on each graphics card is counted. If the number of answers on a graphics card is less than the set value, new answers are sampled from the training set. The sampling process is repeated according to steps 1-5 and then assigned to the graphics card. This process continues until the number of answers on that graphics card exceeds the set value, at which point step 6 is repeated. In this embodiment, the value set for the number of answers on each graphics card is the training batch size, which defaults to 16.
[0024] The key improvements of this invention are twofold: a dominance-based answer pruning strategy and an answer scheduling strategy that fully utilizes graphics card utilization. Traditional group relative strategy optimization requires generating a large number of answers, and the training overhead is proportional to the number of answers, making traditional group relative strategy optimization inefficient. Therefore, the dominance-based answer pruning strategy of this invention can prune unimportant answers, thereby improving training efficiency.
[0025] On the other hand, after adopting the advantage-based answer pruning strategy, the present invention will reduce the number of answers on different graphics cards. In order to fully utilize the parallel computing capabilities of the graphics cards, the answer scheduling strategy is used to assign new answers to each graphics card, thereby improving the utilization rate of the graphics card and thus improving the training efficiency.
[0026] In order to better demonstrate the technical effects achieved by the present invention, the present invention will be described in detail through the following experimental data.
[0027] Table 1
[0028] Table 1 shows the training results of the Qianwen 2.5 model (Qwen2.5-1.5B-Instruct) using the proposed method (Answer Pruning, AP) on the GSM8K dataset. The comparison in Table 1 shows that the proposed method can achieve a 1.76x training speedup on the GSM8K dataset for the Qianwen 2.5 model, without sacrificing the performance of the final trained model. Comparing the results of Random (randomly reduced answers), Largest (removing answers with large absolute advantage), and AP (removing answers with small absolute advantage) demonstrates the rationality of the importance metric used for AP. Comparing AP* (without an answer scheduling strategy that fully utilizes GPU utilization) with AP (with an answer scheduling strategy that fully utilizes GPU utilization) shows that the answer scheduling strategy that fully utilizes GPU utilization can further improve training efficiency. Comparing GRPO (G=8, pruning rate 0%) with AP (G=16, pruning rate 50%) shows that, despite AP using fewer pruned answers than GRPO, the proposed method (AP) achieves better accuracy.
[0029] The present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments. Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure. Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A training optimization and acceleration method for an inference model, characterized by: The method comprises the following steps: Step 1: For each question prompt in the training dataset, let the old strategy model generate a set of G answers; Step 2: Use the reward calculation system to calculate the reward value for each answer; Step 3: Calculate the relative advantage of a set of G responses; Step 4: Calculate the pruning importance index, that is, the absolute value of the relative advantage value; Step 5: Prune the answers based on the pruning importance index and pruning rate, giving priority to the answers with smaller absolute values. The pruning rate is the set value, which defaults to 0.
5. Step 6: Input the remaining answers after pruning into the policy model, reference model, and old policy model to calculate the predicted probabilities of the three models for the answers, calculate the gradient, guide the training of the policy model, and train to obtain the inference model.
2. The method for accelerating the training optimization of an inference model according to claim 1, characterized in that: In step 2, the reward value of each answer consists of two parts: one is the format reward value of the answer, and the other is the accuracy reward value. If the format is correct, the reward is increased by 1, and if the answer matches the answer, the reward value is increased by 1, otherwise the reward value is 0.
3. The method for accelerating the training optimization of an inference model according to claim 1, wherein: In step 3, the relative advantage value of each answer is: the reward value minus the mean reward value of G answers, and then divided by the variance of the reward values of the G answers.
4. The method for accelerating the training optimization of an inference model according to claim 1, wherein: In step 6, the gradient is calculated as follows: obtain the first part of the value, that is, the Kullback-Leibler divergence of the reference model and the policy model for the predicted probability of the answer, obtain the second part of the value, that is, the ratio of the predicted probability of the answer by the policy model and the old policy model, and then subtract the first part from the second part to obtain the final gradient.
5. The method for accelerating the training optimization of an inference model according to claim 1, wherein: The training optimization acceleration method also includes answering the scheduling strategy, which is as follows: After step 5, count the number of answers on each graphics card. If the number of answers on the graphics card is less than the set value, resample the answers to new questions from the training set. The sampling process is carried out according to steps 1-5, and then assigned to the graphics card; until the number of answers on the graphics card is greater than the set value, then proceed to step 6.
6. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of a training optimization acceleration method for an inference model as described in any one of claims 1-5 are implemented.
7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of a training optimization acceleration method for an inference model as described in any one of claims 1-5 are implemented.