Training method and device of student model, electronic equipment and storage medium

Abstract

The present disclosure relates to the technical field of neural network models, in particular to a student model training method and device, electronic equipment and storage medium. The method comprises: obtaining an algorithm optimization model output generated by an algorithm optimization student model based on at least one question, obtaining a basic reward based on the algorithm optimization model output and a preset answer, and obtaining an answer length reward based on the length of the algorithm optimization model output and an answer length threshold. The first word generation probability of the next word of different content in the algorithm optimization model output being generated as a different word is obtained, and the first generation entropy is obtained based on the first word generation probability. The algorithm optimization student model is trained based on the answer length reward and the first generation entropy to obtain a calculation reward optimization student model. If the calculation reward optimization student model converges, it is output. This scheme can ensure that the calculation reward optimization student model generates correct and short answers, improve the convergence speed of the model in the training process, and improve the user experience.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of neural network model, in particular to a student model training method and device, electronic equipment and storage medium. BACKGROUND

[0002] In recent years, large language models (LLMs) are widely used in natural language processing, intelligent question answering and multi-modal understanding, etc. With the continuous expansion of the size of the large language model, the training and inference costs also increase dramatically. In order to improve the model deployment efficiency, model distillation is widely used to build lightweight small models. Model distillation refers to the technology of migrating the knowledge of a large and complex teacher model to a small student model. The student model can inherit the performance advantages of the teacher model while maintaining low computational cost and faster inference speed.

[0003] Among them, the student model obtained by distilling the teacher model will significantly decrease in accuracy, generalization ability and complex task performance due to the sharp reduction in the number of parameters. To make up for this performance degradation, the student model can be trained, such as training the student model based on reinforcement learning optimization, contrast learning constraint, reward modeling, etc. to optimize the performance of the student model. However, in the above-mentioned scheme, the convergence speed of the student model is slow, and the student model obtained by training is prone to generate relatively long answers based on the problem, thereby damaging the user experience. SUMMARY

[0004] In order to solve the problems in the related art, the embodiments of the present disclosure provide a student model training method and device, electronic equipment and storage medium.

[0005] In a first aspect, the embodiments of the present disclosure provide a student model training method, which is implemented by a computer and includes:

[0006] obtaining an algorithm optimization model output generated by an algorithm optimization student model based on at least one question, the algorithm optimization student model being obtained by training a student model based on a group relative strategy optimization (GRPO) algorithm, and the student model being obtained by distilling a teacher model, and the teacher model being a large language model;

[0007] obtaining a basic reward of each question based on the algorithm optimization model output corresponding to each question in the at least one question and a preset answer corresponding to each question , wherein the basic reward is when the algorithm optimization model output matches the preset answer ;

[0008] obtaining a length of the algorithm optimization model output corresponding to each question and a preset answer length threshold , and based on obtaining a length reward of an answer to each question

[0009] When the algorithm optimization student model generates the algorithm optimization model output according to each question, a first word generation probability that a next word of different content in the algorithm optimization model output is generated as a different word is obtained, and a first generation entropy of the algorithm optimization student model is obtained based on the first word generation probability;

[0010] The algorithm optimization student model is trained based on the length reward of the answer to each question and the first generation entropy, to obtain a calculation reward optimization student model;

[0011] If it is determined that the calculation reward optimization student model converges, the calculation reward optimization student model is output.

[0012] In an embodiment of the present disclosure, before obtaining the algorithm optimization model output generated by the algorithm optimization student model based on the at least one question, the method further comprises:

[0013] obtaining a student model, determining the student model as a student model to be optimized, and setting an optimization round to 0;

[0014] training the student model to be optimized based on a group relative policy optimization (GRPO) algorithm to obtain the algorithm optimization student model, and adding 1 to the current optimization round;

[0015] obtaining a basic reward of each question based on the algorithm optimization model output corresponding to each question in the at least one question and a preset answer corresponding to each question , comprising:

[0016] If the current optimization round is greater than or equal to a preset training round threshold, and it is determined that the algorithm optimization student model converges, obtaining the basic reward of each question based on the algorithm optimization model output corresponding to each question and the preset answer corresponding to each question .

[0017] In an embodiment of the present disclosure, before obtaining the basic reward of each question based on the algorithm optimization model output corresponding to each question in the at least one question and the preset answer corresponding to each question , the method further comprises:

[0018] ​If the current optimization round is less than the training round threshold, or it is determined that the algorithm optimization student model does not converge, a format reward and an accuracy reward are obtained, wherein the format reward is positively correlated with a label matching rate of a label in the algorithm optimization model output and a preset label, and the accuracy reward is obtained based on the algorithm optimization model output and a preset answer corresponding to at least one question;

[0019] The algorithm optimization student model is trained based on the accuracy reward and the format reward to obtain a basic reward optimization student model;

[0020] The basic reward optimization student model is determined as the student model to be optimized, and the step of training the student model to be optimized based on the group relative strategy optimization (GRPO) algorithm is returned.

[0021] In an embodiment of the present disclosure, when the algorithm optimization student model generates an algorithm optimization model output according to each question, a next word of different content in the algorithm optimization model output is generated as a first word generation probability of different words, and a first generation entropy of the algorithm optimization student model is obtained based on the first word generation probability, including:

[0022] The generation entropy corresponding to the content is obtained based on the content ; The generation entropy corresponding to the content is obtained based on the content ;

[0023] The generation entropy corresponding to the content is obtained based on the content ;

[0024] The first generation entropy of the algorithm optimization student model is obtained based on the generation entropy corresponding to different content in the algorithm optimization model output corresponding to each question.

[0025] In an embodiment of the present disclosure, before the basic reward optimization student model is determined as the student model to be optimized, the method further includes:

[0026] The basic reward optimization model output generated by the basic reward optimization student model based on at least one question is obtained;

[0027] The answer accuracy rate of the basic reward optimization student model is obtained based on the basic reward optimization model output and a preset answer corresponding to at least one question;

[0028] The historical model output generated according to at least one question after at least one training of the base reward optimization student model is obtained, and a historical answer accuracy of the base reward optimization student model is obtained according to the base reward optimization student model and a preset answer corresponding to the at least one question;

[0029] The base reward optimization student model is determined as the student model to be optimized, including:

[0030] If the answer accuracy is greater than or equal to the historical answer accuracy, the base reward optimization student model is determined as the student model to be optimized, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned.

[0031] In an embodiment of the present disclosure, the base reward optimization student model is determined as the student model to be optimized, further including:

[0032] If the answer accuracy is less than the historical answer accuracy, the base reward optimization student model is determined as the student model to be optimized, the length of the input data and the output data used when training the student model to be optimized is increased, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned.

[0033] In an embodiment of the present disclosure, determining that the calculation reward optimization student model converges includes:

[0034] The degree of change of the loss function value after multiple training of the calculation reward optimization student model is obtained.

[0035] The second generation entropy of the calculation reward optimization student model is obtained based on a second word generation probability of different contents in the reward optimization model output generated by the calculation reward optimization student model.

[0036] If the degree of change of the loss function value is less than or equal to the loss function value change threshold, and the second generation entropy is less than or equal to the generation entropy threshold, it is determined that the calculation reward optimization student model converges.

[0037] In a second aspect, an embodiment of the present disclosure provides a student model training device, including:

[0038] The model output acquisition module is configured to obtain algorithm optimization model output generated by an algorithm optimization student model based on at least one question, the algorithm optimization student model being obtained by training a student model based on a group relative strategy optimization GRPO algorithm, the student model being obtained by distilling a teacher model, and the teacher model being a large language model.

[0039] The basic reward obtaining module is configured to obtain a basic reward of each question based on the algorithm optimization model output corresponding to each question and a preset answer corresponding to each question , wherein the basic reward when the algorithm optimization model output matches the preset answer , the basic reward when the algorithm optimization model output does not match the preset answer ;

[0040] The length reward obtaining module is configured to obtain a length of the algorithm optimization model output corresponding to each question and a preset answer length threshold , and obtain an answer length reward of each question based on ;

[0041] The generation entropy obtaining module is configured to obtain a first word generation probability that a next word of different content in the algorithm optimization model output is generated as a different word when the algorithm optimization model output is generated by the algorithm optimization child model according to each question, and obtain a first generation entropy of the algorithm optimization child model based on the first word generation probability

[0042] The model training module is configured to train the algorithm optimization child model based on the answer length reward of each question and the first generation entropy, to obtain a calculation reward optimization child model

[0043] The model output module is configured to output the calculation reward optimization child model if it is determined that the calculation reward optimization child model converges.In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the method in any one of the first aspect.

[0044] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the method in any one of the first aspect.

[0045] According to the technical scheme provided by the embodiments of the present disclosure, by obtaining the algorithm optimization model output generated by the algorithm optimization child model based on the at least one question, obtaining the basic reward of each question based on the algorithm optimization model output corresponding to each question and the preset answer corresponding to each question in the at least one question , obtaining the length of the algorithm optimization model output corresponding to each question and a preset answer length threshold , and obtaining the answer length reward of each question based on ​The next word of different content in the algorithm optimization model output is generated as the first word generation probability of different words when the algorithm optimization model outputs are generated according to each question, and the first generation entropy of the algorithm optimization child model is obtained based on the first word generation probability; the algorithm optimization child model is trained based on the answer length reward of each question and the first generation entropy, and a calculation reward optimization child model is obtained; if it is determined that the calculation reward optimization child model converges, the calculation reward optimization child model is output. In the above scheme, the algorithm optimization child model is trained based on the answer length reward of each question and the first generation entropy, which can prompt the student model obtained by training to output correct and relatively short answers, and improve the convergence speed of the student model in the training process. Therefore, the technical scheme provided by the embodiment of the disclosure can ensure that the student model after training, that is, the calculation reward optimization child model, generates correct and relatively short answers, and improves the convergence speed of the student model in the training process, thereby improving the user experience.

[0046] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and cannot limit the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0047] Other features, objects, and advantages of the disclosure will become more apparent from the following detailed description of the non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0048] Figure 1 A flowchart of a training method of a student model according to an embodiment of the disclosure is shown.

[0049] Figure 2 A structural block diagram of a training device of a student model according to an embodiment of the disclosure is shown.

[0050] Figure 3 A structural block diagram of an electronic device according to an embodiment of the disclosure is shown.

[0051] Figure 4 A structural schematic diagram of a computer system suitable for implementing the method according to the embodiment of the disclosure is shown. DETAILED DESCRIPTION

[0052] Hereinafter, exemplary embodiments of the disclosure will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement them. In addition, parts irrelevant to the description of the exemplary embodiments are omitted in the drawings for the sake of clarity.

[0053] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there exist the features, numbers, steps, actions, components, parts or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, components, parts or combinations thereof exist or are added.

[0054] It should also be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] In the present disclosure, if the operation of acquiring user information or user data or the operation of showing user information or user data to others is involved, the operation is an operation authorized, confirmed by the user, or actively selected by the user.

[0056] The student model distilled from the teacher model usually significantly decreases in accuracy, generalization ability and complex task performance due to the sharp reduction in the number of parameters. To make up for this performance degradation, the student model can be trained, such as based on reinforcement learning optimization, contrast learning constraint, reward modeling and other strategies to train the student model to optimize the performance of the student model. However, in the above-mentioned scheme, the convergence speed of the student model is slow, and the student model trained based on the problem is easy to generate a relatively long answer, thereby damaging the user experience.

[0057] To solve the above problems, the embodiments of the present disclosure provide a student model training method and device, electronic equipment and storage medium.

[0058] The student model training method is implemented by a computer, which can include terminal equipment, servers and other processing equipment, wherein the terminal equipment can be a user equipment (User Equipment, UE), a personal computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the student model training method can be realized by a processor calling computer readable instructions stored in a memory.

[0059] Figure 1 A flowchart of a student model training method according to an embodiment of the present disclosure is shown. As shown in Figure 1 The student model training method includes the following steps:

[0060] In step S01, an algorithm optimization model output generated by an algorithm optimization student model based on at least one question is acquired.

[0061] The algorithm-optimized student model is obtained by training the algorithm-optimized model based on a group relative policy optimization (GRPO) algorithm. The student model is obtained by distilling a teacher model based on a large language model.

[0062] In an implementation manner of the present disclosure, a large language model (LLM) refers to a deep learning model trained using a large amount of text data, which can be used to generate natural language text or understand the meaning of language text. The large language model can process various natural language tasks such as text classification, question answering, dialogue, etc.

[0063] In an implementation manner of the present disclosure, a group relative policy optimization (GRPO) algorithm is a reinforcement learning algorithm based on policy optimization.

[0064] In step S02, a basic reward of each question is obtained based on an algorithm-optimized model output corresponding to each question in the at least one question and a preset answer corresponding to each question .

[0065] wherein the basic reward is when the algorithm-optimized model output matches the preset answer, and the basic reward is when the algorithm-optimized model output does not match the preset answer.

[0066] In step S03, a length of the algorithm-optimized model output corresponding to each question is obtained and a preset answer length threshold , and a response length reward of each question is obtained based on .

[0067] In step S04, a first word generation probability that a next word of different content in the algorithm-optimized model output is generated as a first word of different words when the algorithm-optimized student model generates the algorithm-optimized model output according to each question is obtained, and a first generation entropy of the algorithm-optimized student model is obtained based on the first word generation probability.

[0068] In an implementation manner of the present disclosure, the first word generation probability that the next word of different content in the algorithm-optimized model output is generated as the first word of different words when the algorithm-optimized student model generates the algorithm-optimized model output according to each question is obtained, and the first generation entropy of the algorithm-optimized student model is obtained based on the first word generation probability, which can be implemented by the following steps:

[0069] obtaining, when the algorithm-optimized student model generates the algorithm-optimized model output according to the question , a content ​a next word of the content is generated as a first word generation probability of a first word in the preset vocabulary wherein the content includes a first word of an algorithm optimization model output , and the preset vocabulary includes A words.

[0070] Based on the generation entropy corresponding to the content corresponding to the content .

[0071] Based on the generation entropy corresponding to different contents in the algorithm optimization model output of each question, a first generation entropy of the algorithm optimization child model is obtained.

[0072] In step S05, the algorithm optimization child model is trained based on the answer length reward of each question and the first generation entropy, to obtain a reward calculation optimization child model.

[0073] In an implementation manner of the present disclosure, the algorithm optimization child model is trained based on the answer length reward of each question and the first generation entropy, which can be understood as training the algorithm optimization child model based on the reward (for example, the sum of the two) calculated based on the answer length reward of each question in the at least one question and the first generation entropy. Wherein, on the premise that the algorithm optimization model output based on the input question of the algorithm optimization child model is correct, the shorter the length of the algorithm optimization model output, the greater the answer length reward. Similarly, when the algorithm optimization child model obtains the algorithm optimization model output based on the input question, a next word of different content is generated as a first word generation probability of different words, and the greater the first word generation probability corresponding to any word (that is, the greater the probability that the algorithm optimization child model determines that the next word of the corresponding content is generated as the corresponding word), the higher the first generation entropy, that is, the first generation entropy is used to indicate the certainty of the algorithm optimization model in the prediction process.

[0074] In step S06, if it is determined that the reward calculation optimization child model converges, the reward calculation optimization child model is output.

[0075] In an implementation manner of the present disclosure, the determination of the convergence of the reward calculation optimization child model can be realized by the following steps:

[0076] Obtain the change degree of the loss function value after multiple training before the reward calculation optimization child model is obtained.

[0077] When the reward calculation optimization child model generates the reward optimization model output, a second word generation probability of a next word of different content in the reward optimization model output is generated as a second word generation probability of different words, and a second generation entropy of the reward calculation optimization child model is obtained based on the second word generation probability.​​​

[0078] If the change degree of the loss function value is less than or equal to the change degree threshold of the loss function value, and the second generation entropy is less than or equal to the generation entropy threshold, it is determined that the calculation reward optimization child model converges.

[0079] On the contrary, it is determined that the calculation reward optimization child model does not converge, which can be achieved by the following steps:

[0080] If the change degree of the loss function value is greater than the change degree threshold of the loss function value, or the second generation entropy is greater than the generation entropy threshold, it is determined that the calculation reward optimization child model does not converge.

[0081] In an implementation manner of the present disclosure, the change degree of the loss function value experienced by the calculation reward optimization child model after multiple training can be understood as obtaining the loss function value of each training of the calculation reward optimization child model in the previous multiple training, and calculating the absolute value of the difference between the maximum loss function value and the minimum loss function value in the multiple loss function values. The absolute value can be understood as indicating the change degree of the first loss function value of the calculation reward optimization child model in the previous multiple training. If the absolute value is greater than the preset change degree threshold of the loss function value, it can be considered that the change degree of the loss function value is greater than the change degree threshold of the loss function value, otherwise, if the absolute value is less than or equal to the preset change degree threshold of the loss function value, it can be considered that the change degree of the loss function value is less than or equal to the change degree threshold of the loss function value.

[0082] In an implementation manner of the present disclosure, the second word generation probability of the next word of different content in the reward optimization model output generated by the calculation reward optimization child model is generated as different words, and the second generation entropy of the calculation reward optimization child model is obtained based on the second word generation probability, which can be achieved by the following steps:

[0083] Obtain the change degree of the loss function value experienced by the reward optimization child model according to the question when generating the reward optimization model output The next word of the content is generated as the second word generation probability of the th word in the preset vocabulary , wherein the content includes the first words of the reward optimization model output, , and the preset vocabulary includes A words. The generation entropy corresponding to the content is obtained based on . Based on the generation entropy corresponding to the different content in the reward optimization model output, the second generation entropy of the reward optimization child model is obtained.

[0084] According to the technical solution provided in this disclosure, the algorithm optimization model output generated by the student model based on at least one question is obtained, and the basic reward for each question is obtained based on the algorithm optimization model output corresponding to each question and the preset answer corresponding to each question. Obtain the length of the algorithm optimization model output for each problem. and preset response length threshold and based on Get a reward for the length of the answer to each question. When the algorithm-optimized student model generates the output of the algorithm-optimized model for each question, the next word of different content in the output of the algorithm-optimized model is generated as the first word generation probability of different words, and the first generation entropy of the algorithm-optimized student model is obtained based on the first word generation probability. The algorithm-optimized student model is trained based on the answer length reward for each question and the first generation entropy to obtain a computational reward-optimized student model. If it is determined that the computational reward-optimized student model has converged, the computational reward-optimized student model is output. In the above scheme, training the algorithm-optimized student model based on the answer length reward for each question and the first generation entropy can prompt the trained student model to output correct and relatively concise answers, and improve the convergence speed of the student model during training. Therefore, the technical solution provided by this embodiment can ensure that the trained student model, i.e., the computational reward-optimized student model, generates correct and relatively concise answers, and improves the convergence speed of the student model during training, thereby improving the user experience.

[0085] In one embodiment of this disclosure, before step S01, obtaining the algorithm optimization model output generated by the algorithm optimization student model based on at least one problem, the method further includes the following steps:

[0086] Obtain the student model, identify it as the student model to be optimized, and set the optimization round to 0.

[0087] The group relative strategy optimization algorithm (GRPO) is used to train the student model to be optimized, obtain the optimized student model, and increment the current optimization round by 1.

[0088] The basic reward for each question is obtained based on the output of the algorithm optimization model corresponding to each question and the preset answer corresponding to each question. ,include:

[0089] If the current optimization round is greater than or equal to the preset training round threshold, and it is determined that the algorithm optimization student model has converged, then the basic reward for each question is obtained based on the output of the algorithm optimization model corresponding to each question and the preset answer corresponding to each question. .

[0090] According to the technical scheme provided by the embodiment of the present disclosure, the student model is obtained, the student model is determined as a to-be-optimized student model, and the optimization round is set to 0. The to-be-optimized student model is trained based on the group relative strategy optimization GRPO algorithm to obtain an algorithm-optimized student model, and the current optimization round is increased by 1. If the current optimization round is greater than or equal to a preset training round threshold, and it is determined that the algorithm-optimized student model converges, the basic reward of each question is obtained based on the algorithm-optimized model output corresponding to each question and the preset answer corresponding to each question The algorithm-optimized student model used to obtain the basic reward can be ensured to converge and the round of training based on the GRPO algorithm to meet the requirements, which helps to improve the accuracy of the output generated by the calculation-reward-optimized student model obtained through final training.

[0091] In an implementation manner of the present disclosure, the basic reward of each question is obtained based on the algorithm-optimized model output corresponding to each question and the preset answer corresponding to each question Previously, the method further comprises:

[0092] If the current optimization round is less than the training round threshold, or it is determined that the algorithm-optimized student model does not converge, the format reward and the accuracy reward are obtained. The format reward is positively correlated with the label matching rate of the label in the algorithm-optimized model output and the preset label, and the accuracy reward is obtained based on the algorithm-optimized model output and the preset answer corresponding to the at least one question.

[0093] The algorithm-optimized student model is trained based on the accuracy reward and the format reward to obtain a basic-reward-optimized student model.

[0094] The basic-reward-optimized student model is determined as a to-be-optimized student model, and the step of training the to-be-optimized student model based on the group relative strategy optimization GRPO algorithm is returned.

[0095] In an implementation manner of the present disclosure, the algorithm-optimized model output can include a thinking output and an answer output. The label in the algorithm-optimized model output can include a label in the thinking output and a label in the answer output, wherein the label in the thinking output can include “thinking start” and “thinking end”, and the label in the answer output can include “answer start” and “answer end”.

[0096] In an implementation form of the present disclosure, the label matching rate can be used to indicate the probability that the label in the algorithm optimization model output corresponding to at least one question matches the preset label. Wherein, when the label in the algorithm optimization model output corresponding to any question includes all preset labels, it is determined that the label in the algorithm optimization model output corresponding to the question matches the preset label; when the label in the algorithm optimization model output corresponding to any question does not include all preset labels, it is determined that the label in the algorithm optimization model output corresponding to the question does not match the preset label; based on whether the label in the algorithm optimization model output corresponding to each question in the at least one question matches the preset label, the label matching rate is obtained.

[0097] In an implementation form of the present disclosure, the format reward positively correlated with the label matching rate can be obtained by initializing the format reward to 0, and then traversing the algorithm optimization model output corresponding to each question in the at least one question, if the label in the algorithm optimization model output corresponding to any question matches the preset label, then the format reward is increased by 1, if the label in the algorithm optimization model output corresponding to any question does not match the preset label, then the format reward is decreased by 1, until the algorithm optimization model output corresponding to the last question.

[0098] In an implementation form of the present disclosure, the accuracy reward based on the algorithm optimization model output and the preset answer corresponding to the at least one question can be understood as: if the algorithm optimization model output corresponding to any question is the same as the preset answer of the question, then the accuracy reward corresponding to the question is set to 1; if the algorithm optimization model output corresponding to the question is different from the preset answer of the question, then the accuracy reward corresponding to the question is set to 0.

[0099] In an implementation form of the present disclosure, training the algorithm optimization student model based on the accuracy reward and the format reward can be understood as training the algorithm optimization student model based on the reward (such as the sum of the two) calculated based on the accuracy reward and the format reward. Wherein, the higher the accuracy rate of the algorithm optimization model output obtained by the algorithm optimization student model based on the input question, the higher the accuracy reward; similarly, the higher the label matching rate of the label in the algorithm optimization model output obtained by the algorithm optimization student model based on the input question and the preset label, the higher the format reward.

[0100] According to the technical scheme provided by the embodiment of the present disclosure, when the current optimization round is less than the training round threshold or it is determined that the algorithm optimization student model does not converge, the format reward and the accuracy reward are taken; the algorithm optimization student model is trained based on the accuracy reward and the format reward to obtain a basic reward optimization student model; the basic reward optimization student model is determined as the student model to be optimized, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned. Wherein, training the algorithm optimization student model based on the accuracy reward and the format reward can prompt the answer output by the trained student model to include a label matched with the preset label, and the output answer can correctly answer the input question, which helps to improve the accuracy of the output generated by the finally trained calculation reward optimization student model.

[0101] In an implementation manner of the present disclosure, before the basic reward optimization student model is determined as the student model to be optimized, the method further comprises:

[0102] The basic reward optimization model output generated by the basic reward optimization student model based on the at least one question is obtained.

[0103] The answer accuracy rate of the basic reward optimization student model is obtained based on the preset answer corresponding to the at least one question and the basic reward optimization model output.

[0104] The historical model output generated by the basic reward optimization student model based on the at least one question after at least one training is obtained, and the historical answer accuracy rate of the basic reward optimization student model is obtained based on the preset answer corresponding to the at least one question and the historical model output.

[0105] The basic reward optimization student model is determined as the student model to be optimized, comprising:

[0106] If the answer accuracy rate is greater than or equal to the historical answer accuracy rate, the basic reward optimization student model is determined as the student model to be optimized, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned.

[0107] In an implementation manner of the present disclosure, obtaining the historical model output generated by the basic reward optimization student model based on the at least one question after at least one training can be understood as, before each training of the basic reward optimization student model, obtaining the model output generated by the model based on the at least one question in history, storing the model output as the historical model output, and generating training number indication information for indicating the training number of the model generating the historical model output. By obtaining the training number indication information, the historical model output generated by the basic reward optimization student model based on the at least one question after at least one training can be obtained based on the training number indication information.

[0108] According to the technical scheme provided by the embodiment of the present disclosure, the base reward optimization student model output generated by the student model based on at least one question is obtained; the answer accuracy of the base reward optimization student model is obtained based on the base reward optimization model output and the preset answer corresponding to the at least one question; the historical model output generated by the student model before the base reward optimization student model is obtained according to at least one question after at least one training is experienced, and the historical answer accuracy of the base reward optimization student model is obtained according to the historical model output and the preset answer corresponding to the at least one question; if the answer accuracy is greater than or equal to the historical answer accuracy, the base reward optimization student model is determined as the student model to be optimized. It can be ensured that before the student model to be optimized is trained based on the group relative strategy optimization GRPO algorithm, the student model to be optimized does not appear the condition that the answer accuracy decreases with the increase of the number of training, which helps to improve the accuracy of the output generated by the calculation reward optimization student model finally trained.

[0109] In an embodiment of the present disclosure, the base reward optimization student model is determined as the student model to be optimized, and the method further comprises:

[0110] If the answer accuracy is less than the historical answer accuracy, the base reward optimization student model is determined as the student model to be optimized, the length of the input data and the output data used when training the student model to be optimized is increased, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned.

[0111] According to the technical scheme provided by the embodiment of the present disclosure, by determining the base reward optimization student model as the student model to be optimized when the answer accuracy is less than the historical answer accuracy, and increasing the length of the input data and the output data used when training the student model to be optimized, it can be avoided that the accuracy of the model output generated by the student model to be optimized continues to decrease due to the truncation of the input data or the output data, the accuracy of the model output generated by the student model to be optimized after training is increased, and it helps to improve the accuracy of the output generated by the calculation reward optimization student model finally trained.

[0112] Figure 2 A structural block diagram of a student model training device according to an embodiment of the present disclosure is shown. The device can be realized as part or all of an electronic device by software, hardware, or a combination of the two.

[0113] As shown in Figure 2 The student model training device 200 comprises:

[0114] The model output obtaining module 201 is configured to obtain algorithm optimization model output generated by an algorithm optimization student model based on at least one question, the algorithm optimization student model being obtained by training a student model based on a group relative strategy optimization GRPO algorithm, the student model being obtained by distilling a teacher model, and the teacher model being a large language model;

[0115] The basic reward obtaining module 202 is configured to obtain a basic reward of each question based on the algorithm optimization model output corresponding to each question in the at least one question and a preset answer corresponding to each question , the basic reward being when the algorithm optimization model output matches the preset answer .

[0116] The length reward obtaining module 203 is configured to obtain a length of the algorithm optimization model output corresponding to each question and a preset answer length threshold , and obtain an answer length reward of each question based on . .

[0117] The generation entropy obtaining module 204 is configured to obtain a first word generation probability that a next word of different content in the algorithm optimization model output generated by the algorithm optimization student model according to each question is generated as a different word, and obtain a first generation entropy of the algorithm optimization student model based on the first word generation probability.

[0118] The model training module 205 is configured to train the algorithm optimization student model based on the answer length reward of each question and the first generation entropy, to obtain a computing reward optimization student model.

[0119] The model output module 206 is configured to output the computing reward optimization student model if it is determined that the computing reward optimization student model converges.

[0120] According to the technical scheme provided by the embodiments of the present disclosure, by obtaining algorithm optimization model output generated by an algorithm optimization student model based on at least one question, obtaining a basic reward of each question based on the algorithm optimization model output corresponding to each question in the at least one question and a preset answer corresponding to each question , obtaining a length of the algorithm optimization model output corresponding to each question and a preset answer length threshold , and obtaining an answer length reward of each question based on . The next word of different content in the algorithm optimization model output is generated as the first word generation probability of different words when the algorithm optimization model output is generated according to each question, and the first generation entropy of the algorithm optimization child model is obtained based on the first word generation probability; the algorithm optimization child model is trained based on the answer length reward of each question and the first generation entropy, to obtain a calculation reward optimization child model; if it is determined that the calculation reward optimization child model converges, the calculation reward optimization child model is output. In the above scheme, the algorithm optimization child model is trained based on the answer length reward of each question and the first generation entropy, which can prompt the student model obtained by training to output correct and relatively short answers, and improve the convergence speed of the student model in the training process, so that the technology provided in the embodiments of the present disclosure can ensure that the student model after training, that is, the calculation reward optimization child model, generates correct and relatively short answers, and improves the convergence speed of the student model in the training process, thereby improving the user experience.

[0121] The present disclosure also discloses an electronic device, Figure 3 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0122] As Figure 3 shown, the electronic device includes a memory and a processor, wherein the memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the method according to the embodiments of the present disclosure.

[0123] The present disclosure provides a student model training method, which is implemented by a computer and includes:

[0124] The algorithm optimization model output generated by the algorithm optimization child model based on at least one question is obtained, the algorithm optimization child model is obtained by training the student model based on the group relative strategy optimization GRPO algorithm, and the student model is obtained by distilling the teacher model, and the teacher model is a large language model.

[0125] The basic reward of each question is obtained based on the algorithm optimization model output corresponding to each question in the at least one question and the preset answer corresponding to each question , wherein the basic reward is when the algorithm optimization model output matches the preset answer .

[0126] The length of the algorithm optimization model output corresponding to each question is obtained and the preset answer length threshold , and the answer length reward of each question is obtained based on .​

[0127] The algorithm optimization student model generates, as a first word generation probability of different words, a next word of different content in the algorithm optimization model output when generating the algorithm optimization model output based on each question, and obtains a first generation entropy of the algorithm optimization student model based on the first word generation probability.

[0128] The algorithm optimization student model is trained based on the answer length reward and the first generation entropy for each question to obtain a reward calculation optimization student model.

[0129] If it is determined that the reward calculation optimization student model converges, the reward calculation optimization student model is output.

[0130] In an embodiment of the present disclosure, before the algorithm optimization student model generates the algorithm optimization model output based on the at least one question, the method further comprises:

[0131] The student model is obtained, the student model is determined as the student model to be optimized, and the optimization round is set to 0.

[0132] The student model to be optimized is trained based on a group relative policy optimization (GRPO) algorithm to obtain the algorithm optimization student model, and the current optimization round is incremented by 1.

[0133] The basic reward of each question is obtained based on the algorithm optimization model output corresponding to each question in the at least one question and the preset answer corresponding to each question , comprising:

[0134] If the current optimization round is greater than or equal to a preset training round threshold, and it is determined that the algorithm optimization student model converges, the basic reward of each question is obtained based on the algorithm optimization model output corresponding to each question and the preset answer corresponding to each question .

[0135] In an embodiment of the present disclosure, before the basic reward of each question is obtained based on the algorithm optimization model output corresponding to each question in the at least one question and the preset answer corresponding to each question , the method further comprises:

[0136] If the current optimization round is less than the training round threshold, or it is determined that the algorithm optimization student model does not converge, the format reward and the accuracy reward are obtained, wherein the format reward is positively correlated with a label matching rate of labels in the algorithm optimization model output and preset labels, and the accuracy reward is obtained based on the algorithm optimization model output and the preset answer corresponding to the at least one question.

[0137] The algorithm optimization student model is trained based on the accuracy reward and the format reward to obtain a basic reward optimization student model.

[0138] The base reward optimization student model is determined as the student model to be optimized, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned.

[0139] In an embodiment of the present disclosure, when the algorithm optimization model output is generated by the algorithm optimization student model according to each question, the next word of different content in the algorithm optimization model output is generated as the first word generation probability of different words, and the first generation entropy of the algorithm optimization student model is obtained based on the first word generation probability, including:

[0140] When the algorithm optimization model output is generated by the algorithm optimization student model according to the question The next word of the content Is generated as the first word generation probability of the th word in the preset vocabulary , wherein the content includes the first word of the algorithm optimization model output, , and the preset vocabulary includes A words.

[0141] Based on The generation entropy corresponding to the content .

[0142] Based on the generation entropy corresponding to different content in the algorithm optimization model output corresponding to each question, the first generation entropy of the algorithm optimization student model is obtained.

[0143] In an embodiment of the present disclosure, before the base reward optimization student model is determined as the student model to be optimized, the method further includes:

[0144] Obtain the base reward optimization model output generated by the base reward optimization student model based on at least one question.

[0145] Based on the base reward optimization model output and the preset answer corresponding to the at least one question, the answer accuracy of the base reward optimization student model is obtained.

[0146] Obtain the historical model output generated by the base reward optimization student model according to at least one question after at least one training, and obtain the historical answer accuracy of the base reward optimization student model according to the preset answer corresponding to the at least one question.

[0147] The base reward optimization student model is determined as the student model to be optimized, including:

[0148] ​If the answer accuracy rate is greater than or equal to the historical answer accuracy rate, the base reward optimization student model is determined as the student model to be optimized, and a step of training the student model to be optimized based on a group relative strategy optimization GRPO algorithm is returned.

[0149] In an embodiment of the present disclosure, determining the base reward optimization student model as the student model to be optimized further includes:

[0150] If the answer accuracy rate is less than the historical answer accuracy rate, the base reward optimization student model is determined as the student model to be optimized, the length of the input data and the output data used when training the student model to be optimized is increased, and a step of training the student model to be optimized based on a group relative strategy optimization GRPO algorithm is returned.

[0151] Figure 4 A structural diagram of a computer system suitable for implementing the method according to the embodiments of the present disclosure is shown.

[0152] As shown in Figure 4 , the computer system includes a processing unit that can execute various methods in the above embodiments according to programs stored in a read-only memory (ROM) or loaded from a storage section into a random access memory (RAM). Various programs and data required for the operation of the computer system are also stored in the RAM. The processing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0153] The following components are connected to the I / O interface: an input section including a keyboard, a mouse, etc.; an output section including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a LAN card, a modem, etc. The communication section performs communication processes via a network such as the Internet. A drive is also connected to the I / O interface as necessary. A removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive as necessary, so that a computer program read out from the removable medium is installed into the storage section as necessary. The processing unit can be implemented as a CPU, a GPU, a TPU, a FPGA, a NPU, etc.

[0154] In particular, according to embodiments of the present disclosure, the methods described above can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for executing the methods described above. In such embodiments, the computer program can be downloaded and installed from a network via the communication section, and / or installed from a removable medium.

[0155] The flow and block diagrams in the drawings represent possible architectural, functional, and operational architectures of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block can represent a module, a segment, or a portion of code that comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0156] The units or modules described in the embodiments of the present disclosure can be implemented by software, or by programmable hardware. The described units or modules can also be provided in a processor, and the names of the units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0157] As another aspect, the present disclosure also provides a computer readable storage medium, which can be the computer readable storage medium included in the electronic device or the computer system in the above embodiments, or can exist separately and not be assembled into the device. The computer readable storage medium stores one or more programs for being executed by one or more processors to perform the method described in the present disclosure.

[0158] The above description is merely preferred embodiments of the present disclosure and a description of the applied technical principles. Those skilled in the art should understand that the inventive scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and also includes other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.

Claims

1. A method for training a student model, said method being implemented by a computer, characterized in that, The method comprises the following steps: obtaining an algorithm optimization model output generated by an algorithm optimization student model based on at least one question, wherein the algorithm optimization student model is obtained by training a student model based on a group relative strategy optimization GRPO algorithm, and the student model is obtained by distilling a teacher model, wherein the teacher model is a large language model, and the large language model is a deep learning model trained using text data and used for processing a natural language task, and the natural language task comprises a text question and answer task; obtaining a basic reward of each question in the at least one question based on an algorithm optimization model output corresponding to each question and a preset answer corresponding to each question , wherein the basic reward when the algorithm optimization model output matches the preset answer , the basic reward when the algorithm optimization model output does not match the preset answer ; acquiring a length of an output of an algorithm optimization model corresponding to each question and a preset answer length threshold , and based on acquiring an answer length reward of the question ; obtaining a first word generation probability of a next word of different content in the algorithm optimization model output generated by the algorithm optimization student model based on each question, and obtaining a first generation entropy of the algorithm optimization student model based on the first word generation probability; training the algorithm optimization student model based on a reply length reward of each question and the first generation entropy to obtain a calculation reward optimization student model; if it is determined that the calculation reward optimization student model converges, outputting the calculation reward optimization student model.

2. The training method of a student model according to claim 1, wherein Before the step of obtaining the algorithm optimization model output generated by the algorithm optimization student model based on at least one question, the method further comprises the following steps: obtaining a student model, determining the student model as a student model to be optimized, and setting an optimization round to 0; training the student model to be optimized based on a group relative strategy optimization GRPO algorithm to obtain an algorithm optimization student model, and adding 1 to a current optimization round; The base reward of each question is obtained based on an algorithm optimization model corresponding to each question in the at least one question and a preset answer corresponding to each question comprising: If the current optimization round is greater than or equal to the preset training round threshold, and it is determined that the algorithm optimization child model converges, a basic reward of each question is obtained based on the algorithm optimization model output corresponding to each question and the preset answer corresponding to each question .

3. The training method of a student model according to claim 2, wherein The base reward of each question is obtained based on an algorithm optimization model corresponding to each question in the at least one question and a preset answer corresponding to each question Previously, the method further comprises: if the current optimization round is less than a training round threshold or it is determined that the algorithm optimization student model does not converge, obtaining a format reward and an accuracy reward, wherein the format reward is positively correlated with a label matching rate of a label in the algorithm optimization model output and a preset label, and the accuracy reward is obtained based on a preset answer corresponding to the at least one question and the algorithm optimization model output; training the algorithm optimization student model based on the accuracy reward and the format reward to obtain a basic reward optimization student model; determining the basic reward optimization student model as the student model to be optimized, and returning to the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm.

4. The training method of a student model according to claim 1, wherein The step of obtaining a first word generation probability of a next word of different content in the algorithm optimization model output generated by the algorithm optimization student model based on each question, and obtaining a first generation entropy of the algorithm optimization student model based on the first word generation probability, comprises the following steps: Obtain the student model optimized by the algorithm based on the problem. When generating the output of the algorithm optimization model, the content The next word is generated as the first word in the preset vocabulary. The probability of generating the first word of each word The content of Including the output of the algorithm optimization model One word, The preset vocabulary includes A words; based on acquiring content corresponding to the generated entropy ; obtaining the first generation entropy of the algorithm optimization student model based on a generation entropy corresponding to different content in the algorithm optimization model output corresponding to each question.

5. The training method of a student model according to claim 3, wherein Before the step of determining the basic reward optimization student model as the student model to be optimized, the method further comprises the following steps: obtaining a basic reward optimization model output generated by the basic reward optimization student model based on the at least one question; obtaining a reply accuracy rate of the basic reward optimization student model based on a preset answer corresponding to the at least one question and the basic reward optimization model output; The historical model output generated according to the at least one question after at least one training before obtaining the base reward-optimized student model, and the historical answer accuracy of the base reward-optimized student model according to a preset answer corresponding to the at least one question according to the historical model output are obtained. The base reward-optimized student model is determined as the student model to be optimized, including: If the answer accuracy is greater than or equal to the historical answer accuracy, the base reward-optimized student model is determined as the student model to be optimized, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned.

6. The training method of a student model according to claim 5, wherein The base reward-optimized student model is determined as the student model to be optimized, further including: If the answer accuracy is less than the historical answer accuracy, the base reward-optimized student model is determined as the student model to be optimized, the length of the input data and the output data used when training the student model to be optimized is increased, and the step of training the student model to be optimized based on the group relative strategy optimization GRPO algorithm is returned.

7. The training method of a student model according to claim 1, wherein The determination of the convergence of the calculation reward-optimized student model includes: The change degree of the loss function value after multiple trainings before the calculation reward-optimized student model is obtained; The second word generation probability that the next word of different contents in the reward-optimized model output generated by the calculation reward-optimized student model is generated as a different word is obtained, and the second generation entropy of the calculation reward-optimized student model is obtained based on the second word generation probability; If the change degree of the loss function value is less than or equal to the change degree threshold of the loss function value, and the second generation entropy is less than or equal to the generation entropy threshold, it is determined that the calculation reward-optimized student model converges.

8. A training device for a student model, characterized by Including: The model output acquisition module is configured to obtain algorithm optimization model output generated by the algorithm optimization student model based on at least one question, the algorithm optimization student model is obtained by training the student model based on the group relative strategy optimization GRPO algorithm, the student model is obtained by distilling the teacher model, the teacher model is a large language model, the large language model is a deep learning model trained using text data, and is used for processing natural language tasks, the natural language tasks include text question and answer tasks; The basic reward obtaining module is configured to obtain a basic reward of each question in the at least one question based on an algorithm optimization model output corresponding to the each question and a preset answer corresponding to the each question , wherein the basic reward when the algorithm optimization model output matches the preset answer , the basic reward when the algorithm optimization model output does not match the preset answer ; a length reward obtaining module configured to obtain a length of an output of an algorithm optimization model corresponding to each question and a preset answer length threshold , and based on obtain an answer length reward of the question ; The generation entropy acquisition module is configured to obtain the first word generation probability that the next word of different contents in the algorithm optimization model output generated by the algorithm optimization student model according to each question is generated as a different word, and the first generation entropy of the algorithm optimization student model is obtained based on the first word generation probability; The model training module is configured to train the algorithm optimization student model based on the answer length reward of each question and the first generation entropy, and obtain a calculation reward-optimized student model; The model output module is configured to output the calculation reward-optimized student model if it is determined that the calculation reward-optimized student model converges.

9. An electronic device, comprising: A computer program product comprising a memory and a processor; wherein the memory is to store one or more computer instructions, wherein the one or more computer instructions are to be executed by the processor to implement the method of any one of claims 1-7.

10. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions, when executed by a processor, implement the method of any one of claims 1-7.

Images