Method and system for improving large language model code generation capability based on fine-grained distillation

Abstract

The invention discloses a method and system for improving large language model code generation capability based on fine-grained distillation, and the method comprises the steps: calculating a score on a token level, and controlling a whole distillation process in a finer-grained manner; the data set processing module is used for processing the data set, enabling the student model to generate codes and logits, and enabling the teacher model to generate logits; in the agent module, centralized management is carried out on a Q table and resources, so that an agent can be used as a plug-in; in the agent module, firstly, a Q table is constructed; then evaluating codes generated by the student model by using the test set; updating the Q table according to the evaluation result; finally, the distillation temperature T is dynamically updated according to the Q table; in the loss function module, a cross entropy part in a standard distillation loss function is removed, and only a KL divergence part is reserved. According to the method, the distillation effect of generating the large language model by the code can be improved, and the quality of the generated code is improved.

Description

Technical Field

[0001] The present invention belongs to the field of knowledge distillation and code generation, and mainly relates to code generation tasks in natural language processing, and specifically to a method and system for improving the code generation capability of a large language model based on fine-grained distillation. Background Art

[0002] In recent years, many excellent large language models have been proposed. Some of them are closed source models, such as ChatGPT and GPT-4. Some are open source models, such as CodeT5+, CodeGen, CodeLlama, DeepSeek-Coder-Instruct. Although these large language models are powerful, the increasing number of parameters has become a problem. Although closed source models can be called through interfaces, they often have problems such as high price, low security, data privacy, and network restrictions. Model distillation can alleviate these problems to a certain extent. The knowledge of larger models can be extracted into smaller open source models, thereby improving the performance of smaller models. In code generation tasks, general knowledge distillation methods need to rely on reinforcement learning to determine rewards through feedback from unit tests, thereby improving the distillation effect of the model.

[0003] From sequence-level distillation to internal-level distillation, and now to personalized distillation, many excellent distillation methods have been proposed. There are also some studies that try to extract knowledge from closed-source models into open-source models. However, most of these methods are general and ignore the characteristics of the domain itself. In the case of code generation of large language models, if only general distillation methods are used for these models, the distilled model may not be the optimal model, or the distillation effect is not good. In the general distillation method, the optimal model is generally selected by loss, but we found that the reduction in loss does not mean that the performance of the model has improved. Of course, there are also distillation methods that combine the characteristics of the code itself. These methods combine reinforcement learning with compiler feedback to improve the quality of code generation. Although these methods are effective, they still stay on the surface of the application and do not delve into the process of code generation from large compound models.

[0004] For code generation tasks, the general distillation method is guided by reinforcement learning, which adjusts the parameters of the student model by evaluating the quality of the code generated by the student model. In this case, the state of reinforcement learning is the generated code fragment x = {x 1 ,x 2 ,...,x n}(n is the maximum length), action y={y 1 ,y 2 ,....,y m} (m is the number of tokens) is each token in the tokenizer, and the reward function t(x,y) is related to the result of the unit test. For example, if all unit tests pass, the reward function is +1.0. This design seems reasonable, but it actually does not take into account the code generation process and is not accurate to the token level. Therefore, a surface code generation method that can go deep into the token layer is needed. Summary of the invention

[0005] Purpose of the invention: The present invention proposes a method and system for improving the code generation capability of a large language model based on fine-grained distillation, and improves the code generation capability of the model by dynamically adjusting the distillation temperature T.

[0006] Invention content: The method of improving the code generation capability of a large language model based on fine-grained distillation according to the present invention comprises the following steps:

[0007] Collect various code generation datasets as training datasets. Each sample in the dataset includes a problem, a feasible solution, a function signature, a task ID, and multiple unit tests.

[0008] Design a loss function based on the student model and the corresponding teacher model; when the word segmenters of the student model and the teacher model are inconsistent, the loss function is the cross entropy loss function;

[0009] According to the code generation process, a Q table is constructed, including a Q value table and a Q mapping table; the Q value table is used to record the scores of the most likely k tokens at each position for a code snippet; the Q mapping table is used to record the most likely k tokens at each position for a code snippet, and find their subscripts in the Q value table through the Q mapping table;

[0010] Unit test the code actually generated by the student model, and evaluate the student model based on the results of the unit test;

[0011] Update the Q table based on the results of the unit test evaluation; for each token in the code, find its position in the Q value table based on the Q mapping table, and add the corresponding score if it exists;

[0012] The distillation temperature T is updated according to the Q value table, and the optimization direction of the model is determined according to the Q value.

[0013] Furthermore, the question is what the user wants the model to do; the feasible solution is specific to the problem and meets the needs of the user; the task id is unique and is used to identify the sample; the one function entry is the name of the function expected to be generated by the model; and the multiple unit tests are used to evaluate the code generated by the model.

[0014] Furthermore, the student model is an open source model; the teacher model is an open source model or a closed source model. When it is an open source model, it is necessary to ensure that the hardware conditions are sufficient to load the teacher model and generate code, and at the same time, it is necessary to satisfy that the student model and the teacher model have the same word segmenter; when the teacher model is a closed source model, there is no need to consider the hardware conditions, but the logits generated by the teacher model must be obtained. If not, the part of the model that uses the Q value optimization will take effect.

[0015] Furthermore, when the teacher model is an open source model or logits are available, the loss function is KL divergence:

[0016] KL divergence is used to calculate the similarity of the output probabilities of the student model and the teacher model. The calculation formula is as follows:

[0017]

[0018] Among them, P is the probability distribution of the student model output, Q is the probability distribution of the teacher model output, and p(x i ) is one of the probability distributions output by the student model, q(x i ) is one of the probability distributions output by the teacher model, and n is the number of samples in the training data set; KL divergence is used to calculate the similarity between the output probabilities of the student model and the teacher model. The more similar they are, the closer the code generated by the student model is to the teacher model;

[0019] For the case where the teacher model is a closed-source model and logits cannot be obtained, the loss function is the cross entropy loss function:

[0020]

[0021] The cross entropy loss function is used to calculate the similarity between the outputs of the student model and the teacher model. The more similar they are, the closer the code generated by the student model is to the teacher model.

[0022] Furthermore, the Q-value table records the score at each position of a token sequence, and calculates the reward accurately to the token level. First, a (n, m, k) Q-values ​​array is initialized, and all values ​​are set to 0, where n is the number of samples in the evaluation data set, m is the length of the generated sequence, and k refers to the number of tokens with the highest probability. Then, a (n, m, k) Q-mappings array is initialized, and according to the evaluation data set, for each sample, the first k most likely tokens at each position are recorded.

[0023] Furthermore, the unit test of the code actually generated by the student model is:

[0024] For a code snippet actually generated by the student model, if it can pass all unit tests, the reward is set to 1.0; if it passes any unit test, the reward is set to 0.3; if it can compile successfully but cannot pass all unit tests, the reward is set to "-0.3"; if the compilation fails, the reward is set to "-1.0".

[0025] Further, the updating of the distillation temperature T according to the Q value table is specifically:

[0026]

[0027] Among them, q old It is the sum of each element in the Q value table after the last update of the Q table, q is the sum of each element in the Q value table after the Q table is updated this time, and T is the distillation temperature; the Q table corresponds to the top k tokens with the highest probability at each position in the output. A larger Q value means a better model; if the Q value increases, T should be appropriately reduced, and if the Q value decreases, T should be appropriately increased.

[0028] Furthermore, the distillation temperature T is a variable in the KL divergence, which is used to control the smoothness of the probability distribution obtained by the teacher model. If T is set higher, the probability distribution is smoother and each probability is closer. For the probability distribution of the teacher model, the calculation formula is as follows:

[0029]

[0030] Among them, logits teacer It is the last output of the fully connected layer of the teacher model. Dim = -1 means the object of calculation is logits. teacher The last one in dimension.

[0031] The system for improving the code generation capability of a large language model based on fine-grained distillation according to the present invention comprises:

[0032] Data processing and prediction module: used to process the code generation data set and convert it into token form; and use the student model to predict the prompt to obtain logits and the generated code; use the teacher model to predict the prompt to obtain logits; and then pass the logits obtained by the student and the logits obtained by the teacher to the loss function module for training;

[0033] Proxy module: used to centrally manage various resources and act as a plug-in; including evaluation data sets, Q value tables and Q mapping tables; the evaluation data sets are used to evaluate the effect of distillation after a round of training; the Q value table is used to record the score at each position of a token sequence and accurately calculate the reward to the token level; the Q mapping table is used to find the subscript of the token at each position in the Q value table based on the actual generated code;

[0034] Loss function module: Updates model parameters; uses KL divergence to calculate the similarity of the output probabilities of the student model and the teacher model, and adjusts the temperature T in the KL divergence through the proxy module; if the word segmenters of the student model and the teacher model do not match, the loss function will become only cross entropy, and the Q value in the proxy will be used to optimize the model.

[0035] Furthermore, the reward is accurately implemented at the token level as follows:

[0036] First, initialize an (n, m, k) Q-values ​​array and set all values ​​to 0, where n is the number of samples in the evaluation data set, m is the length of the generated sequence, and k refers to the number of tokens with the highest probability. Then initialize an (n, m, k) Q-mappings array and record the top k most likely tokens at each position for each sample according to the evaluation data set.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: the present invention deepens the rewards in reinforcement learning to the token level, and combines it with the code generation process, alleviating the problem in previous methods that the reinforcement learning rewards are too discrete, resulting in poor distillation effect; at the same time, the present invention uses a dynamic adjustment of T for distillation, which allows users to not consider the T value setting problem, reduces the amount of experiments, and makes the entire distillation process more stable; at the same time, the present invention uses a special Q value to optimize the student model, and uses an optimization method based on Q value instead of an optimization method based on loss, which can ensure that the best distillation result is saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a framework diagram of a system that improves the code generation capability of large language models based on fine-grained distillation;

[0039] Figure 2 It is a schematic diagram of the Q table structure;

[0040] Figure 3 It is the Q value table update flow chart. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, the present invention provides a method for improving the code generation capability of a large language model based on fine-grained distillation, comprising the following steps:

[0043] Step 1: Collect various types of data as training data sets. Each sample includes a problem, a feasible solution, a function entry, multiple unit tests, and a task id.

[0044] A problem refers to natural language or a part of the code, such as "Please generate a function that can add two numbers", or "defadd(a,b):". A feasible solution refers to the code that can solve the problem, such as "defadd(a,b):\nreturn a+b". A feasible solution must be correct and pass all unit tests. Function entry refers to the function signature corresponding to the problem, such as "add". The function entry is used to facilitate unit testing. If the problem is not clearly defined, the function name in the generated code is also uncertain. The function name of the generated code needs to be replaced with the function entry. Multiple unit tests refer to multiple inputs and outputs for the problem, such as the input is "35", the output is "8", the input is "15", and the output is "6". Task id refers to the number corresponding to a piece of data, generally represented by a number, which is used to uniquely identify a piece of data. The specific data structure is shown in Table 1:

[0045] Table 1 Dataset structure

[0046]

[0047] Step 2: Collect multiple types of models as student models and teacher models. The word segmenters of the student model and the teacher model should be as consistent as possible, because the loss function used for distillation requires logits, which is the last layer output of the fully connected layer of the model. If the word segmenters are different, the shape of logits will be different, and KL divergence cannot be used. Instead, the cross entropy loss function should be used.

[0048] Pass the problem to the student model and the teacher model to get the logits of the student model and the logits of the teacher model. The purpose of getting logits is to serve the loss function. If there is no logits, the KL divergence part of the loss function will not work. Logits refers to the last layer output of the fully connected layer obtained by forward propagation after the model receives the input. In the code generation task, it represents the score of each token at each position of the output. If it is a positive score, it means that this token may be generated. If it is a negative score, it means that this token may not be generated.

[0049] KL divergence is used to calculate the similarity between two probability distributions. The higher the KL divergence, the lower the similarity. The formula is as follows:

[0050]

[0051] Among them, P is the probability distribution of the student model output, Q is the probability distribution of the teacher model output, p(x) is one of the probability distributions of the student model output, and q(x) is one of the probability distributions of the teacher model output.

[0052] The cross entropy loss function is used to calculate the similarity between two distributions. The higher the cross entropy, the lower the similarity. The formula is as follows:

[0053]

[0054] Where p is the distribution of the student model output, q is the distribution of the teacher model output, and p(x i ) is one of the distributions output by the student model, q(x i ) is one of the output distributions of the teacher model.

[0055] Step 3: Design a loss function based on the specific situation.

[0056] The design of the loss function is a very important step, which directly affects the effect of the model after distillation. The loss function should be KL divergence only by default, because there may be no feasible solution in the training data set, or the feasible solution is too different from the output of the model, resulting in reduced generalization ability of the model after distillation. And only KL divergence can more intuitively discover the impact of distillation temperature T, and can better apply the method of dynamically adjusting T. If the word segmenters of the student model and the teacher model used are inconsistent, then the loss function should be designed to be cross entropy only. In standard distillation, the loss function consists of cross entropy and KL divergence, and the formula is as follows:

[0057] loss = a*CrossEntropy(labels student ,labels)+(1-a)*KL(probs student ,probs teacher )

[0058] Among them, CrossEntropy is the cross entropy, which represents the similarity between the labels of the student model's true output and the labels in the training set. KL is the KL divergence, which represents the similarity between the probability distribution of the student model's output and the probability distribution of the teacher model's output. a is the weight, between (0,1), which is used to balance the optimization direction of the model.

[0059] In the present invention, the loss function is designed to be only KL divergence, because in practical applications, there is a huge difference between the labels in the training set and the labels originally output by the model, which will cause a huge change in the parameters of the model after distillation, resulting in a decrease in generalization ability. Only KL divergence can alleviate this problem to a large extent, and can intuitively understand the impact of the distillation temperature T. The formula is as follows:

[0060]

[0061] loss = KL(probs student ,probs teacher )

[0062] Among them, logits teacher It is the last output of the fully connected layer of the teacher model. Dim = -1 means the object of calculation is logits. teacher The last dimension, softmax can get the probability distribution, and the temperature T is used to control the smoothness of the probability distribution. The larger T is, the smoother it is and the closer the probability distribution is.

[0063] Step 4: According to the code generation process, construct Figure 2 The Q table shown. The Q table is divided into two tables, one is the Q value table and the other is the Q mapping table. Q value table and Q mapping table. The Q value table and Q mapping table are constructed based on the evaluation data set. The function of the Q value table is to record the rewards of the most likely top k tokens at each position according to the characteristics of code generation, and obtain the new distillation temperature T through the rewards. The function of the Q mapping table is to record the most likely top k tokens at each position according to the evaluation data set. If the actually generated token exists in the Q mapping table, its subscript is obtained and the Q value table is updated.

[0064] The structure of both the Q value table and the Q mapping table should be (n, m, k), where n is the number of samples in the evaluation dataset, m is the maximum length of a sequence in code generation, and k is used to plan how many tokens with the highest probability will be considered.

[0065] Reconstruct the Q table based on the evaluation data set. Give the questions in the evaluation data set to the student model, perform forward propagation, obtain logits, and use logits to obtain the top k most likely tokens at each position, put these tokens into the Q mapping table, and the Q mapping table is constructed. At the same time, initialize each value in the Q value table to 0.0.

[0066] Step 5: Evaluate the student model based on the evaluation dataset.

[0067] Put a sample from the evaluation data set into the student model for code generation, and then use the unit test set to test the code. If the code passes all unit tests, set the reward to 1.0; if it passes any unit test, set the reward to 0.3; if the compilation is successful, set the reward to -0.3; if the compilation fails, set the reward to -1.0.

[0068] Step 6: Update the Q value table based on the evaluation data set.

[0069] The questions in the evaluation data set are given to the student model for code generation. The code generated by the student model is then serialized using the word segmenter corresponding to the model to obtain a token sequence. The token sequence is compared with the corresponding position in the Q mapping table. If the token exists in the Q mapping table, its index is obtained and the reward is used to update the Q value table. The Q value table update process is as follows: Figure 3 shown.

[0070] Step 7: According to the Q value table, obtain the new distillation temperature T.

[0071] Add up each element in the Q value table to get the total reward, then use the formula to get the new T value, and finally pass the T value to the loss function. The T value update formula is as follows:

[0072]

[0073] Among them, q old is the sum of each element in the Q value table after the last update of the Q table, q is the sum of each element in the Q value table after this update of the Q table, T is the distillation temperature, and the formula ensures that T is not less than 0.1. There should be a certain relationship between Q value and T. If the Q value increases, try to reduce T so that the top k tokens with the highest probability still appear in the Q mapping table.

[0074] The present invention also proposes a system for improving the code generation capability of a large language model based on fine-grained distillation, comprising:

[0075] The data processing and prediction module is used to process the code generation data set and convert it into the form of tokens; use the student model to predict the prompt to obtain logits and the generated code; use the teacher model to predict the prompt to obtain logits; and then pass the logits obtained by the student and the logits obtained by the teacher to the loss function module for training.

[0076] The proxy module is used to centrally manage various resources and acts as a plug-in. The module in the code contains resources such as evaluation datasets, Q-value tables, and Q-mapping tables. The evaluation dataset is used to evaluate the effect of distillation after a round of training. The function of the Q-value table is to record the score at each position of a token sequence and to accurately calculate the reward to the token level. First, initialize a (n, m, k) Q-values ​​array and set all values ​​to 0, where n is the number of samples in the evaluation dataset, m is the length of the generated sequence, and k refers to the number of tokens with the highest probability among the first k, which is generally set to 50. Then initialize a (n, m, k) Q-mappings array. According to the evaluation dataset, for each sample, record the first k most likely tokens at each position. The function of the Q mapping table is to find the subscript of the token at each position in the Q-value table based on the actual generated code.

[0077] Loss function module, the loss function has a great impact on the quality of the code generated after the model distillation, and is used to update the model parameters; the loss function in the standard distillation is cross entropy + KL divergence, but because the labels in the training set often do not match the results of the model output, the effect is not good; this method changes the loss function to only KL divergence, and adjusts the temperature T in the KL divergence through the proxy module to achieve a better distillation effect; if the word segmenters of the student model and the teacher model do not match, then the loss function will become only cross entropy, and the Q value in the proxy will be used to optimize the model.

[0078] The model obtained by using this method is evaluated using the evaluation dataset, which is a verified benchmark dataset, including HumanEval, a code completion dataset containing 164 python samples; MBPP, a code generation dataset containing 974 python samples; MBPP-S, a code completion dataset containing 476 python samples. The experimental results show that after using this method, the quality of the code generated by the distilled model is improved compared with the original model on the above three benchmark datasets. If the model is a model with a small number of parameters, such as codet5p-220m, then there is a 2% to 3% improvement in pass@k on the three datasets. Here, pass@k refers to the total percentage of success in the case where the model generates k codes for a problem and one of the k codes can pass all unit tests. If the model is a model with a large number of parameters, such as deepseek-coder-instruct-1.3B, there is a 7% to 10% improvement in pass@k on HumanEval and MBPP-S, and a 4% to 5% improvement on MBPP.

[0079] This method is also compared with other distillation methods. The results show that this method has good results on most models, and in most cases, it has a 2% to 3% improvement in pass@k. In addition, this method does not require manual adjustment of the distillation temperature T, which can save distillation time.

[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for improving the code generation capability of a large language model based on fine-grained distillation, characterized in that: The implementation process is as follows: Collect various code generation datasets as training datasets. Each sample in the dataset includes a problem, a feasible solution, a function signature, a task ID, and multiple unit tests. Design a loss function based on the student model and the corresponding teacher model; when the word segmenters of the student model and the teacher model are inconsistent, the loss function is the cross entropy loss function; According to the code generation process, a Q table is constructed, including a Q value table and a Q mapping table; the Q value table is used to record the scores of the most likely k tokens at each position for a code snippet; the Q mapping table is used to record the most likely k tokens at each position for a code snippet, and find their subscripts in the Q value table through the Q mapping table; Unit test the code actually generated by the student model, and evaluate the student model based on the results of the unit test; Update the Q table based on the results of the unit test evaluation; For each token of the code, find its position in the Q value table according to the Q mapping table, and add the corresponding score if it exists; The distillation temperature T is updated according to the Q value table, and the optimization direction of the model is determined according to the Q value.

2. The method for improving the code generation capability of a large language model based on fine-grained distillation according to claim 1, characterized in that: The question is what the user wants the model to do; the feasible solution is specific to the problem and meets the user's needs; the task id is unique and is used to identify the sample; the one function entry is the name of the function expected to be generated by the model; the multiple unit tests are used to evaluate the code generated by the model.

3. The method for improving the code generation capability of a large language model based on fine-grained distillation according to claim 1, characterized in that: The student model is an open source model; the teacher model is an open source model or a closed source model. When it is an open source model, it is necessary to ensure that the hardware conditions are sufficient to load the teacher model and generate code, and at the same time, it is necessary to ensure that the student model and the teacher model have the same word segmenter; when the teacher model is a closed source model, there is no need to consider the hardware conditions, but the logits generated by the teacher model must be obtained. If not, the part of the model that uses the Q value optimization will take effect.

4. The method for improving the code generation capability of a large language model based on fine-grained distillation according to claim 3, characterized in that: If the teacher model is an open source model or logits are available, the loss function is KL divergence: KL divergence is used to calculate the similarity of the output probabilities of the student model and the teacher model. The calculation formula is as follows: Among them, P is the probability distribution of the student model output, Q is the probability distribution of the teacher model output, and p(x i ) is one of the probability distributions output by the student model, q(x i ) is one of the probability distributions output by the teacher model, and n is the number of samples in the training data set; KL divergence is used to calculate the similarity between the output probabilities of the student model and the teacher model. The more similar they are, the closer the code generated by the student model is to the teacher model; For the case where the teacher model is a closed-source model and logits cannot be obtained, the loss function is the cross entropy loss function: The cross entropy loss function is used to calculate the similarity between the outputs of the student model and the teacher model. The more similar they are, the closer the code generated by the student model is to the teacher model.

5. The method for improving the code generation capability of a large language model based on fine-grained distillation according to claim 4, characterized in that: The Q-value table records the score at each position of a token sequence, and calculates the reward accurately to the token level. First, initialize a (n, m, k) Q-values ​​array and set all values ​​to 0, where n is the number of samples in the evaluation data set, m is the length of the generated sequence, and k refers to the number of tokens with the highest probability. Then initialize a (n, m, k) Q-mappings array and record the top k most likely tokens at each position for each sample according to the evaluation data set.

6. The method for improving the code generation capability of a large language model based on fine-grained distillation according to claim 1, characterized in that: The unit test of the code actually generated by the student model is as follows: For a code snippet actually generated by the student model, if it can pass all unit tests, the reward is set to 1.0; if it passes any unit test, the reward is set to 0.3; if it can compile successfully but cannot pass all unit tests, the reward is set to "-0.3"; if the compilation fails, the reward is set to "-1.0".

7. The method for improving the code generation capability of a large language model based on fine-grained distillation according to claim 1, characterized in that: The specific method of updating the distillation temperature T according to the Q value table is: Among them, q old It is the sum of each element in the Q value table after the last update of the Q table, q is the sum of each element in the Q value table after the Q table is updated this time, and T is the distillation temperature; the Q table corresponds to the top k tokens with the highest probability at each position in the output. A larger Q value means a better model; if the Q value increases, T should be appropriately reduced, and if the Q value decreases, T should be appropriately increased.

8. The method for improving the code generation capability of a large language model based on fine-grained distillation according to claim 1, characterized in that: The distillation temperature T is a variable in the KL divergence, which is used to control the smoothness of the probability distribution obtained by the teacher model. If T is set higher, the probability distribution is smoother and each probability is closer. For the probability distribution of the teacher model, the calculation formula is as follows: Among them, logits teacher It is the last output of the fully connected layer of the teacher model. Dim = -1 means the object of calculation is logits. teacher The last one in dimension.

9. A system for improving the code generation capability of a large language model based on fine-grained distillation using the method as claimed in claims 1 to 8, characterized in that: include: Data processing and prediction module: used to process the code generation data set and convert it into token form; The student model is used to predict the prompt to obtain logits and the generated code; the teacher model is used to predict the prompt to obtain logits; the logits obtained by the student and the logits obtained by the teacher are then passed to the loss function module for training; Agent module: used to centrally manage various resources and act as a plug-in; including evaluation data sets, Q value tables, and Q mapping tables; The evaluation dataset is used to evaluate the effect of distillation after a round of training. The Q-value table is used to record the score at each position of a token sequence and accurately calculate the reward to the token level. The Q mapping table is used to find the subscript of the token at each position in the Q-value table based on the actual generated code. Loss function module: Updates model parameters; uses KL divergence to calculate the similarity of the output probabilities of the student model and the teacher model, and adjusts the temperature T in the KL divergence through the proxy module; if the word segmenters of the student model and the teacher model do not match, the loss function will become only cross entropy, and the Q value in the proxy will be used to optimize the model.

10. The system for improving code generation capability of large language models based on fine-grained distillation according to claim 9, characterized in that: The rewards are accurately implemented at the token level as follows: First, initialize an (n, m, k) Q-values ​​array and set all values ​​to 0, where n is the number of samples in the evaluation data set, m is the length of the generated sequence, and k refers to the number of tokens with the highest probability. Then initialize an (n, m, k) Q-mappings array and record the top k most likely tokens at each position for each sample according to the evaluation data set.

Images