Method and system for improving code generation capability of large language model based on fine-grained distillation

Abstract

The application discloses a method and system for improving the code generation capability of a large language model based on fine-grained distillation, calculates scores at the token level to control the entire distillation process in a more fine-grained manner, processes a data set in a data set processing module, and then lets a student model generate code and logits and lets a teacher model generate logits, centrally manages Q tables and resources in a proxy module, so that the proxy can be used as a plug-in, first constructs a Q table in the proxy module, then evaluates the code generated by the student model using a test set, updates the Q table according to the evaluation result, and finally dynamically updates the distillation temperature T according to the Q table, and removes the cross-entropy part in the standard distillation loss function in a loss function module, and only keeps the KL divergence part. The application can improve the distillation effect of the code generation large language model, and improve the quality of the generated code.

Description

Technical Field

[0001] This invention belongs to the field of knowledge distillation and code generation, and mainly relates to code generation tasks in natural language processing. Specifically, it relates to a method and system for improving the code generation capability of large language models based on fine-grained distillation. Background Technology

[0002] In recent years, many excellent large language models have been proposed. Some of these are closed-source models, such as ChatGPT and GPT-4. Others are open-source models, such as CodeT5+, CodeGen, CodeLlama, and DeepSeek-Coder-Instruct. While these large language models are powerful, the ever-increasing number of parameters has become a problem. Although closed-source models can be accessed via APIs, they often suffer from high costs, low security, data privacy concerns, and network limitations. Model distillation can alleviate these problems to some extent by extracting knowledge from larger models into smaller, open-source models, thereby improving the performance of the smaller model. In code generation tasks, general knowledge distillation methods often rely on reinforcement learning, using unit test feedback to determine rewards, thus improving the model's distillation effectiveness.

[0003] From sequence-level distillation to internal-level distillation, and now to personalized distillation, many excellent distillation methods have been proposed. Some research has also attempted to extract knowledge from closed-source models to open-source models. However, most of these methods are general and ignore the specific characteristics of the domain. In the case of generating large language models from code, if only general distillation methods are applied to these models, the resulting models may not be optimal, or the distillation effect may be poor. In general distillation methods, the optimal model is usually selected through loss, but we have found that reducing the loss does not necessarily mean improved model performance. Of course, there are also distillation methods that incorporate the characteristics of the code itself. These methods combine reinforcement learning with compiler feedback to improve the quality of code generation. While these methods are effective, they remain at the surface level of application and do not delve into the process of code generation from large-scale models.

[0004] For code generation tasks, a common distillation method is reinforcement learning-guided, adjusting the student model's parameters by evaluating the quality of the code generated. In this case, the state of the reinforcement learning is the generated code fragment x = {x1, x2, ..., x...}. n} (where n is the maximum length), action y = {y1, y2, ..., y} m} (where m is the number of tokens) represents each token in the tokenizer, and the reward function t(x,y) is related to the results of unit tests; for example, if all unit tests pass, the reward function is +1.0. This design seems reasonable, but it doesn't actually consider the code generation process and isn't precise enough to the token level. Therefore, a more nuanced code generation method that can delve into the token layer is needed. Summary of the Invention

[0005] Purpose of the invention: This invention proposes a method and system for improving the code generation capability of large language models based on fine-grained distillation. By dynamically adjusting the distillation temperature T, the code generation capability of the model is improved.

[0006] Summary of the Invention: The present invention discloses a method for improving the code generation capability of large language models based on fine-grained distillation, comprising the following steps:

[0007] We collected various code generation datasets as training datasets. Each sample in the dataset includes a problem, a feasible solution, a function signature, a task ID, and multiple unit tests.

[0008] The loss function is designed based on the student model and the corresponding teacher model; when the word segmenters of the student model and the teacher model are inconsistent, the loss function is the cross-entropy loss function.

[0009] Based on the code generation process, a Q-table is constructed, including a Q-value table and a Q-mapping table. The Q-value table is used to record the scores of the k most likely tokens at each position for a code snippet. The Q-mapping table is used to record the index of the k most likely tokens at each position for a code snippet in the Q-value table.

[0010] Perform unit tests on the actual code generated by the student model, and evaluate the student model based on the results of the unit tests;

[0011] Update the Q table based on the results of unit test evaluation; for each token in the code, find its position in the Q value table according to the Q mapping table, and add the corresponding score if it exists;

[0012] The distillation temperature T is updated based on the Q-value table, and the optimization direction of the model is determined based on the Q-value.

[0013] Furthermore, the problem is what the user wants the model to do; the feasible solution is problem-oriented and meets the user's needs; the task ID is unique and used to identify samples; the function entry point is the expected function name generated by the model; and the multiple unit tests are used to evaluate the code generated by the model.

[0014] Furthermore, the student model is an open-source model; the teacher model is either an open-source model or a closed-source model. When it is an open-source model, it is necessary to ensure that the hardware conditions are sufficient to load the teacher model and generate code, and at the same time, the student model and the teacher model need to have the same word segmenter. When the teacher model is a closed-source model, there is no need to consider the hardware conditions, but it is necessary to obtain the logits generated by the teacher model. If not, the part of the model that uses Q-value optimization will be implemented.

[0015] Furthermore, for cases where the teacher model is an open-source model or where logits can be obtained, the loss function is the KL divergence:

[0016] KL divergence is used to calculate the similarity between the output probabilities of the student model and the teacher model. The calculation formula is as follows:

[0017]

[0018] Where P is the probability distribution of the student model output, Q is the probability distribution of the teacher model output, and p(x) i q(x) is one of the probability distributions output by the student model. i ) is one of the probability distributions output by the teacher model, and n is the number of samples in the training dataset; KL divergence is used to calculate the similarity between the output probabilities of the student model and the teacher model. The greater the similarity, the closer the code generated by the student model is to the teacher model.

[0019] For cases where the teacher model is a closed-source model and logits cannot be obtained, the loss function is the cross-entropy loss function:

[0020]

[0021] The cross-entropy loss function is used to calculate the similarity between the outputs of the student model and the teacher model. The greater the similarity, the closer the code generated by the student model is to that of the teacher model.

[0022] Furthermore, the Q-value table records the score at each position of a token sequence, refining the reward to the token level. First, an (n, m, k) Q-values ​​array is initialized with all values ​​set to 0, where n is the number of samples in the evaluation dataset, m is the length of the generated sequence, and k refers to the number of the first few tokens with the highest probability. Then, an (n, m, k) Q-mappings array is initialized, and based on the evaluation dataset, for each sample, the k most likely tokens at each position are recorded.

[0023] Furthermore, the unit testing of the code actually generated by the student model is as follows:

[0024] For a code snippet actually generated by the student model, if it passes all unit tests, the reward is set to 1.0; if it passes any one unit test, the reward is set to 0.3; if it compiles successfully but fails all unit tests, the reward is set to "-0.3"; if it fails to compile, the reward is set to "-1.0".

[0025] Furthermore, the step of updating the distillation temperature T according to the Q-value table specifically involves:

[0026]

[0027] Where, q old It is the sum of each element in the Q-table after the last update, q is the sum of each element in the Q-table after this update, and T is the distillation temperature. The Q-table corresponds to the top k tokens with the highest probability at each position in the output. A larger Q-value means a better model. If the Q-value increases, T should be reduced appropriately, and if the Q-value decreases, T should be increased appropriately.

[0028] Furthermore, the distillation temperature T is a variable in the KL divergence used to control the smoothness of the probability distribution obtained from the teacher model. A higher T value results in a smoother probability distribution and closer approximations of each probability. The probability distribution of the teacher model is calculated using the following formula:

[0029]

[0030] Among them, logits teacer This is the output of the last fully connected layer in the teacher model. dim = -1 indicates that the object being computed is logits. teacher The last one in the dimension.

[0031] The system for improving code generation capabilities of large language models based on fine-grained distillation, as described in this invention, includes:

[0032] The data processing and prediction module is used to process the code generation dataset and convert it into tokens; it uses the student model to predict the prompt, obtaining logits and the generated code; it uses the teacher model to predict the prompt, obtaining logits; and then it passes the logits obtained by the student and the teacher to the loss function module for training.

[0033] The proxy module is used for centralized management of various resources and acts as a plugin. It includes an evaluation dataset, a Q-value table, and a Q-map table. The evaluation dataset is used to evaluate the effect of distillation after one round of training. The Q-value table records the score at each position of a token sequence, refining the reward to the token level. The Q-map table finds the index of the token at each position in the Q-value table based on the actual generated code.

[0034] Loss function module: updates model parameters; calculates the similarity of output probabilities between the student and teacher models using KL divergence, and adjusts the temperature T in the KL divergence through the proxy module; if the word segmenters of the student and teacher models do not match, the loss function will become cross-entropy only, and the Q value in the proxy will be used to optimize the model.

[0035] Furthermore, the reward is implemented with precision down to the token level as follows:

[0036] First, initialize a (n, m, k) Q-values ​​array with all values ​​set to 0. n is the number of samples in the evaluation dataset, m is the length of the generated sequence, and k refers to the number of the first k most probable tokens. Then, initialize a (n, m, k) Q-mappings array. Based on the evaluation dataset, for each sample, record the first k most likely tokens at each position.

[0037] Beneficial Effects: Compared with existing technologies, the beneficial effects of this invention are as follows: This invention delves into the reward in reinforcement learning at the token level and combines it with the code generation process, alleviating the problem of poor distillation results caused by overly discrete reinforcement learning rewards in previous methods; at the same time, this invention uses a dynamically adjusted T method for distillation, which eliminates the need for users to consider the T value setting, reduces the amount of experimentation, and makes the entire distillation process more stable; furthermore, this invention uses a special Q value to optimize the student model, using a Q-value-based optimization method instead of a loss-based optimization method, which ensures that the best distillation result is preserved. Attached Figure Description

[0038] Figure 1 This is a framework diagram of a system that improves the code generation capability of large language models based on fine-grained distillation;

[0039] Figure 2 This is a schematic diagram of the Q-table structure;

[0040] Figure 3 This is a flowchart of the Q-value table update process. Detailed Implementation

[0041] The invention will now be further described with reference to the accompanying drawings.

[0042] like Figure 1 As shown, this invention provides a method for improving the code generation capability of large language models based on fine-grained distillation, comprising the following steps:

[0043] Step 1: Collect various types of data as training datasets. Each sample includes a problem, a feasible solution, a function entry point, multiple unit tests, and a task ID.

[0044] The problem refers to natural language or a portion of code, such as "Please generate a function that can add two numbers" or "defadd(a,b):". A feasible solution is code that can complete the problem, such as "defadd(a,b):\nreturn a+b". A feasible solution must be correct and pass all unit tests. The function entry point is the function signature corresponding to the problem, such as "add". The function entry point facilitates unit testing. If the problem definition is unclear, the function name in the generated code will also be uncertain, requiring the function name in the generated code to be replaced with the function entry point. Multiple unit tests refer to multiple inputs and outputs for the problem, such as input "35" and output "8", or input "15" and output "6". The task ID is the number corresponding to a piece of data, usually represented by a number, used to uniquely identify a piece of data. The specific data structure is shown in Table 1.

[0045] Table 1 Dataset Structure

[0046]

[0047] Step 2: Collect multiple types of models as student and teacher models. The word segmenters of the student and teacher models should be as consistent as possible because the loss function used for distillation requires logits, which is the output of the last layer of the fully connected layer of the model. If the word segmenters are different, the shape of the logits will be different, so KL divergence cannot be used, and the cross-entropy loss function should be used instead.

[0048] The problem is passed to the student and teacher models, yielding their respective logits. The purpose of obtaining logits is to support the loss function; without logits, the KL divergence component of the loss function will not function. Logits refer to the output of the last layer of the fully connected layer obtained through forward propagation after the model receives the input. In code generation tasks, it represents the score of each token at each position in the output; a positive score indicates that the token is likely to be generated, while a negative score indicates that the token is unlikely to be generated.

[0049] KL divergence is used to calculate the similarity between two probability distributions. The higher the KL divergence, the lower the similarity. The formula is as follows:

[0050]

[0051] Where P is the probability distribution of the student model output, Q is the probability distribution of the teacher model output, p(x) is one of the probability distributions of the student model output, and q(x) is one of the probability distributions of the teacher model output.

[0052] The cross-entropy loss function is used to calculate the similarity between two distributions. The higher the cross-entropy, the lower the similarity. The formula is as follows:

[0053]

[0054] Where p is the distribution of the student model output, q is the distribution of the teacher model output, and p(x) i q(x) is one of the distributions in the student model output. i ) is one of the output distributions of the teacher model.

[0055] Step 3: Design the loss function according to the specific circumstances.

[0056] The design of the loss function is crucial, directly impacting the model's performance after distillation. The default loss function should be KL divergence only, as feasible solutions may not exist in the training dataset, or the feasible solutions may differ significantly from the model's output, leading to reduced generalization ability after distillation. Furthermore, KL divergence only provides a more direct observation of the effect of distillation temperature T, making it better suited for dynamically adjusting T. If the student and teacher models use different segmenters, the loss function should be designed as cross-entropy only. In standard distillation, the loss function consists of cross-entropy and KL divergence, as shown in the following formula:

[0057] loss = a * CrossEntropy(labels) student ,labels)+(1-a)*KL(probs student ,probs teacher )

[0058] Here, CrossEntropy represents the similarity between the labels output by the student model and the labels in the training set. KL divergence represents the similarity between the probability distributions output by the student model and the probability distributions output by the teacher model. 'a' is the weight, ranging from (0,1), used to balance the optimization direction of the model.

[0059] In this invention, the loss function is designed as KL divergence only. This is because in practical applications, there is a significant difference between the labels in the training set and the labels originally output by the model, leading to substantial changes in the model's parameters after distillation, thus reducing generalization ability. KL divergence only can largely alleviate this problem and provides a direct understanding of the effect of distillation temperature T. The formula is as follows:

[0060]

[0061] loss = KL(probs) student ,probs teacher )

[0062] Among them, logits teacher This is the output of the last fully connected layer in the teacher model. dim = -1 indicates that the object being computed is logits. teacher The last dimension, softmax, yields the probability distribution. Temperature T controls the smoothness of the probability distribution; the larger T is, the smoother the distribution and the closer the probability distribution is to the nearest integer.

[0063] Step 4: Based on the code generation process, construct as follows Figure 2 The Q-table is shown below. The Q-table consists of two tables: the Q-value table and the Q-map table. The Q-value table and Q-map table are constructed based on the evaluation dataset. The purpose of the Q-value table is to record the reward of the top k most likely tokens at each position, based on the characteristics of code generation, and to obtain the new distillation temperature T based on the reward. The purpose of the Q-map table is to record the top k most likely tokens at each position based on the evaluation dataset. If the actually generated token exists in the Q-map table, its index is obtained, and the Q-value table is updated.

[0064] Both the Q-value table and the Q-map table should have a structure of (n, m, k), where n is the number of samples in the evaluation dataset, m is the maximum length of a sequence in code generation, and k is used to plan how many tokens with the highest probability will be considered.

[0065] Based on the evaluation dataset, the Q-table is reconstructed. The questions from the evaluation dataset are fed to the student model, and forward propagation is performed to obtain logits. The top k most likely tokens at each position are then obtained from the logits and added to the Q-map table, completing the Q-map table construction. Simultaneously, each value in the Q-value table is initialized to 0.0.

[0066] Step 5: Evaluate the student model based on the evaluation dataset.

[0067] A sample from the evaluation dataset is fed into the student model to generate code. The code is then tested using a unit test set. If the code passes all unit tests, the reward is set to 1.0. If it passes any one unit test, the reward is set to 0.3. If the compilation is successful, the reward is set to -0.3. If the compilation fails, the reward is set to -1.0.

[0068] Step 6: Update the Q-value table based on the evaluation dataset.

[0069] The questions in the evaluation dataset are given to the student model for code generation. The generated code is then serialized using the model's corresponding token segmenter to obtain a token sequence. This token sequence is compared to its corresponding position in the Q-mapping table. If the token exists in the Q-mapping table, its index is obtained, and the Q-value table is updated using the reward. The Q-value table update process is as follows: Figure 3 As shown.

[0070] Step 7: Obtain the new distillation temperature T according to the Q value table.

[0071] The total reward is obtained by summing the elements in the Q-value table, then the new T-value is calculated using a formula, and finally, the T-value is passed to the loss function. The T-value update formula is as follows:

[0072]

[0073] Where, q old The sum of each element in the Q-value table after the last update is q, and the sum of each element in the Q-value table after the current update is q. T is the distillation temperature, and the formula guarantees that T is not less than 0.1. The Q-value and T should be related; if the Q-value increases, T should be decreased to ensure that the top k tokens with the highest probability still appear in the Q-map.

[0074] This invention also proposes a system for improving the code generation capability of large language models based on fine-grained distillation, comprising:

[0075] The data processing and prediction module processes the code generation dataset, transforming it into tokens; it then uses the student model to predict the prompt, obtaining logits and the generated code; it uses the teacher model to predict the prompt, obtaining logits; and finally, it passes the logits obtained by the student and the teacher to the loss function module for training.

[0076] The proxy module is used for centralized management of various resources and acts as a plugin. The module contains resources such as an evaluation dataset, a Q-value table, and a Q-mapping table. The evaluation dataset is used to evaluate the distillation effect after one round of training. The Q-value table records the score at each position in a token sequence, refining the reward to the token level. First, an (n, m, k) Q-values ​​array is initialized with all values ​​set to 0. n is the number of samples in the evaluation dataset, m is the length of the generated sequence, and k refers to the number of the first k most probable tokens, typically set to 50. Then, an (n, m, k) Q-mappings array is initialized. Based on the evaluation dataset, for each sample, the first k most likely tokens at each position are recorded. The Q-mapping table finds the index of the token at each position in the Q-value table based on the actual generated code.

[0077] The loss function module significantly impacts the quality of the code generated after model distillation and is used to update model parameters. The standard distillation loss function is cross-entropy + KL divergence, but its effectiveness is poor because the labels in the training set often don't match the model's output. This method changes the loss function to KL divergence only, adjusting the temperature T in the KL divergence through a proxy module to achieve better distillation results. If the word segmenters of the student and teacher models don't match, the loss function will become cross-entropy only, and the Q-value in the proxy will be used to optimize the model.

[0078] The model obtained after applying this method was evaluated using evaluation datasets, which were validated benchmark datasets, including HumanEval (a code completion dataset containing 164 Python samples), MBPP (a code generation dataset containing 974 Python samples), and MBPP-S (a code completion dataset containing 476 Python samples). Experimental results show that, after applying this method, the distilled model improves the quality of the generated code on all three benchmark datasets compared to the original model. For models with fewer parameters, such as codet5p-220m, there is a 2%–3% improvement in pass@k across all three datasets. Here, pass@k refers to the percentage of k code generated for a given problem, where success is achieved if at least one of the k code passes all unit tests. For models with more parameters, such as deepseek-coder-instruct-1.3B, there is a 7%–10% improvement in pass@k on HumanEval and MBPP-S, and a 4%–5% improvement on MBPP.

[0079] This method was also compared with other distillation methods, and the results showed that it performed well on most models, achieving a 2%–3% improvement in pass@k in most cases. Furthermore, this method eliminates the need for manual adjustment of the distillation temperature T, saving distillation time.

[0080] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for improving the code generation capability of large language models based on fine-grained distillation, characterized in that, The implementation process is as follows: We collected various code generation datasets as training datasets. Each sample in the dataset includes a problem, a feasible solution, a function signature, a task ID, and multiple unit tests. The loss function is designed based on the student model and the corresponding teacher model; when the word segmenters of the student model and the teacher model are inconsistent, the loss function is the cross-entropy loss function. Based on the code generation process, a Q-table is constructed, including a Q-value table and a Q-mapping table. The Q-value table is used to record the scores of the k most likely tokens at each position for a code snippet. The Q-mapping table is used to record the index of the k most likely tokens at each position for a code snippet in the Q-value table. Perform unit tests on the actual code generated by the student model, and evaluate the student model based on the results of the unit tests; Update the Q table based on the results of unit test evaluation; For each token in the code, find its position in the Q-value table according to the Q-mapping table, and if it exists, add the corresponding score; The distillation temperature T is updated based on the Q-value table, and the optimization direction of the model is determined based on the Q-value.

2. The method for improving the code generation capability of large language models based on fine-grained distillation according to claim 1, characterized in that, The problem is what the user wants the model to do; the feasible solution is specific to the problem and meets the user's needs; the task ID is unique and used to identify samples; the function entry point is the expected function name generated by the model; and the multiple unit tests are used to evaluate the code generated by the model.

3. The method for improving the code generation capability of large language models based on fine-grained distillation according to claim 1, characterized in that, The student model is an open-source model; the teacher model can be either an open-source or closed-source model. When it is an open-source model, it is necessary to ensure that the hardware conditions are sufficient to load the teacher model and generate code, and the student model and the teacher model must have the same word segmenter. When the teacher model is a closed-source model, hardware conditions are not required, but the logits generated by the teacher model must be obtained. If not, the part of the model optimized using Q-values ​​will be applied.

4. The method for improving the code generation capability of large language models based on fine-grained distillation according to claim 3, characterized in that, For cases where the teacher model is an open-source model or where logits can be obtained, the loss function is the KL divergence: KL divergence is used to calculate the similarity between the output probabilities of the student model and the teacher model. The calculation formula is as follows: Where P is the probability distribution of the student model output, Q is the probability distribution of the teacher model output, and p(x) i q(x) is one of the probability distributions output by the student model. i ) is one of the probability distributions output by the teacher model, and n is the number of samples in the training dataset; KL divergence is used to calculate the similarity between the output probabilities of the student model and the teacher model. The greater the similarity, the closer the code generated by the student model is to the teacher model. For cases where the teacher model is a closed-source model and logits cannot be obtained, the loss function is the cross-entropy loss function: The cross-entropy loss function is used to calculate the similarity between the outputs of the student model and the teacher model. The greater the similarity, the closer the code generated by the student model is to that of the teacher model.

5. The method for improving the code generation capability of large language models based on fine-grained distillation according to claim 4, characterized in that, The Q-value table records the score at each position in a token sequence, refining the reward to the token level. First, an (n, m, k) Q-values ​​array is initialized with all values ​​set to 0. n is the number of samples in the evaluation dataset, m is the length of the generated sequence, and k refers to the number of the first few tokens with the highest probability. Then, an (n, m, k) Q-mappings array is initialized. Based on the evaluation dataset, for each sample, the k most likely tokens at each position are recorded.

6. The method for improving the code generation capability of large language models based on fine-grained distillation according to claim 1, characterized in that, The unit testing of the code actually generated by the student model is as follows: For a code snippet actually generated by the student model, if it passes all unit tests, the reward is set to 1.0; if it passes any one unit test, the reward is set to 0.3; if it compiles successfully but fails all unit tests, the reward is set to "-0.3"; if it fails to compile, the reward is set to "-1.0".

7. The method for improving the code generation capability of large language models based on fine-grained distillation according to claim 1, characterized in that, The specific steps for updating the distillation temperature T based on the Q-value table are as follows: Where, q old It is the sum of each element in the Q-table after the last update, q is the sum of each element in the Q-table after this update, and T is the distillation temperature. The Q-table corresponds to the top k tokens with the highest probability at each position in the output. A larger Q-value means a better model. If the Q-value increases, T should be reduced appropriately, and if the Q-value decreases, T should be increased appropriately.

8. The method for improving the code generation capability of large language models based on fine-grained distillation according to claim 1, characterized in that, The distillation temperature T is a variable in the KL divergence, used to control the smoothness of the probability distribution obtained from the teacher model. A higher T value results in a smoother probability distribution and closer approximations of each probability. The probability distribution of the teacher model is calculated using the following formula: Among them, logits teacher This is the output of the last fully connected layer in the teacher model. dim = -1 indicates that the object being computed is logits. teacher The last one in the dimension.

9. A system for improving the code generation capability of large language models based on fine-grained distillation using the method described in any one of claims 1 to 8, characterized in that, include: Data processing and prediction module: used to process the code-generated dataset and convert it into tokens; The student model is used to predict the prompt, obtaining logits and the generated code; the teacher model is used to predict the prompt, obtaining logits; and the logits obtained by the student and the teacher are then passed to the loss function module for training. Agent module: Used for centralized management of various resources, acting as a plugin; including evaluation dataset, Q-value table and Q-mapping table; The evaluation dataset is used to evaluate the effect of distillation after one round of training; the Q-value table is used to record the score at each position of a token sequence, refining the reward to the token level; the Q-map table is used to find the index of the token at each position in the Q-value table based on the actual generated code; Loss function module: updates model parameters; calculates the similarity of output probabilities between the student and teacher models using KL divergence, and adjusts the temperature T in the KL divergence through the proxy module; if the word segmenters of the student and teacher models do not match, the loss function will become cross-entropy only, and the Q value in the proxy will be used to optimize the model.

10. The system for improving the code generation capability of large language models based on fine-grained distillation according to claim 9, characterized in that, The reward system, accurate to the token level, is implemented as follows: First, initialize a (n, m, k) Q-values ​​array with all values ​​set to 0. n is the number of samples in the evaluation dataset, m is the length of the generated sequence, and k refers to the number of the first k most probable tokens. Then, initialize a (n, m, k) Q-mappings array. Based on the evaluation dataset, for each sample, record the first k most likely tokens at each position.

Images