Code generation task-oriented multilayer structured pruning method and device

Abstract

The embodiment of the invention provides a multi-layer structured pruning method and device for a code generation task. The multi-layer structured pruning method for the code generation task comprises the steps that a to-be-pruned model and a pruning target are determined; determining a use vocabulary based on the programming corpus, performing vocabulary pruning on the to-be-pruned model based on the use vocabulary, and determining a first pruning model; determining a target layer based on the first pruning model, determining a first pruning influence based on the target layer and a pruning target, performing layer selection pruning based on the first pruning influence, and determining a second pruning model; determining a target neuron based on the second pruning model, determining a second pruning influence based on the target neuron and the pruning target, performing feedforward network pruning based on the second pruning influence, and determining a third pruning model; and performing performance recovery based on the third pruning model, and determining a target model. Therefore, the model structure is optimized from multiple dimensions, multi-dimensional pruning is realized in the field of code generation, and the model performance is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and particularly to a multi-layer structured pruning method for code generation tasks. Background Art

[0002] In recent years, large language models (LLMs) have performed excellently in many fields and have been widely deployed, and software engineering is no exception. LLMs perform well in tasks such as code generation, code summarization, and vulnerability detection. However, the huge scale and intensive computational requirements of these models pose challenges in resource-constrained environments. In addition, the energy consumption associated with training and inference leads to high carbon emissions, raising concerns about environmental sustainability.

[0003] To improve energy efficiency and sustainability, green software engineering has promoted the exploration of model pruning, quantization, and knowledge distillation, aiming to reduce the computational and environmental impact of LLMs. Quantization accelerates inference by converting high-precision weights to low-precision. Knowledge distillation transfers knowledge from a larger model to a smaller model, which requires additional computational resources. Among them, model pruning stands out as a promising strategy, and there are two main methods: one is unstructured pruning, which only zeros specific weights while maintaining the original number of parameters; the other is structured pruning, which effectively reduces the model size by removing entire structural components (such as neurons, layers) while retaining the integrity and functionality of the model. For example, the unstructured pruning method SparseGPT targets individual weights, achieving sparsity but not significantly reducing hardware requirements, so its use is limited in constrained environments. In contrast, the structured pruning method ShortGPT improves operation efficiency while maintaining model integrity by reducing the number of parameters and hardware dependence.

[0004] Although these pruning methods can theoretically be applied to code generation models, through careful analysis, three basic limitations of the existing methods have been identified: (1) Inconsistent pruning objectives: Current pruning methods mainly focus on inter-layer similarity metrics (such as angular distance, cosine similarity, and Taylor score between layers), aiming to retain the general language modeling ability of the model while ignoring the specific requirements of downstream tasks. (2) Limited pruning scope: Existing methods usually adopt a single-component pruning strategy (such as only focusing on reducing layer redundancy), and fail to utilize the potential synergistic effects that can be achieved by an integrated multi-granularity pruning method across different model components. (3) Lack of post-adjustment methods for code: Existing pruning methods rely on general supervised fine-tuning on downstream datasets to restore performance, without considering domain-specific adaptation for code-related tasks. This general method fails to utilize the unique characteristics and requirements of the code generation task, which may limit the effectiveness of the post-pruning recovery process.

[0005] Therefore, a better solution is urgently needed. Summary of the Invention

[0006] In view of this, embodiments of this specification provide a multi-layer structured pruning method for code generation tasks. One or more embodiments of this specification also relate to a multi-layer structured pruning apparatus for code generation tasks, a computing device, a computer-readable storage medium, and a computer program, so as to solve the technical defects existing in the prior art.

[0007] According to the first aspect of the embodiments of this specification, a multi-layer structured pruning method for code generation tasks is provided, including: Determine the model to be pruned and the pruning target; Determine the used vocabulary based on the programming corpus, perform vocabulary pruning on the model to be pruned based on the used vocabulary, and determine the first pruned model; Determine the target layer based on the first pruned model, determine the first pruning impact based on the target layer and the pruning target, perform layer selection pruning based on the first pruning impact, and determine the second pruned model; Determine the target neurons based on the second pruned model, determine the second pruning impact based on the target neurons and the pruning target, perform feedforward network pruning based on the second pruning impact, and determine the third pruned model; Perform performance recovery based on the third pruned model to determine the target model.

[0008] In a possible implementation, the pruning target includes:

[0009]

[0010] where θ original and θ pruned respectively represent the parameters of the model to be pruned and the target model, x represents the model input, y i represents the generated code of the model, KL represents the Kullback-Leibler divergence between their output probability distributions, represents the pruning target loss function.

[0011] In a possible implementation, performing vocabulary pruning on the model to be pruned based on the used vocabulary to determine the first pruned model includes: Determine the usage frequency based on the used vocabulary; Perform vocabulary pruning on the model to be pruned based on the usage frequency to determine the first pruned model.

[0012] In a possible implementation, determining the first pruning impact based on the target layer and the pruning target, and performing layer selection pruning based on the first pruning impact to determine the second pruned model includes: Remove the target layer from the first pruning model to determine the first test model, where the target layer is any layer in the first pruning model. Determine the first pruning impact based on the first test model and the pruning target. Determine the first target pruning impact from the first pruning impact. Delete the target layer corresponding to the first target pruning impact to determine the second pruning model.

[0013] In a possible implementation, determine the second pruning impact based on the target neuron and the pruning target, and perform feed-forward network pruning based on the second pruning impact to determine the third pruning model, including: Determine the heuristic rule, where the heuristic rule is a processing rule for the target neuron, and the target neuron is any neuron in the second pruning model. Remove the target neuron from the second pruning model based on the heuristic rule to determine the second test model. Determine the second pruning impact based on the second test model and the pruning target. Determine the second target pruning impact from the second pruning impact. Delete the target neuron corresponding to the second target pruning impact to determine the third pruning model.

[0014] In a possible implementation, perform performance recovery based on the third pruning model to determine the target model, including: Obtain the training data of the model to be pruned. Train the third pruning model based on the training data to determine the target model.

[0015] In a possible implementation, it further includes: Determine at least two code intelligent tasks. Evaluate the performance of the target model based on at least two code intelligent tasks to determine the evaluation result. Wherein, the evaluation result includes the efficiency evaluation result and the robustness evaluation result.

[0016] According to the second aspect of the embodiments of the present specification, a multi-layer structured pruning device for code generation tasks is provided, including: A target determination module configured to determine the model to be pruned and determine the pruning target. A vocabulary pruning module configured to determine the used vocabulary based on the programming corpus, and perform vocabulary pruning on the model to be pruned based on the used vocabulary to determine the first pruning model. A layer selection pruning module configured to determine the target layer based on the first pruning model, determine the first pruning impact based on the target layer and the pruning target, and perform layer selection pruning based on the first pruning impact to determine the second pruning model. A neuron pruning module, configured to determine target neurons based on a second pruning model, determine second pruning impacts based on the target neurons and pruning objectives, perform feed-forward network pruning based on the second pruning impacts, and determine a third pruning model; A performance recovery module, configured to perform performance recovery based on the third pruning model and determine a target model.

[0017] According to a third aspect of the embodiments of the present specification, a computing device is provided, including: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above multi-layer structured pruning method for code generation tasks are implemented.

[0018] According to a fourth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the above multi-layer structured pruning method for code generation tasks are implemented.

[0019] According to a fifth aspect of the embodiments of the present specification, a computer program is provided. When the computer program is executed on a computer, the computer is made to execute the steps of the above multi-layer structured pruning method for code generation tasks.

[0020] The embodiments of the present specification provide a multi-layer structured pruning method and device for code generation tasks. The multi-layer structured pruning method for code generation tasks includes: determining a model to be pruned and determining pruning objectives; determining used vocabulary based on a programming corpus, performing vocabulary pruning on the model to be pruned based on the used vocabulary, and determining a first pruning model; determining a target layer based on the first pruning model, determining first pruning impacts based on the target layer and pruning objectives, performing layer selection pruning based on the first pruning impacts, and determining a second pruning model; determining target neurons based on the second pruning model, determining second pruning impacts based on the target neurons and pruning objectives, performing feed-forward network pruning based on the second pruning impacts, and determining a third pruning model; performing performance recovery based on the third pruning model and determining a target model. Thus, the model structure is optimized from multiple dimensions, multi-dimensional pruning is achieved in the field of code generation, and the model performance is improved. Description of the Drawings

[0021] Figure 1 is a flowchart of a multi-layer structured pruning method for code generation tasks provided by an embodiment of the present specification; Figure 2 is an architecture diagram of a multi-layer structured pruning method for code generation tasks provided by an embodiment of the present specification; Figure 3It is a schematic structural diagram of a multi-layer structured pruning device for code generation tasks provided by an embodiment of this specification; Figure 4 It is a structural block diagram of a computing device provided by an embodiment of this specification. Detailed implementation manners

[0022] In the following description, many specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the connotation of this specification. Therefore, this specification is not limited by the specific implementations disclosed below.

[0023] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first can also be referred to as the second, and similarly, the second can also be referred to as the first. Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining".

[0025] This application proposes a multi-layer structured pruning method for code generation tasks, which reduces the high computational requirements and energy consumption problems of large language models in code generation tasks. Traditional methods fail to utilize the unique characteristics and requirements of code generation tasks, which may limit the effectiveness of pruning techniques. The technology developed in this application not only significantly reduces the computational and environmental costs of the model, but also maintains the high performance and high robustness of the model in code generation tasks. Its innovation points include: First, a unified structured pruning method is proposed, which combines vocabulary, layer, and FFN pruning to optimize the model structure from multiple dimensions, and multi-dimensional pruning is realized in the code generation field for the first time; Second, aiming at minimizing the KL divergence between the pruned model and the original model, the code generation behavior is directly optimized, which is more in line with the task requirements; Third, a customized code instruction data strategy is introduced, and by replacing the code in the training data, the performance recovery efficiency of the pruned model is improved, which is better than the traditional fine-tuning method. This application provides an efficient solution for model pruning in resource-constrained environments.

[0026] In this specification, a multi-layer structured pruning method for code generation tasks is provided. This specification also relates to a multi-layer structured pruning device for code generation tasks, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0027] See Figure 1 , Figure 1 which shows a flowchart of a multi-layer structured pruning method for code generation tasks according to an embodiment of this specification, specifically including the following steps.

[0028] Step 101: Determine the model to be pruned and the pruning target; In practical applications, determine the pruning target with the goal of minimizing the KL divergence between the pruned model and the original model, ensuring that the output probability distribution of the pruned model remains similar to that of the original model in code generation tasks.

[0029] In a possible implementation, the pruning target includes:

[0030]

[0031] where θ original and θ pruned represent the parameters of the model to be pruned and the target model respectively, x represents the model input, y i represents the code generated by the model, KL represents the Kullback-Leibler divergence between their output probability distributions, represents the pruning target loss function.

[0032] In practical applications, determine the pruning target with the goal of minimizing the KL divergence between the pruned model and the original model, ensuring that the output probability distribution of the pruned model remains similar to that of the original model in code generation tasks.

[0033] Specifically, the pruning target can be expressed as , where θ original and θ pruned represent the parameters of the original model and the pruned model respectively, x represents the model input, y i represents the code generated by the model, KL represents the Kullback-Leibler divergence between their output probability distributions, represents the pruning target loss function; Formally, the KL divergence can be expressed as: , this goal ensures that the pruned model retains the token generation probability of the original model and directly optimizes the code generation behavior of the model.

[0034] Step 102: Determine the used vocabulary based on the programming corpus, perform vocabulary pruning on the model to be pruned based on the used vocabulary, and determine the first pruned model.

[0035] In practical applications, refer to Figure 2 , by analyzing the vocabulary actually used in the given programming corpus, remove those words that do not appear, and reduce the embedding matrix and output layer weights of the model to adapt to the pruned vocabulary list.

[0036] In a possible implementation, performing vocabulary pruning on the model to be pruned based on the used vocabulary to determine the first pruned model includes: determining the usage frequency based on the used vocabulary; performing vocabulary pruning on the model to be pruned based on the usage frequency to determine the first pruned model.

[0037] In practical applications, by analyzing the vocabulary actually used in the given programming corpus, remove those words that do not appear, thereby reducing the embedding size of the model.

[0038] Specifically, given a vocabulary list V and a usage statistic C, the pruned vocabulary list V′ is defined as: , where is a threshold, indicating the minimum frequency that a token must have to be retained in the vocabulary list. Accordingly, the embedding matrix is reduced to , removing unnecessary token embeddings and reducing the complexity of the model. At the same time, the output layer weight is also reduced to .

[0039] Step 103: Determine the target layer based on the first pruned model, determine the first pruning impact based on the target layer and the pruning goal, and perform layer selection pruning based on the first pruning impact to determine the second pruned model.

[0040] In practical applications, refer to Figure 2 , evaluate the redundancy between layers in the model, and find and remove the layer that has the least impact on the model output through an iterative method.

[0041] In a possible implementation, determining the first pruning impact based on the target layer and the pruning goal, and performing layer selection pruning based on the first pruning impact to determine the second pruned model includes: removing the target layer from the first pruned model to determine the first test model; where the target layer is any layer in the first pruned model; determining the first pruning impact based on the first test model and the pruning goal; determining the first target pruning impact from the first pruning impact; deleting the target layer corresponding to the first target pruning impact to determine the second pruned model.

[0042] In practical applications, remove one layer from the model; calculate the KL divergence between the pruned model and the original model; select the layer with the least impact on the KL divergence for removal; repeat the above steps until the predetermined number of pruned layers is reached.

[0043] Step 104: Determine the target neurons based on the second pruned model, determine the second pruning impact based on the target neurons and the pruning objective, perform feed-forward network pruning based on the second pruning impact, and determine the third pruned model.

[0044] In practical applications, refer to Figure 2 , for feed-forward network (FFN) pruning, determine which neurons can be removed based on heuristic rules (such as retaining the top, bottom, middle neurons or random selection), where the best pruning rule is selected by calculating the similarity between the pruned model and the original model.

[0045] In one possible implementation, determining the second pruning impact based on the target neurons and the pruning objective, performing feed-forward network pruning based on the second pruning impact, and determining the third pruned model includes: determining a heuristic rule; where the heuristic rule is a processing rule for the target neurons, and the target neurons are any neurons in the second pruned model; removing the target neurons from the second pruned model based on the heuristic rule to determine the second test model; determining the second pruning impact based on the second test model pair and the pruning objective; determining the second target pruning impact from the second pruning impact; deleting the target neurons corresponding to the second target pruning impact to determine the third pruned model.

[0046] In practical applications, select a heuristic rule (such as retaining the top K neurons); remove the selected neurons; calculate the similarity between the pruned model and the original model; select the pruning rule with the highest similarity.

[0047] Step 105: Perform performance recovery based on the third pruned model to determine the target model.

[0048] In practical applications, use a customized code instruction data strategy (replacing the code generated by the original model with the code in the training dataset), perform post-training on the pruned model, and use the LoRA technique for efficient parameter adjustment to recover the model performance.

[0049] In one possible implementation, performing performance recovery based on the third pruned model to determine the target model includes: obtaining the training data of the model to be pruned; training the third pruned model based on the training data to determine the target model.

[0050] In practical applications, evaluate the performance of the original model and the pruned model on the training set; replace the code generated by the original model with the code in the training dataset; use the LoRA technique to perform post-training on the pruned model and adjust the model parameters to recover the performance.

[0051] In one possible implementation, it further includes: determining at least two code intelligence tasks; evaluating the performance of the target model based on the at least two code intelligence tasks to determine the evaluation result; wherein, the evaluation result includes an efficiency evaluation result and a robustness evaluation result.

[0052] In practical applications, comprehensively evaluate the pruned model on multiple code intelligence tasks (such as code generation, CoT generation, and code output prediction), and analyze the performance of the pruned model in terms of efficiency (such as storage requirements, GPU usage, computational efficiency, and environmental impact) and robustness.

[0053] Specifically, evaluate the performance of the pruned model on multiple tasks; analyze the improvements of the pruned model in terms of storage requirements, GPU usage, computational efficiency, and environmental impact; evaluate the robustness of the pruned model under different perturbation scenarios.

[0054] In one embodiment, the optimal parameter settings of the multi-layer structured pruning method for code generation tasks are as shown in Table 1 below. Table 1 is the hyperparameter setting, and Hyper-parameter is the hyperparameter: Table 1

[0055] Compared with the prior art, the beneficial effects of this application are as follows: A multi-layer structured pruning technique for code generation tasks proposed in this application first proposes a unified structured pruning method, combines vocabulary, layer, and FFN pruning, optimizes the model structure from multiple dimensions, and realizes multi-dimensional pruning in the field of code generation for the first time; secondly, with the goal of minimizing the KL divergence between the pruned model and the original model, it directly optimizes the code generation behavior, which is more in line with the task requirements; then it introduces a customized code instruction data strategy, and improves the performance recovery efficiency of the pruned model by replacing the code in the training data, which is superior to the traditional fine-tuning method.

[0056] Furthermore, to evaluate the effectiveness of the proposed solution in reducing the model size while maintaining performance, a series of experiments were designed to compare the proposed solution with other structured pruning methods and the original dense model. Five representative pruning methods, including ShortGPT, UIDL, Linearity, SLEB, and LLM-pruner were considered as baselines. Each baseline method has its unique focus and explores different aspects of model pruning. To ensure fairness, the number of model parameters of each baseline method after pruning was controlled to be similar to that of the proposed solution. The data in the table measures the model performance using Pass@1, BLEU-4, and Exact Match (EM) metrics. HE represents HumanEval, OE represents OpenEval, CH represents CodeHarmony, Dense is the original dense model, FT is the traditional fine-tuning method, PT is the post-training method proposed in this application, and Flab-Pruner is the method of this application.

[0057] Table 2

[0058] Through experiments, Table 2 shows the performance comparison table of the method in this embodiment and other baseline methods, where the values are in percentages. The multi-layer structured pruning technology for code generation tasks proposed in this embodiment can better retain the model performance compared with the baseline methods. Specifically, compared with the single-component pruning method, the method of this application performs best in code generation tasks. Taking HumanEval as an example, the Pass@1 of the method of this application on the CodeQwen model is 75.00%, while the best Pass@1 of other baseline methods is 42.68%, and the relative performance improvement is 75.73%. In the inference generation task, the method of this application also performs excellently. Its BLEU-4 on the CodeQwen model is 31.42%, while the best BLEU-4 of other baseline methods is 15.92%, and the relative performance improvement is 97.98%. For the code output prediction task, the method of this application is also superior to other methods. Taking Crux-O as an example, the EM of the method of this application on the CodeQwen model is 31.75%, while the best EM of other baseline methods is 27.63%, and the relative performance improvement is 14.93%.

[0059] Advantages of the performance recovery strategy: Compared with traditional fine-tuning methods, the proposed method can achieve better performance recovery. As shown in Table 2, in all tasks of all three models, the method of this application after post-training has achieved the best performance. Taking HumanEval as an example, the Pass@1 of the method of this application on the CodeQwen model is 78.05%, while the best Pass@1 of the traditional fine-tuning method is 76.22%, and the relative performance improvement is 2.40%. This shows the efficiency of the method and also shows the strong competitiveness of the method proposed in this embodiment.

[0060] Furthermore, the improvements of the method of this application in terms of storage requirements, GPU usage, computing efficiency, and environmental impact are verified. The specific contents are as follows: (1) First, compare the efficiency of the pruned model with the original model, focusing on key metrics such as GPU utilization rate, speed, carbon dioxide emissions, and FLOPs. These metrics are crucial for practical deployment considerations in software engineering. Among them, the GPU utilization rate and the number of tokens processed per second are calculated by gpu_poor10, the carbon dioxide emissions are calculated using codecarbon11, and the FLOPs are calculated using Calflops12; (2) Further consider comparing the efficiency of the pruned model with the original model under different precisions, focusing on BF16, FP8, and INT4 (INT4 is implemented through AutoAWQ).

[0061] Table 3

[0062] Experiments show that, as shown in Table 3, Table 3 is the improvement table of the method of this embodiment in terms of efficiency under three precisions. The method of this application is significantly better than the original dense model in many aspects.

[0063] GPU utilization rate: The original dense model requires 13.55G of GPU memory during inference, while the method of this application (whether post-trained or not) only requires 10.72G. This means that the FP16 precision model that could originally only run on devices with more than 12G of GPU memory can run on GPUs with less than 12G after pruning. Further, when using INT4 precision, the GPU utilization rate of the method of this application is only 3.02G, enabling it to run on a GPU with 4G of memory.

[0064] Number of tokens processed per second (Token / s): The dense model can process 30 tokens per second, while the throughput of the method of this application reaches 38 tokens / second, indicating that it is more efficient in using computing resources.

[0065] Carbon dioxide emissions (CO2): When evaluating HumanEval, the carbon dioxide emissions of the dense model were 2.14 grams, while the emissions of the method of this application were 1.84 grams, which reflects the advantage of the method of this application in terms of environmental friendliness.

[0066] Floating-point operations (FLOPs): The dense model requires 7.04 trillion floating-point operations during inference, while the computational intensity of the method of this application has been reduced to 5.64 trillion FLOPs.

[0067] Generally speaking, the method of this application is significantly superior to the original dense model in many aspects. This further demonstrates the improvements of the method of this application in terms of storage requirements, GPU usage, computational efficiency, and environmental impact.

[0068] Furthermore, in order to comprehensively evaluate the robustness of the method of this application, a code generation task was designed as a key benchmark to further verify the robustness of the method of this application under different perturbation scenarios. The specific contents are as follows: (1) Four token-level perturbation methods were constructed in HumanEval: Format, Func_name, Natgen, and Nlaugenter. Each perturbation method was specifically designed to detect the model's resistance to specific types of noise, so as to comprehensively evaluate the model's robustness when facing real-world interference from multiple angles.

[0069] (1-1) Format: This perturbation method introduces noise into the code format, such as inserting line breaks or replacing space indents with tabs; (1-2) Func_name: This perturbation method changes the function names in the code, including applying character-level or word-level natural text transformations to component words; (1-3) Natgen: This perturbation method introduces code syntax noise, such as inserting useless code or swapping operands; (1-4) Nlaugenter: This perturbation method introduces natural language noise into the docstring, such as applying synonym replacement or back-translation.

[0070] Table 4

[0071] After experiments, Table 4 is a comparison table of the robustness of the method in this embodiment and the dense model. The values are in percentages, indicating that the performance of the pruned model slightly decreases compared to the dense model under some label-level perturbation conditions. Under the four perturbation methods, the performance of the three pruned models is slightly lower than that of the dense model, and the maximum decrease is less than 10%. However, under certain perturbations, the performance of the pruned model after post-training even exceeds that of the dense model. Taking the CodeQwen model as an example, after post-training, under function name perturbation, the performance of the method of this application is 77.44%, while the performance of the dense model is 75.61%. This shows that the method of this application performs excellently in terms of robustness and can maintain or even improve the model performance under certain interferences.

[0072] The embodiments of this specification provide a multi-layer structured pruning method and device for code generation tasks. The multi-layer structured pruning method for code generation tasks includes: determining the model to be pruned and determining the pruning target; determining the used vocabulary based on the programming corpus, pruning the model to be pruned based on the used vocabulary, and determining the first pruned model; determining the target layer based on the first pruned model, determining the first pruning impact based on the target layer and the pruning target, performing layer selection pruning based on the first pruning impact, and determining the second pruned model; determining the target neurons based on the second pruned model, determining the second pruning impact based on the target neurons and the pruning target, performing feed-forward network pruning based on the second pruning impact, and determining the third pruned model; performing performance recovery based on the third pruned model and determining the target model. Thus, the model structure is optimized from multiple dimensions, multi-dimensional pruning is achieved in the field of code generation, and the model performance is improved.

[0073] Corresponding to the above method embodiments, this specification also provides embodiments of a multi-layer structured pruning device for code generation tasks. Figure 3 Fig. shows a schematic structural diagram of a multi-layer structured pruning device for code generation tasks provided by an embodiment of this specification. As Figure 3 shown, the device includes: A target determination module 301, configured to determine the model to be pruned and determine the pruning target; A vocabulary pruning module 302, configured to determine the used vocabulary based on the programming corpus, prune the model to be pruned based on the used vocabulary, and determine the first pruned model; A layer selection pruning module 303, configured to determine the target layer based on the first pruned model, determine the first pruning impact based on the target layer and the pruning target, perform layer selection pruning based on the first pruning impact, and determine the second pruned model; A neuron pruning module 304, configured to determine the target neurons based on the second pruned model, determine the second pruning impact based on the target neurons and the pruning target, perform feed-forward network pruning based on the second pruning impact, and determine the third pruned model; The performance recovery module 305 is configured to perform performance recovery based on the third pruning model and determine a target model.

[0074] In a possible implementation, the pruning objectives include:

[0075]

[0076] where θ original and θ pruned respectively represent the parameters of the model to be pruned and the target model, x represents the model input, y i represents the code generated by the model, KL represents the Kullback-Leibler divergence between their output probability distributions, represents the pruning objective loss function.

[0077] In a possible implementation, determining a first pruning model based on pruning the model to be pruned using vocabulary includes: Determining the usage frequency based on the vocabulary used; Pruning the model to be pruned based on the usage frequency to determine the first pruning model.

[0078] In a possible implementation, determining a first pruning impact based on the target layer and the pruning objective, and performing layer selection pruning based on the first pruning impact to determine a second pruning model includes: Removing the target layer from the first pruning model to determine a first test model; where the target layer is any layer in the first pruning model; Determining a first pruning impact based on the first test model and the pruning objective; Determining a first target pruning impact from the first pruning impact; Deleting the target layer corresponding to the first target pruning impact to determine the second pruning model.

[0079] In a possible implementation, determining a second pruning impact based on the target neuron and the pruning objective, and performing feedforward network pruning based on the second pruning impact to determine a third pruning model includes: Determining a heuristic rule; where the heuristic rule is a processing rule for the target neuron, and the target neuron is any neuron in the second pruning model; Removing the target neuron from the second pruning model based on the heuristic rule to determine a second test model; Determining a second pruning impact based on the second test model and the pruning objective; Determining a second target pruning impact from the second pruning impact; Deleting the target neuron corresponding to the second target pruning impact to determine the third pruning model.

[0080] In a possible implementation, performance recovery is performed based on a third pruning model to determine a target model, including: Obtain the training data of the model to be pruned; Train the third pruning model based on the training data to determine the target model.

[0081] In a possible implementation, it further includes: Determine at least two code intelligent tasks; Evaluate the performance of the target model based on at least two code intelligent tasks to determine an evaluation result; Among them, the evaluation result includes an efficiency evaluation result and a robustness evaluation result.

[0082] The embodiments of the present specification provide a multi-layer structured pruning method and device for code generation tasks. The multi-layer structured pruning device for code generation tasks includes: determining a model to be pruned and a pruning target; determining used vocabulary based on a programming corpus, and performing vocabulary pruning on the model to be pruned based on the used vocabulary to determine a first pruning model; determining a target layer based on the first pruning model, determining a first pruning impact based on the target layer and the pruning target, and performing layer selection pruning based on the first pruning impact to determine a second pruning model; determining target neurons based on the second pruning model, determining a second pruning impact based on the target neurons and the pruning target, and performing feed-forward network pruning based on the second pruning impact to determine a third pruning model; performing performance recovery based on the third pruning model to determine the target model. Thus, the model structure is optimized from multiple dimensions, multi-dimensional pruning is achieved in the field of code generation, and the model performance is improved.

[0083] The above is a schematic solution of a multi-layer structured pruning device for code generation tasks in this embodiment. It should be noted that the technical solution of the multi-layer structured pruning device for code generation tasks belongs to the same concept as the technical solution of the above multi-layer structured pruning method for code generation tasks. For the details not described in the technical solution of the multi-layer structured pruning device for code generation tasks, reference can be made to the description of the technical solution of the above multi-layer structured pruning method for code generation tasks.

[0084] Figure 4 FIG. shows a structural block diagram of a computing device 400 according to an embodiment of the present specification. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 through a bus 430, and a database 450 is used to store data.

[0085] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of such networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface.

[0086] In one embodiment of the present specification, the above components of the computing device 400, as well as Figure 4 other components not shown, may also be connected to each other, for example, via a bus. It should be understood that Figure 4 the block diagram of the computing device shown is for illustrative purposes only and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0087] The computing device 400 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.) or other types of mobile devices, or a stationary computing device such as a desktop computer or a Personal Computer (PC). The computing device 400 can also be a mobile or stationary server.

[0088] Among them, the processor 420 is used to execute the following computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned multi-layer structured pruning method for code generation tasks are implemented. The above is a schematic solution of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-mentioned multi-layer structured pruning method for code generation tasks belong to the same concept. For the details not described in detail in the technical solution of the computing device, reference can be made to the description of the technical solution of the above-mentioned multi-layer structured pruning method for code generation tasks.

[0089] An embodiment of this specification also provides a computer-readable storage medium that stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the above-mentioned multi-layer structured pruning method for code generation tasks are implemented.

[0090] The above is a schematic solution of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-mentioned multi-layer structured pruning method for code generation tasks belong to the same concept. For the details not described in detail in the technical solution of the storage medium, reference can be made to the description of the technical solution of the above-mentioned multi-layer structured pruning method for code generation tasks.

[0091] An embodiment of this specification also provides a computer program. When the computer program is executed on a computer, the computer is made to execute the steps of the above-mentioned multi-layer structured pruning method for code generation tasks.

[0092] The above is a schematic solution of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the above-mentioned multi-layer structured pruning method for code generation tasks belong to the same concept. For the details not described in detail in the technical solution of the computer program, reference can be made to the description of the technical solution of the above-mentioned multi-layer structured pruning method for code generation tasks.

[0093] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] The computer instructions include computer program code, which may be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, removable hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0095] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of this specification are not limited by the described action sequence, because according to the embodiments of this specification, some steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0096] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0097] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The alternative embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. The embodiments selected and specifically described in this specification are to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can understand and utilize this specification well. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A multi-layer structured pruning method for code generation tasks, characterized in that, Including: Determine the model to be pruned and the pruning target; Determine the used vocabulary based on a programming corpus, perform vocabulary pruning on the model to be pruned based on the used vocabulary, and determine the first pruned model; Determine the target layer based on the first pruned model, determine the first pruning impact based on the target layer and the pruning target, and perform layer selection pruning based on the first pruning impact to determine the second pruned model; Determine the target neuron based on the second pruned model, determine the second pruning impact based on the target neuron and the pruning target, and perform feedforward network pruning based on the second pruning impact to determine the third pruned model; Perform performance recovery based on the third pruned model to determine the target model.

2. The method according to claim 1, wherein The pruning target includes: Among them, θ original and θ pruned respectively represent the parameters of the model to be pruned and the target model, x represents the model input, and y i represents the generated code of the model, KL represents the Kullback-Leibler divergence between their output probability distributions, represents the pruning objective loss function.

3. The method according to claim 1, wherein The performing vocabulary pruning on the model to be pruned based on the used vocabulary to determine the first pruned model includes: Determine the usage frequency based on the used vocabulary; Perform vocabulary pruning on the model to be pruned based on the usage frequency to determine the first pruned model.

4. The method according to claim 1, wherein The determining the first pruning impact based on the target layer and the pruning target, and performing layer selection pruning based on the first pruning impact to determine the second pruned model includes: Remove the target layer from the first pruned model to determine the first test model; wherein, the target layer is any layer in the first pruned model; Determine the first pruning impact based on the first test model and the pruning target; Determine the first target pruning impact from the first pruning impact; Delete the target layer corresponding to the first target pruning impact to determine the second pruned model.

5. The method according to claim 1, wherein The determining the second pruning impact based on the target neuron and the pruning target, and performing feedforward network pruning based on the second pruning impact to determine the third pruned model includes: Determine a heuristic rule; wherein, the heuristic rule is a processing rule for the target neuron, and the target neuron is any neuron in the second pruned model; Remove the target neuron from the second pruned model based on the heuristic rule to determine the second test model; Determine the second pruning impact based on the second test model and the pruning target; Determine the second target pruning impact from the second pruning impact; Delete the target neuron corresponding to the second target pruning impact to determine the third pruned model.

6. The method according to claim 1, characterized in that The performing performance recovery based on the third pruned model to determine the target model includes: Obtain the training data of the model to be pruned; Train the third pruned model based on the training data to determine the target model.

7. The method according to claim 1, wherein It further includes: Determine at least two code intelligent tasks; Evaluate the performance of the target model based on the at least two code intelligent tasks to determine the evaluation result; Wherein, the evaluation result includes an efficiency evaluation result and a robustness evaluation result.

8. A multi-layer structured pruning device for code generation tasks, characterized in that, Including: A target determination module configured to determine the model to be pruned and the pruning target; A vocabulary pruning module configured to determine the used vocabulary based on a programming corpus, perform vocabulary pruning on the model to be pruned based on the used vocabulary, and determine the first pruned model; A layer selection and pruning module, configured to determine a target layer based on the first pruning model, determine a first pruning impact based on the target layer and the pruning objective, perform layer selection and pruning based on the first pruning impact, and determine a second pruning model; A neuron pruning module, configured to determine target neurons based on the second pruning model, determine a second pruning impact based on the target neurons and the pruning objective, perform feed-forward network pruning based on the second pruning impact, and determine a third pruning model; A performance recovery module, configured to perform performance recovery based on the third pruning model to determine a target model.

9. A computing device, characterized in that, Comprising: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multi-layer structured pruning method for code generation tasks according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which when executed by a processor implement the steps of the multi-layer structured pruning method for code generation tasks according to any one of claims 1 to 7.

Images