A cross-language code comment generation method based on knowledge distillation

Abstract

The application provides a cross-language code annotation generation method based on knowledge distillation, aiming at solving the problem of poor annotation generation effect of low-resource programming languages in the prior art. The method first constructs a code-annotation pair dataset of a high-resource programming language (i.e. a language with a large amount of training data), pre-trains and fine-tunes the high-quality annotation generation to form a teacher model. Then, through the knowledge distillation technology, the knowledge of the teacher model is transferred to a student model for a low-resource programming language (i.e. a language with scarce training data), and the student model gradually improves its annotation generation ability by learning the soft label and hard label generated by the teacher model. The trained student model can efficiently generate high-quality code annotations for low-resource programming languages and has cross-language adaptability. The method effectively improves the accuracy and readability of low-resource language code annotations, greatly reduces the dependence on low-resource data, and has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and software engineering, and in particular to a cross-language code annotation generation method based on knowledge distillation. This method is applicable to multi-programming language scenarios, especially between high-resource programming languages ​​(i.e., languages ​​with abundant training data) and low-resource programming languages ​​(i.e., languages ​​with scarce training data), improving the annotation generation quality for low-resource programming languages ​​through knowledge transfer. This invention can be widely applied to multilingual software development, automated document generation, and cross-language code understanding and maintenance. Background Technology

[0002] In software development and maintenance, code comments are one of the most important tools for understanding and maintaining code. High-quality code comments can significantly improve code readability and provide important references for subsequent development and debugging. However, manually writing code comments is time-consuming and laborious, and is prone to omissions or inaccuracies. Therefore, automating the generation of code comments has become an important research direction.

[0003] Currently, deep learning-based code annotation generation technology has made significant progress, especially for high-resource programming languages ​​(such as Python and Java). Due to the abundance of open-source projects and training data, existing automatic annotation generation models can generate annotations corresponding to code with relatively high accuracy. These models, by learning from large-scale code-annotation pairs, can generate natural language descriptions that conform to the semantics of the code. However, for low-resource programming languages ​​(such as Ruby), the scarcity of training data often results in less than ideal annotation generation performance.

[0004] Cross-language code annotation generation is a potential solution to this problem. By acquiring knowledge from high-resource languages ​​and transferring that knowledge to low-resource languages, the code annotation generation capability of low-resource languages ​​can be effectively improved. Knowledge distillation is a commonly used knowledge transfer technique that can compress and transfer knowledge from large, high-precision teacher models to smaller student models, enabling them to achieve high performance under limited resource conditions.

[0005] However, existing methods typically employ multilingual joint training, attempting to train the model simultaneously on multiple languages. While this approach shows some effectiveness in multilingual environments, when the goal is to optimize annotation generation for a specific low-resource language, the model may fail to achieve the desired results due to interference from high-resource languages. Specifically, in multilingual joint training, the model simultaneously learns features from both high-resource and low-resource languages. In this case, the rich features and grammatical structures of the high-resource language may dominate the model's learning process, causing the model to focus more frequently on patterns from these high-resource languages ​​while ignoring the unique features of the low-resource language. Due to the uneven distribution of data, the model's processing of the low-resource language becomes less precise, resulting in poor annotation generation. Therefore, designing a novel knowledge distillation method specifically addressing the differences between high-resource and low-resource languages ​​is an effective way to improve the annotation generation performance of low-resource language code. The proposed solution in this invention uses knowledge distillation to gradually transfer the knowledge of the teacher model trained in the high-resource language to the student model in the low-resource language. This method can effectively reduce the interference of imbalanced data distribution in joint training because the student model only focuses on absorbing the essential knowledge from the teacher model of the high-resource language, and does not directly learn the data distribution of the high-resource language, thereby preserving the unique features of the low-resource language and improving the quality of the generated annotations. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-language code annotation generation method based on knowledge distillation. It trains a teacher model using at least one high-resource programming language, Python or Java, and a student model using Ruby code snippets. Through knowledge distillation, it transfers knowledge from high-resource programming languages ​​to the Ruby code annotation generation task, improving the accuracy and efficiency of Ruby code annotation generation. Specifically, it includes the following steps:

[0007] Step 1: Construct a dataset for high-resource programming languages;

[0008] Specifically, the process involves collecting code snippets and their corresponding comments from high-resource programming languages, constructing a code-comment pair dataset, and preprocessing the data.

[0009] Step 2: Train a teacher model for a high-resource programming language;

[0010] Prior to this, an LLaMA pre-trained model is used as the teacher model. The dataset is fine-tuned by supervised learning based on code-annotations in at least one high-resource programming language, Python or Java. The cross-entropy loss function is used to optimize the teacher model parameters so that it can generate corresponding code annotations.

[0011] Step 3: Using knowledge distillation techniques, knowledge from the teacher model is transferred to the student model to reduce reasoning time and computational resources;

[0012] Prior to this, after the teacher model is trained, knowledge distillation is employed to transfer the knowledge from the teacher model to a smaller student model. By setting a distillation loss function—an optimization objective that comprehensively considers the probability distribution (soft labels) of the teacher model and the true annotation labels (hard labels)—the student model is able to learn the annotation features and structure generated by the teacher model.

[0013] Step 4: Train and optimize the student model to enable it to generate cross-language code annotations;

[0014] Firstly, during the distillation process, Ruby code snippets are used in conjunction with the output of the teacher model to train the student model, enabling it to generate accurate annotations. Specifically, the trained student model is evaluated using metrics such as BLEU and ROUGE to assess the quality of the generated annotations. Based on the evaluation results, the model is fine-tuned to further improve the generation effect.

[0015] Step 5: Deploy the student model and generate Ruby code comments in the development environment;

[0016] Prioritizing and specifically after the model passes evaluation, the student model is deployed to the development environment to generate corresponding code comments for the input Ruby code snippets. Using code snippets provided by developers, the model can automatically generate comments related to the code logic, supporting code understanding and maintenance during the Ruby code development process.

[0017] In addition, the cross-language code annotation generation technology provided by the present invention may also have the following additional technical features:

[0018] Furthermore, in step one, the preprocessing includes limiting the code length to no more than 512 characters. This limitation aims to ensure that the input code snippets are not too long, thereby avoiding performance degradation and memory overflow issues caused by the model processing excessively long sequences. In addition, appropriately truncating code snippets can help the model focus on generating function or method-level annotations, improving the relevance and accuracy of the generated annotations.

[0019] Furthermore, in step three, during the distillation process, to enable the student model to better learn the annotation-generated features of the teacher model, the distillation temperature parameter T is initially set to 5.0. A higher temperature value makes the probability distribution of the teacher model's output smoother, thus providing the student model with more nuanced guidance signals. As training progresses, the temperature parameter is gradually reduced, for example, by 0.5 to 1.0 every few rounds, so that it approaches 1.0 at final convergence. This temperature adjustment strategy helps the student model learn the global knowledge of the teacher model in the early stages, while finely mimicking the specific behaviors of the teacher model in the later stages.

[0020] Furthermore, in step five, the annotations generated by the student model during the inference phase can be continuously optimized based on developer feedback, enabling dynamic model improvement. Specifically, developers can evaluate and modify the generated annotations, collecting feedback data (such as modified annotations, feedback labels, and evaluation scores) for retraining. To this end, the model has a feedback collection mechanism that periodically fine-tunes and updates the accumulated developer feedback data, especially optimizing inaccurate, verbose, or semantically incorrect annotations, gradually improving the generation effect and accuracy.

[0021] The cross-language code comment generation method proposed in this invention has the following beneficial effects:

[0022] This invention provides a cross-language code annotation generation method based on knowledge distillation, which transfers annotation generation knowledge from at least one high-resource programming language, Python and Java, to the Ruby code annotation generation task, significantly improving the accuracy and quality of Ruby code annotations.

[0023] The technical solution of this invention uses only a high-resource programming language during the training process of the teacher model, avoiding the complexity and data imbalance problems in multi-language parallel training, and reducing the difficulty and resource consumption of model training.

[0024] By employing knowledge distillation technology, this invention enables student models to learn key generative features from teacher models, significantly improving the annotation generation capabilities of low-resource programming languages ​​while ensuring the model's adaptability and generalization ability across different programming languages.

[0025] Since the student model is a simplified version of the teacher model after distillation, the student model generated by this invention has a smaller scale and higher computational efficiency, and can be effectively applied in resource-constrained environments, making it suitable for a variety of practical application scenarios.

[0026] By transferring knowledge from high-resource programming languages ​​to low-resource languages, this invention effectively reduces the need for large-scale training data in low-resource programming languages ​​and solves the problem that low-resource languages ​​cannot obtain high-performance models due to data scarcity.

[0027] The cross-language annotation generation method of the present invention can improve the efficiency of developers in understanding and maintaining code in different programming languages, and is helpful for the development, debugging and maintenance of cross-language code projects. Attached Figure Description

[0028] Figure 1 This is a flowchart of the present invention;

[0029] Figure 2 This is a flowchart of the knowledge distillation technique in this invention. Detailed Implementation

[0030] To make the objectives, features, and advantages of this invention more apparent and understandable, the following description will be provided in conjunction with the accompanying drawings. Figure 1 , 2 The embodiments of the present invention will be described in more detail below. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the invention.

[0031] This invention provides a cross-language code annotation generation method based on knowledge distillation. It includes the following steps:

[0032] Step S01: Construction and preprocessing of high-resource language datasets

[0033] We collected code snippets and their corresponding comments from at least one high-resource programming language, Python and Java, to construct a cross-language code-annotation dataset. These datasets primarily consist of code snippets at the function or method level, as functions and methods are typically the most representative parts of the code.

[0034] Data cleaning involves removing invalid or redundant code and comment pairs to prevent erroneous comments or outdated code from affecting model learning. Specific steps for data cleaning include:

[0035] Using regular expressions and syntax parsing tools, extract code snippets from function or method definitions to ensure that the dataset is focused on generating comments at the function or method level.

[0036] The extracted code snippets are segmented to reduce noise data introduced by different programming styles, ensure code input consistency, and reduce unnecessary noise data.

[0037] Step S02: Pre-training and fine-tuning of the teacher model

[0038] Build and fine-tune the teacher model on a high-resource programming language. Select an LLaMA pre-trained model as the teacher model and fine-tune it on a large number of high-resource language code snippets and their annotations to ensure the model can generate accurate annotations. Specific steps include:

[0039] Using supervised learning, the teacher model takes code snippets of a high-resource language as input and outputs comments corresponding to the code as output. The model parameters are optimized using the cross-entropy loss function.

[0040] The fine-tuned teacher model underwent rigorous evaluation on a high-resource programming language, generating annotations with high accuracy and fluency, and good readability.

[0041] Step S03: Knowledge Distillation Process

[0042] Knowledge distillation is the core step of this invention. By transferring knowledge from the teacher model to the student model, the student model can generate high-quality code comments under low-resource programming language conditions. The specific process is as follows:

[0043] (1) Input and output relationship:

[0044] The teacher model takes code snippets from at least one high-resource programming language, either Python or Java, as input, while the student model takes code snippets from Ruby. The teacher model's output consists of two parts: comment text (as hard labels) and an intermediate probability distribution (as soft labels).

[0045] (2) Calculation of distillation loss:

[0046] Hard label loss:

[0047] L hard =CE(c true c student ),

[0048] That is, the cross-entropy loss between the annotations generated by the student model and the real annotations is used to ensure that the annotations generated by the student model are consistent with the real annotations.

[0049] Soft label loss:

[0050] L soft =T 2 ×KL(p teacher p student ),

[0051] That is, the distillation loss is calculated by comparing the probability distributions output by the teacher model and the student model. The probability distribution is smoothed using a temperature parameter T, and by minimizing this loss function, the student model gradually approximates the performance of the teacher model.

[0052] (3) Temperature regulation and optimization:

[0053] Adjusting the temperature T during the distillation process can smooth the output probability of the teacher model, which is beneficial for the student model to learn more from the teacher model. The higher the temperature, the smoother the probability distribution, and the easier it is for the student model to capture subtle differences in the annotations.

[0054] The optimization objective of the loss function is to combine the losses of hard labels and soft labels in a weighted manner:

[0055] L=αL hard +(1-α)L soft ,

[0056] Where α is a coefficient that controls the weighting between hard label and soft label losses.

[0057] Step S04: Training and Optimization of the Low-Resource Language Student Model

[0058] The student model is primarily trained on Ruby code snippets, using knowledge acquired from the teacher model to improve the accuracy of annotation generation. Specific steps include:

[0059] During training, Ruby code snippets are input, and knowledge distillation techniques are used to ensure that the annotations generated by the student model maintain similarity to the output of the teacher model.

[0060] By evaluating metrics such as BLEU and ROUGE, the quality of annotations generated by student models can be monitored, and model parameters can be optimized based on the evaluation results to improve the accuracy and readability of annotation generation.

[0061] Step S05: Reasoning and Deployment of Low-Resource Languages

[0062] The trained student model is deployed in a real development environment to generate accurate and efficient code comments for Ruby code snippets. Specific steps include:

[0063] Input Ruby code snippets into the student model, generate corresponding comments, and provide real-time feedback to the developers;

[0064] Based on developer needs, annotations of different styles and granularities are generated. Through automated model deployment, developers can access these annotations in real time, aiding in code understanding and maintenance.

[0065] The above description is merely one specific embodiment of the present invention. It should be noted that, within the framework of the principles and methods of the present invention, those skilled in the art can make equivalent substitutions and modifications. For example, the teacher model can employ a large-scale pre-trained model such as LLaMA to ensure the accuracy of cross-language code annotation generation; while the student model can use an optimized small model, such as DistilBERT, to meet the needs of low-resource languages ​​with more efficient inference and resource friendliness. Without departing from the technical concept of the present invention, related substitutions and adjustments are still within the scope of protection of the present invention.

Claims

1. A method for generating cross-language code annotations based on knowledge distillation, characterized in that, Includes the following steps: Step 1: Collect code snippets and corresponding comments from at least one high-resource programming language, such as Python and Java, to construct a code-annotation pair dataset, and preprocess the code-annotation pair dataset; Step 2: Use the LLaMA pre-trained model as the teacher model, take the code snippets of the high-resource programming language as input and the corresponding comments as output, fine-tune the teacher model through supervised learning, and update the teacher model parameters using the cross-entropy loss function to obtain the teacher model; Step 3: Generate annotation text and corresponding probability distributions using the trained teacher model. Use the annotation text as hard labels and the probability distributions as soft labels. Use Ruby code snippets as input to the student model and train the student model using a joint distillation loss function. The joint distillation loss function is: L=αL hard +(1-α)L soft Among them, L hard The cross-entropy loss between the generated annotations and the actual annotations for the student model; L soft The probability distribution loss is the difference between the output probability distribution of the teacher model and the output probability distribution of the student model; α is the weighting coefficient of the hard label loss and the soft label loss. Step 4: Train the student model using Ruby code snippets, update the student model parameters according to the joint distillation loss function, and evaluate the code comments generated by the student model using the BLEU and ROUGE evaluation metrics. Step 5: Deploy the trained student model in the development environment and generate corresponding code comments for the input Ruby code snippets.

2. The method for generating cross-language code annotations based on knowledge distillation according to claim 1, characterized in that, In step 1, the preprocessing of the dataset for code-comments includes: extracting function-level or method-level code fragments using regular expressions and syntax parsing tools; performing word segmentation on the extracted code fragments; and limiting the length of the code fragments to within 512 characters.

3. The method for generating cross-language code annotations based on knowledge distillation according to claim 1, characterized in that, In step 2, the high-resource programming languages ​​include Python and Java, and the teacher model uses the corresponding code and comments in Python and Java to perform supervised learning and fine-tuning training on the dataset.

4. The method for generating cross-language code annotations based on knowledge distillation according to claim 1, characterized in that, In step 3, the knowledge distillation training sets a temperature parameter to smooth the probability distribution of the teacher model output. The initial value is 5, and it is gradually reduced to 1 during the training process.

5. The method for generating cross-language code annotations based on knowledge distillation according to claim 1, characterized in that, In step 4, the student model is trained using Ruby code snippets, and the BLEU and ROUGE evaluation metrics are used to evaluate the code comments generated by the student model.

6. The method for generating cross-language code annotations based on knowledge distillation according to claim 1, characterized in that, In step 5, the deployed student model receives Ruby code snippets input by developers, generates corresponding code comments, collects modification data, feedback annotations, or evaluation scores from developers regarding the generated code comments, and uses the collected data to periodically fine-tune the student model.

Images