Cross-language code annotation generation method based on knowledge distillation

Abstract

The invention provides a cross-language code annotation generation method based on knowledge distillation, and aims to solve the problem of poor annotation generation effect of a low-resource programming language in the prior art. According to the method, firstly, a code-annotation pair data set of a high-resource programming language (namely a language with a large amount of training data) is constructed, high-quality annotation generation is subjected to pre-training and fine tuning, and a teacher model is formed; and then, through a knowledge distillation technology, knowledge of the teacher model is migrated to a student model for a low-resource programming language (namely a language with scarce training data), and the student model gradually improves the annotation generation ability by learning a soft tag and a hard tag generated by the teacher model. The trained student model can efficiently generate high-quality code annotations for low-resource programming languages, and has cross-language adaptability. According to the method, the accuracy and readability of low-resource language code annotation are effectively improved, the dependency of low-resource data is greatly reduced, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and software engineering, and in particular to a cross-language code annotation generation method based on knowledge distillation. The method is applicable to multi-programming language scenarios, especially between high-resource programming languages ​​(i.e., languages ​​with a large amount of training data) and low-resource programming languages ​​(i.e., languages ​​with scarce training data), and improves the annotation generation quality of low-resource programming languages ​​through knowledge transfer. The invention can be widely used in the fields of multi-language software development, automated document generation, and cross-language code understanding and maintenance. Background Art

[0002] In the process of software development and maintenance, code comments are one of the important tools for understanding and maintaining code. High-quality code comments can significantly improve the readability of the code and provide an important reference for subsequent development and debugging. However, manually writing code comments is time-consuming and laborious, and is prone to omissions or inaccuracies. Therefore, automatically generating code comments has become an important direction of current research.

[0003] At present, code comment generation technology based on deep learning has made significant progress, especially for high-resource programming languages ​​(such as Python, Java, etc.). Due to their large number of open source projects and rich training data, existing automatic comment generation models can generate comments corresponding to the code more accurately. These models can generate natural language descriptions that conform to the semantics of the code by learning large-scale code-comment pairs. However, for low-resource programming languages ​​(such as Ruby), due to the scarcity of training data, the effect of comment generation is often not ideal.

[0004] Cross-language code comment generation is a potential solution to this problem. By acquiring knowledge from high-resource languages ​​and transferring this knowledge to low-resource languages, the code comment generation capability of low-resource languages ​​can be effectively improved. Knowledge distillation is a commonly used knowledge transfer technique that can compress and transfer knowledge from a large, high-precision teacher model to a smaller student model, enabling it to achieve higher performance under limited resource conditions.

[0005] However, existing methods usually use a multi-language joint training method to try to train the model directly on multiple languages ​​at the same time. Although this method has a certain effect in a multi-language environment, when the goal is to optimize the annotation generation of a specific low-resource language, the model may not achieve the expected effect due to the interference of high-resource languages. Specifically, in multi-language joint training, the model learns the features of high-resource languages ​​and low-resource languages ​​at the same time. In this case, the rich features and grammatical structures of high-resource languages ​​may dominate the learning process of the model, causing the model to tend to pay more attention to the patterns of these high-resource languages ​​more frequently and ignore the unique features of low-resource languages. Due to the uneven distribution of data, the model's processing of low-resource languages ​​will become less accurate, resulting in poor annotation generation. Therefore, a new knowledge distillation method is designed specifically for the differences between high-resource languages ​​and low-resource languages, which becomes an effective way to improve the effect of low-resource language code annotation generation. The solution proposed in the present invention gradually transfers the knowledge of the teacher model trained in high-resource languages ​​to the student model of low-resource languages ​​through knowledge distillation. This method can effectively reduce the interference of uneven data distribution in joint training, because the student model only focuses on absorbing the essence of knowledge from the high-resource language teacher model, and does not directly learn the data distribution of the high-resource language, thereby retaining the unique features of the low-resource language and improving the quality of the generated annotations. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-language code annotation generation method based on knowledge distillation, construct a teacher model for high-resource programming languages ​​(i.e., a large amount of training data), generate a student model for low-resource programming languages ​​(i.e., a small amount of training data), and optimize the accuracy and efficiency of cross-language annotation generation. Specifically, the method includes the following steps:

[0007] Step 1: Build a dataset of high-resource programming languages;

[0008] Prioritize, specifically collect code snippets and their corresponding comments of high-resource programming languages, construct a code-comment pair dataset, and preprocess the data.

[0009] Step 2: Train a teacher model for a high-resource programming language;

[0010] Prioritize using large-scale pre-trained models, fine-tuned on high-resource language datasets, to enable them to generate accurate annotations given a code snippet.

[0011] Step 3: Use knowledge distillation technology to transfer the knowledge of the teacher model to the student model to reduce inference time and computing resources;

[0012] Prioritize, after the teacher model is trained, the knowledge distillation technique is used to transfer the knowledge of the teacher model to a smaller student model. By setting the distillation loss function, an optimization objective that comprehensively considers the probability distribution of the teacher model (soft labels) and the true annotation labels (hard labels), the student model can learn the annotation features and structures generated by the teacher model.

[0013] Step 4: Train and optimize the student model to enable it to generate cross-language code comments;

[0014] Prioritize, during the distillation process, code snippets in low-resource languages ​​are used in combination with the output of the teacher model to train the student model so that it has the ability to generate accurate annotations. Specifically, the trained student model is evaluated, and the quality of the generated annotations is evaluated using indicators such as BLEU and ROUGE. The model is tuned based on the evaluation results to further improve the generation effect.

[0015] Step 5: Deploy the student model and generate code comments in multiple programming languages ​​in a real development environment;

[0016] Prioritize, specifically after the model is evaluated as qualified, deploy the student model to the development environment to generate real-time annotations for multiple programming languages ​​(including low-resource languages). Through the code snippets provided by the developer, the model can automatically generate annotations related to the code logic and support application in cross-language projects.

[0017] In addition, the cross-language code comment generation technology provided by the present invention may also have the following additional technical features:

[0018] Furthermore, in step 1, the preprocessing process includes limiting the length of the code to no more than 512 characters. This restriction is intended to ensure that the code snippet input is not too long, thereby avoiding performance degradation and memory overflow caused by the model processing overly long sequences. In addition, moderate code snippet truncation can also help the model focus on function or method-level annotation generation, improving the relevance and accuracy of generated annotations.

[0019] Furthermore, in the distillation process in step 3, in order to enable the student model to better learn the annotation generation features of the teacher model, the distillation temperature parameter T is initially set to 5.0. A higher temperature value will make the probability distribution of the teacher model output smoother, thereby providing a more delicate guidance signal for the student model. As the training progresses, the temperature parameter is gradually reduced, for example, by 0.5 to 1.0 every several rounds, so that it is close to 1.0 at the final convergence. This temperature adjustment strategy helps the student model learn the global knowledge of the teacher model in the early stage, and finely imitate the specific behavior of the teacher model in the later stage.

[0020] Furthermore, in step 5, the annotations generated by the student model during the reasoning phase support continuous optimization based on developer feedback, thus achieving dynamic improvement of the model. Specifically, developers can evaluate and modify the generated annotations, collect feedback data (such as modified annotations, feedback annotations, and evaluation scores) and use them for retraining. To this end, the model sets up a feedback collection mechanism, which periodically fine-tunes and updates the accumulated developer feedback data, especially optimizing the generated annotations that are inaccurate, lengthy, or inconsistent with the code semantics, and gradually improves the generation effect and accuracy.

[0021] The cross-language code comment generation method proposed in the present invention has the following beneficial effects:

[0022] The present invention provides a cross-language code annotation generation method based on knowledge distillation, which effectively transfers the rich annotation generation experience of high-resource programming languages ​​to low-resource programming languages, significantly improving the accuracy and quality of low-resource language code annotations.

[0023] The technical solution of the present invention only uses high-resource programming languages ​​in the training process of the teacher model, avoiding the complexity and data imbalance problems in multi-language parallel training, and reducing the difficulty and resource consumption of model training.

[0024] Through knowledge distillation technology, the present invention enables the student model to learn key generation features from the teacher model, significantly improving the annotation generation capability of low-resource programming languages ​​while ensuring the adaptability and generalization ability of the model between different programming languages.

[0025] Since the student model is a simplified version of the teacher model after distillation, the student model generated by the present invention has a smaller scale and higher computational efficiency, can be effectively applied in resource-constrained environments, and is suitable for a variety of practical application scenarios.

[0026] By migrating knowledge from high-resource programming languages ​​to low-resource languages, the present invention effectively reduces the demand for large-scale training data in low-resource programming languages, and solves the problem that low-resource languages ​​cannot obtain high-performance models due to data scarcity.

[0027] The cross-language annotation generation method of the present invention can improve the efficiency of developers in understanding and maintaining codes in different programming languages, and is helpful for the development, debugging and maintenance of cross-language code projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the present invention;

[0029] Figure 2 Flow chart of the knowledge distillation technology in the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, features and advantages of the present invention more clearly understood, the following Figure 1 , 2 The embodiments of the present invention are described in more detail. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be interpreted as limiting the invention.

[0031] The present invention provides a cross-language code annotation generation method based on knowledge distillation. The method comprises the following steps:

[0032] Step S01: Construction and preprocessing of high-resource language datasets

[0033] We collect code snippets and their corresponding comments for high-resource programming languages ​​(such as Python, Java, etc.) to build cross-language code-comment pair datasets. These datasets mainly include code snippets at the function or method level, because functions and methods are usually the most representative parts of the code.

[0034] Clean the data set to remove invalid or redundant code and comment pairs to prevent incorrect comments or outdated code from affecting model learning. The specific steps of data cleaning include:

[0035] Using regular expressions and syntax parsing tools, we extract code snippets from the function or method definition part to ensure that the dataset focuses on the comment generation at the function or method level.

[0036] Segment the extracted code snippets to reduce the noise data introduced by different programming styles, ensure the consistency of code input, and reduce unnecessary noise data.

[0037] Step S02: Pre-training and fine-tuning of the teacher model

[0038] Build and fine-tune the teacher model on a high-resource programming language. Choose a large-scale pre-trained model (such as LLaMA) and fine-tune it on a large number of high-resource language code snippets and their annotations to ensure that the model can generate accurate annotations. The specific steps include:

[0039] Using supervised learning methods, the teacher model input is a code snippet in a high-resource language, and the output is a comment corresponding to the code. The model parameters are optimized through the cross entropy loss function;

[0040] The fine-tuned teacher model is rigorously evaluated on high-resource programming languages, and the generated annotations have high accuracy, fluency, and good readability.

[0041] Step S03: Knowledge distillation process

[0042] Knowledge distillation is the core step of the present invention. By transferring the knowledge of the teacher model to the student model, the student model can generate high-quality code comments under the condition of low-resource programming languages. The specific process is as follows:

[0043] (1) Relationship between input and output:

[0044] The teacher model is fed with code snippets from a high-resource language, and the student model is fed with code snippets from a low-resource language.

[0045] The output of the teacher model consists of two parts: annotation text (as hard labels) and intermediate probability distributions (as soft labels).

[0046] (2) Calculation of distillation loss:

[0047] Hard label loss:

[0048] L hard =CE(c true , c student ),

[0049] That is, the cross entropy loss between the annotations generated by the student model and the true annotations is used to ensure that the generated annotations of the student model are consistent with the true annotations.

[0050] Soft label loss:

[0051] L soft =T 2 ×KL(p teacher , p student ),

[0052] That is, by comparing the probability distributions of the teacher model and the student model output, the distillation loss is calculated. The temperature parameter T is used to smooth the probability distribution, and by minimizing this loss function, the student model gradually approaches the performance of the teacher model.

[0053] (3) Temperature regulation and optimization:

[0054] Adjusting the temperature T during the distillation process can make the output probability of the teacher model smoother, which is conducive to the student model learning more knowledge from the teacher model. The higher the temperature, the smoother the probability distribution, and the easier it is for the student model to capture the subtle differences in the annotations.

[0055] The optimization goal of the loss function is to combine the losses of hard labels and soft labels by weight:

[0056] L=ɑL hard +(1-α)L soft ,

[0057] Among them, ɑ is the coefficient that controls the weight between hard label and soft label losses.

[0058] Step S04: Training and Optimization of Low-Resource Language Student Model

[0059] The student model is mainly trained for low-resource programming languages ​​and improves the accuracy of annotation generation by acquiring knowledge from the teacher model. The specific steps include:

[0060] During the training process, code snippets in low-resource languages ​​are input, and through knowledge distillation technology, the annotations generated by the student model can maintain similarity with the output of the teacher model;

[0061] Through evaluation indicators (such as BLEU and ROUGE), the quality of annotations generated by the student model is monitored, and the model parameters are optimized according to the evaluation results to improve the accuracy and readability of annotation generation.

[0062] Step S05: Reasoning and deployment for low-resource languages

[0063] The trained student model is deployed in the actual development environment to generate comments for the code of low-resource programming languages. The model can handle a variety of low-resource languages, and the generated comments are accurate and efficient. The specific steps include:

[0064] Input low-resource language code snippets into the student model, generate corresponding annotations, and provide real-time feedback to developers;

[0065] The student model supports multiple programming languages ​​and can generate comments of different styles and granularity according to the needs of developers. Through the automatic deployment of the model, developers can obtain comments in real time to assist in code understanding and maintenance.

[0066] The above description is only one of the specific implementation schemes of the present invention. It should be pointed out that under the framework of the principles and methods of the present invention, technicians in the field can make equivalent replacements and variations. For example, the teacher model can adopt large-scale pre-trained models such as LLaMA to ensure the accuracy of cross-language code annotation generation; and the student model can use optimized small models, such as DistilBERT, to meet the needs of low-resource languages ​​with more efficient reasoning and resource-friendliness. Under the premise of not deviating from the technical ideas of the present invention, the relevant replacements and adjustments are still within the scope of protection of the present invention.

Claims

1. A cross-language code comment generation method based on knowledge distillation, characterized in that: The following steps are involved: Step 1: Collect and preprocess code snippets and their corresponding comments in high-resource programming languages ​​to construct a code-comment pair dataset; Step 2: Preliminary training and fine-tuning of a large-scale pre-trained model on a high-resource programming language dataset to obtain a teacher model that specializes in generating code comments. The teacher model is enabled to generate accurate comments through supervised learning. Step 3: Use knowledge distillation technology to transfer the teacher model’s annotations and their generated probability distribution as soft labels to the student model; Step 4: Train the student model for low-resource programming languages, and optimize the generation ability of the student model through evaluation indicators so that the annotations it generates gradually approach the annotations of the teacher model; Step 5: Deploy the low-resource language student model in a real environment to realize low-resource language code annotation generation.

2. The cross-language code annotation generation method according to claim 1, characterized in that: In step 1, regular expressions and syntax parsing tools are used to extract code snippets at the function or method level to ensure that the comment generation task focuses on the code snippets of functions or methods, and the code snippets are further segmented to ensure the consistency of code input and reduce unnecessary noise data.

3. The cross-language code annotation generation method according to claim 1, characterized in that: The pre-training and fine-tuning of the teacher model in step 2 includes: Select a large-scale pre-trained model (such as LLaMA) and fine-tune it on a large number of high-resource language (such as Python, Java, etc.) code snippets and their annotations; Using supervised learning methods, the input of the teacher model is a code snippet in a high-resource language, and the output is the comments corresponding to the code; Use the cross entropy loss function to optimize the model parameters to generate accurate annotations; After fine-tuning, the teacher model has excellent annotation generation performance on high-resource languages ​​and has the ability to generate high-quality annotations on high-resource programming languages, laying the foundation for the knowledge distillation process.

4. The cross-language code annotation generation method according to claim 1, characterized in that: The knowledge distillation process in step 3 adopts distillation loss, which includes a weighted combination of hard label loss and soft label loss. The specific formula is: L=αL hard +(1-α)L soft Among them, L hard is the cross entropy loss between the annotations generated by the student model and the true annotations, L soft is the probability distribution loss output by the teacher model and the student model, α is the weight coefficient of the hard label and soft label losses. By adjusting this weight, the loss balance is achieved and the student model’s imitation of the teacher model’s annotation generation behavior is enhanced.

5. The cross-language code annotation generation method according to claim 1, characterized in that: The student model training and optimization process in step 4 includes: Train the student model for low-resource languages ​​(such as Ruby) and improve the accuracy of annotation generation by acquiring knowledge from the teacher model; During the training process, code snippets in low-resource languages ​​are input, and through knowledge distillation technology, the annotations generated by the student model can maintain similarity with the output of the teacher model; The quality of annotations generated by the student model is monitored through evaluation indicators (such as BLEU and ROUGE), and the model parameters are optimized based on the evaluation results to improve the accuracy and readability of the generated annotations.

6. The cross-language code annotation generation method according to claim 1, characterized in that: In step 5, the deployment of the student model can adapt to multiple low-resource programming languages. By inputting code snippets in the corresponding language, high-quality annotations can be generated in real time, and consistent annotation standards can be maintained in cross-language development projects.

Images