A Program Automatic Repair Method Based on Perturbation and Frozen Pre-trained Models

Abstract

The present invention relates to a program automatic repair method based on a perturbed and frozen pre-trained model. First, a defect repair dataset is constructed through an open-source code repository, and defective code, code context, and repair code are extracted from historical commits containing defect repairs, and the dataset is cleaned; noise is added to the pre-trained model through a matrix-based perturbation method; the training parameter quantity is reduced by freezing the encoder network of the model; in addition, through a checkpoint integration strategy, multiple checkpoints during the model training process are saved; finally, a candidate patch list is generated by inference using multiple checkpoints. After reordering the patch list, a final candidate patch list is obtained. Finally, the Top-N patches in the list are run through a test suite to check whether the patches can pass all test cases, and the correct patches are output. This method not only reduces the training cost but also improves the repair effect of the model.

Description

Technical Field

[0001] The present invention belongs to the technical fields of software debugging and artificial intelligence, and particularly relates to a method for automatically repairing programs based on a perturbed and frozen pre-trained model. Background Art

[0002] Automatic program repair refers to generating a repaired patch for defective code through an automated tool, reducing the cost of developers debugging code and improving development efficiency.

[0003] Most existing studies train models through deep learning methods to automatically generate patches, such as models based on neural machine translation and pre-trained models. However, the repair scheme using a neural machine translation-based model is limited by the lack of training data sets and a simple model architecture. The model cannot fully understand the semantics of the program, and the repair effect is not good. For the scheme using a pre-trained model, due to the overfitting problem between the pre-trained model and the program repair task, the model can still only achieve a sub-optimal repair effect, and because the number of parameters of the pre-trained model is huge, a large amount of time and computing resources are required during training. Summary of the Invention

[0004] To solve the problems of poor repair effect and high training cost of the current program repair scheme, the present invention proposes a method for automatically repairing programs based on a perturbed and frozen pre-trained model, including the following 4 steps:

[0005] Step 1, construct a program defect repair data set;

[0006] In Step 1, constructing a program defect repair data set includes the following steps:

[0007] Step 1.1, in the code submission history of the open-source code repository, query the historical records whose submission information contains defect repair;

[0008] Step 1.2, extract defective code, code context, and correct patches from the defect repair historical records to construct a data set;

[0009] Step 1.3, perform data cleaning on the data set to filter out duplicate and data with missing repair code in the data set;

[0010] Step 2, fine-tune the pre-trained model by means of perturbation and freezing;

[0011] In Step 2, fine-tuning the pre-trained model includes the following steps:

[0012] Step 2.1 Perturb the parameters of the pre-trained model;

[0013] According to the variance of the parameter matrix, different intensities of noise are added to different parameter matrices to help the model better adapt to the data of downstream tasks and alleviate the overfitting problem of the model in the pre-training stage;

[0014] Step 2.2, freeze the encoder of the model;

[0015] The reason for performing this step is that the code repair task is a type of generation task. By only freezing the encoder, the semantic information and repair ability learned by the model on the original dataset can be retained, while reducing the number of training parameters, accelerating the training speed, and saving computing resources.

[0016] Step 3, generate patches through multiple checkpoints of the model;

[0017] The said Step 3, generating patches through multiple checkpoints of the model, includes the following steps:

[0018] Step 3.1, during the model training stage, save the checkpoints at k (k = 5) steps;

[0019] Step 3.2, during the model inference stage, for the defective code to be repaired, use the defective code and the code context as the input to the checkpoints, and use beam search on the k checkpoints to generate n (n = 100) candidate patches for the defective code at each checkpoint.

[0020] Step 4, patch sorting and verification;

[0021] The said Step 4, patch sorting and verification, includes the following steps:

[0022] Step 4.1, merge the k candidate patch lists, first filter out the duplicate patches, and then sort according to the ranking and confidence score of the patches to obtain the final candidate patch list;

[0023] Step 4.2, for the Top-N (N = 100) patches in the candidate list, run the corresponding test suite to check whether all test cases can be passed.

[0024] The reason for performing this step is that the patches ranked higher in the list have a higher probability of being correct patches. By only verifying the Top-N patches, the burden in the verification stage can be alleviated;

[0025] Finally, output all the patches that pass the test cases to complete the program repair.

[0026] Compared with the prior art, the beneficial effects of the present invention:

[0027] In the model training stage of the present invention, noise is added to the parameters of the pre-trained model through a matrix-based perturbation method to alleviate the overfitting problem between the pre-trained model and the downstream tasks, improve the repair performance of the model. By freezing the encoder network, the number of parameters to be trained during fine-tuning is reduced, and the time and computing resources consumed by model training are decreased. In addition, during the model training process, a checkpoint integration strategy is applied to save the checkpoints of multiple models, and multiple checkpoints are used to generate patches and verify during the inference stage, so as to better capture the diversity in the defect repair data.

[0028] Generally speaking, the method of the present invention can reduce the cost of model training while improving the repair success rate. Brief Description of the Drawings

[0029] Figure 1 It is a schematic diagram of the overall process of the method of the present invention. Detailed Embodiments

[0030] In order to more clearly show the purpose and technical solutions of the present invention, the detailed embodiments of the present invention will be described in more detail below with reference to the accompanying drawings.

[0031] A program automatic repair method based on perturbing and freezing a pre-trained model includes the following steps:

[0032] Step 1, construct a program defect repair data set;

[0033] In this step, real defects and repair results in the open-source repository are collected, and the data is cleaned to construct a large-scale high-quality data set for fine-tuning the pre-trained model.

[0034] The said Step 1, constructing a program defect repair data set, as Figure 1 shown in Step 1 of

[0035] Step 1.1, in the code commit history of the open-source code repository, query the historical records whose commit information contains defect repairs;

[0036] The said open-source code repository refers to open-source projects on GitHub and GitLab;

[0037] Step 1.2, extract defective code, code context, and correct patches from the defect repair historical records to construct a data set;

[0038] The said code context refers to the function where the defective code is located.

[0039] Step 1.3, perform data cleaning on the data set to filter out duplicate data and data with missing repair code in the data set.

[0040] The repeated data refers to the same defect repair data that has appeared in multiple warehouses;

[0041] The data for repairing code defects means that the developer completed the repair task by directly deleting the defective code without corresponding repair code;

[0042] Step 2, fine-tune the pre-trained model by the methods of perturbation and freezing;

[0043] In this step, the pre-trained model CodeT5 is fine-tuned. First, the parameters of the pre-trained model are perturbed to alleviate the overfitting problem and enhance the repair effect. Then, the encoder of the model is frozen to reduce the number of training parameters and save time and computing resources;

[0044] The above Step 2, fine-tuning the pre-trained model by the methods of perturbation and freezing, as Figure 1 shown in Step 2 of [reference], includes the following steps:

[0045] Step 2.1, perturb the parameters of the pre-trained model;

[0046] According to the variance of the parameter matrix, different intensities of noise are added to different parameter matrices to help the model better adapt to the data of downstream tasks and alleviate the overfitting problem of the model in the pre-training stage;

[0047] For the parameter matrices [W1, W2, …, W N of the pre-trained model, where N is the number of types of parameter matrices, the perturbed parameter matrix is:

[0048]

[0049] where std represents the standard deviation, U(a, b) represents the uniform distribution noise of (a, b), and λ is a hyperparameter controlling the noise intensity;

[0050] Step 2.2, freeze the encoder of the model;

[0051] The freezing means that during the model training, the requires_grad attribute of the encoder network is set to False.

[0052] Step 3, generate patches through multiple checkpoints of the model;

[0053] The above Step 3, generating patches through multiple checkpoints of the model, as Figure 1 shown in Step 3 of [reference], includes the following steps:

[0054] Step 3.1, during the model training stage, save the checkpoints at k (k = 5) steps;

[0055] Step 3.2, in the model inference stage, for the defective code to be repaired, use the defective code and the code context as the input of the checkpoint, perform beam search on k checkpoints, and generate n (n = 100) candidate patches for the defective code at each checkpoint;

[0056] During inference, set the beam size in beam search to 100 and the batch size to 8.

[0057] Step 4, Patch sorting and verification;

[0058] The said Step 4, Patch sorting and verification, as Figure 1 shown in Step 4, includes the following steps:

[0059] Step 4.1, for the k candidate patch lists, first remove the duplicate patches, and then sort them in the ranking order in the original list. For patches with the same ranking, sort them in reverse order according to the confidence scores of the patches to obtain the final candidate patch list;

[0060] The said patch ranking refers to the order of the patches in the list after sorting the patches according to the confidence scores for each patch list generated by checkpoint inference.

[0061] The said confidence score refers to the probability value given by the model for the generated result;

[0062] Step 4.2, for the Top-n (N = 100) patches in the candidate list, run the corresponding test suite to check whether all test cases can pass.

[0063] Finally, output all the patches that pass the test cases to complete the program repair.

Claims

1. A program automatic repair method based on a perturbed and frozen pre-trained model, characterized in that It includes the following steps: Step 1, constructing a program defect repair dataset, and the said Step 1 includes the following steps: Step 1.1, querying the historical records with defect repair in the code submission history of the open-source code repository; Step 1.2, extracting defective code, code context, and correct patches from the defect repair historical records to construct a dataset; Step 1.3, performing data cleaning on the dataset to filter out duplicate data and data with missing repair code in the dataset; Step 2, fine-tuning the pre-trained model by the methods of perturbation and freezing, and the said Step 2 includes the following steps: Step 2.1, perturbing the parameters of the pre-trained model CodeT5; Step 2.2, freezing the encoder of the pre-trained model CodeT5; Step 3, generating patches through multiple checkpoints of the model, and the said Step 3 includes the following steps: Step 3.1, saving k checkpoints during the model training phase, where k = 5; Step 3.2, during the model inference phase, for the defective code to be repaired, taking the defective code and code context as the input of the checkpoints, using beam search on k checkpoints, and generating n candidate patches for the defective code on each checkpoint, where n = 100; Step 4, patch sorting and verification, and the said Step 4 includes the following steps: Step 4.1, for the k candidate patch lists, first removing duplicate patches, and then arranging them in the ranking order in the original list. For patches with the same ranking, arranging them in reverse order according to the confidence scores of the patches to obtain the final candidate patch list; Step 4.2, for the top 100 patches in the candidate list, running the corresponding test suite to check whether all test cases can pass.

2. The method according to claim 1, wherein The said Step 2, fine-tuning the pre-trained model by the methods of perturbation and freezing, further includes: Perturbing model parameters: adding different intensities of noise to different parameter matrices according to the variances of the parameter matrices; Freezing the model network: during the model training process, setting the requires_grad attribute of the encoder network to freeze the encoder network of the model.

Images