Man-machine collaboration program repairing method based on large language model

Abstract

The invention provides a man-machine collaborative program repairing method based on a large language model, relates to the technical field of automatic program repairing, and remarkably improves the accuracy and success rate of program repairing. According to the method, a large language model is guided through preliminary analysis of manual teaching, errors in a program can be more accurately positioned and repaired by combining a repairing process of multiple iterations and more information introduced step by step, misjudgment and omission are reduced, and the program repairing efficiency is greatly improved. Compared with a traditional method, blind exploration of a large language model on complex problems is avoided through a man-machine cooperation mode, time and computing resources are saved, and adaptability of program repair is enhanced. According to the method, program defects of various types and complexity can be handled, and simple grammar errors or complex logic errors can be effectively handled through a step-by-step deep repair process.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic program repair, and more specifically, particularly relates to a human-computer collaborative program repair method based on a large language model. Background Art

[0002] As the scale of software development expands and the complexity increases, the frequency of bugs in programs also increases. Traditional manual debugging methods are time-consuming and labor-intensive, while fully automated program repair methods have the problem of insufficient accuracy. In recent years, program repair methods based on large language models (LLMs) have attracted widespread attention due to their powerful code understanding and generation capabilities. However, existing LLM-based program repair methods still have many shortcomings.

[0003] Some interactive repair methods require multiple independent conversations to fix an incorrect code, which is computationally expensive. For example, the ChatRepair method requires an average of 10 independent conversation sessions to fix an incorrect code, which may lead to inefficiency and waste of resources in practical applications. At the same time, insufficient utilization of augmented information is also a common problem. Existing methods mainly rely on problem descriptions and failed test cases to generate repairs, and do not make full use of other beneficial information such as solution descriptions, guidance from human mentors, etc., which limits the depth of LLM's understanding of the program and the accuracy of repairs. Insufficient human-computer collaboration is another significant flaw of current methods. Most methods regard the repair process as an automated program, ignoring the inherent collaborative and interactive characteristics of programming. The lack of an effective human-computer interaction mechanism makes it impossible to fully utilize the knowledge and experience of human experts, which may lead to a decrease in the quality of repair results in practical applications. Finally, insufficient adaptability to specific scenarios is also a problem worthy of attention. In specific scenarios such as online programming education, existing program repair methods based on large language models often ignore the importance of maintaining the original logic of the program during the program repair process, which makes it easy for the tools to generate repaired programs that are correct but too different from the students' original defective programs. This makes it difficult for students to directly understand how to modify their own programs through the repaired programs.

[0004] Although some representative works have attempted to solve the above problems, such as the interactive repair method proposed by Gao et al., the conversational prompt enhancement explored by Sobania et al., and the ChatRepair method proposed by Xia et al., these methods have not yet fully solved the above problems. Therefore, there is an urgent need for a program repair method that can overcome existing defects and is suitable for practical application scenarios (such as online programming education). This method should be able to effectively utilize enhanced information, achieve efficient human-computer collaboration, and show excellent repair performance on large-scale uncontaminated datasets.

[0005] In order to solve the above problems, the present invention proposes a human-computer collaborative program repair method CREF (Conversational Program Repair Framework) based on a large language model. This method fully utilizes human expert knowledge and the conversation ability of a large language model through conversational interaction, and also utilizes the failure repair records in past conversations to achieve a more efficient and accurate program repair process, thereby improving the efficiency of programming assistants in debugging programs and answering questions for students in online programming education scenarios, and significantly reducing labor costs. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a human-computer collaborative program repair method based on a large language model to solve the above problems.

[0007] A human-computer collaborative program repair method based on a large language model comprises the following steps:

[0008] S1: The human assistant gives a preliminary analysis of the defective code;

[0009] S2: The procedure after the repair of large language model generation;

[0010] S3: Automatic testing verifies program correctness;

[0011] S4: Provide solution documents, and the large language model will generate the repaired program again;

[0012] S5: Automatic testing verifies program correctness;

[0013] S6: Provide a failed test case, and the large language model will generate the repaired program again;

[0014] S7: Automatic testing verifies program correctness.

[0015] Preferably, in step S1, the human teaching assistant performs a preliminary analysis of the defect code in combination with the question information, and the analysis content is a brief statement of one to two sentences.

[0016] Preferably, in step S2, the large language model generates a repaired program in combination with the preliminary analysis results and defective code provided by the human teaching assistant.

[0017] Preferably, in step S3, if the repaired program passes all test cases, the repair process is terminated and recorded as successful repair, otherwise the subsequent steps are continued.

[0018] Preferably, in step S4, the large language model combines the solution document of the problem corresponding to the defect code and the record of the previous incorrect repair to regenerate the repaired program.

[0019] Preferably, in step S5, if the repaired program generated again passes all test cases, the repair process is terminated and recorded as a successful repair, otherwise the subsequent steps are continued.

[0020] Preferably, in step S6, the large language model combines the failed test cases and the records that were not correctly repaired before to regenerate the repaired program.

[0021] Preferably, in step S7, if the repaired program finally generated passes all test cases, it is recorded as a successful repair, otherwise it is recorded as a failed repair, and it can be decided whether to restart execution from step S1.

[0022] Preferably, when constructing an evaluation data set, it is necessary to collect and organize the manually written defect brief analysis, program title description, program test case, program solution document, and benchmark repair code corresponding to each defect code, and the test case design should avoid problems caused by patch overfitting and excessive size of a single test case.

[0023] Preferably, the evaluation indicators include Top5 and RPSR. Top5 indicates the proportion of all test cases that pass at least one of the five generated results correctly. RPSR is calculated based on the tree edit distance of the abstract syntax tree. The calculation formula is:

[0024] The numerator is the tree edit distance between the defective code and the fixed code generated by the large language model, and the denominator is the tree edit distance between the defective code and the baseline fixed code.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. This human-computer collaborative program repair method based on a large language model significantly improves the accuracy and success rate of program repair. The initial analysis of the human assistant provides guidance for the large language model, combined with multiple iterations of the repair process and the gradual introduction of more information, it can more accurately locate and repair errors in the program and reduce misjudgments and omissions.

[0027] 2. Greatly improve the efficiency of program repair. Compared with traditional methods, the human-computer collaborative approach avoids the blind exploration of large language models on complex problems, saving time and computing resources.

[0028] 3. Enhanced the adaptability of program repair. It can deal with program defects of various types and complexities. Whether it is a simple syntax error or a complex logic error, it can be effectively handled through a step-by-step repair process.

[0029] 4. Improved the effectiveness of programming education. In online programming education scenarios, it can provide students with timely and accurate repair suggestions, helping them to understand the causes and solutions of errors more quickly, thereby improving learning efficiency and programming ability.

[0030] 5. Reduce the cost of software development. Reduce the problems of extended development cycle and increased manpower input caused by program errors, and improve the overall benefits of software development.

[0031] 6. Promoted the effective application of large language models in the field of program repair. It fully utilized the language processing capabilities of large language models, and at the same time made up for its possible limitations through human-computer collaboration, thus opening up new directions for the application of large language models.

[0032] 7. The quality evaluation system of program repair has been optimized. By introducing clear evaluation indicators such as Top5 and RPSR, the repair effect can be evaluated more objectively and accurately, providing a strong basis for the improvement and optimization of the method.

[0033] 8. Improved the maintainability and readability of the program. The repaired program is more in line with programming standards and logic, and is easy to maintain and expand in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0035] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0036] As the scale and complexity of software development continue to increase, errors in programs are becoming more and more frequent. Traditional program repair methods have many drawbacks. In order to improve the efficiency and accuracy of program repair, we propose a human-computer collaborative program repair method based on a large language model. This method aims to fully combine the knowledge and experience of human experts with the powerful language processing capabilities of large language models to achieve more efficient and accurate program repair.

[0037] The technical solution adopted by the present invention is a human-computer collaborative program repair method CREF (Conversational Program Repair Framework) based on a large language model. Considering the specific educational scenario, CREF first asks the human assistant to read the defective code given by the student and give a preliminary analysis result. This is usually a very simple question or two, which may not fully point out all the errors in the program and how to modify these errors, and may even give inaccurate guidance. Therefore, it only takes a short time for human experts to complete. The large language model combines the preliminary analysis results of human experts to give the program after the first repair, and can automatically verify the correctness of the repaired program with the help of test cases. Next, if all test cases have not been passed correctly, the solution document of the corresponding question of the defective code will be automatically provided to the large language model again on the basis of keeping the existing dialogue records, and the repair program will be tried again. After the generation, the automatic test verification is also performed. If it still fails, it will be provided to the failed test case just generated by the automatic test of the large language model at the end, and the repair program will be tried again to complete the entire repair process. The processing flow of CREF is shown in the figure below:

[0038]

[0039] Step 1: The human assistant gives a preliminary analysis of the defective code;

[0040] First, for the defective code given by the students, the human teaching assistant gives a preliminary analysis based on the question description and other information. It is usually a short content of one to two sentences, which requires very little manpower and time cost.

[0041] Step 2: The large language model generates the repaired program;

[0042] The large language model combines the preliminary analysis information provided by the current human teaching assistant to generate repaired code based on the user's defective code.

[0043] Step 3: Automatically test and verify the correctness of the program;

[0044] After the repaired code is generated, the repaired code is automatically tested with the help of test cases. If the test passes, the repair process ends and is recorded as a successful repair. Otherwise, the subsequent process continues.

[0045] Step 4: Provide a solution document, and the large language model will generate the repaired program again;

[0046] The large language model combines the solution document information of the problem corresponding to the defect code with the incorrectly repaired records generated in the previous steps to generate the repaired code again.

[0047] Step 5: Automatic testing to verify program correctness;

[0048] After the repaired code is generated, the repaired code is automatically tested with the help of test cases. If the test passes, the repair process ends and is recorded as a successful repair. Otherwise, the subsequent process continues.

[0049] Step 6: Provide a failed test case, and the large language model will generate the repaired program again;

[0050] The large language model combines the test case information that the defective code does not meet expectations during the automated testing process, and combines it with the records of incorrect repairs generated in the previous steps to generate the repaired code again.

[0051] Step 7: Automatically test and verify the correctness of the program;

[0052] After the repair code is generated, the repair code is automatically tested with the help of test cases. If the test passes, it is recorded as a successful repair, otherwise it is recorded as a failed repair. You can decide whether to execute again from step 1 based on the actual situation.

[0053] CREF uses key technologies such as large language models, prompt word engineering, and conversational program repair. The CREF method was compared with the Baseline method that directly repairs programs using a large language model, the T&S&F method that provides assistant guidance information, problem-solving documents, and failed test cases in a single conversation, and the Multi Regenerate method that does not use the failure repair records of the previous steps during the conversation. The Top 5 repair accuracy and RPSR repair precision were evaluated, showing that the CREF method has a significant advantage in terms of accuracy:

[0054]

[0055] In this method, since it is necessary to select a suitable large language model to reason about the repaired code in the subsequent program repair process, it is necessary to construct a more reasonable evaluation dataset and select appropriate evaluation indicators in order to evaluate the program repair performance of the large language model.

[0056] The process of constructing the dataset focuses on collecting and organizing the manually written defect brief analysis corresponding to each defect code, as well as the program title description, program test cases, program solution documents, and benchmark repair code. The design of test cases needs to avoid the problem of patch overfitting (the program passes all test cases but still has defects), and in order for the large language model to better understand the information, it is necessary to avoid a single test case being too large, thereby exceeding the context length limit that the large language model can accept, or causing forgetting caused by the problem of long prompts.

[0057] Evaluation indicators need to start from two dimensions, namely, repair accuracy and patch precision, which respectively describe whether the generated repaired code can pass the verification of all test cases and the relative size of the patch. The evaluation indicators used in this method include Top5 and RPSR (Release Patch Size Rate), where Top5 indicates the proportion of at least one of the five generated results that correctly passes all test cases, and RPSR is calculated based on the tree edit distance of the abstract syntax tree:

[0058]

[0059] Here, the numerator is the tree edit distance between the defective code and the fixed code generated by LLM, and the denominator is the tree edit distance between the defective code and the baseline fixed code.

[0060] In this method, there is no explicit restriction on the selection of a large language model, but choosing a large language model that performs better in various code tasks will bring higher repair accuracy and precision in the subsequent program repair process.

[0061] In the program repair task, the large language model mainly needs to set the following hyperparameters, and the specific values ​​can be determined according to the actual situation:

[0062] 1. Temperature: determines the randomness of the content generated by the large language model. The lower the value, the weaker the randomness. A value of 0 means that a greedy selection method is used, and it can be considered that the randomness of the generated content is extremely low. The temperature value used in this method is 1.0, which is also the default temperature value of most large language models.

[0063] 2. Candidate list size (Top_p): The tokens with the highest probability whose sum of probability does not exceed a certain value are included in the candidate list, with a value between 0 and 1. The candidate list size used in this method is 1.0, which means that all tokens with probability will be included in the candidate list;

[0064] 3. Context length: In order to avoid the forgetting problem caused by too long prompt words in large language models, this method sets the context length to 2048 tokens.

[0065] The prompt word format is designed as follows:

[0066]

[0067] In this method, the prompt words adopt the principle of minimalist design, and the defect code and the description of the problem associated with the code, the solution document, the manual preliminary analysis information, and the error test case are input into the large language model through the simplest few sentences. Since the understanding ability of the large language model will decrease when processing large text content, this method provides each category of information as a separate conversation record to the large language model, that is, there may be multiple consecutive conversation records from users at the beginning, so that the large language model can more accurately capture the key information.

[0068] When CREF conducts multiple rounds of repair dialogues, it is necessary to input the dialogue records of the previous round of failed repairs each time the large language model generates the next repair program. In this process, the original dialogue information will be maintained, that is, each round of dialogue is based on the dialogue information added on the basis of the previous round. Because the number of tokens of failed test cases is usually significantly greater than other types of context information, the failed test cases will be provided to the large language model for repair program generation reasoning in the last round of CREF.

[0069] For online education scenarios, the present invention proposes a human-computer interactive program repair method CREF based on a large language model, with the help of technologies and methods such as a large language model, a prompt word project, and a conversational program repair method. The accuracy of the CREF method Top5 and the precision RPSR on the test set are significantly improved compared with the baseline base line, the T&S&F method that provides all information at once, and the MutliRegenerate method that ignores the error repair in the previous step, and can be put into practical online programming education applications. At the same time, the method also has application potential in other fields of software engineering. For example, in the code review process, the method can be used to use the code review records for subsequent automated program repair to improve the development efficiency of software engineering.

[0070] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A human-computer collaborative program repair method based on a large language model, characterized in that: The following steps are involved: S1: The human assistant gives a preliminary analysis of the defective code; S2: The procedure after the repair of large language model generation; S3: Automatic testing verifies program correctness; S4: Provide solution documents, and the large language model will generate the repaired program again; S5: Automatic testing verifies program correctness; S6: Provide a failed test case, and the large language model will generate the repaired program again; S7: Automatic testing verifies program correctness.

2. The human-computer collaborative program repair method based on a large language model as claimed in claim 1, characterized in that: In step S1, the human assistant performs a preliminary analysis of the defect code based on the question information. The analysis content is a brief statement of one to two sentences.

3. The human-computer collaborative program repair method based on a large language model as claimed in claim 1, characterized in that: In step S2, the large language model combines the preliminary analysis results given by the human assistant and the defective code to generate a repaired program.

4. The human-computer collaborative program repair method based on a large language model as claimed in claim 1, characterized in that: In step S3, if the repaired program passes all test cases, the repair process ends and is recorded as successful, otherwise the subsequent steps are continued.

5. The human-computer collaborative program repair method based on a large language model as claimed in claim 1, characterized in that: In step S4, the large language model combines the solution document of the problem corresponding to the defect code and the previous record of incorrect repair to generate the repaired program again.

6. The human-computer collaborative program repair method based on a large language model as claimed in claim 1, characterized in that: In step S5, if the repaired program generated again passes all test cases, the repair process ends and is recorded as successful repair, otherwise the subsequent steps are continued.

7. The human-computer collaborative program repair method based on a large language model as claimed in claim 1, characterized in that: In step S6, the large language model combines the failed test cases and the records that were not correctly repaired before to generate the repaired program again.

8. The human-computer collaborative program repair method based on a large language model as claimed in claim 1, characterized in that: In step S7, if the repaired program finally generated passes all test cases, it is recorded as a successful repair, otherwise it is recorded as a failed repair, and it can be decided whether to restart the execution from step S1.

9. The human-machine collaborative program repair method based on a large language model according to any one of claims 1 to 8, characterized in that: When constructing an evaluation dataset, it is necessary to collect and organize the manually written defect brief analysis, program title description, program test case, program solution document, and benchmark repair code corresponding to each defect code. The test case design must avoid problems caused by patch overfitting and a single test case being too large.

10. The human-machine collaborative program repair method based on a large language model according to any one of claims 1 to 9, characterized in that: Evaluation indicators include Top5 and RPSR. Top5 indicates the proportion of all test cases that pass at least one of the five generated results correctly. RPSR is calculated based on the tree edit distance of the abstract syntax tree. The calculation formula is: The numerator is the tree edit distance between the defective code and the fixed code generated by the large language model, and the denominator is the tree edit distance between the defective code and the baseline fixed code.

Images