Code generation method based on consistency enhancement and user interaction
Through the code generation method of consistency enhancement and user interaction, the co-evolution strategy of code and test cases is used to improve the code reliability and test cases generated by large language models, reduce the difficulty and workload of user intervention, and improve the efficiency of code development.
Patent Information
- Application Number
- CN202510465997.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-15
- Publication Date
- 2025-08-01
AI Technical Summary
The code results generated by existing large language models are unreliable, and the existing consistency technology is limited by the incorrect test metrics, which leads to the unreliable majority of voting results, and frequent user intervention and large workload.
Using a code generation method based on consistency enhancement and user interaction, we introduce user feedback through a lightweight interaction framework, and use the co-evolution strategy of code and test cases to gradually improve the quality of code and test cases, and reduce user intervention rounds and workload.
It significantly improves the accuracy of test cases and code reliability of large language models, reduces the difficulty and frequency of user intervention, and improves the efficiency and quality of code development.
Smart Images
Figure CN120407403A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of artificial intelligence and software engineering, and relates to a testing and interaction framework for improving the reliability of code generation by large language models, and particularly relates to a code generation method based on consistency enhancement and user interaction. Background Art
[0002] Code generation technology automatically generates code snippets that implement specific functions according to natural language requirements, which can reduce the workload of developers writing code and improve development efficiency. In recent years, large language models (LLMs) represented by GPT-4 (Achiam et al., 2023) [1], DeepSeek-Coder (Guo et al., 2024) [2], and CodeGen (Nijkamp et al., 2022) [3] have significantly improved the performance in the field of code generation and reached the current state-of-the-art level due to their large-scale parameters and rich code corpus pre-training.
[0003] Although LLMs perform well, the results they generate are not always reliable. Existing research has proposed consistency techniques to improve the reliability of code generation, such as the Self-Consistency technique CodeT (Chen et al., 2022) [4], MPSC (Huang et al., 2023) [5] which improve output consistency by generating multiple candidate solutions and using majority voting to improve the reliability of the results. However, the prerequisite for such techniques is that the test metrics used to evaluate consistency must themselves be highly accurate. Since the test metrics are usually also generated by LLMs, they may contain a high error rate, resulting in unreliable majority voting results based on incorrect tests. Experiments show that in three commonly used code generation datasets (HumanEval, HumanEvalPlus, and MBPP), the average error rate of test cases generated by LLMs is as high as 37.7%, seriously affecting the effectiveness of existing Self-Consistency techniques. Therefore, it is of great significance to study a code generation method based on consistency enhancement and user interaction.
[0004] [1] Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report [J]. arXiv preprint arXiv:2303.08774, 2023.
[0005] [2] Guo D, Zhu Q, Yang D, et al. DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence[J]. arXiv preprint arXiv:2401.14196, 2024.
[0006] [3] Nijkamp E, Pang B, Hayashi H, et al. Codegen: An open large language model for code with multi-turn program synthesis[J]. arXiv preprint arXiv:2203.13474, 2022.
[0007] [4] Chen B, Zhang F, Nguyen A, et al. CodeT: Code Generation with Generated Tests[C] / / The Eleventh International Conference on Learning Representations.
[0008] [5] Huang B, Lu S, Wan X, et al. Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency[C] / / Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers). 2024:1429-1450. Summary of the Invention
[0009] In view of the problems existing in the above prior art, the present invention proposes a code generation method based on consistency enhancement and user interaction. By introducing user feedback through a lightweight interaction framework, the accuracy of test cases generated by a large language model (LLM) is improved, and the reliability of the code is further enhanced. The present invention adopts a co-evolution strategy for code and test cases, and uses two-way consistency voting (from code to test cases, from test cases to code) to gradually improve the quality of code and test cases, significantly reducing the number of rounds and workload of user intervention. Through consistency voting from code to test cases, select the test cases that pass the least amount of code, that is, the test cases that are most likely to be incorrect, and let the user correct them, which can reduce the number of rounds of user intervention. Through the consistency from test cases to code, as the quality of test cases improves, the code that passes more test cases is also more likely to be correct.
[0010] The technical solution of the present invention is as follows:
[0011] A method for code generation based on consistency enhancement and user interaction, comprising the following steps:
[0012] A. Generation of the initial code set and test case set, the implementation method is:
[0013] A1. Based on the natural language problem description and method signature, use a large language model (LLM) to generate a set of initial code collections as the original code set.
[0014] A2. Use the same large language model (LLM) to generate a corresponding initial test case set as the test set to be confirmed according to the same natural language problem description and method signature.
[0015] B. Consistency voting and sorting, the implementation method is:
[0016] B1. Execute each code in the original code set on each test case in the test set to be confirmed, and calculate the consistency voting relationship between the code and the test case.
[0017] B2. Use the consistency voting results from code to test cases to sort the test cases, and sort the test cases in ascending order according to the number of codes passed, and give priority to identifying the test cases with the least number of passed codes as the most likely incorrect test cases.
[0018] C. User feedback and test case correction, the implementation method is:
[0019] C1. For the most likely incorrect test cases identified in step B2, introduce user feedback through an interactive method to confirm and correct them.
[0020] C2. Update the test set, and move the corrected test cases from the test set to be confirmed to the corrected test set.
[0021] D. Code repair and code testing based on the corrected test set are implemented as follows:
[0022] D1. Execute all the code on the corrected test set obtained in step C to get the code that fails to execute. Use the same large language model (LLM) to repair the code that fails to pass the corrected test cases to obtain a set of repaired code.
[0023] D2. Test whether the repaired code can pass all the corrected test cases. The code that cannot pass all the corrected test cases will be moved from the original code set to the discarded code set.
[0024] E. Termination condition judgment is implemented as follows:
[0025] Judge whether the termination condition is met. When any of the following termination conditions is met, terminate the process; the termination conditions are:
[0026] (1) In a certain iteration, if there is no repaired code that can pass all the corrected test cases, select the code that passes the most test cases and output it; terminate the iteration.
[0027] (2) When there is a piece of code that passes all the corrected and to-be-confirmed test cases, terminate the iteration process and output the code.
[0028] (3) When all the to-be-confirmed test cases are corrected, terminate the iteration process and output the code that passes the most test cases.
[0029] When none of the termination conditions is met, repeat the iteration: from step B to this step.
[0030] Furthermore, the test cases in step A2 are represented in the form of assertions (assert).
[0031] Furthermore, the code running test cases in step B1 are executed in parallel.
[0032] Furthermore, in step C1, the user feedback only needs to provide the correct output value corresponding to the test case input, without modifying the code, as it is easier to modify the test case than the code.
[0033] Furthermore, when using the large language model (LLM) to repair the code in step D1, the prompts provided to the large language model (LLM) include:
[0034] The corrected test cases that the current code fails to pass;
[0035] If there are other previously corrected test cases, they will also be provided;
[0036] The model is required to modify the code so that the repaired code can pass all the corrected test cases.
[0037] The technical effects of the present invention are as follows:
[0038] A code generation method based on consistency enhancement and user interaction according to the present invention adopts a co-evolution strategy of code and test cases, accurately identifies test cases that may be incorrect through consistency voting, efficiently corrects test cases in combination with user feedback, and further effectively repairs the code using the corrected test cases, significantly improving the accuracy of test cases generated by large language models (LLMs) and the reliability of code generation, reducing the difficulty and frequency of developer intervention, thereby improving code development efficiency and generation quality. Brief Description of the Drawings
[0039] Figure 1 is the overall framework diagram of the code generation method based on consistency enhancement and user interaction of the present invention;
[0040] Figure 2 is the flowchart of the method for consistency enhancement in a specific embodiment of the present invention. Detailed Embodiments
[0041] The present invention will be further clearly and completely described below with reference to the accompanying drawings and specific embodiments.
[0042] The overall framework diagram of a code generation method based on consistency enhancement and user interaction of the present invention is as Figure 1 shown. Among them, the large language model (LLM) generates an initial test case set and a code set, calculates the consistency voting relationship (Consistency Voting) between the code set and the test set, corrects the test cases through user feedback, corrects the code based on the corrected test cases, and finally outputs a more reliable and accurate code, and the quality of this code is better than any single code in the original code set. The flowchart of the method of the present invention is as Figure 2 shown. The main body of the method is a loop iteration process. During the iteration process, the test set is divided into two subsets: the corrected test set and the to-be-confirmed test set. The to-be-confirmed test set contains test cases in the generated test cases that have not been corrected by the user; the corrected test set contains test cases that have been corrected by the user; when all to-be-confirmed test cases are corrected, or a code that can pass all test cases is found, the iteration terminates. Each iteration includes: consistency voting and sorting, user feedback and test case correction, code repair and code testing, and termination condition judgment, reflecting the co-evolution process of test cases and code. The specific steps of the method are as follows:
[0043] A. Generation of the initial code set and test case set
[0044] A1. Based on the natural language problem description and method signature, use a large language model (LLM) to generate an initial set of code as the original code set.
[0045] A2. Use the same LLM to generate an initial set of test cases corresponding to the initial code set as the test set to be confirmed, where the test cases are represented in the form of assertions (assert).
[0046] B. Consistency Voting and Sorting
[0047] B1. Run each test case in the test set to be confirmed on each code in the original code set to collect preliminary execution information. Specifically, record whether each code can pass each test. It should be noted that this process can be executed in parallel, so the efficiency is relatively high. After the execution is completed, calculate the consistency voting relationship between the code and the test case, that is, the consistency from the test case to the code is how many test cases each code can pass, and the consistency from the code to the test case is how many codes each test case can pass.
[0048] B2. Use the consistency voting results from the code to the test case to sort the test cases. Sort the test cases in ascending order according to the number of codes that pass, to identify the test cases that are most likely to be incorrect, that is, the test cases passed by the smallest number of codes, called suspicious test cases, which are selected as the objects to be corrected in the next step. As the iteration progresses, the quality of the code gradually improves, so the accuracy of identifying incorrect test cases also continuously increases.
[0049] As Figure 2 in this stage, t1, t2, t3, t4 are the test cases in the test set to be confirmed:
[0050] t4 is passed by 3 codes; t2 and t3 are each passed by 2 codes; t1 is passed by only 1 code.
[0051] Therefore, according to the consistency voting results from the code to the test case, t1 has the fewest passing codes and is most likely to be incorrect, so it is selected as the object to be corrected in the next step.
[0052] C. User Feedback and Test Case Correction
[0053] C1. For the test cases that are most likely to be incorrect identified in step B2, introduce user feedback through an interactive method to confirm and correct the suspicious test cases. Specifically, the user will be requested to check and correct the test case results. The user only needs to provide the correct test output according to the problem description and test input to complete the correction. The corrected test cases are called corrected test cases.
[0054] C2. Update the test set, remove the corrected test case from the to-be-confirmed test set, and add it to the corrected test set.
[0055] As Figure 2 in this stage, t1 is corrected by the user to t1', then it is removed from the to-be-confirmed test set and added to the corrected test set.
[0056] D. Code repair and code testing based on the corrected test set
[0057] D1. Execute all the code on the corrected test set obtained in step C to get the code that fails to execute. Use the large language model (LLM) to repair the code that fails to pass the corrected test cases to obtain the repaired code set. The same large language model (LLM) used to initially generate the code is still used in the specific implementation, which is convenient for clearly demonstrating the improvement brought by co-evolution.
[0058] As Figure 2 in this stage, the original codes c1 and c3 cannot pass the just-corrected t1', while c2 can pass. Therefore, the same LLM model is used to repair the failed codes to obtain the repaired codes c1' and c3'.
[0059] When repairing the code, the prompts provided to the large language model (LLM) include:
[0060] The corrected test cases that the current code fails to pass (e.g., t1');
[0061] If there are other previously corrected test cases, they will also be provided;
[0062] Request the model to modify the code so that the repaired code can pass all the corrected test cases.
[0063] D2. After the code repair is completed, test whether the repaired code can pass all the corrected test cases. The code that cannot pass all the corrected test cases will be moved from the original code set to the discarded code set. This process highlights the importance of code diversity: some codes may have serious logical errors and are difficult to repair, while others may only fail in special boundary cases, and such codes can be improved through repair. Diversity increases the chance of obtaining the correct code.
[0064] E. Termination condition judgment
[0065] Judge whether the termination condition is met. When any of the following termination conditions is met, terminate the process; the termination conditions are:
[0066] (1) In a certain iteration, if no repaired code can pass all the corrected test cases, then select the code that passes the most test cases and output it; terminate the iteration.
[0067] (2) When there is a piece of code that passes all the corrected and to-be-confirmed test cases, terminate the iteration process and output the code.
[0068] (3) When all the to-be-confirmed test cases are corrected, terminate the iteration process and output the code that passes the most test cases.
[0069] When none of the termination conditions are met, repeat the iteration: from step B to this step.
[0070] In the method of the present invention, as more and more test cases are corrected, the consistency voting of the test cases on the code becomes more and more reliable. Therefore, the code that passes all the test cases usually has a higher probability of correctness. On the other hand, as the code quality continuously improves, the consistency voting of the code on the test also becomes more accurate. Therefore, the identification of incorrect tests in the next iteration will be more precise.
[0071] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art can understand that: without departing from the spirit and scope of the present invention and the appended claims, various substitutions and modifications are possible. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present invention is defined by the scope defined in the claims.
Claims
1. A method for code generation based on consistency enhancement and user interaction, characterized in that, It includes the following steps: A. Generation of Initial Code Set and Test Case Set A1. Based on the natural language problem description and method signature, use a large language model to generate a set of initial code collections as the original code set; A2. Use the same large language model to generate the corresponding initial test case set according to the same natural language problem description and method signature as the test set to be confirmed; B. Consistency Voting and Sorting B1. Execute each code in the original code set on each test case in the test set to be confirmed, and calculate the consistency voting relationship between the code and the test case; B2. Use the consistency voting results from code to test cases to sort the test cases. Sort the test cases from the least number of passing codes to the most, and preferentially identify the test case with the least number of passing codes as the most likely incorrect test case; C. User Feedback and Test Case Correction C1. For the most likely incorrect test case identified in step B2, introduce user feedback through an interactive method to confirm and correct it; C2. Update the test set, and move the corrected test cases from the test set to be confirmed to the corrected test set; D. Code Repair and Code Testing Based on the Corrected Test Set D1. Execute all codes on the corrected test set obtained in step C to get the codes that fail to execute. Use the same large language model to repair the codes that fail to pass the corrected test cases to obtain the repaired code set; D2. Test whether the repaired codes can pass all the corrected test cases. The codes that cannot pass all the corrected test cases will be moved from the original code set to the discarded code set; E. Termination Condition Judgment Judge whether the termination condition is met. When any of the following termination conditions is met, terminate the process; the termination conditions are: (1) In a certain iteration, if no repaired code can pass all the corrected test cases, then select the code that passes the most test cases and output it; Terminate the iteration; (2) When there is a code that passes all the corrected and to-be-confirmed test cases, terminate the iteration process and output the code; (3) When all the to-be-confirmed test cases are corrected, terminate the iteration process and output the code that passes the most test cases; When none of the termination conditions is met, repeat the iteration: from step B to this step.
2. The method according to claim 1, wherein The test cases in step A2 are represented in the form of assertions (assert).
3. The method according to claim 1, wherein The code running test cases in step B1 are executed in parallel.
4. The method according to claim 1, characterized in that In step C1, the user feedback only needs to provide the correct output value corresponding to the test case input, without modifying the code. It is easier to modify the test case than to modify the code.
5. The method according to claim 1, characterized in that, When using the large language model to repair the code in step D1, the prompts provided to the large language model include: The corrected test cases that the current code fails to pass; If there are other previously corrected test cases, they will also be provided; Request the model to modify the code so that the repaired code can pass all the corrected test cases.