A large model-based code generation method and system

Abstract

The application belongs to the technical field of code automatic generation, and provides a code generation method and system based on a large model. A pre-trained large model is used to convert a natural language description problem into a plan including multiple sequentially executed subtask steps, generate a logical deduction set of the plan and a deduction output set of each test case, compare the logical deduction set and the deduction output set, verify the plan, and correct the plan that fails to pass the verification. According to the problem, the verified plan, and the logical deduction process of each test case, multiple code segments are generated in parallel. The generated code segments are tested using a test case set, the test pass rate of each code segment on the test case set is calculated, the code segment with the highest pass rate is selected as the optimal code segment, and the selected code segment is higher than the set threshold value. The application significantly reduces the required iteration number, and greatly improves the overall efficiency and success rate of code generation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of code automatic generation, and particularly relates to a code generation method and system based on a large model. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Large language models have shown great capabilities in code generation tasks, but when dealing with complex tasks, they still face great challenges in accurately understanding requirements, logical reasoning, and generating high-quality code. In recent years, multi-agent technology has become a research hotspot to break this bottleneck by simulating different roles in software engineering (such as product managers, architects, development engineers, and test engineers) to work together to better handle complex development tasks. However, existing multi-agent solutions are still in the exploratory stage and have obvious shortcomings in the reliability and efficiency of the process.

[0004] Existing code generation techniques generally decompose task requirements into multi-step task planning, but generally treat it as a simple text generation problem, ignoring the logical rigor required as a basis for subsequent code generation, lacking effective verification mechanisms, lacking verification of the correctness of the planning itself, and unable to guarantee the accuracy and reasonableness of the generated task planning. An incorrect plan will directly lead to the code generated subsequently failing to meet the original requirements.

[0005] On the other hand, in the code generation and testing phase, the existing method generally adopts a simple serial loop of generation-testing-optimization in the iterative optimization process, with strong randomness in the iteration starting point and limited search space, resulting in slow convergence speed, multiple iterations, and ineffective parallel exploration of the solution space. It also lacks sufficient use of feedback information, is blind and inefficient, and requires a large number of repeated iterations to generate code that meets the requirements, resulting in excessive time consumption and seriously hindering the large-scale application of the technology in actual business scenarios. SUMMARY

[0006] To solve the above problems, the present application proposes a code generation method and system based on a large model. The present application verifies the correctness of the planning phase by testing the task planning with test cases, ensuring the quality of code generation from the source. In the code generation phase, multiple candidate codes are generated in parallel to explore the diversity of solutions, and the optimal candidate is selected for serial iteration, balancing the exploration breadth and optimization efficiency, significantly reducing the number of iterations required, and greatly improving the overall efficiency and success rate of code generation.

[0007] According to some embodiments, the present application adopts the following technical solutions:

[0008] A large model-based code generation method, comprising the following steps:

[0009] Converting a natural language description problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model;

[0010] Introducing a test case set-based verification mechanism, using a pre-trained large model to generate a set of logical deductions of the plan and a set of deduction outputs of each test case, verifying the plan by comparing the set of logical deductions and the set of deduction outputs, and correcting the plan that fails the verification;

[0011] Using a pre-trained large model to generate a plurality of code snippets in parallel using the problem, the verified plan, and the logical deduction process of each test case as input;

[0012] Using the test case set to test each code snippet generated in the current round, calculating the test pass rate of each code snippet on the test case set, and selecting the code snippet with the highest pass rate as the optimal code snippet in the current round;

[0013] Determine whether the pass rate of the optimal code snippet in the current round is higher than the set threshold, if yes, take the optimal code snippet in the current round as the final result, otherwise enter the next round of code snippet iteration optimization until the pass rate of the optimal code snippet in the current round is higher than the set threshold.

[0014] As an optional implementation, the process of converting a natural language description problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model includes: using a pre-trained large model to convert a natural language description problem Q into a multi-step plan P through a prompt word engineering, the prompt word engineering adopts a few-shot prompt technique, and the prompt word contains a plurality of problem and plan pairs (Q, P).

[0015] As an optional implementation, the process of introducing a test case set-based verification mechanism includes: defining a test case set, each test case in the set consisting of an input-output pair, inputting the plan, the problem, and all test cases into the large model, and guiding the large model to generate a set of logical deductions based on the plan and a set of deduction outputs for each test case through a prompt word.

[0016] Compare the set of deduction outputs with the set of expected outputs of the samples one by one, if the deduction outputs of all test cases are consistent with the expected outputs, the plan passes all test cases.

[0017] The process of modifying the planning that fails to pass the verification is that if the derived output of the test case is inconsistent with the expected output, it is considered as a failure, and the failure information is fed back to the process of converting the problem described in natural language into the planning including multiple sequentially executed subtask steps by using the pre-trained large model, and the planning is iteratively modified until the planning passes all test cases or reaches the maximum number of iterations.

[0018] As an alternative embodiment, the process of generating multiple code snippets in parallel by using the pre-trained large model as input includes: taking the problem description, the verified planning, and the logical derivation process of each test case as input, guiding the pre-trained large model to generate multiple functionally complete code snippets by using prompt words, and the logical derivation process of each test case provides logical constraints for code generation, guiding the pre-trained large model to generate code with correct running logic.

[0019] As an alternative embodiment, the process of entering the next round of iterative optimization of the code snippet includes: randomly selecting a test case from the failed test cases, feeding the test case, its running error information, the problem description, the logical derivation process of the test case, and the code snippet to be optimized to the code generation agent, and the code generation agent generates a new round of multiple optimized code snippets according to the above information, recalculates the test passing rate of each code snippet on the test case set for each newly generated code snippet, and selects the code snippet with the highest passing rate as the optimal code snippet of the current round.

[0020] As a further embodiment, if the passing rate of the code snippet with the highest passing rate in the new round is still not higher than the threshold, the next round of iterative optimization of the code snippet is entered again until the passing rate of the optimal code snippet of the current round is higher than the threshold or the maximum number of iterations is reached.

[0021] As a further embodiment, if the maximum number of iterations has been reached and the passing rate of the optimal code snippet of the current round is still not higher than the threshold, the code snippet with the highest global test passing rate is selected from all rounds of optimal snippets as the final result.

[0022] As an alternative embodiment, the pre-trained large model is a large language model, and each step is implemented by different agents.

[0023] A code generation system based on a large model, comprising:

[0024] a planning generation module configured to convert a problem described in natural language into a planning including multiple sequentially executed subtask steps by using a pre-trained large model;

[0025] The planning verification module is configured to introduce a verification mechanism based on the test case set, generate a logical derivation set of the planning and a derivation output set of each test case by using the pre-trained large model, verify the planning by comparing the logical derivation set and the derivation output set, and correct the planning that fails to pass the verification;

[0026] The candidate code snippet generation module is configured to take the question, the planning that passes the verification, and the logical derivation process of each test case as input, and generate multiple code snippets in parallel by using the pre-trained large model;

[0027] The code testing module is configured to test each code snippet generated in the current round by using the test case set, calculate the test passing rate of each code snippet on the test case set, and select the code snippet with the highest passing rate as the optimal code snippet in the current round.

[0028] The final code snippet determination module is configured to determine whether the passing rate of the optimal code snippet in the current round is higher than a set threshold value, and if yes, take the optimal code snippet in the current round as the final result, or if not, enter the iteration optimization of the code snippet in the next round until the passing rate of the optimal code snippet in the current round is higher than the set threshold value.

[0029] An electronic device includes a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the steps in the above method are completed.

[0030] Compared with the prior art, the beneficial effects of the present application are:

[0031] The present application cooperates the task planning agent, the code generation agent and the test agent to work, and efficiently generates high-quality code through the self-adaptive optimization mechanism, and the present application is oriented to functionally complete function-level code generation, and is suitable for solving the code generation problem in a complex task scenario.

[0032] In the planning optimization and code optimization two stages, the feedback of the test case is used to drive iteration, wherein, in the planning stage, the planning verification mechanism is introduced to ensure the correctness of the generated planning; in the code generation and testing stage, a parallel-serial hybrid optimization strategy is adopted, multiple code snippets are generated in parallel first, the optimal solution is selected from multiple solutions, and the feedback information of the failed test case is used for targeted serial optimization of the optimal code, so that the target code is efficiently and high-quality generated.

[0033] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which constitute a part of this specification, are incorporated herein to provide further understanding of the present application, and are incorporated for explanation of the exemplary embodiments of the present application and to provide further explanation of the present application, and do not constitute an improper limitation of the present application.

[0035] Figure 1 is a schematic diagram of a code generation process of an embodiment;

[0036] Figure 2 is a schematic diagram of a task planning process of an embodiment;

[0037] Figure 3 is a schematic diagram of a code generation and testing process of an embodiment. DETAILED DESCRIPTION

[0038] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed description is illustrative only, and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0041] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0042] Embodiment One

[0043] A code generation method based on a large model, as shown in Figure 1 includes the following steps:

[0044] Converting a natural language description problem into a plan including a plurality of sequentially executed subtask steps using a pre-trained large model;

[0045] Introducing a verification mechanism based on a test case set, generating a logical derivation set of the plan and a derivation output set of each test case using the pre-trained large model, verifying the plan by comparing the logical derivation set and the derivation output set, and correcting the plan that fails the verification;

[0046] Taking the problem, the verified plan, and the logical derivation process of each test case as input, generating a plurality of code fragments in parallel using the pre-trained large model;

[0047] The code snippets generated in this round are tested using a set of test cases. The pass rate of each code snippet on the set of test cases is calculated, and the code snippet with the highest pass rate is selected as the optimal code snippet in this round.

[0048] Determine if the pass rate of the best code snippet in this round is higher than the set threshold. If it is, the best code snippet in this round is taken as the final result. Otherwise, proceed to the next round of code snippet iteration optimization until the pass rate of the best code snippet in the current round is higher than the set threshold.

[0049] Each step is implemented by an independent intelligent agent, which will be described in detail below:

[0050] Utilizing task planning agents, such as Figure 2 As shown, complex requirements are decomposed and verified before code generation to generate a reliable task plan.

[0051] Specifically, it includes:

[0052] Plan generation:

[0053] Using a Large Language Model (LLM), the problem Q (or problem requirement Q, problem description Q) described in natural language is transformed into a multi-step plan P through prompt word engineering. The prompt words employ few-shot prompting technology, containing multiple problem-plan pairs (Q, P) to guide the large model in generating a well-formatted and logically clear plan. The plan should consist of multiple sequentially executed sub-task steps, with concise and clear descriptions, and may include control logic such as conditional branches and loops.

[0054] Planning, Validation, and Iteration:

[0055] To ensure planning quality, a verification mechanism based on test case sets is introduced to verify the accuracy and rationality of the planning. A test case set for a given sample is defined. Each test case It consists of input-output pairs, that is For the planning P, it is compared with the problem description Q and the input set of all test cases { Input the large model together, and guide it with prompts to generate a set of logical derivations based on the plan. and the derivation output set of each test case : ;

[0056] Derivation output set Expected output set of the sample A one-to-one comparison is made. If the derived output of all test cases is consistent with the expected output, the plan P passes all test cases, otherwise the failure information is fed back to the plan generation step, and the plan P is iteratively corrected until the plan P passes all test cases or reaches the maximum number of iterations, that is:

[0057] ;

[0058] The verified task planning and logical deduction examples are used to generate candidate code based on the code generation agent.

[0059] The problem description Q, the verified plan P, and the logical deduction process of the k test cases are input to guide the large model to generate n functionally complete code snippets (functions) in parallel through prompt words:

[0060] ;

[0061] wherein, is the n code snippets generated by the large model, is the logical deduction process of the k test cases, which provides logical constraints for code generation and guides the model to generate code with correct running logic.

[0062] The test agent is used to test the code and manage the iterative optimization process.

[0063] Code testing and optimization:

[0064] As shown in Figure 3 , the n code snippets generated in this round of iteration (superscript (1) represents the first round of iteration) are tested using the test case set T, and the test pass rate of each code snippet on the test case set is calculated . The code snippet with the highest pass rate is selected as the optimal code snippet in this round, and if its test pass rate is higher than the preset pass rate threshold (e.g., 0.9), the code snippet is output as the final result, and the process terminates.

[0065] Code iterative optimization:

[0066] If the test pass rate is not higher than the threshold , the next round of code snippet iterative optimization is entered.

[0067] A test case is randomly selected from the failed test cases , and the test case and its running error information , problem description Q, logical deduction of this use case , and code snippet to be optimized Feedback to the code generation agent, as Figure 3 shown, the code generation agent generates a new round of n optimized code snippets accordingly

[0068] ;

[0069] The superscript (2) means the second round, and similarly, the newly generated code snippets repeat the testing and optimization process. This cycle continues until the test pass rate of the optimal code snippet is higher than the threshold , or the maximum number of iterations m is reached.

[0070] If the threshold is not met after reaching the maximum number of iterations, the code snippet with the highest global test pass rate is selected from all m rounds of optimal snippets as the final output.

[0071] Embodiment Two

[0072] A code generation system based on a large model, comprising:

[0073] a planning generation module configured to convert a natural language description problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model;

[0074] a planning verification module configured to introduce a verification mechanism based on a set of test cases, generate a set of logical deductions of the plan and a set of deduction outputs of each test case using a pre-trained large model, verify the plan by comparing the set of logical deductions and the set of deduction outputs, and correct the plan that fails the verification;

[0075] a candidate code snippet generation module configured to use the problem, the verified plan, and the logical deduction process of each test case as input, and use a pre-trained large model to generate a plurality of code snippets in parallel;

[0076] a code testing module configured to test each code snippet generated in the current round using the set of test cases, calculate the test pass rate of each code snippet on the set of test cases, and select the code snippet with the highest pass rate as the optimal code snippet in the current round;

[0077] a final code snippet determination module configured to determine whether the pass rate of the optimal code snippet in the current round is higher than a set threshold, and if so, take the optimal code snippet in the current round as the final result, otherwise enter the next round of code snippet iteration optimization until the pass rate of the optimal code snippet in the current round is higher than the set threshold.​

[0078] Example Three

[0079] An electronic device including memory and a processor and computer instructions stored on the memory and running on the processor that, when executed by the processor, complete the steps of the above method.

[0080] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code. CD - ROM

[0081] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1

[0082] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1

[0084] ​​​​​​​The above merely describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art without departing from the spirit and principle of the present application shall fall within the protection scope of the present application.

Claims

1. A large model-based code generation method, characterized by, The method comprises the following steps: Converting a natural language description of a problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model; Introducing a verification mechanism based on a set of test cases, generating a set of logical deductions of the plan and a set of deduction outputs of each test case using the pre-trained large model, verifying the plan by comparing the set of logical deductions and the set of deduction outputs, and correcting the plan that fails the verification; Using the pre-trained large model to generate a plurality of code snippets in parallel as input, using the pre-trained large model to generate a plurality of code snippets in parallel, and using the pre-trained large model to generate a plurality of code snippets in parallel; Testing each code snippet generated in the current round using the set of test cases, calculating the test pass rate of each code snippet on the set of test cases, and selecting the code snippet with the highest pass rate as the optimal code snippet in the current round; Determining whether the pass rate of the optimal code snippet in the current round is higher than a set threshold, and if so, taking the optimal code snippet in the current round as the final result, otherwise entering the next round of code snippet iterative optimization until the pass rate of the optimal code snippet in the current round is higher than the set threshold; The process of introducing a verification mechanism based on a set of test cases includes defining a set of test cases, each test case in the set consisting of an input-output pair, inputting the plan, the problem and all test cases into the large model, and guiding the large model to generate a set of logical deductions based on the plan and a set of deduction outputs for each test case through prompt words; Comparing the set of deduction outputs with the set of expected outputs of the samples one by one, and if the deduction outputs of all test cases are consistent with the expected outputs, the plan passes all test cases; If the deduction outputs of a test case are inconsistent with the expected outputs, it is considered a failure, the failure information is fed back to the process of converting a natural language description of a problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model, and the plan is iteratively corrected until the plan passes all test cases or the maximum number of iterations is reached.

2. The large model-based code generation method of claim 1, wherein, The process of converting a natural language description of a problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model includes converting a natural language description of a problem Q into a multi-step plan P using a pre-trained large model through prompt word engineering, the prompt word engineering uses a few-shot prompting technique, and the prompt words contain a plurality of problem and plan pairs (Q, P).

3. The large model-based code generation method of claim 1, wherein, The process of generating a plurality of code snippets in parallel using the pre-trained large model as input, the verified plan, and the logical deduction process of each test case includes inputting the problem description, the verified plan, and the logical deduction process of each test case, guiding the pre-trained large model to generate a plurality of functionally complete code snippets through prompt words, and providing logical constraints for code generation using the logical deduction process of the test cases to guide the pre-trained large model to generate code with correct running logic.

4. The large model-based code generation method of claim 1, wherein, The process of the next round of code snippet iterative optimization includes: randomly selecting a test case from the failed test cases, feeding the test case, its running error information, problem description, logical derivation process of the test case, and the code snippet to be optimized to the code generation agent, and the code generation agent generates a plurality of optimized code snippets in the new round according to the above information, recalculates the test passing rate of each code snippet on the test case set for the newly generated code snippets, and selects the code snippet with the highest passing rate as the optimal code snippet in the current round.

5. The large model-based code generation method of claim 4, wherein, If the passing rate of the code snippet with the highest passing rate in the new round is still not higher than the threshold, the next round of code snippet iterative optimization is entered again until the passing rate of the optimal code snippet in the current round is higher than the threshold or the maximum iteration number is reached.

6. The large model-based code generation method of claim 5, wherein, If the maximum iteration number has been reached and the passing rate of the optimal code snippet in the current round is still not higher than the threshold, the code snippet with the highest global test passing rate is selected from the optimal snippets in all rounds as the final result.

7. The large model-based code generation method of claim 1, wherein, The pre-trained large model is a large language model, and each step is implemented by different agents.

8. A large model-based code generation system, characterized by, It comprises: a plan generation module configured to convert a natural language description problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model; a plan verification module configured to introduce a verification mechanism based on a test case set, generate a logical derivation set of the plan and a derivation output set of each test case using the pre-trained large model, verify the plan by comparing the logical derivation set and the derivation output set, and correct the plan that fails the verification; The process of introducing a verification mechanism based on a test case set includes: defining a test case set, each test case in the set consisting of an input-output pair, inputting the plan, the problem, and all test cases into the large model together, and guiding the large model to generate a logical derivation set based on the plan and a derivation output set of each test case through a prompt word; The derivation output set is compared with the expected output set of the sample one by one, and if the derivation output of all test cases is consistent with the expected output, the plan passes all test cases; The process of correcting the plan that fails the verification, if the derivation output of a test case is inconsistent with the expected output, is regarded as a failure, the failure information is fed back to the process of converting a natural language description problem into a plan comprising a plurality of sequentially executed subtask steps using a pre-trained large model, and the plan is iteratively corrected until the plan passes all test cases or the maximum iteration number is reached; a candidate code snippet generation module configured to input the problem, the verified plan, and the logical derivation process of each test case into the pre-trained large model to generate a plurality of code snippets in parallel; a code test module configured to test each code snippet generated in the current round using the test case set, calculate the test passing rate of each code snippet on the test case set, and select the code snippet with the highest passing rate as the optimal code snippet in the current round. The final code snippet determining module is configured to judge whether the passing rate of the optimal code snippet of the current round is higher than a set threshold value, and if yes, the optimal code snippet of the current round is taken as the final result, otherwise, the code snippet iteration optimization of the next round is entered until the passing rate of the optimal code snippet of the current round is higher than the set threshold value.

9. An electronic device, characterized by comprising: A computer program product, comprising a memory and a processor, and computer instructions stored on the memory and run on the processor, when the computer instructions are run by the processor, steps in the method of any one of claims 1-7 are completed.

Images