A Unit Test Generation Method Based on a Large Language Model

Abstract

This invention relates to a unit test generation method based on a large language model, comprising: collecting code fine-tuning data to obtain an original dataset; preprocessing the original dataset to obtain an optimized dataset; generating unit test data based on query questions and code solutions in the optimized dataset; training a large language model using supervised fine-tuning with the optimized dataset and the generated unit test data to obtain a unit test generator; and generating unit test results for the program to be tested using the unit test generator. This invention improves the efficiency, accuracy, and flexibility of unit test generation methods, providing support for quality assurance and test optimization in the software development process.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for generating unit tests based on a large language model. Background Technology

[0002] Large Language Models (LLMs) have shown significant potential in code generation tasks, but their ability to generate correct answers in a single inference iteration remains limited, especially in scenarios requiring complex logical reasoning or precise verification. Current mainstream techniques typically employ a Best-of-N strategy to improve generation quality, which involves generating candidate answers multiple times and then using a validator or reward model to select the optimal result.

[0003] Existing research typically uses unit tests as validators to screen candidate code. Specifically, for a specific problem and code, test cases are first generated using a large language model, including the input parameters and expected output of the function being tested. Then, based on the code's performance on the test cases, the best-performing program is selected. However, this method faces many challenges in practical applications. The most significant problem is that, due to the inherent "illusion" phenomenon of large language models, the generated unit tests may be unreliable, often resulting in discrepancies between expected output and input parameters, thus weakening the validator's effectiveness in selecting the optimal program. Summary of the Invention

[0004] This invention provides a unit test generation method based on a large language model to address the shortcomings of existing technologies.

[0005] This invention provides a unit test generation method based on a large language model, comprising:

[0006] S1: Collect code fine-tuning data to obtain the original dataset;

[0007] S2: Preprocess the original dataset to obtain an optimized dataset;

[0008] S3: Generate unit test data based on the query problems and code solutions in the optimized dataset;

[0009] S4: Using the optimized dataset and the unit-generated data, train the large language model based on supervised fine-tuning to obtain a unit test generator;

[0010] S5: Generate unit test results for the program to be tested using the unit test generator.

[0011] According to the unit test generation method based on a large language model provided by the present invention, in step S1, code fine-tuning data is collected from the CodeFeedback-Filtered-Instruction dataset and the TACO dataset.

[0012] According to the unit test generation method based on a large language model provided by the present invention, step S2 further includes:

[0013] S21: Remove problematic data from the original dataset to obtain a filtered dataset;

[0014] S22: Format the code solutions in the filtered dataset into function form to obtain the optimized dataset.

[0015] According to the unit test generation method based on a large language model provided by the present invention, the problem data in step S21 includes random data, non-deterministic data, and data lacking test standards.

[0016] According to the unit test generation method based on a large language model provided by the present invention, step S3 further includes:

[0017] S31: Using the Llama3.1-70B model, the query problems and code solutions in the optimized dataset are sampled multiple times to obtain basic unit test data;

[0018] S32: Verify the basic unit test data by removing erroneous unit test data to obtain filtered unit test data;

[0019] S33: The filtered unit test data is distinguished by removing erroneous unit test data to obtain unit test data.

[0020] According to the unit test generation method based on a large language model provided by the present invention, step S32 further includes:

[0021] S321: Apply the basic unit test data to the code solution in the optimized dataset to obtain the execution result;

[0022] S322: Based on the execution result, the corresponding basic unit test data is judged. When the execution result allows the code solution to pass, the corresponding basic unit test data is correct basic unit test data. When the execution result does not allow the code solution to pass, the corresponding basic unit test data is incorrect basic unit test data.

[0023] S323: Remove the erroneous basic unit test data from the basic unit test data to obtain filtered unit test data.

[0024] According to the unit test generation method based on a large language model provided by the present invention, step S322 further includes:

[0025] By capturing execution feedback information generated by the Python interpreter, erroneous basic unit test data can be repaired.

[0026] According to the unit test generation method based on a large language model provided by the present invention, step S33 further includes:

[0027] S331: Solution for generating error codes;

[0028] S332: Apply the filtered unit test data to the error code solution. When the filtered unit test data does not distinguish between error code solutions, the corresponding filtered unit test data is error filtered unit test data. When the filtered unit test data distinguishes between error code solutions, the corresponding filtered unit test data is correct filtered unit test data.

[0029] S333: Remove the erroneous filtered unit test data from the filtered unit test data to obtain unit test data.

[0030] According to the unit test generation method based on a large language model provided by the present invention, the unit test generator in step S4 takes the query problem and code solution in the optimized dataset as input and the unit test data as output.

[0031] According to the unit test generation method based on a large language model provided by the present invention, the expression of the loss function of the unit test generator in step S4 is as follows:

[0032]

[0033] Where L(x,y) is the loss function of the unit test generator, x is the data in the input optimized dataset, y is the unit test result generated by the unit test generator, i is the index value of the obtained token, j is the index value of the given token, and Model(y i |x,y j<i) Given the j=i-1th tokens of x and y, let y be the probability of generating the i-th token of y.

[0034] This invention provides a unit test generation method based on a large language model. By collecting and preprocessing code fine-tuning data, it can generate unit test data that more closely resembles actual programming scenarios. Furthermore, by utilizing a large language model for supervised fine-tuning, it can significantly improve the accuracy and efficiency of unit test generation, making the generated unit tests more reflective of the code's actual functionality and potential problems. In the data preprocessing stage, this invention ensures the high quality of the large language model dataset used for training and optimization by eliminating random, non-deterministic, and problematic data lacking test standards. This helps reduce noise and misjudgments during unit test generation, improving the reliability and effectiveness of the generated unit tests. Secondly, this invention, through multiple sampling and verification to remove erroneous unit test data and repairing erroneous basic unit test data, can generate... More targeted and discriminative unit test data not only verifies code correctness but also effectively distinguishes solutions to erroneous code, thereby improving software test coverage and effectiveness. Furthermore, this invention trains a large language model based on optimized datasets and generated unit test data, resulting in a unit test generator with stronger generalization capabilities. This generator can generate compliant unit test data for different types of code and testing needs, further enhancing the level of automated testing in the software development process. The unit test generation method of this invention automates and intelligentizes the software testing process, reducing the workload of testers and improving testing efficiency. Simultaneously, continuous optimization and improvement of the unit test generation method can drive the software development process towards a higher level of automation and intelligence. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 A schematic diagram of a unit test generation method based on a large language model provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the optimized dataset acquisition method provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the unit test data acquisition method provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of this invention, and should not be construed as limiting the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. In the description of this invention, it should be understood that the terminology used is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0040] The embodiments of the present invention are described below with reference to the figures.

[0041] like Figure 1 As shown, this invention provides a unit test generation method based on a large language model, comprising:

[0042] S1: Collect data for code fine-tuning to obtain the original dataset.

[0043] In step S1, the existing high-quality code fine-tuning data serves as the core foundation of the entire process, and a rigorously selected dataset is chosen.

[0044] In step S1, code fine-tuning data is collected from the CodeFeedback-Filtered-Instruction dataset and the TACO dataset.

[0045] Furthermore, the collected datasets include the training sets of the CodeFeedback-Filtered-Instruction and TACO datasets. These datasets are designed to provide stable and efficient support for code generation and instruction tuning tasks, thereby ensuring that the model's performance in various complex scenarios meets or exceeds expectations.

[0046] As a fundamental part of the unit testing method of this invention, CodeFeedback-Filtered-Instruction is a highly optimized and meticulously constructed set of code instruction queries, derived from four well-known open-source code instruction fine-tuning datasets. These datasets are obtained through cleaning, integrating, and deeply optimizing a large number of open-source code resources, specifically including Magicoder-OSS-Instruct, Magicoder-Evol-Instruct, Evol-Instruct-Code, and a high-quality subset of Python code related to ShareGPT. The CodeFeedback-Filtered-Instruction dataset ensures that each instruction possesses high relevance, accuracy, and applicability, providing a reliable foundation for code generation and instruction tuning in this invention.

[0047] Meanwhile, the TACO dataset provides another important data source for this invention, focusing on algorithm code generation. This dataset integrates a large number of programming problems from multiple well-known programming platforms (such as LeetCode and Codeforces), covering not only traditional algorithm problems but also more complex and innovative programming scenarios. Through in-depth analysis and integration of these problems, the TACO dataset can provide diverse training materials for code generation models, thereby significantly improving the accuracy, stability, and versatility of the models in solving algorithmic problems.

[0048] S2: Preprocess the original dataset to obtain an optimized dataset.

[0049] In step S2, a large language model is used to filter out problems that are not suitable for unit test generation (such as problems involving random number generation). Then, the code program is formatted to convert the non-standard code in the original dataset into function form for subsequent unit test generation and processing.

[0050] like Figure 2 As shown, step S2 further includes:

[0051] S21: Remove problematic data from the original dataset to obtain a filtered dataset.

[0052] To optimize data quality, this invention employs a principle-based screening method in step S21, refining and filtering the dataset at a higher level. Specifically, this invention utilizes the Llama 3.1-70B-Instruct model to accurately identify and eliminate data unsuitable for unit testing. For example, data involving randomness, non-determinism, or lacking clear testing standards are systematically excluded. This principle-driven screening process not only ensures high testability and consistency of the data but also lays a solid foundation for reliability in subsequent test generation.

[0053] The problem data in step S21 includes random data, non-deterministic data, and data lacking test standards.

[0054] Furthermore, random data refers to data that contains random elements or is generated by a random process. In the context of unit testing, random data can lead to inconsistent and unpredictable test results. For example, if the code solution includes a random number generator and the tests do not properly control or anticipate these random elements, different results may be obtained each time the tests are run, making it impossible for the tests to consistently verify the correctness of the code.

[0055] Nondeterministic data is similar to random data, but the difference is that nondeterministic data does not depend on random processes, but may be affected by external factors (such as system state, timestamps, network requests, etc.). These factors are usually difficult to control or predict in unit testing, which may lead to inconsistent test results. Nondeterministic data makes tests difficult to repeat, thereby reducing the reliability of the tests.

[0056] Lack of test criterion data refers to data where test criteria or expected results are not clearly defined. In unit testing, each test should have a clearly defined expected result to verify whether the code works as intended. If the test data does not provide these criteria or results, then the tests cannot effectively evaluate the correctness of the code. This situation may be caused by incomplete test descriptions, improper test data design, or unclear test requirements.

[0057] S22: Format the code solutions in the filtered dataset into function form to obtain the optimized dataset.

[0058] After data filtering is completed in step S21, this invention further performs a comprehensive functional refactoring of the original code solution. Through this process, the structure of the original code is transformed into a standardized function form. This transformation not only improves the modularity and readability of the code but also significantly enhances its applicability in unit test generation tasks. The core objective of this functional code refactoring is to facilitate subsequent automated unit test generation, ensuring that each piece of code can be parsed and tested in a consistent manner, thereby greatly improving the efficiency and accuracy of unit test generation.

[0059] S3: Generate unit test data based on the query problems and code solutions in the optimized dataset.

[0060] In step S3, based on the code data preprocessed in step S2, unit test data is generated using a large language model. To ensure the correctness of the code program, the present invention also uses existing correct code programs in the dataset as verifiers to check whether the generated unit tests meet expectations. If there are errors in the generated unit tests, the incorrect unit tests are repaired by combining the return status of the code executor with the large language model. In addition, to further improve the quality of unit tests, the present invention filters out false positive test cases, that is, unit tests that can pass the incorrect program, thereby ensuring that the generated unit tests are more reliable.

[0061] like Figure 3 As shown, step S3 further includes:

[0062] S31: Using the Llama3.1-70B model, the query problems and code solutions in the optimized dataset are sampled multiple times to obtain basic unit test data.

[0063] In step S31, based on the query and code solutions provided in the dataset, this invention utilizes the Llama 3.1-70B model to generate diverse unit tests through multiple samplings. This generation strategy aims to expand the coverage of unit tests, thereby improving the verification capability and robustness of the code solutions.

[0064] S32: Verify the basic unit test data by removing erroneous unit test data to obtain filtered unit test data.

[0065] Step S32 further includes:

[0066] S321: Apply the basic unit test data to the code solution in the optimized dataset to obtain the execution result.

[0067] S322: Based on the execution result, the corresponding basic unit test data is judged. When the execution result allows the code solution to pass, the corresponding basic unit test data is correct basic unit test data. When the execution result does not allow the code solution to pass, the corresponding basic unit test data is incorrect basic unit test data.

[0068] After generating unit tests in step S31, the present invention further performs execution verification on these tests. Specifically, the present invention applies each unit test to the ground truth code solution provided in the dataset and records its execution results. The execution results serve as the basis for labeling unit tests as true or false: unit tests that enable the ground truth code solution to pass are considered as correct unit tests, while unit tests that fail are marked as incorrect unit tests.

[0069] Specifically, each generated unit test is first applied to the real code solution provided in the dataset to verify its effectiveness and accuracy. After executing the unit tests, the results of each test are recorded. These results will be used to evaluate the correctness of the unit tests.

[0070] The next step is to determine whether a unit test is correct: if a unit test allows the real code solution to pass successfully, that is, without causing errors or exceptions, then the unit test is considered correct, meaning that it correctly verifies a part of the functionality of the code.

[0071] Faulty unit tests: If a unit test fails to pass the actual code solution, for example because the expected result of the test does not match the actual result, then the unit test is marked as faulty, indicating that the test itself has a logical error or that the part of the code it tests has a defect.

[0072] Step S322 further includes:

[0073] By capturing execution feedback information generated by the Python interpreter, erroneous basic unit test data can be repaired.

[0074] For certain complex code instructions, generating a small number of correct unit tests often requires significant computational resources. These complex instructions not only place higher demands on test generation, but also rely on inefficient methods such as multiple sampling, making it difficult to meet the need for efficient generation.

[0075] To more efficiently collect more correct unit tests when tackling challenging problems, this invention introduces an optimization method based on Python interpreter execution feedback. For erroneous unit tests, this invention captures execution feedback information generated by the Python interpreter and uses this feedback to guide the repair of unit tests. Specifically, this invention again utilizes the Llama 3.1-70B model to repair erroneous unit tests based on this execution feedback. This execution feedback-based repair method is more efficient than the multiple sampling strategy and can significantly reduce the time and resources required to generate correct unit tests.

[0076] S323: Remove the erroneous basic unit test data from the basic unit test data to obtain filtered unit test data.

[0077] S33: The filtered unit test data is distinguished by removing erroneous unit test data to obtain unit test data.

[0078] Step S33 further includes:

[0079] S331: Solution for generating error codes.

[0080] S332: Apply the filtered unit test data to the error code solution. When the filtered unit test data does not distinguish between error code solutions, the corresponding filtered unit test data is error filtered unit test data. When the filtered unit test data distinguishes between error code solutions, the corresponding filtered unit test data is correct filtered unit test data.

[0081] S333: Remove the erroneous filtered unit test data from the filtered unit test data to obtain unit test data.

[0082] Following step S32, in order to select high-quality unit tests from all correct unit tests, this invention further believes that high-quality unit tests should not only ensure that correct code solutions pass the tests successfully, but also exclude erroneous code solutions as much as possible. Based on this principle, this invention introduces an additional filtering process: first, erroneous code solutions are generated using a less powerful model; then, the discriminative power of the unit tests is tested using these erroneous code solutions. "False positive unit tests" that cannot exclude erroneous code solutions, i.e., unit tests that fail to reject erroneous solutions, are further filtered out.

[0083] Specifically, in step S33, the present invention uses a model with weaker performance than Llama 3.1-70B used to generate unit tests to generate a series of erroneous code solutions. These erroneous code solutions include logical errors, syntax errors, performance issues, or other types of defects. Subsequently, the previously generated correct unit tests are applied to these erroneous code solutions to observe whether these unit tests can correctly identify and reject these erroneous code solutions. If a unit test fails to reject one or more erroneous code solutions (i.e., it incorrectly considers these erroneous codes to be correct), then the unit test is considered a "false positive unit test." These tests may not be able to effectively distinguish between correct and erroneous code solutions and will therefore be further filtered out.

[0084] S4: Using the optimized dataset and the unit-generated data, train the large language model based on supervised fine-tuning to obtain a unit test generator.

[0085] In step S4, the unit test generator takes the query problem and code solution in the optimized dataset as input and the unit test data as output.

[0086] The expression for the loss function of the unit test generator in step S4 is as follows:

[0087]

[0088] Where L(x, y) is the loss function of the unit test generator, x is the data in the input optimized dataset, y is the unit test result generated by the unit test generator, i is the index value of the obtained token, j is the index value of the given token, and Model(y i |x, y j <i)) is the probability of generating the i-th token of y given the j=i-1-th token of x and y.

[0089] In step S4, this invention utilizes Supervised Fine-Tuning (SFT) to train the unit test generator. During the construction of the training data, this invention uses the problem description and code solution as input instructions to the model, and high-quality unit tests as the target output (answer). This process aims to improve the overall performance and applicability of the unit test generator by enabling the model to generate more accurate unit tests for specific code solutions through supervised learning.

[0090] S5: Generate unit test results for the program to be tested using the unit test generator.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0092] The following specific embodiment illustrates a unit test generation method based on a large language model according to the present invention.

[0093] To evaluate the generalization ability of this invention across multiple code generation models, this invention selected several open-source and closed-source models of different sizes for testing, including Llama3-8B, Llama3-70B, GPT-3.5-turbo, and GPT-4o-mini.

[0094] To demonstrate the performance advantages of this invention, several baseline methods using unit testing as the verifier are selected, including MBR-Exec based on minimum Bayesian risk decoding, CodeT based on dual execution agreement, and MPSC based on multi-angle consistency.

[0095] This invention selects the HumanEval Plus dataset as the evaluation benchmark. The HumanEval Plus dataset is a well-known code generation dataset, which includes 164 manually written data points and detailed and accurate unit tests, used to evaluate the correctness of code programs generated by the model.

[0096] Table 1. Accuracy results on multiple code generation models.

[0097]

[0098] This invention was replicated on the HumanEval Plus dataset and involved multiple code generation models. For each data point, 100 candidate code programs and 100 unit tests were generated. The unit test generation of this invention was compared with baseline methods using a standard majority voting method, and the results are shown in Table 1.

[0099] As shown in Table 1, this invention achieves the highest accuracy for multiple open-source and closed-source models of different scales, including Llama3-8B, Llama3-70B, GPT-3.5-turbo, and GPT-4o-mini, demonstrating superior performance compared to baseline methods.

[0100] In addition, to evaluate the quality of the unit tests generated by the model after training, this invention compared and analyzed the accuracy of the unit tests generated by the model before training, and the specific results are shown in Table 2.

[0101] Table 2 Unit Test Quality

[0102]

[0103] As shown in Table 2, the unit tests generated by the model trained in this invention have higher accuracy and higher quality compared to the Llama 3.1-8B model, proving the high performance and effectiveness of this invention.

[0104] In summary, the unit test generation method based on a large language model of the present invention designs an automated synthetic data framework, uses existing code instruction fine-tuning data to generate high-quality unit tests, and then uses the coding problems in the code instruction fine-tuning data, program code, and synthetic high-quality unit test data to combine as training data to train a high-quality unit test generation model.

[0105] First, this invention, based on an automated data synthesis framework, can fine-tune data using existing code instructions to generate higher-quality unit test data. Second, by training a high-quality unit test generation model, this invention significantly improves the reliability and quality coverage of the generated unit tests. Experimental results show that the unit test generation model trained using this invention has a significantly higher accuracy than existing baseline methods.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A unit test generation method based on a large language model, characterized in that, include: S1: Collect code fine-tuning data to obtain the original dataset; S2: Preprocess the original dataset to obtain an optimized dataset; S3: Generate unit test data based on the query problems and code solutions in the optimized dataset; Step S3 further includes: S31: using the Llama3.1-70B model, sampling the query problem and code solution in the optimized dataset multiple times to obtain basic unit test data; S32: removing verification error unit test data from the basic unit test data to obtain filtered unit test data; S33: removing differentiation error unit test data from the filtered unit test data to obtain unit test data. Step S32 further includes: S321: applying the basic unit test data to the code solution in the optimized dataset to obtain an execution result; S322: judging the corresponding basic unit test data based on the execution result; when the execution result allows the code solution to pass, the corresponding basic unit test data is correct basic unit test data; when the execution result does not allow the code solution to pass, the corresponding basic unit test data is incorrect basic unit test data; S323: removing the incorrect basic unit test data from the basic unit test data to obtain filtered unit test data; S4: Using the optimized dataset and the unit test data, train the large language model based on supervised fine-tuning to obtain a unit test generator; In step S4, the unit test generator takes the query problem and code solution in the optimized dataset as input and the unit test data as output. S5: Generate unit test results for the program to be tested using the unit test generator.

2. The unit test generation method based on a large language model according to claim 1, characterized in that, In step S1, code fine-tuning data is collected from the CodeFeedback-Filtered-Instruction dataset and the TACO dataset.

3. The unit test generation method based on a large language model according to claim 1, characterized in that, Step S2 further includes: S21: Remove problematic data from the original dataset to obtain a filtered dataset; S22: Format the code solutions in the filtered dataset into function form to obtain the optimized dataset.

4. The unit test generation method based on a large language model according to claim 3, characterized in that, The problem data mentioned in step S21 includes random data, nondeterministic data, and data lacking test standards.

5. The unit test generation method based on a large language model according to claim 1, characterized in that, Step S322 also includes: By capturing execution feedback information generated by the Python interpreter, erroneous underlying unit test data can be repaired.

6. The unit test generation method based on a large language model according to claim 1, characterized in that, Step S33 further includes: S331: Solution for generating error codes; S332: Apply the filtered unit test data to the error code solution. When the filtered unit test data does not distinguish between error code solutions, the corresponding filtered unit test data is error filtered unit test data. When the filtered unit test data distinguishes between error code solutions, the corresponding filtered unit test data is correct filtered unit test data. S333: Remove the erroneous filtered unit test data from the filtered unit test data to obtain unit test data.

7. The unit test generation method based on a large language model according to claim 1, characterized in that, The expression for the loss function of the unit test generator in step S4 is: in, The loss function for the unit test generator. The data in the optimized input dataset, The unit test results generated by the unit test generator. The index value of the obtained token. For a given token, the index value is where , For a given and Under the conditions, generate The The probability of each token.

Images