Test case automatic generation method and device fusing multi-source human programming experience and medium

Abstract

This invention relates to a method, apparatus, and medium for automatically generating test cases by integrating multi-source human programming experience. The method includes: a preparation step: based on a target programming problem P, searching for existing correct human programming code for the target programming problem P, as well as human error submission code and human correction code for each error submission; a multi-dimensional analysis step: obtaining a first analysis result generated by a large language model; a difference analysis step: obtaining a second analysis result generated by the large language model; and a test case generation step: based on the first and second analysis results, generating executable test script construction prompts to obtain a test script generated by the large language model, and running the script to generate test cases for the programming problem P. Compared with existing technologies, this invention has advantages such as significantly improving the discriminative power of test cases.

Description

Technical Field

[0001] This invention relates to the field of test case generation, and in particular to a method, apparatus and medium for automatically generating test cases that integrates multi-source human programming experience. Background Technology

[0002] With the development of pre-training and fine-tuning techniques, large-scale language models (LLMs), such as Google DeepMind's AlphaCode, OpenAI's Codex and its successors, DeepSeek Coder, and CodeLlama, have demonstrated performance comparable to or even surpassing that of human programmers in automated code generation tasks. However, scientifically and comprehensively evaluating the capabilities of these complex models to ensure the quality and reliability of their generated code remains a key technological bottleneck. Code verifiers, typically driven by test suites, play a central role in the evaluation process.

[0003] Traditional test case generation methods involve manual planning and rule-based test sets. For example, in programming competition platforms (such as Codeforces, AtCoder, and LeetCode), test cases are typically meticulously designed by experienced problem setters or community members, following specific rules to cover typical scenarios and boundary conditions of common algorithms and data structures. These test sets are of high quality, but they are labor-intensive and difficult to automate on a large scale and dynamically expand.

[0004] Some improved technical solutions attempt to generate test cases using large language models. For example, Chinese patents CN119105965A, CN119576799A, and CN119917421A all disclose schemes that use large language models as the main driver and prompt words to generate test cases. These are summarized as follows:

[0005] 1. LLM-based test case generation: Direct generation paradigm: LLM directly generates complete test cases (inputs and expected outputs) based on the problem description. Examples include CodeRM and AceCoder. This type of method relies on LLM's deep understanding of the problem, but may be limited by the cognitive biases of LLM, easily resulting in homogenized or simplistic test cases.

[0006] 2. Input Interpreter Paradigm: The LLM (Local Level Manager) generates test inputs and then uses a "ground-truth solution" or reference implementation as an interpreter to compute the corresponding outputs. For example, LiveCodeBench uses this paradigm, generating a large number of random or synthetic inputs through the LLM. EvalPlus also partially adopts this approach by mutating the seed input. This type of method can generate a large number of tests, but random inputs may lack specificity and fail to address deep logical flaws.

[0007] 3. Customized Test Generation Based on Program Analysis: For example, methods like TestEval utilize static or dynamic program analysis techniques (such as symbolic execution and fuzzing) to generate customized test cases for specific code solutions, aiming to maximize code coverage or trigger specific errors. These methods are highly targeted but have high analysis costs, poor real-time performance, and are difficult to apply to large-scale evaluations or integrate into RL training loops.

[0008] Existing solutions for generating test cases using large language models generally suffer from one or more of the following problems:

[0009] 1. Test Case Homogenization and LLM-Centric Bias: Existing automated or LLM-based TCG methods, especially those relying solely on the LLM itself to generate tests, are prone to producing test cases that resemble the error patterns and cognitive biases inherent in the LLM, creating a "homogenization trap." This leads to test sets that may focus heavily on common error types in the LLM while neglecting the diverse errors that human programmers are more likely to make in complex logic, boundary conditions, integer overflows, and other areas.

[0010] 2. Verifier Ineffectiveness and Persistent Blind Spots: Validators built on homogeneous test cases have blind spots regarding the types of errors human programmers might make, making it difficult to rigorously evaluate code robustness. Especially when generating tests with complex boundary conditions and interactive scenarios, the design of an LLM-centric approach can lead to validator failure.

[0011] 3. Reward Hacking and Optimization Misdirection: In frameworks like RLVR, if the validator (i.e., part of the reward function) has flaws, LLMs may learn to exploit these flaws (i.e., "reward hacking") to obtain high scores instead of genuinely improving code quality. This causes the optimization process to deviate from its intended goal.

[0012] 4. Dynamic integration and scalability challenges: Most existing TCG methods, especially customized methods based on program analysis, have high computational costs and are difficult to efficiently generate test cases dynamically and integrate them into large-scale online evaluation or RL training loops, which hinders the continuous and adaptive improvement of model robustness. Summary of the Invention

[0013] The purpose of this invention is to address the shortcomings of the prior art by providing a method, apparatus, and medium for automatically generating test cases that integrates multi-source human programming experience.

[0014] The objective of this invention can be achieved through the following technical solutions:

[0015] A method for automatically generating test cases that integrates multi-source human programming experience includes:

[0016] Preparation steps: Based on the target programming problem P, search for existing correct human programming code for the target programming problem P, as well as human error submission code and human correction code for each human error submission code;

[0017] Multidimensional analysis steps: The target programming problem P and correct human programming code are used as input to the large language model. Constraint handling difference analysis and defense pattern deconstruction are used to construct prompt words, and the first analysis result generated by the large language model is obtained.

[0018] Differential analysis steps: The target programming problem P, human error submission codes, and human correction codes for each human error submission code are used as input to the large language model. Constraint processing differential analysis, defense completeness defect analysis, and targeted error pattern reproduction are used to construct prompt words to obtain the second analysis result generated by the large language model.

[0019] Test case generation steps: Based on the first and second analysis results, construct prompts to generate executable test scripts, obtain test scripts generated by the large language model, and run the scripts to generate test cases for programming problem P.

[0020] The correct human programming code mentioned comes from high-scoring or acceptable code on programming competition platforms.

[0021] The human error submission code refers to programming code that results in incorrect execution or times out.

[0022] The multidimensional analysis steps include:

[0023] The target programming problem P and correct human programming code are used as input to the large language model. The first prompt words, which are constructed based on constraint handling difference analysis and defense pattern deconstruction, are further input to obtain the boundary values, equivalence classes and defense pattern descriptions obtained by the large language model parsing.

[0024] Further input of a fourth prompt word, designed based on structured extraction and storage according to a predefined format, yields a predefined data structure.

[0025] The difference analysis steps include:

[0026] The target programming problem P, human error submission codes, and human correction codes for each human error submission are used as input to the large language model. Further input is a second cue word constructed based on constraint handling difference analysis, defense completeness defect analysis, and targeted error pattern reproduction. This yields the error causes and I obtained from the large language model's analysis. diff Description of features;

[0027] Further input of a fourth prompt word designed based on structured extraction and storage according to a predefined format yields a data structure in a predefined format.

[0028] The use case generation steps include:

[0029] Based on the first and second analysis results as prerequisite information, the code generation template is used as input, and a third prompt word constructed based on generating an executable test script is further input to obtain the test script generated by the large language model.

[0030] Run the test script to generate test input;

[0031] Perform self-validation on the test input, whereby the self-validation content should at least include: data type and data range;

[0032] All self-verified test inputs are processed to obtain the expected results based on a pre-defined interpreter, and the test cases and the corresponding expected results are combined to obtain test cases.

[0033] In the test case generation step, the input is a fifth prompt word constructed based on the design idea of ​​the test script, the key test points covered, and the analytical insights on which it is based, and an explanation document with human scale generated by a large language model is obtained.

[0034] The multidimensional analysis step and the difference analysis step use different large language models, which improves the diversity of test case composition and achieves stronger test case diversity and coverage.

[0035] An automatic test case generation device that integrates multi-source human programming experience includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0036] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. By utilizing existing correct human programming code for a similar target programming problem P, as well as incorrect human submissions and corrected human code that addresses these errors, boundary values, equivalence classes, defense patterns, descriptions of error causes, and Idiff characteristics are obtained through a combination of multidimensional analysis and difference analysis. This effectively improves the discriminative power of the test code against some typical errors.

[0039] 2. Multidimensional analysis steps and differential analysis are performed in two separate steps to generate object files, thus taking into account both common code problem testing and highly challenging boundary testing.

[0040] 3. By generating objects such as JSON, it is easier to automatically extract test case generation scripts.

[0041] 4. The generated test input undergoes self-verification, ensuring the validity of subsequent tests. The self-verification module automatically validates the test input generated by the large language before testing, ensuring it strictly adheres to the problem constraints (such as data range and format), thereby filtering out all invalid "noise" test cases. This crucial step ensures that subsequent tests focus entirely on the core algorithmic logic of the code, rather than its ability to handle invalid input. Therefore, when a validated test case causes one piece of code to fail while another passes, it precisely exposes the differences in their core logic, greatly improving the ability to differentiate between different code implementations.

[0042] 5. The key to this invention lies in the fact that the discovery and extraction of such high-discrimination test cases is not accidental, but rather accomplished systematically and automatically through this framework. It transforms the tacit knowledge of human experts (how to write robust code, what common errors are) into explicit, machine-executable instructions for generating high-discrimination tests. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the main steps of the method of the present invention;

[0044] Figure 2 This is a schematic diagram of the technical route of the method of the present invention. Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0046] A method for automatically generating test cases that integrates multi-source human programming experience, such as... Figure 1 and Figure 2 As shown, it includes:

[0047] Preparation steps: Based on the target programming problem P, search for existing correct human-written code S for the target programming problem P. human And human error code submissions S wrong And human-corrected code S for each human error submission. correct ;

[0048] In this context, the target programming problem P belongs to a specific category. For this category, relevant examples can be retrieved from various platforms or proprietary code repositories. For instance, in this embodiment, correct human programming code comes from high-scoring or acceptable code on programming competition platforms. In this embodiment, at least 10 AC (Accepted) codes from different users on competition platforms are collected. Furthermore, incorrect human submissions are those that result in errors or time out.

[0049] The acceptable code above can be selected from generally accepted correct code or code that has been verified as correct by experts. This data reveals common programming pitfalls and error patterns.

[0050] For a programming problem P, it has a problem description D, an input space Xp, and a true solution function f. p :X p →Y p The goal of the test case generation method is to produce a set of test cases T = (I... i O i ), where each input I i ∈X p Its corresponding output O i =f p (I i ).

[0051] Multidimensional analysis steps: The target programming problem P and correct human programming code are used as input to the large language model. Constraint handling difference analysis and defense pattern deconstruction are used to construct prompt words, and the first analysis result generated by the large language model is obtained.

[0052] In this embodiment, the multidimensional analysis steps specifically include:

[0053] The target programming problem P and correct human programming code are used as inputs to a large language model. Further input is a first cue word constructed based on constraint handling difference analysis and defense pattern deconstruction. This yields the boundary values, equivalence classes, and defense pattern descriptions obtained from the large language model parsing. The cue words are designed to guide LLM beyond simply understanding S. human Instead of focusing on functionality, the goal is to deconstruct the code logic from specific dimensions (such as boundary conditions, exception handling, equivalence classes, and defensive coding patterns) and extract structured information that can be used to generate challenging tests. For example, the prompt might include: "Analyze the following correct code S..." human Identify and list all boundary value checking logic for processing input parameter X, and explain the expected behavior for each boundary value. Alternatively: Extract Shuhuman's equivalence class partitioning strategy for input data structure Y, and provide a representative input example for each equivalence class.

[0054] Further input of a fourth prompt word, designed for structured extraction and storage according to a predefined format, yields a predefined data structure. The LLM analysis output (such as identified boundary values, equivalence classes, and defense pattern descriptions) is then extracted and stored in a structured format, for example, as a JSON object or a specific data structure, for precise recall during subsequent test script generation.

[0055] Differential analysis steps: The target programming problem P, human error submission codes, and human correction codes for each human error submission code are used as input to the large language model. Constraint processing differential analysis, defense completeness defect analysis, and targeted error pattern reproduction are used to construct prompt words to obtain the second analysis result generated by the large language model.

[0056] In this embodiment, the difference analysis step specifically includes:

[0057] The target programming problem P, human error submission codes, and human correction codes for each human error submission are used as input to the large language model. Further input is a second cue word constructed based on constraint handling difference analysis, defense completeness defect analysis, and targeted error pattern reproduction. This yields the error causes and I obtained from the large language model's analysis. diff The description of the features, in which the core of the cue word design is: the cue words are designed to guide the LLM to focus on S. wrong and S correct The key differences between them, and inferring the relationship between these differences and S. wrong The goal of identifying causal relationships within errors is to ultimately find the specific input condition (Idiff) that precisely triggers the error. For example, the prompt might include: "Compare the following error codes S..." wrong And the corrected code S correct Find out the cause of Swrong The key code difference that led to the failure on problem P. Based on this difference, S is inferred. wrong Describe the specific error types that exist in the system, and describe an error that can trigger this error but can be detected by S. correct Correctly processed test input feature I diff .

[0058] Further input of a fourth prompt word, designed based on structured extraction and storage according to a predefined format, yields a predefined data structure, where LLM relates to error reasons and I. diff The descriptions of the features will also be structured and parsed to extract key parameters and logic that can be used to generate targeted test scripts.

[0059] Test case generation steps: Based on the first and second analysis results, construct prompts to generate executable test scripts, obtain test scripts generated by the large language model, and run the scripts to generate test cases for programming problem P.

[0060] In this embodiment, the use case generation step specifically includes:

[0061] Based on the first and second analysis results as prerequisite information, the code generation template is used as input, and a third prompt word constructed based on generating an executable test script is further input. This results in a test script generated by a large language model, which utilizes insights extracted through LLM analysis and structure (such as boundary values, equivalence class partitioning, I...). diff (Features) Through another set of carefully designed prompts or code generation templates, the LLM (or a dedicated code generation module) is guided to automatically generate executable Python test scripts. These scripts can dynamically and strategically construct test inputs that conform to specific test intentions. For example, if the extracted insight is "boundary value N=1", the generated script will contain logic for input_N=1.

[0062] Run the test script to generate test input;

[0063] Perform self-validation on the test input, whereby the self-validation content should at least include: data type and data range;

[0064] All self-verified test inputs are processed using a pre-defined interpreter to obtain expected results, and test cases are then combined with the corresponding expected results to form test cases. This process can involve using prompts to guide the LLM to automatically generate corresponding self-verifying code snippets for the generated test scripts or their inputs. This code is used to check whether the generated test inputs meet the basic constraints of the problem (such as data type and range) and the expected test scenario before test execution. This is a crucial quality assurance step, ensuring the effectiveness of subsequent tests.

[0065] In this embodiment, during the test case generation step, a fifth prompt word is input, which is constructed based on the design idea of ​​the test script, the key test points covered, and the analytical insights on which it is based, to obtain an explanatory document of human scale generated by a large language model.

[0066] Furthermore, in this embodiment, the multidimensional analysis step and the differential analysis step are implemented by combining multiple large language models, which improves the diversity of test case composition and achieves stronger test case diversity and coverage.

[0067] Based on the above method, this application utilizes the SAGA framework to process a large number of programming problems, generating structured data pairs containing problem descriptions, Python test scripts, mathematical explanations, and self-verifying code. These data pairs encapsulate the specific human-machine collaborative analysis logic and test generation strategies within the SAGA framework. Using these structured data pairs, a pre-trained base code LLM is fine-tuned. The goal of this fine-tuning is to enable the model to learn to directly output high-quality test scripts and related metadata that meet the requirements of the SAGA framework based on new problem descriptions, thereby internalizing SAGA's core test generation capabilities.

[0068] Finally, the following metrics are used as evaluation metrics for the test suite composed of test cases:

[0069] 1. Detection Rate (DR): The probability that a test suite T can detect an incorrect candidate solution S ≠ fp.

[0070] 2. Verifier Accuracy (VAcc): For a given problem P, the test suite T successfully identifies all known incorrect solutions S. correct (P)'s ability. If T can refuse S correct For each solution in (P), VAcc(T) = 1.

[0071] 3. Distinct Error Pattern Coverage (DEPC): The number of different error modes (or error solutions) that test suite T can detect.

[0072] 4. Area Under the Accuracy-Number of Test Cases Curve (AUC-AccN): Measures the average efficiency of the validator's accuracy improvement when the number of test cases is increased.

[0073] Finally, using defined metrics such as DR, VAcc, DEPC, and AUC-AccN, the quality of the SAGA framework and the test suite generated by TCGCoder-7B is evaluated on standard test sets (such as TCGBench and TCGBench-Lite), and compared with existing baseline methods.

[0074] Compared with existing best-in-class technologies (such as the LLM input interpreter paradigm represented by LiveCodeBench, the mutation enhancement paradigm represented by EvalPlus, and the customized analysis paradigm represented by TestEval), the SAGA framework and TCGCoder-7B of this invention have the following significant advantages:

[0075] 1. Enhanced defect detection capabilities and assessment accuracy:

[0076] Significantly improved detection rate (DR): Experiments show that the test suite generated by SAGA achieves a DR@50 of 90.62% on TCGbench-Lite, significantly higher than Input-Interpreter (81.07%), EvalPlus (71.12%), and TestChain (68.31%). This means that SAGA can more effectively detect errors in the code.

[0077] The validator accuracy (VAcc) is significantly improved: SAGA's VAcc@50 is 32.58%, far exceeding Input-Interpreter (16.72%), EvalPlus (15.15%), and TestChain (11.88%). This indicates that the validator built by SAGA can more accurately identify all known flawed solutions.

[0078] Superior overall performance (AUC@50): SAGA's AUC@50 is 0.2228, which is a significant advantage over Input-Interpreter (0.1234), EvalPlus (0.1139) and TestChain (0.0841). This indicates that SAGA generates test cases of higher overall quality and requires fewer test cases to achieve the same accuracy.

[0079] 2. Greater diversity and coverage of test cases:

[0080] By integrating the complex logic of correct human solutions with the specific patterns of incorrect solutions, SAGA can generate test cases that cover a wider range of scenarios (including boundaries, anomalies, and complex interactions) and have more diverse patterns, effectively overcoming the homogenization and bias problems of the LLM-centric TCG method. This is reflected in higher DEPC and diversity ratio (DivRatio@50 reaches 94.06%).

[0081] 3. Effectively suppress reward cheating and improve RLVR:

[0082] Because SAGA generates higher-quality test suites with fewer blind spots, validators built upon this foundation can provide more accurate and robust reward signals for RLVR, reducing the chances of LLMs exploiting evaluation vulnerabilities for "reward cheating" and thus guiding the model towards optimization that truly improves code quality.

[0083] 4. It combines systematicity and scalability:

[0084] SAGA provides a systematic framework to leverage human experience, and its core logic can be applied to specialized models such as TCGCoder-7B through model distillation. TCGCoder-7B (AUC@50:0.1890) already outperforms all baseline methods, and as a specialized model, it is more suitable for large-scale dynamic integration and deployment in terms of inference efficiency.

[0085] 5. Deep utilization of human programming intelligence:

[0086] Unlike simply using human code as a seed or true solution, SAGA uses LLM to deeply analyze, deconstruct, and differentiate human correct and incorrect programming practices, thereby more intelligently guiding the generation of test cases, which is lacking in existing technical solutions.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for automatically generating test cases that integrates multi-source human programming experience, characterized in that, include: Preparation steps: Based on the target programming problem P, search for existing correct human programming code for the target programming problem P, as well as human error submission code and human correction code for each human error submission code; Multidimensional analysis steps: The target programming problem P and correct human programming code are used as input to the large language model. Constraint handling difference analysis and defense pattern deconstruction are used to construct prompt words, and the first analysis result generated by the large language model is obtained. Differential analysis steps: The target programming problem P, human error submission codes, and human correction codes for each human error submission code are used as input to the large language model. Constraint processing differential analysis, defense completeness defect analysis, and targeted error pattern reproduction are used to construct prompt words to obtain the second analysis result generated by the large language model. Test case generation steps: Based on the first and second analysis results, construct prompts to generate executable test scripts, obtain test scripts generated by the large language model, and run the scripts to generate test cases for programming problem P.

2. The method for automatically generating test cases by integrating multi-source human programming experience as described in claim 1, characterized in that, The correct human programming code mentioned comes from high-scoring or acceptable code on programming competition platforms.

3. The method for automatically generating test cases by integrating multi-source human programming experience as described in claim 1, characterized in that, The human error submission code refers to programming code that results in incorrect execution or times out.

4. The method for automatically generating test cases by integrating multi-source human programming experience as described in claim 1, characterized in that, The multidimensional analysis steps include: The target programming problem P and correct human programming code are used as input to the large language model. The first prompt words, which are constructed based on constraint handling difference analysis and defense pattern deconstruction, are further input to obtain the boundary values, equivalence classes and defense pattern descriptions obtained by the large language model parsing. Further input of a fourth prompt word designed based on structured extraction and storage according to a predefined format yields a predefined data structure.

5. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that, The difference analysis steps include: The target programming problem P, human error submission codes, and human correction codes for each human error submission are used as input to the large language model. Further input is a second cue word constructed based on constraint handling difference analysis, defense completeness defect analysis, and targeted error pattern reproduction. This yields the error causes and I obtained from the large language model's analysis. diff Description of features; Further input of a fourth prompt word designed based on structured extraction and storage according to a predefined format yields a data structure in a predefined format.

6. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that, The use case generation steps include: Based on the first and second analysis results as prerequisite information, the code generation template is used as input, and a third prompt word constructed based on generating an executable test script is further input to obtain the test script generated by the large language model. Run the test script to generate test input; Perform self-validation on the test input, whereby the self-validation content should at least include: data type and data range; All self-verified test inputs are processed to obtain the expected results based on a pre-defined interpreter, and the test cases and the corresponding expected results are combined to obtain test cases.

7. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 6, characterized in that, In the test case generation step, the input is a fifth prompt word constructed based on the design idea of ​​the test script, the key test points covered, and the analytical insights on which it is based, and an explanation document with human scale generated by a large language model is obtained.

8. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that, The large language models used in the multidimensional analysis step and the difference analysis step are different large language models.

9. A test case automatic generation device integrating multi-source human programming experience, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images