Automatic test case generation method and device fusing multi-source human programming experience and medium

Abstract

The invention relates to an automatic test case generation method and device fusing multi-source human programming experiences and a medium, and the method comprises the steps: a preparation step: based on a target programming problem P, searching human correct programming codes existing for the target programming problem P, the human error submission codes and the human correction codes are corrected aiming at the human error submission codes; a multi-dimensional analysis step: obtaining a first analysis result generated by the large language model; a differential analysis step: obtaining a second analysis result generated by the large language model; and a case generation step: based on the first analysis result and the second analysis result, generating an executable test script construction cue word to obtain a test script generated by the large language model, and running the script to generate a test case for the programming problem P. Compared with the prior art, the method has the advantages that the distinguishing performance of the test case is remarkably improved, and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of test case generation, and in particular to a test case automatic generation method and device fusing multi-source human programming experience and a medium. BACKGROUND

[0002] With the development of pre-training and fine-tuning techniques, large-scale language models (LLMs), such as Google DeepMind's AlphaCode, OpenAI's Codex and its subsequent models, DeepSeek Coder, CodeLlama, etc., have shown comparable or even superior performance to human programmers in automated code generation tasks. However, scientifically and comprehensively evaluating the capabilities of these complex models to ensure the quality and reliability of the generated code remains a key technical bottleneck. Code verifiers, usually driven by test suites, play a crucial role in the evaluation process.

[0003] Traditional test case generation methods are manual planning and rule-based test sets. For example, in programming competition platforms such as Codeforces, AtCoder, and LeetCode, test cases are usually designed by experienced problem setters or community members, following specific rules and covering typical scenarios and boundary conditions for common algorithms and data structures. This type of test set has high quality, but the cost of manpower is high, and it is difficult to generate and dynamically expand on a large scale.

[0004] Some improved technical solutions attempt to use large language models to generate test cases. For example, Chinese patents CN119105965A, CN119576799A, and CN119917421A all disclose schemes for generating test cases by large language models using prompt words. These schemes can be summarized as follows:

[0005] 1. LLM-based test case generation: Direct generation paradigm: LLM directly generates complete test cases (input and expected output) according to problem description. For example, CodeRM, AceCoder, etc. This method relies on the LLM's deep understanding of the problem, but may be limited by the LLM's cognitive bias, and is prone to produce homogeneous or simple test cases.

[0006] 2. Input interpreter paradigm: LLMs are responsible for generating test inputs, and then an "ground-truth solution" or reference implementation is used as an interpreter to calculate the corresponding output. For example, LiveCodeBench uses this paradigm to generate a large number of random or synthetic inputs through LLMs. EvalPlus also partially uses this idea by mutating seed inputs. This method can generate a large number of tests, but random inputs may lack pertinence and be difficult to touch deep logic defects.

[0007] 3. Customized test generation based on program analysis: For example, TestEval and other methods use static or dynamic program analysis techniques (such as symbolic execution, fuzz testing guidance) to generate customized test cases for specific code solutions, aiming to maximize code coverage or trigger specific errors. This method is highly targeted, but the analysis cost is high, real-time performance is poor, and it is difficult to apply to large-scale evaluation or integrated into the RL training loop.

[0008] In this regard, the existing solutions for generating test cases using large language models generally have one or more of the following problems:

[0009] 1. Test Case Homogenization and LLM-Centric Bias: Existing automated or LLM-based TCG methods, especially those that rely solely on LLMs to generate tests, are prone to generating test cases similar to LLM's own error patterns and cognitive biases, forming a "homogenization trap". This leads to test sets that may focus highly on LLM's common error types, while ignoring the diverse errors that human programmers are more prone to make in complex logic, boundary conditions, and integer overflow.

[0010] 2. Verifier Ineffectiveness and Persistent Blind Spots: Verifiers built based on homogenized test cases have blind spots for the types of errors that human programmers are likely to make, making it difficult to rigorously evaluate the robustness of the code. In particular, when generating tests for complex boundary conditions and interaction scenarios, LLM-centric designs may lead to ineffective verifiers.

[0011] 3. Reward Hacking and Optimization Misdirection: In RLVR and other frameworks, if the verifier (part of the reward function) has defects, the LLM may learn to exploit these defects (i.e. "reward hacking") to get high scores, rather than truly improving code quality. This can cause the optimization process to deviate from the intended goal.

[0012] 4. Dynamic integration and scalability challenges: Most existing TCG methods, especially program analysis-based customization methods, have high computational costs, making it difficult to efficiently generate test cases dynamically and integrate them into large-scale online evaluation or RL training loops, hindering continuous and adaptive improvement of model robustness. SUMMARY

[0013] The present application aims to solve the above-mentioned defects of the prior art and provides a test case automatic generation method, device and medium that integrates multi-source human programming experience.

[0014] The object of the present application can be achieved by the following technical solutions:

[0015] A test case automatic generation method that integrates multi-source human programming experience, comprising:

[0016] Preparation step: based on the target programming problem P, search for existing human correct programming code for the target programming problem P, and human error submission code and human correction code for each human error submission code;

[0017] Multi-dimensional analysis step: taking the target programming problem P and the human correct programming code as inputs of a large language model, to constrain the difference analysis and defense mode deconstruction construction prompt words, to obtain a first analysis result generated by the large language model;

[0018] Difference analysis step: taking the target programming problem P, human error submission code and human correction code for each human error submission code as inputs of a large language model, to constrain the difference analysis, defense completeness defect analysis and targeted error mode reproduction construction prompt words, to obtain a second analysis result generated by the large language model;

[0019] Test case generation step: based on the first analysis result and the second analysis result, to generate an executable test script construction prompt word, to obtain a test script generated by the large language model, and to run the script to generate a test case for the programming problem P.

[0020] The human correct programming code comes from high-score code or acceptable code on a programming competition platform.

[0021] The human error submission code is a programming code with incorrect running results or running timeout.

[0022] The multi-dimensional analysis step comprises:

[0023] Taking the target programming problem P and the human correct programming code as inputs of a large language model, and further inputting the first prompt word constructed based on the constraint processing difference analysis and defense mode deconstruction construction, to obtain boundary values, equivalence classes and defense mode descriptions parsed by the large language model;

[0024] Further input the fourth prompt word designed on the basis of structured extraction and storage in a predefined format, get the data structure in the predefined format.

[0025] The difference analysis step includes:

[0026] The target programming problem P, the human error submission code and the human correction code corrected for each human error submission code are input into the large language model, and further input the second prompt word constructed on the basis of difference analysis, defense completeness defect analysis and targeted error mode reproduction, get the error reason and I diff Description of the features;

[0027] Further input the fourth prompt word designed on the basis of structured extraction and storage in a predefined format, get the data structure in the predefined format.

[0028] The use case generation step includes:

[0029] Based on the first analysis result and the second analysis result as the front information, the code generation template is input, and further input the third prompt word constructed on the basis of generating executable test scripts, get the test script generated by the large language model;

[0030] Running the test script generates test input;

[0031] Self-verify the test input, wherein the content of self-verification at least includes: data type and data range;

[0032] All self-verified test inputs are based on pre-set interpreters to get expected results, and test cases and corresponding expected results are combined to get test cases.

[0033] In the use case generation step, input the fifth prompt word constructed on the basis of the design idea of the test script, the key test points covered, and the analysis insights relied on, get the human-scaled explanation document generated by the large language model.

[0034] The large language model used in the multi-dimensional analysis step and the difference analysis step is different, which improves the diversity of test case composition and realizes stronger test case diversity and coverage.

[0035] A test case automatic generation device fusing multi-source human programming experience, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to realize the method as described above.

[0036] A storage medium having stored thereon a program which, when executed, implements the method as described above.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] 1. By using the human correct programming codes for the same type of target programming problem P, the human error submission codes and the human correction codes for each human error submission code, the boundary value, the equivalence class, the defense mode, the description about the error cause and the Idiff characteristics are obtained based on the combined mode of multi-dimensional analysis and differential analysis, thereby effectively improving the distinguishability of the test code on some typical errors.

[0039] 2. The multi-dimensional analysis step and the differential analysis are performed in two steps to generate the object file, thereby taking into account the commonality test of the code problem and the high-difficulty boundary test.

[0040] 3. By generating objects such as JSON, the automation extraction of the test case generation script is facilitated.

[0041] 4. The generated test input is self-verified to ensure the effectiveness of the subsequent test. The self-verification module automatically checks the test input generated by the large language before testing to ensure that it strictly follows the problem constraints (such as data range and format), thereby filtering out all invalid "noise" use cases. This key step ensures that the focus of the subsequent test is completely concentrated on the core algorithm logic of the code, rather than the processing ability of illegal input. Therefore, when a verified use case can make a code fail while another passes, it precisely exposes the difference between the two in the core logic, thereby greatly improving the distinguishability of different code implementations.

[0042] 5. The key of the present application is that the discovery and extraction of such high-distinguishability use cases is not accidental, but is systematically and automatically completed by the framework. It converts the tacit knowledge of human experts (how to write robust code, what are the common errors) into machine-executable explicit instructions for generating high-distinguishability tests. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The figure is a schematic diagram of the main step flow of the method of the present application;

[0044] Figure 2 The figure is a schematic diagram of the technical route of the method of the present application. DETAILED DESCRIPTION

[0045] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present application, and detailed implementation modes and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0046] A test case automatic generation method fusing multi-source human programming experience, as shown in Figure 1 and Figure 2 , comprising:

[0047] Preparation step: based on the target programming problem P, search for human correct programming code S for the target programming problem P human , and human error submission code S wrong and human correction code S correct corrected for each human error submission code;

[0048] Among them, for the target programming problem P, it belongs to a category, for this category, some cases can be retrieved in some platforms or self-owned code library, for example, in the embodiment, the human correct programming code comes from the high score code or acceptable code on the programming competition platform. In the embodiment, at least 10 AC codes from different users on the competition platform are collected, in addition, the human error submission code is the programming code with incorrect running result or running timeout.

[0049] For the above acceptable code, the commonly recognized correct code can be selected, or the code that is correct after expert inspection, and these data reveal common programming pitfalls and error patterns.

[0050] For a programming problem P, it has a problem description D, an input space Xp, and a true solution function f p : X p →Y p The goal of the test case generation method is to output a test case set T = (I i , O i ), where each input I i ∈X p , and the corresponding output O i = f p (I i ).

[0051] Multi-dimensional analysis step: taking the target programming problem P and the human correct programming code as the input of the large language model to constrain the difference analysis and defense mode deconstruction construction prompt word, and obtaining the first analysis result generated by the large language model;

[0052] In the embodiment, the multi-dimensional analysis step specifically comprises:

[0053] The target programming problem P and the human correct programming code are input into the large language model, and further input is input to build the first prompt word based on the constraint processing difference analysis and defense mode deconstruction. The boundary value, equivalence class and defense mode description analyzed by the large language model are obtained, wherein the prompt word is designed to guide the LLM to not only understand the function of S human , but also to deconstruct its code logic from specific dimensions (such as boundary conditions, exception handling, equivalence classes, defensive coding patterns), and extract structured information that can be used to generate challenging tests. For example, the prompt word contains: “Analyze the following correct code S human , identify and list all boundary value check logic for processing input parameter X, and explain the expected behavior corresponding to each boundary value.” Or: “Extract the equivalence class division strategy of Shuman for input data structure Y, and give a representative input example for each equivalence class.”

[0054] Further input is input to design the fourth prompt word based on the pre-defined format for structured extraction and storage, and the data structure Ben in the pre-defined format is obtained. The analysis output of the LLM (such as the identified boundary value, equivalence class, and defense mode description) will be structured extracted and stored in a pre-defined format, for example, stored as a JSON object or a specific data structure, so that the test script can be accurately called when generating the test script later.

[0055] Difference analysis step: input the target programming problem P, human error submission code and human correction code for each human error submission code into the large language model, and build prompt words for difference analysis, defense completeness defect analysis and targeted error mode reproduction, to obtain the second analysis result generated by the large language model;

[0056] In this embodiment, the difference analysis step specifically includes:

[0057] The target programming problem P, human error submission code and human correction code for each human error submission code are input into the large language model, and further input is input to build the second prompt word based on the constraint processing difference analysis, defense completeness defect analysis and targeted error mode reproduction. The error reason and I diff characteristic description analyzed by the large language model are obtained, wherein the prompt word design core: the prompt word is designed to guide the LLM to focus on the key differences between S wrong and S correct , and infer the causal relationship between these differences and the errors in S wrong , and the ultimate goal is to find a specific input condition (Idiff) that can accurately trigger the error. For example, the prompt word contains: “Compare the following error code S wrong and the corrected code S correct , find the cause of Swrong Key code differences that failed on problem P. Based on this difference, infer the specific error type present in S wrong and describe a test input feature I that can trigger this error but be handled correctly by S correct diff .”

[0058] Further input to structure and store in a predefined format, the fourth prompt word designed based on this, get the data structure in the predefined format, where the LLM is about the error cause and I diff feature description, which will also be structured parsed, extracting key parameters and logic that can be used to generate targeted test scripts.

[0059] Use case generation step: based on the first analysis result and the second analysis result, build prompt words to generate executable test scripts, get test scripts generated by large language models, and run scripts to generate test cases for programming problem P.

[0060] In this embodiment, the use case generation step specifically includes:

[0061] Based on the first analysis result and the second analysis result as pre-information, generate templates as input, and further input the third prompt word built based on generating executable test scripts, get test scripts generated by large language models, where the insights (such as boundary values, equivalence class division, I diff features) analyzed and structured extracted by LLM, through another set of carefully designed prompt words or code generation templates, guide LLM (or a dedicated code generation module) to automatically generate executable Python test scripts. These scripts can dynamically and strategically construct test inputs that meet specific test intentions. For example, if the extracted insight is "boundary value N = 1", the generated script will contain the logic of input_N = 1.

[0062] Run the test script to generate test input;

[0063] Self-verify the test input, where the self-verification content at least includes: data type and data range;

[0064] Get the expected result based on the pre-set interpreter for all self-verified test inputs, and combine the test cases and corresponding expected results to get the test cases. In this process, you can use prompt words to guide LLM to automatically generate corresponding self-verification code snippets for the generated test scripts or their generated inputs. These codes are used to check whether the generated test input meets the basic constraints (such as data type, range) and expected test scenarios before executing the test. This is a key quality assurance link that ensures the effectiveness of subsequent testing.​

[0065] In this embodiment, in the use case generation step, the fifth prompt word constructed based on the design idea of the test script, the key test points covered, and the analysis insight relied on is input to obtain a human-scaled explanation document generated by a large language model.

[0066] In addition, in this embodiment, the multi-dimensional analysis step and the differential analysis step are implemented by a plurality of large language models, which improves the diversity of the test cases and achieves stronger test case diversity and coverage.

[0067] Based on the above method, the SAGA framework is used to process a large number of programming problems, and structured data pairs containing problem descriptions, Python test scripts, mathematical explanations, and self-verification codes are generated. These data pairs encapsulate specific human-computer collaborative analysis logic and test generation strategies in the SAGA framework. Using these structured data pairs, a pre-trained basic code LLM is fine-tuned. The goal of fine-tuning is to enable the model to learn to directly output high-quality test scripts and related metadata that meet the requirements of the SAGA framework based on new problem descriptions, thereby internalizing the core test generation capabilities of SAGA.

[0068] Finally, the following indicators are used as the measurement indicators of the test suite composed of test cases:

[0069] 1. Detection rate (DR): The probability that a test suite T can detect the error of a candidate solution S ≠ fp for an error, i.e.

[0070] 2. Verifier accuracy (VAcc): The ability of a test suite T to successfully identify all known error solutions S correct (P). If T can reject each solution in S correct (P), then VAcc(T) = 1.

[0071] 3. Distinct error pattern coverage (DEPC): The number of different error patterns (or error solutions) that a test suite T can detect.

[0072] 4. Normalized area under the accuracy-number of test cases curve (AUC-AccN): Measures the average improvement efficiency of verifier accuracy when increasing the number of test cases in the test suite.

[0073] Finally, the quality of the test suites generated by the SAGA framework and TCGCoder-7B is evaluated on standard test sets (e.g., TCGBench, TCGBench-Lite) using defined metrics such as DR, VAcc, DEPC, AUC-AccN, etc., and compared with existing baseline methods.

[0074] Compared with the existing best technical solutions (such as the LLM input interpreter paradigm represented by LiveCodeBench, the mutation enhancement paradigm represented by EvalPlus, and the customized analysis paradigm represented by TestEval), the SAGA framework and TCGCoder-7B of the present application have the following significant advantages:

[0075] 1. Higher defect detection capability and evaluation accuracy:

[0076] Significant improvement in detection rate (DR): Experiments show that the test suite generated by SAGA has a DR@50 of 90.62% on TCGBench-Lite, which is significantly higher than Input-Interpreter (81.07%), EvalPlus (71.12%), and TestChain (68.31%). This means that SAGA can more effectively find errors in the code.

[0077] Significant improvement in verifier accuracy (VAcc): SAGA has a VAcc@50 of 32.58%, which is much higher than Input-Interpreter (16.72%), EvalPlus (15.15%), and TestChain (11.88%). This indicates that the verifier constructed by SAGA can more accurately identify all known error solutions.

[0078] Better comprehensive performance (AUC@50): SAGA has an AUC@50 of 0.2228, which is a significant advantage over Input-Interpreter (0.1234), EvalPlus (0.1139), and TestChain (0.0841), indicating that the test cases generated by SAGA have higher overall quality and require fewer test cases to achieve the same accuracy.

[0079] 2. Stronger test case diversity and coverage:

[0080] SAGA can generate test cases that cover a wider range of scenarios (including boundaries, exceptions, and complex interactions) and have more diverse patterns by combining the complex logic of correct human solutions and the specific patterns of incorrect solutions, effectively overcoming the homogeneity and bias problems of LLM-centric TCG methods. This is reflected in higher DEPC and diversity ratio (DivRatio@50 reaching 94.06%).

[0081] 3. Effectively suppress reward cheating and improve RLVR:

[0082] · Since the test suite generated by SAGA is of higher quality and has fewer blind spots, the verifier built based on it can provide more accurate and robust reward signals for RLVR, reducing the chances of LLM exploiting evaluation loopholes for "reward cheating", thereby guiding the model to optimize in the direction of truly improving code quality.

[0083] 4. Systematic and scalable:

[0084] SAGA provides a systematic framework for utilizing human experience, and its core logic can be imparted to TCGCoder-7B and other specialized models through model distillation. TCGCoder-7B (AUC@50: 0.1890) itself outperforms all baseline methods, and as a specialized model, it is more suitable for large-scale dynamic integration and deployment in terms of inference efficiency.

[0085] 5. Deep utilization of human programming wisdom:

[0086] Unlike simply using human code as seeds or true solutions, SAGA deeply analyzes, deconstructs, and differentially mines correct and incorrect programming practices of humans through LLM, thereby more intelligently guiding the generation of test cases, which is lacking in existing technical solutions.

[0087] The above functions, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of software products, which are stored in a storage medium and include instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various program code storage media.

Claims

1. A method for automatically generating test cases by integrating multi-source human programming experience, characterized in that: include: Preparation step: Based on the target programming problem P, search for existing human-corrected programming codes for the target programming problem P, as well as human-erroneous submitted codes and human-corrected codes for correcting each human-erroneous submitted code; Multi-dimensional analysis step: The target programming problem P and the correct programming code of humans are used as inputs of the large language model. The constraint processing difference analysis and defense mode deconstruction construct prompt words are used to obtain the first analysis result generated by the large language model. Differential analysis step: The target programming problem P, the human error submission code, and the human correction code for each human error submission code are used as input to the large language model. The large language model is used to construct prompt words based on constraint processing difference analysis, defense completeness defect analysis, and targeted error pattern reproduction to obtain the second analysis result generated by the large language model. Use case generation step: Based on the first analysis result and the second analysis result, construct prompt words to generate an executable test script, obtain a test script generated by the large language model, and run the script to generate a test case for the programming problem P.

2. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that: The human-correct programming code comes from a high-scoring code or an acceptable code on a programming competition platform.

3. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that: The human error submission code is a programming code that has an error in running result or runs in timeout.

4. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that: The multidimensional analysis step comprises: The target programming problem P and the correct human programming code are used as inputs to the large language model. Furthermore, the first prompt word constructed based on constraint processing difference analysis and defense pattern deconstruction is input to obtain the boundary value, equivalence class, and defense pattern description obtained by the large language model analysis. A fourth prompt word designed based on structured extraction and storage in a predefined format is further input to obtain a data structure book in the predefined format.

5. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that: The differential analysis step includes: The target programming problem P, human error submission code and human correction code for each human error submission code are used as the input of the large language model, and the second prompt word constructed based on constraint processing difference analysis, defense completeness defect analysis and targeted error pattern reproduction is further input to obtain the error cause and I obtained by the large language model analysis. diff Description of the characteristics; A fourth prompt word designed based on structured extraction and storage in a predefined format is further input to obtain a data structure in the predefined format.

6. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that: The use case generation step includes: Based on the first analysis result and the second analysis result as pre-information, the code generation template is used as input, and a third prompt word constructed based on the generation of an executable test script is further input, thereby obtaining a test script generated by the large language model; Run the test script to generate test input; Perform self-verification on the test input, wherein the self-verification content includes at least: data type and data range; All self-verified test inputs are used to obtain expected results based on a pre-set interpreter, and the test cases and the corresponding expected results are combined to obtain test cases.

7. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 6, characterized in that: In the use case generation step, a fifth prompt word is input based on the design ideas of the test script, the key test points covered, and the analytical insights based thereon, to obtain a human-scaled explanation document generated by a large language model.

8. The method for automatically generating test cases by integrating multi-source human programming experience according to claim 1, characterized in that: The large language models used in the multidimensional analysis step and the differential analysis step are different large language models.

9. A device for automatically generating test cases by integrating multi-source human programming experience, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images