Software test case generation method and system based on large language model

Abstract

The invention discloses a software test case generation method and system based on a large language model, and relates to the technical field of software testing, and the system comprises a to-be-tested module detection subsystem which is used for constructing a to-be-tested module set; the test case generation subsystem is used for constructing a prompt template according to the to-be-tested module and generating a test case by utilizing a program generation model based on an adaptive focus context generation mechanism and the prompt template; the correction verification subsystem is used for performing multi-stage correction processing on the test case, eliminating errors and generating an initial test suite; and the test case supplement subsystem is used for carrying out coverage rate analysis and variation test on the initial test suite, constructing a model input text according to a result, and generating a supplement test case by utilizing a program generation model so as to update the initial test suite and obtain a final test suite. By combining the functional annotation of the to-be-tested module with the logical reasoning ability of the LLM, misjudgment caused by assertion errors can be reduced, and the accuracy of test feedback is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and more specifically to a method and system for generating software testing cases based on a large language model. Background Art

[0002] At present, search-based testing technology (SBST) and symbolic execution technology have made significant progress in the field of unit test generation for statically typed languages. SBST tools represented by Evosuite and Randoop generate test cases that meet coverage indicators through evolutionary algorithms and feedback-driven methods, while symbolic execution tools such as Java PathFinder and KLEE generate test data by solving path constraints. These technologies perform well in quantitative indicators such as branch coverage and statement coverage, but the test cases they generate generally have problems such as poor interpretability and excessive assertion redundancy. For dynamically typed languages ​​such as Python, due to the lack of static type constraints and runtime polymorphism, traditional methods face the problem of path explosion, resulting in significant defects in the efficiency of dynamic language test generation and case quality of existing tools such as Pynguin.

[0003] However, the emerging LLM-driven test generation technology faces two challenges in actual engineering applications: First, there are systematic "illusions" in the assertions generated by large language models. In the HumanEval benchmark test, the accuracy of the true value of the assertions generated by gpt-3.5-turbo-0301 is only 67.95%, which means that nearly one-third of the assertions may contain incorrect expectations that are irrelevant to the code under test; second, for engineering-level complex methods with cyclomatic complexity > 10, the test case coverage rate generated by existing LLM is generally low. What's more serious is that these two problems will have a superposition effect - when the test case contains both incorrect assertions and fails to cover the critical path, it will be difficult for developers to distinguish whether the test failure is due to code defects or errors in the test case itself. Therefore, how to develop an intelligent test generation framework with both high assertion accuracy and high coverage of complex methods is a major problem that technicians in this field need to solve urgently. Summary of the invention

[0004] In view of this, the present invention provides a method and system for generating software test cases based on a large language model, which overcomes the above-mentioned defects.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A software test case generation system based on a large language model, comprising:

[0007] The module detection subsystem to be tested is used to parse and extract information from the project files and requirement documents to generate a set of modules to be tested;

[0008] A test case generation subsystem, configured to construct a prompt template according to each module to be tested in the module set to be tested, and generate a plurality of test cases covering conventional scenarios, boundary conditions and abnormal inputs by using a program generation model based on an adaptive focus context generation mechanism and the prompt template;

[0009] A correction and verification subsystem, used for performing multi-level correction processing on the test cases, eliminating syntax errors, runtime errors and assertion errors, and generating an initial test suite;

[0010] The test case supplement subsystem is used to perform coverage analysis on the initial test suite to determine whether a preset threshold is met; if not, construct a model input text based on uncovered data, and use the program generation model to generate supplementary test cases to update the initial test suite; if yes, perform mutation testing on the initial test suite, and if the test fails, construct the model input text based on surviving variants, and use the program generation model to generate the supplementary test cases to update the initial test suite; if the test passes, generate a final test suite.

[0011] Optionally, the module-to-be-tested detection subsystem includes:

[0012] A lexical analysis unit, used for performing lexical analysis on the project file to be tested and generating a token sequence;

[0013] A syntax analysis unit, used to construct an abstract syntax tree containing a complete syntax structure according to the token sequence;

[0014] A module set forming unit, used for extracting top-level import statements, function definitions and class definitions according to the abstract syntax tree, and forming an initial module set to be tested;

[0015] A relevant information extraction unit, used to extract relevant information of each initial module to be tested in the initial module to be tested set;

[0016] The module set generating unit is used to automatically match the corresponding definition according to the calling information of each of the initial modules to be tested, and associate the relevant information with the corresponding initial modules to be tested to generate the module set to be tested.

[0017] Optionally, the test case generation subsystem includes:

[0018] A module data extraction unit, used to extract module data from the module to be tested, wherein the module data includes module basic data, module characteristic data and module function data;

[0019] A prompt template generating unit, used for constructing the prompt template according to the module characteristic data;

[0020] A dynamic context generation unit, configured to obtain module requirements according to the module data, and generate dynamic context information using the adaptive focus context generation mechanism according to the module requirements and the prompt template;

[0021] A context filtering unit, used to remove information irrelevant to the current test target in the dynamic context information to obtain the model input text;

[0022] A generating unit is used to input the model input text into the program generation model to generate test cases covering conventional scenarios, boundary conditions and abnormal inputs.

[0023] Optionally, the module basic data includes code, interface documentation, function definition and annotation information of the module to be tested.

[0024] Optionally, the module characteristic data includes input and output types, functional requirements, boundary conditions, abnormal conditions and known potential problems of the module to be tested.

[0025] Optionally, the dynamic context generating unit includes:

[0026] A cyclomatic complexity calculation subunit, configured to obtain a decision point of the project file to be tested according to the abstract syntax tree, and calculate the cyclomatic complexity of the module to be tested;

[0027] The requirement selection subunit is used to dynamically select a test type according to the cyclomatic complexity; if the cyclomatic complexity exceeds a preset threshold, a test requirement is automatically added in combination with the requirement document.

[0028] Optionally, the test case supplement subsystem includes:

[0029] A coverage analysis unit, used to perform coverage analysis on the initial test suite, obtain uncovered line numbers in the module to be tested, and generate an uncovered line number list;

[0030] An uncovered code snippet division unit, used for mapping the uncovered line numbers in the uncovered line number list to corresponding functions or methods according to the structure of the module to be tested, extracting code snippets related to the uncovered line numbers, and storing the related code snippets and corresponding functions or methods in the form of tuples;

[0031] An uncovered code snippet optimization unit, used to extract relevant function or method signatures, function documentation or annotation information, and context of relevant code snippets in the module to be tested according to relevant code snippets;

[0032] A supplementary test case generation unit, used for inputting the context of function or method signature / function document or annotation information / code snippet into the program generation model to obtain preliminary supplementary test cases;

[0033] The semantic integrity judgment unit is used to perform a semantic integrity check on the preliminary supplementary test case. If the preliminary supplementary test case does not meet the semantic integrity, the preliminary supplementary test case is adjusted; if the preliminary supplementary test case meets the semantic integrity, the preliminary supplementary test case is used as the final supplementary test case.

[0034] Optionally, the correction verification subsystem includes:

[0035] A grammar correction unit, used to parse the test case and identify grammatical errors, correct the identified grammatical errors, and make grammatical error judgment on the corrected test case. If a grammatical error exists, the corrected test case is transmitted to a runtime error correction unit;

[0036] A runtime error correction unit, used for iteratively correcting the corrected test case according to the collected runtime error information;

[0037] The assertion error correction unit is used to correct the assertion errors of the test case according to the extracted body of the assertion expression and the functional annotation information of the module to be tested.

[0038] Optionally, a feedback subsystem is also included, which is used to execute the test suite, identify and separate erroneous test cases, and generate modification suggestions based on the test cases and the project files to be tested.

[0039] A method for generating software test cases based on a large language model, the specific steps are:

[0040] Build a set of modules to be tested based on the project files to be tested and the requirements documents;

[0041] Constructing a prompt template according to each module to be tested in the module set to be tested, and using an adaptive focus context generation mechanism and the prompt template to generate a program generation model to generate multiple test cases covering conventional scenarios, boundary conditions and abnormal inputs;

[0042] Performing multi-level correction processing on the test cases to eliminate syntax errors, runtime errors and assertion errors to generate an initial test suite;

[0043] Perform coverage analysis on the initial test suite to determine whether it meets a preset threshold; if not, construct a model input text based on uncovered data, and use the program generation model to generate supplementary test cases to update the initial test suite; if so, perform mutation testing on the initial test suite, and if the test fails, construct the model input text based on surviving variants, and use the program generation model to generate the supplementary test cases to update the initial test suite; if the test passes, generate a final test suite.

[0044] It can be seen from the above technical solutions that the present invention provides a method and system for generating software test cases based on a large language model, which has the following beneficial effects compared with the prior art:

[0045] 1. By combining the functional annotations of the module to be tested with the logical reasoning ability of LLM, the correctness of the expected value of the assertion can be dynamically verified, which can not only reduce the misjudgment caused by assertion errors, but also avoid covering up the real defects of the module to be tested due to blindly modifying the assertion, thereby ensuring the accuracy of the test feedback.

[0046] 2. By extracting the context information of the function where the uncovered line is located and combining it with LLM's code understanding ability, it dynamically generates semantically complete code snippets as test targets and guides LLM to generate targeted test cases. This not only effectively covers complex conditional branches, exception handling and other code areas that are difficult for traditional tools to reach, but also improves generation efficiency.

[0047] 3. By building a fully automated closed-loop unit test suite generation tool designed specifically for Python with a complete project as input, integrating syntax correction, runtime error repair and assertion authenticity verification modules, it can independently complete the conversion from original code to high-quality test suites without human intervention, greatly reducing the repetitive workload of test engineers. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the system structure provided by the present invention;

[0049] Figure 2 The present invention provides a flow chart of the method. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The embodiment of the present invention discloses a software test case generation system based on a large language model, such as Figure 1 As shown, including:

[0052] The module detection subsystem to be tested is used to parse and extract information from the project files and requirement documents to generate a set of modules to be tested;

[0053] A test case generation subsystem is used to construct a prompt template according to each module under test in the module set under test, and generate multiple test cases covering regular scenarios, boundary conditions and abnormal inputs by using a program generation model based on an adaptive focus context generation mechanism and the prompt template;

[0054] The correction and verification subsystem is used to implement multi-level correction processing on test cases, eliminate syntax errors, runtime errors and assertion errors, and generate initial test suites;

[0055] The test case supplement subsystem is used to perform coverage analysis on the initial test suite to determine whether it meets the preset threshold; if not, a model input text is constructed based on the uncovered data, and a program generation model is used to generate supplementary test cases to update the initial test suite; if yes, a mutation test is performed on the initial test suite. If the test fails, a model input text is constructed based on the surviving variants, and a program generation model is used to generate supplementary test cases to update the initial test suite; if the test passes, a final test suite is generated.

[0056] In one embodiment, the module detection subsystem includes:

[0057] The lexical analysis unit is used to perform lexical analysis on the project file to be tested and generate a token sequence;

[0058] The syntax analysis unit is used to construct an abstract syntax tree containing a complete syntax structure based on the token sequence;

[0059] A module set building unit, used to extract top-level import statements, function definitions and class definitions according to the abstract syntax tree, and form an initial module set to be tested;

[0060] A relevant information extraction unit, used to extract relevant information of each initial module to be tested in the initial module to be tested set;

[0061] The module set generation unit is used to automatically match the corresponding definition according to the call information of each initial module to be tested, and associate the relevant information with the corresponding initial module to be tested to generate a module set to be tested.

[0062] Furthermore, the module detection subsystem to be tested is used to analyze the structure of the source code in the project file to be tested, extract the module composition such as functions, methods, classes, etc., and analyze the calling relationship and dependency packages between modules; by parsing the code and extracting information, the relevant function and requirement description information (such as module comments) is associated with the corresponding module, and finally a set of modules to be tested suitable for unit testing is constructed.

[0063] Furthermore, the execution steps of the detection subsystem of the module under test are:

[0064] First, perform lexical analysis (decomposition into Tokens, including keywords, identifiers, literals, and operators, etc.) and syntax analysis (parse Tokens into abstract syntax trees AST according to the grammatical rules of the language) on the source code. Using AST, extract all top-level import statements, function definitions, and class definitions from the source code and organize them into an initial set of modules to be tested. At the same time, extract the comment information related to the initial modules to be tested (such as the description in the multi-line string '''...''') and save it separately for subsequent requirements and functional information association. According to the call information of each initial module to be tested in the initial set of modules to be tested (such as function calls, class instantiations, etc.), use the node information in AST to find the corresponding definition in the set. If the target definition is elsewhere in the current project file to be tested, the relevant code snippet is automatically completed; if it is defined in other project files to be tested, insert the corresponding import statement to introduce the definition; finally, a set of modules to be tested that can be directly used for unit testing is formed, providing a basis for subsequent test case generation.

[0065] Among them, the Abstract Syntax Tree (AST) is a structured intermediate representation that displays the grammatical structure of the source code in a hierarchical manner for easy subsequent processing.

[0066] In one embodiment, the test case generation subsystem includes:

[0067] A module data extraction unit, used to extract module data from the module to be tested, where the module data includes module basic data, module characteristic data and module function data;

[0068] A prompt template generating unit, used for building a prompt template according to module characteristic data;

[0069] A dynamic context generation unit, used for obtaining module requirements according to module data, and generating dynamic context information by using an adaptive focus context generation mechanism according to the module requirements and the prompt template;

[0070] A context filtering unit is used to remove information irrelevant to the current test target in the dynamic context information to obtain the model input text;

[0071] The generation unit is used to generate the model by inputting the model text into the program to generate test cases covering regular scenarios, boundary conditions and abnormal inputs.

[0072] In one embodiment, the module basic data includes the code, interface document, function definition and annotation information of the module to be tested.

[0073] In one embodiment, the module characteristic data includes the input and output types, functional requirements, boundary conditions, abnormal conditions, and known potential problems of the module to be tested.

[0074] In one embodiment, the dynamic context generation unit includes:

[0075] A cyclomatic complexity calculation subunit, used for obtaining the decision points of the project file to be tested according to the abstract syntax tree, and calculating the cyclomatic complexity of the module to be tested;

[0076] The requirement selection subunit is used to dynamically select the test type based on the cyclomatic complexity; if the cyclomatic complexity exceeds the preset threshold, the test requirements are automatically added in combination with the requirement document.

[0077] Furthermore, in this embodiment, a large language model (LLM) is used to automatically generate test cases for the module to be tested. An adaptive focus context generation mechanism and a prompt template are used to construct input prompts for the LLM. The adaptive mechanism dynamically adds high-value context content to the prompt information according to the characteristics and requirements of the module to be tested, enriching the input information of the LLM, thereby helping it to generate higher-quality unit test cases and form a test suite.

[0078] Furthermore, the execution steps of the test case generation subsystem are:

[0079] 1) Analyze and extract characteristics of the module to be tested, specifically:

[0080] Module information collection: collect the code, interface documentation, function definition and comment information of the module to be tested;

[0081] Module feature extraction: Extract the input and output types, functional requirements, boundary conditions, abnormal situations, and known potential problems (for example, common boundary conditions or logical errors) of the module to be tested;

[0082] Functional analysis: Use static analysis tools to analyze the control flow, data flow, possible branch conditions, and loop structure of the module under test.

[0083] 2) Design a prompt template, specifically:

[0084] According to the characteristics of the module to be tested, a set of standardized prompt templates are designed, including module function description, test constraints, and input format limitations.

[0085] 3) Automatically select context information: Dynamically select relevant context information according to the different requirements of the module to be tested; specifically:

[0086] First, calculate the cyclomatic complexity of the module to be tested. For complex test cases with a cyclomatic complexity greater than 10, assume that the possibility of abnormal situations is high, and make functional judgments based on the requirements document and choose to add test requirements, for example:

[0087] Abnormal input testing: Pass illegal or abnormally formatted data (such as null values, negative numbers, numbers outside the expected range, etc.) to the module under test to verify the robustness of the module when facing boundary conditions.

[0088] Performance bottleneck test: For complex calculations or loop structures, add performance testing requirements to check the performance of the module under high load or long-term operation, such as whether there is a memory leak or slow response.

[0089] Concurrency testing: If the module to be tested contains multi-threading or concurrent operations, concurrent condition testing will be automatically added to simulate the multi-threaded access scenario to check whether there are problems such as data competition, deadlock or resource conflict.

[0090] Resource limitation test: When resources are limited (such as memory, CPU, etc.), add resource limitation test to verify the stability and performance of the module under test in a low-resource environment.

[0091] Network or external dependency testing: If the module under test depends on external services (such as databases, APIs, network connections, etc.), simulations such as network delays and service unavailability will be automatically inserted to check the behavior of the module under test when external services fail or the network is unstable.

[0092] 4) Context filtering and optimization, specifically: removing irrelevant information to ensure that the LLM input is concise and focused on the test objectives.

[0093] In one embodiment, the test case supplement subsystem includes:

[0094] A coverage analysis unit is used to perform coverage analysis on the initial test suite, obtain uncovered line numbers in the module to be tested, and generate a list of uncovered line numbers;

[0095] An uncovered code snippet partitioning unit is used to map uncovered line numbers in the uncovered line number list to corresponding functions or methods according to the structure of the module to be tested, extract code snippets related to the uncovered line numbers, and store the related code snippets and corresponding functions or methods in the form of tuples;

[0096] The uncovered code snippet optimization unit is used to extract the relevant function or method signatures, function documentation or annotation information, and the context of the relevant code snippets in the module to be tested based on the relevant code snippets;

[0097] A supplementary test case generation unit, used for inputting the context of function or method signature / function document or annotation information / code snippet into the program generation model to obtain preliminary supplementary test cases;

[0098] The semantic integrity judgment unit is used to perform semantic integrity check on the preliminary supplementary test case. If it does not meet the semantic integrity, the preliminary supplementary test case is adjusted; if it meets the semantic integrity, it is used as the final supplementary test case.

[0099] Furthermore, the main function of the test case supplement subsystem is to analyze the coverage of the existing test suite and check whether it reaches the preset threshold. If the coverage is insufficient, the coverage is improved by adding new test cases. Specifically, first use Python's Coverage tool to analyze the coverage of the test suite to determine the line numbers in the module to be tested that have not been covered by the test. Considering that generating test cases in small fragments helps LLM achieve higher coverage, these uncovered line numbers are combined with the module context to obtain uncovered code slices with complete semantic information. Subsequently, the function code containing the uncovered code slices is used as a prompt prefix, and these uncovered code slices are used as key requirements for prompt word design.

[0100] In this process, by appropriately reducing the amount of overall context information provided to LLM and increasing the detail of information in certain key parts, LLM is guided to better understand the functions and intentions of uncovered code segments in a "step-by-step" manner, thereby generating more targeted test cases. If the expected coverage target is still not achieved after the first supplement, the supplement process can be repeated multiple times until a satisfactory coverage level is reached.

[0101] Furthermore, the execution steps of the test case supplement subsystem are:

[0102] 1) Collect uncovered line numbers, specifically:

[0103] Get coverage report: Use the test coverage tool coverage.py to perform coverage analysis on the module to be tested and generate a coverage report;

[0104] Extract uncovered line numbers: Extract all uncovered line numbers in the module to be tested from the coverage report. These uncovered line numbers will be used as core data for processing in subsequent steps.

[0105] Organize uncovered line numbers: Generate a list of all line numbers that have not been tested to facilitate further analysis in subsequent steps.

[0106] 2) Roughly divide the uncovered code slices into:

[0107] Based on function and class division: According to the structure of the module to be tested (for example, function definition, class structure, etc.), the uncovered line numbers are matched with the corresponding functions or methods; each function or method will contain one or more uncovered code lines;

[0108] Roughly identify code snippets: For each uncovered line number, locate the function or code block where the line is located, and extract the code snippets related to the uncovered line number in the function. The code snippets obtained at this time are usually incomplete, including only the relevant uncovered line and the lines before and after it, and may lack necessary context or key information;

[0109] Structured uncovered fragments: These roughly divided code fragments are stored in the form of tuples such as (code fragment, corresponding function or method) to prepare for subsequent semantic supplementation and correction.

[0110] 3) Use LLM to obtain semantically complete uncovered code fragments, specifically:

[0111] 3.1. Prepare complete prompt information, specifically:

[0112] Function or method signature: Extract the function or method signature related to the roughly uncovered code snippet from the module under test as part of the hint. This helps LLM understand the context in which the code snippet is located;

[0113] Function documentation and comments: If the function or method has relevant documentation or comments, pass this information to LLM as context. This documentation information helps LLM understand the function's functionality, input and output requirements, and design goals.

[0114] Context of uncovered snippets: Provide uncovered code snippets (coarse snippets) to LLM with additional context information, such as the overall logic of the function or the context (if any).

[0115] 3.2、 LLM analysis and supplement, with the following features:

[0116] The extracted function signatures, document comments, uncovered code snippets and other information are input to LLM, which will analyze the information, supplement the missing parts in the uncovered snippets, and generate more complete code content. For example, LLM may infer the function of the uncovered part based on the provided function signature and context, and generate related code logic, including variable initialization, control flow structure, exception handling, etc.

[0117] 3.3, Semantic integrity check:

[0118] The code snippets generated by LLM need to be checked for semantic integrity. At this point, you can use static analysis to check the code logic to ensure that the newly generated code meets the original function design requirements and avoid introducing unnecessary logical errors or inconsistencies. If you find that there are problems with the generated code snippets, further adjust the prompt content and regenerate it to ensure that each generated code is reasonable and consistent with the original function.

[0119] In one embodiment, mutation testing is a means for evaluating the quality and efficiency of test cases. Even if 100% statement and branch coverage has been achieved in unit testing, mutation testing is still of great significance. By injecting artificial errors into the code to be tested (i.e., constructing "mutants"), the validity of existing test cases can be verified: if a variant is not discovered by the test case (i.e., the test does not fail on the variant), the variant is considered "survival". Surviving variants indicate that the current test cases are insufficient and need to be supplemented or improved.

[0120] In the implementation process of this embodiment, a combination of MutPy and a large language model (LLM) is used to generate variants. Since the modules to be tested may be very diverse and complex, MutPy sometimes fails to generate variants. In this case, LLM can be used to generate variants. Subsequently, the unittest framework is used to test the generated multiple variants and calculate the mutation score (i.e., the percentage of the number of variants that have been killed to the total number of variants). If the mutation score reaches 100%, it means that the mutation test has fully met the standard.

[0121] If there are surviving variants, new prompts will be designed for these surviving variants, and corresponding supplementary test cases will be generated through LLM and added to the test suite. This cyclic process helps to continuously improve the quality and coverage of test cases.

[0122] In one embodiment, the correction verification subsystem includes:

[0123] A grammar correction unit is used to parse the test case and identify grammar errors, correct the identified grammar errors, and make grammar error judgment on the corrected test case. If there is a grammar error, the corrected test case is transmitted to the runtime error correction unit;

[0124] A runtime error correction unit, used for iteratively correcting the corrected test cases according to the collected runtime error information;

[0125] The assertion error correction unit is used to correct the assertion errors of the test case according to the extracted body of the assertion expression and the functional annotation information of the module to be tested.

[0126] Furthermore, the correction verification subsystem executes the following steps:

[0127] The correction method of the grammar correction unit is:

[0128] When using LLM to generate test cases, some potential syntax errors are often encountered, such as modules not imported, incomplete statement structures, undeclared variables, and undefined functions. These syntax errors will significantly reduce the effectiveness and execution efficiency of test cases. In order to solve this problem, this embodiment uses a syntax correction unit to automatically detect and correct possible syntax errors before executing the test case. Specifically, Python is first used to parse the unit test suite to be tested and convert it into an abstract syntax tree to confirm its compliance with Python syntax rules. If a syntax error is detected during the parsing process, a prompt word is constructed according to the error type (for example, "Please modify the syntax error in the following code:") and the syntax repair capability of LLM is called for preliminary correction. If there are still syntax errors after the correction, the Runtime Error Corrector will further repair them during the execution phase to ensure that the test case can finally run successfully.

[0129] The correction method of the runtime error correction unit is as follows: a test suite that can be successfully compiled may not necessarily be executed completely correctly at runtime; in fact, it may fail due to runtime errors (such as division by zero errors, array out of bounds, null pointer references and type conversion errors, etc.), missing environment dependencies, concurrency conflicts and other runtime exceptions. These runtime errors usually constitute the main challenge to the effectiveness and reliability of test cases. To deal with such problems, it is observed that several high-performance LLMs can automatically generate relatively professional and accurate repair solutions based on given error information, and in most cases, the correction operation does not change the logical function of the code itself. Specifically, after executing the test suite, the runtime error information is tracked and collected; then, prompt words are constructed based on the extracted error information, and LLM is called for targeted repair. After the repair is completed, the test suite is executed again and the above process is iterated. Experiments show that this "detection-repair-retest" cycle mechanism can effectively solve most runtime errors, thereby greatly improving the operability and stability of the test suite.

[0130] The correction method of the assertion error correction unit is: based on the experience of professional test engineers in writing unit tests, an "assertion truth checker" is designed and the inspection indicator "non-assertion error test pass rate" is proposed.

[0131] ;

[0132] In the formula, The number of runtime errors (division by zero, array out of bounds, null pointer reference, type conversion error), missing environment dependencies, and concurrency conflicts; A quantity that asserts that the true value is not equal to the actual value.

[0133] At the implementation level, we first extract the body of the assertion expression (i.e., the left side of the assertion), and combine it with the functional annotations of the module to be tested to form prompt words, so that LLM can simulate the reasoning process of professional test engineers and regenerate the true value of the assertion based on the functional description, thereby obtaining a higher quality and more targeted unit test suite.

[0134] In one embodiment, a feedback subsystem is also included, which is used to execute the test suite, identify and separate erroneous test cases, and generate modification suggestions based on the test cases and the project files to be tested.

[0135] Furthermore, the test suite is first executed to identify and isolate test cases that contain assertion truth value errors. Subsequently, the system combines these test cases with assertion errors with the code to be tested to design corresponding prompt words. By using these test cases, a large language model (LLM) is used to provide feedback detection for the code to be tested, generate code modification suggestions, and finally complete the complete process of unit testing.

[0136] Another aspect of this embodiment includes a method for generating software test cases based on a large language model, such as Figure 2 As shown, the specific steps are:

[0137] Build a set of modules to be tested based on the project files to be tested and the requirements documents;

[0138] Construct prompt templates according to each module under test in the module set under test, and use an adaptive focus context generation mechanism and prompt templates to generate multiple test cases covering regular scenarios, boundary conditions and abnormal inputs using a program generation model;

[0139] Perform multi-level correction processing on test cases to eliminate syntax, runtime and assertion errors and generate initial test suites;

[0140] Perform coverage analysis on the initial test suite to determine whether it meets the preset threshold; if not, build a model input text based on the uncovered data, and use the program generation model to generate supplementary test cases to update the initial test suite; if so, perform mutation testing on the initial test suite. If the test fails, build a model input text based on the surviving mutants, and use the program generation model to generate supplementary test cases to update the initial test suite; if the test passes, generate the final test suite.

[0141] Furthermore, the project file to be tested and the corresponding requirement document are passed to the module to be tested detection subsystem to obtain the module to be tested. The module to be tested uses a large language model (LLM) to generate complete test cases according to the given modules and methods, covering common scenarios, boundary conditions and abnormal inputs, and uses mocking to isolate external dependencies. Subsequently, the semantic errors and assertions in the code are corrected to ensure that the code meets the functional requirements and logical consistency. The validity and coverage of the test cases are then verified through coverage analysis and mutation test generation. If the test case meets the requirements, the feedback subsystem will verify and confirm the final test results to ensure that the generated test cases are efficient and comprehensive; in this embodiment, unified text is used for test case generation and coverage analysis, and the comparison results of this application with existing tools are shown in Tables 1 and 2.

[0142] Table 1: Non-assertion error pass rate and number of generated test cases

[0143]

[0144] Table 2: Statement coverage and branch coverage

[0145]

[0146] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0147] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A software test case generation system based on a large language model, characterized in that: include: The module detection subsystem to be tested is used to parse and extract information from the project files and requirement documents to generate a set of modules to be tested; A test case generation subsystem, configured to construct a prompt template according to each module to be tested in the module set to be tested, and generate a plurality of test cases covering conventional scenarios, boundary conditions and abnormal inputs by using a program generation model based on an adaptive focus context generation mechanism and the prompt template; A correction and verification subsystem, used for performing multi-level correction processing on the test cases, eliminating syntax errors, runtime errors and assertion errors, and generating an initial test suite; A test case supplement subsystem, used to perform coverage analysis on the initial test suite to determine whether a preset threshold is met; If not, constructing a model input text based on uncovered data, and generating supplementary test cases using the program generation model to update the initial test suite; If yes, then mutation testing is performed on the initial test suite, and if the test fails, the model input text is constructed according to the surviving variants, and the supplementary test cases are generated using the program generation model to update the initial test suite; If the tests pass, a final test suite is generated.

2. A software test case generation system based on a large language model according to claim 1, characterized in that: The module detection subsystem to be tested includes: A lexical analysis unit, used for performing lexical analysis on the project file to be tested and generating a token sequence; A syntax analysis unit, used to construct an abstract syntax tree containing a complete syntax structure according to the token sequence; A module set forming unit, used for extracting top-level import statements, function definitions and class definitions according to the abstract syntax tree, and forming an initial module set to be tested; A relevant information extraction unit, used to extract relevant information of each initial module to be tested in the initial module to be tested set; The module set generating unit is used to automatically match the corresponding definition according to the calling information of each of the initial modules to be tested, and associate the relevant information with the corresponding initial modules to be tested to generate the module set to be tested.

3. A software test case generation system based on a large language model according to claim 2, characterized in that: The test case generation subsystem includes: A module data extraction unit, used to extract module data from the module to be tested, wherein the module data includes module basic data, module characteristic data and module function data; A prompt template generating unit, used for constructing the prompt template according to the module characteristic data; A dynamic context generation unit, configured to obtain module requirements according to the module data, and generate dynamic context information using the adaptive focus context generation mechanism according to the module requirements and the prompt template; A context filtering unit, used to remove information irrelevant to the current test target in the dynamic context information to obtain the model input text; A generating unit is used to input the model input text into the program generation model to generate test cases covering conventional scenarios, boundary conditions and abnormal inputs.

4. A software test case generation system based on a large language model according to claim 3, characterized in that: The module basic data includes the code, interface document, function definition and annotation information of the module to be tested.

5. A software test case generation system based on a large language model according to claim 3, characterized in that: The module characteristic data includes the input and output types, functional requirements, boundary conditions, abnormal conditions and known potential problems of the module to be tested.

6. A software test case generation system based on a large language model according to claim 3, characterized in that: The dynamic context generating unit comprises: A cyclomatic complexity calculation subunit, configured to obtain a decision point of the project file to be tested according to the abstract syntax tree, and calculate the cyclomatic complexity of the module to be tested; The requirement selection subunit is used to dynamically select a test type according to the cyclomatic complexity; if the cyclomatic complexity exceeds a preset threshold, a test requirement is automatically added in combination with the requirement document.

7. A software test case generation system based on a large language model according to claim 4, characterized in that: The test case supplement subsystem includes: A coverage analysis unit, used to perform coverage analysis on the initial test suite, obtain uncovered line numbers in the module to be tested, and generate an uncovered line number list; An uncovered code snippet division unit, used for mapping the uncovered line numbers in the uncovered line number list to corresponding functions or methods according to the structure of the module to be tested, extracting code snippets related to the uncovered line numbers, and storing the related code snippets and corresponding functions or methods in the form of tuples; An uncovered code snippet optimization unit, used to extract relevant function or method signatures, function documentation or annotation information, and context of relevant code snippets in the module to be tested according to relevant code snippets; A supplementary test case generation unit, used for inputting the context of function or method signature / function document or annotation information / code snippet into the program generation model to obtain preliminary supplementary test cases; The semantic integrity judgment unit is used to perform a semantic integrity check on the preliminary supplementary test case. If the preliminary supplementary test case does not meet the semantic integrity, the preliminary supplementary test case is adjusted; if the preliminary supplementary test case meets the semantic integrity, the preliminary supplementary test case is used as the final supplementary test case.

8. A software test case generation system based on a large language model according to claim 4, characterized in that: The correction verification subsystem comprises: A grammar correction unit, used to parse the test case and identify grammatical errors, correct the identified grammatical errors, and make grammatical error judgment on the corrected test case. If a grammatical error exists, the corrected test case is transmitted to a runtime error correction unit; A runtime error correction unit, used for iteratively correcting the corrected test case according to the collected runtime error information; The assertion error correction unit is used to correct the assertion errors of the test case according to the extracted body of the assertion expression and the functional annotation information of the module to be tested.

9. A software test case generation system based on a large language model according to claim 1, characterized in that: It also includes a feedback subsystem for executing the test suite, identifying and separating erroneous test cases, and generating modification suggestions based on the test cases and the project files to be tested.

10. A method for generating software test cases based on a large language model, characterized in that: The specific steps are: Build a set of modules to be tested based on the project files to be tested and the requirements documents; Constructing a prompt template according to each module to be tested in the module set to be tested, and using an adaptive focus context generation mechanism and the prompt template to generate a program generation model to generate multiple test cases covering conventional scenarios, boundary conditions and abnormal inputs; Performing multi-level correction processing on the test cases to eliminate syntax errors, runtime errors and assertion errors to generate an initial test suite; Performing coverage analysis on the initial test suite to determine whether a preset threshold is met; If not, constructing a model input text based on uncovered data, and generating supplementary test cases using the program generation model to update the initial test suite; If yes, then mutation testing is performed on the initial test suite, and if the test fails, the model input text is constructed according to the surviving variants, and the supplementary test cases are generated using the program generation model to update the initial test suite; If the tests pass, a final test suite is generated.

Images