A test case generation method based on LLM and SMT solver

Abstract

The present invention relates to the field of software testing technology, and discloses a test case generation method based on LLM and SMT solver. With search-type testing as the core driver, when coverage stagnation occurs, the execution path is semantically analyzed with the help of LLM, and a logical expression describing the target path condition is generated. Taking into account the limitations of LLM in logical reasoning, the system further introduces an SMT solver to verify and reason the logical expression output by LLM, and finally generates feasible test inputs, thereby improving test efficiency and coverage. The entire process forms a closed-loop optimization process that coordinates semantic understanding and logical solving. This method not only improves the efficiency and accuracy of path coverage, but also has good adaptability and versatility. It is applicable to multiple languages ​​and complex program structures, effectively improving the intelligence level and engineering practical value of automated testing. It effectively alleviates the problem of insufficient accuracy of large language models in complex path constraint reasoning.

Description

Technical Field

[0001] The present invention relates to the field of software testing technology, and in particular to a test case generation method based on LLM and SMT solver. Background Art

[0002] Test case generation is a critical step in software testing. Its core goal is to construct input data that can effectively discover program defects and verify that program behavior meets expectations. Traditional test case generation methods, such as manual writing, random testing, and search-based software testing (SBST), have improved testing efficiency to a certain extent, but they still face many challenges in dealing with complex logical paths and cannot fully cover all branches of the program.

[0003] In recent years, large language models (LLMs), driven by rapid advancements in natural language processing (NLP), have demonstrated powerful capabilities in text generation and semantic understanding. LLMs based on the Transformer architecture, such as the GPT series, BERT, and T5, have demonstrated exceptional performance in processing natural language and code, demonstrating capabilities such as understanding program structure, generating semantically sound text, and automatically completing code. These technologies have given LLMs significant potential in software engineering, particularly test case generation. The Transformer architecture, through its self-attention mechanism, can capture long-range dependencies and contextual information, demonstrating excellent sequence modeling capabilities. Consequently, LLMs demonstrate exceptional capabilities in understanding and generating program code, particularly in automated testing, where they can generate test cases that meet requirements based on input program code and prompts (cue words). Furthermore, LLMs possess powerful semantic reasoning capabilities, enabling them to generate reasonable variable relationships and path constraints based on program context.

[0004] To overcome these bottlenecks, recent research has attempted to introduce large language models (LLMs) into the testing process to enhance path analysis and test generation capabilities. However, because LLMs excel at semantic understanding rather than precise logical reasoning, they still have certain limitations in handling complex path constraints and generating high-precision test data. Specifically:

[0005] (1) DynsaMOSA is the most advanced search-based software testing (SBST) algorithm. Figure 1As shown in Figure 2, the core concept of DynaMOSA is to dynamically adjust the focus objectives during the testing process to improve test resource utilization and coverage. In the initial phase of the algorithm, DynaMOSA randomly initializes a set of test cases and iteratively optimizes them through evolutionary operations such as crossover and mutation. To select high-quality test cases, the algorithm introduces a non-dominated sorting mechanism. This constructs a non-dominated solution set based on the test case's performance in the multi-objective space and combines it with crowding distance to assess the diversity of the individual solutions, ensuring that the solution set strikes a balance between optimization accuracy and diversity.

[0006] In terms of target prioritization, DynaMOSA adopts a "prioritization"-based strategy, prioritizing test cases that are closer to uncovered targets, building an initial set of high-value solutions, and guiding the search to focus on critical coverage paths. Furthermore, to avoid wasting computational resources by indiscriminately attempting to cover all targets in each iteration, DynaMOSA uses a control dependency graph (CDG) to identify dependencies between targets and dynamically adjust the current target set. Specifically, for test targets that depend on uncovered targets, the algorithm temporarily removes them from the target vector and attempts to cover only those targets that are currently reachable or have a higher priority. When the upper-level targets they depend on are successfully covered, the relevant targets are re-added to the target set, thereby achieving dynamic adjustment of test focus and efficient resource allocation.

[0007] However, despite its widespread application in multi-target test case generation and its effective improvement in resource utilization and coverage by introducing a control dependency graph to dynamically prioritize test targets, DynaMOSA can still encounter coverage stagnation when dealing with logically complex paths within programs with deeply nested conditional branches. Its core mechanism relies on heuristic search and mutation operations, lacking precise modeling and systematic solution of path conditions. Consequently, it often struggles to improve coverage when faced with paths that are difficult to reach through random generation or evolution.

[0008] (2) The core idea of ​​CODAMOSA is to introduce a large language model auxiliary mechanism in the search-based software testing process to break through the bottleneck of stagnant test coverage and improve the overall test effect and code coverage capability. In the initial stage of the algorithm, CODAMOSA is consistent with traditional search algorithms, randomly initializes a set of test cases, and searches through operations such as crossover and mutation. When it is detected that the coverage has not improved within several generations, the algorithm will call a large language model (such as Codex) to generate sample test cases for low-coverage functions and parse them into a standard format that can participate in evolution. To ensure the effective use of the generated test cases, CODAMOSA constructs a special deserialization mechanism and grammatical structure processing module, introduces representation methods such as "uninterpretable statements", thereby expanding the search space and improving path accessibility. Through this search and prompt collaborative strategy, CODAMOSA can accurately guide the search direction while maintaining diversity and variability, effectively improving the coverage capability of complex paths.

[0009] However, while CODAMOSA demonstrates promising results in addressing coverage stagnation, it still exhibits limitations when dealing with branch targets with complex logical conditions. Because its test case generation relies primarily on the prompt-based code generation capabilities of a large language model, it lacks precise modeling of path conditions and constraint-solving mechanisms, making it difficult to precisely control and cover specific logical paths. When target branches involve complex Boolean expressions, numerical relationships, or deep dependencies, the test code generated by LLM often struggles to meet the triggering conditions, resulting in reduced coverage.

[0010] Therefore, how to improve the testing efficiency and coverage capability of software testing, alleviate the deficiencies in logical reasoning in test case generation scenarios based on LMM, and improve the coverage of complex branch paths and the effectiveness of test cases is a technical problem that needs to be solved urgently. Summary of the Invention

[0011] The main purpose of the present invention is to provide a test case generation method based on LLM and SMT solver, aiming to solve at least one of the above technical problems.

[0012] To achieve the above object, the present invention provides a test case generation method based on LLM and SMT solver, the method comprising the following steps:

[0013] S1: When using SBST to generate test cases, the system performs coverage stagnation detection after each round of test case execution.

[0014] S2: Based on the uncovered branches in the coverage stagnation detection results, static path analysis is performed. Combining CFG and breadth-first algorithm, the shortest path from the program start node to the uncovered branch is searched, capturing branch correlation information during the path construction process.

[0015] S3: using the LLM, based on the structured input encapsulated by the path information of the shortest path and the branch association information, generating an SMT-LIB expression capable of triggering a branch corresponding to the shortest path;

[0016] S4: Use the SMT solver to solve the SMT-LIB expression output by the LLM, and generate new test cases based on the solution results.

[0017] Optionally, step S1 specifically includes:

[0018] S11: In the process of generating test cases using SBST, the system collects and updates branch coverage information after each round of test case execution.

[0019] S12: Analyze the coverage growth trend through a sliding window to determine whether a trigger condition is met. If so, trigger coverage stagnation detection.

[0020] Optionally, in step S12, the triggering condition specifically includes: in several consecutive generations of testing, the improvement of coverage is less than a preset threshold, or the target branch has never been covered within a set number of iterations.

[0021] Optionally, in step S2, the shortest path from the program start node to the uncovered branch is searched by combining the CFG and the breadth-first algorithm, specifically including:

[0022] S21: Perform lexical analysis, syntax analysis, and semantic analysis on the program source code to divide it into basic blocks, create nodes for each basic block, add directed edges based on the control transfer relationship, build a CFG, and map uncovered branches to edges in the CFG;

[0023] S22: Add the starting point of the CFG to the queue and mark it as visited. Take out the head node of the queue and traverse all outgoing edges. Perform access judgment on the node pointed by each outgoing edge. If the node is not visited, add the node to the queue as the head node of the queue and re-execute the outgoing edge traversal and access judgment.

[0024] S23: When it is detected that the queue head node belongs to the target uncovered branch, the search is terminated, the shortest path is generated by backtracking the predecessor chain and converted into a statement sequence in the program source code.

[0025] Optionally, in step S2, the branch association information specifically includes: a code snippet of the target branch and a code snippet that calls an external function; wherein, when the shortest path contains uncontrollable environment variables or external function dependencies, the system generates an alternative path suggestion or introduces a path instability identifier in the prompt word.

[0026] Optionally, step S3 specifically includes:

[0027] S31: using LLM to identify and define all variables involved in the shortest path, and perform type declaration and initialization in a format that complies with SMT-LIB syntax;

[0028] S32: Based on the statement sequence corresponding to the shortest path, the LLM is driven to construct path constraints that describe the program execution conditions and convert them into standardized SMT-LIB code;

[0029] S33: Considering the external function calls contained in the shortest path, the LLM is driven to expand the function body logic, perform function inline expansion, and incorporate the function behavior into the overall path constraints to generate a complete SMT-LIB expression that can trigger the shortest path of the target branch.

[0030] Optionally, step S4 specifically includes:

[0031] S41: Solve the SMT-LIB expression output by the LLM using the SMT solver to verify whether the SMT-LIB expression is satisfiable. If so, return the specific variable assignment in the SMT-LIB expression as test data, and generate a new test case using the test data.

[0032] S42: If not, return to step S3, use LLM to optimize the generated SMT-LIB expression, and re-use the SMT solver to solve the SMT-LIB expression output by LLM.

[0033] Optionally, step S42 specifically includes:

[0034] S421: When an error occurs during the execution of an SMT-LIB expression in the SMT solver, the error information is returned to the LLM as feedback, allowing the LLM to optimize the generated SMT-LIB expression and fix format errors, undeclared variables, or logical conflicts in the SMT-LIB expression.

[0035] S422: Reuse the SMT solver to solve the SMT-LIB expression output by the LLM.

[0036] Optionally, step S42 further includes:

[0037] S423: When the SMT runs normally in the SMT solver but the generated test case fails to reach the target branch, a structured input for guiding the LLM to optimize the SMT-LIB expression is generated based on the current SMT-LIB expression and path coverage information and the variable assignment result returned by the SMT solver;

[0038] S434: Utilize the LLM to optimize the generated SMT-LIB expression for the target branch that fails to be reached based on the structured input, and reuse the SMT solver to solve the SMT-LIB expression output by the LLM.

[0039] Optionally, in step S423, based on the current SMT-LIB expression and path coverage information and the variable assignment result returned by the SMT solver, a structured input for guiding the LLM to optimize the SMT-LIB expression is generated, specifically including:

[0040] S4231: Based on the current path coverage information and the variable assignment results returned by the SMT solver, analyze the difference nodes between the current path coverage information and the untriggered target branches, and generate several first hint words for optimizing the SMT-LIB expression;

[0041] S4232: Based on the current SMT-LIB expression, all constraints in the current SMT-LIB expression are analyzed, a set of contradictory constraints in the SMT-LIB expression is extracted, and several second prompt words for the user to optimize the SMT-LIB expression are generated;

[0042] S4233: Based on several of the first prompt words and several of the second prompt words, generate a structured input for guiding LLM to optimize the SMT-LIB expression; wherein, the structured input is configured to drive the SMT-LIB expression to select a path that does not trigger the target branch at the difference node and correct the constraint contradictions in the previously generated SMT-LIB expression.

[0043] The beneficial effects of the present invention are as follows: a test case generation method based on LLM and SMT solver is proposed. By having the language model be responsible for the extraction and expression of conditions at the semantic level, and then handing it over to the SMT solver for precise logical reasoning and solution, the system realizes the division of labor and cooperation between semantic understanding and logical solution, significantly reducing the computational and reasoning burden of the language model in path reachability analysis. This method not only improves the efficiency and accuracy of path coverage, but also has good adaptability and versatility, is applicable to multiple languages ​​and complex program structures, and effectively improves the intelligence level and engineering practical value of automated testing. It effectively alleviates the problem of insufficient accuracy of large language models in complex path constraint reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flowchart of the test case generation method based on LLM and SMT solver of the present invention;

[0045] Figure 2 This is a schematic diagram of the test case generation method based on LLM and SMT solver of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] Definitions of key terms and abbreviations:

[0048] (1) SBST: Search-based software testing (SBST) is an automated software testing technique that aims to select appropriate test cases from a large number of possible input spaces through search algorithms (such as genetic algorithms, simulated annealing, etc.) to cover different execution paths of the program. The goal of SBST is to generate effective test cases through intelligent search to achieve the highest possible code coverage. In the SBST process, a test case is usually regarded as an "individual" and its execution effect is evaluated through interaction with the target program. SBST repeatedly adjusts these test cases through optimization algorithms (such as evolutionary algorithms or genetic algorithms) to achieve coverage of the target path. It is particularly suitable for programs with complex paths and no clear constraints.

[0049] (2) LLM: Large Language Model (LLM) is a large-scale artificial intelligence model trained based on deep learning technology. It has the ability to understand, generate and process natural language by analyzing and learning the language structure, semantics and contextual information in massive text data.

[0050] (3) Prompt: When using a large language model (LLM) for natural language processing or code generation tasks, a prompt is provided to the model as input to guide it to generate the desired output.

[0051] (4) SMT: Satisfiability Modulo Theories (SMT) is an automated decision method that combines logical reasoning with specific mathematical theories (such as integer arithmetic, arrays, bit vectors, etc.). It introduces background theory based on propositional logic and is used to determine whether a logical formula with theoretical constraints has an interpretation that satisfies the conditions.

[0052] (5) SMT solver: An SMT solver is an automated tool used to determine whether a set of logical constraints is satisfiable (i.e., whether there is a set of variable assignments that can make the constraints hold). It is widely used in software verification, test generation, program analysis, formal methods, and other fields. SMT solvers can handle constraints that are more complex than traditional Boolean satisfiability (SAT) problems. They not only support Boolean logic, but also handle variables and relations in theoretical domains such as integers, real numbers, arrays, strings, and bit vectors.

[0053] (6) SMT-LIB: SMT-LIB is the standard input language used by SMT solvers. It is used to represent logical expressions, constraints, and solution goals. SMT-LIB provides a unified format for constraint solving, facilitating exchange and execution between different SMT solvers.

[0054] (7) Branch Coverage: Branch coverage is a common test coverage metric in software testing, used to assess whether test cases sufficiently cover all conditional branches in a program. Branch coverage requires that each branch of each conditional statement (usually the "true" and "false" paths) is executed at least once, ensuring that the program's behavior is fully verified under different execution paths.

[0055] (8) CFG: A control flow graph (CFG) is a graph structure commonly used in static program analysis to represent the control flow path of a program during execution. In a control flow graph, a node usually represents a basic block, which is a sequence of statements that are executed sequentially without interruption, while an edge represents a possible control jump or execution flow in the program, such as an if branch, a loop jump, a function call return, etc.

[0056] Test case generation is a key step in ensuring software quality. Search-Based Software Testing (SBST) is widely used in existing technologies for automated test case generation. This approach models the testing process as an optimization problem, leveraging various optimization algorithms to search the input space for test data that covers the target path. However, SBST often struggles with complex branching paths and is prone to falling into local optima, resulting in limited test coverage. To overcome these bottlenecks, recent research has attempted to incorporate large language models (LLMs) into the testing process to enhance path analysis and test generation capabilities. However, because LLMs excel at semantic understanding rather than precise logical reasoning, they still have limitations in handling complex path constraints and generating high-precision test data.

[0057] To this end, this paper proposes a test case generation method that integrates LLM with a satisfiability module theory (SMT) solver. When SBST encounters path coverage stagnation, this method uses LLM to analyze and extract current path information and convert path conditions into SMT format. The SMT solver then generates valid test inputs that satisfy complex conditional constraints. Furthermore, a closed-loop feedback mechanism is incorporated to dynamically optimize the generation process, improving test efficiency and coverage. This method effectively mitigates the logical reasoning limitations of LLM by offloading the path solving task to the SMT solver, significantly improving coverage of complex branching paths and test case effectiveness.

[0058] Specifically, the embodiment of the present invention provides a test case generation method based on LLM and SMT solver, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of a test case generation method based on LLM and SMT solver of the present invention.

[0059] In this embodiment, a test case generation method based on LLM and SMT solver includes the following steps:

[0060] S1: When using SBST to generate test cases, the system performs coverage stagnation detection after each round of test case execution.

[0061] S2: Based on the uncovered branches in the coverage stagnation detection results, static path analysis is performed. Combining CFG and breadth-first algorithm, the shortest path from the program start node to the uncovered branch is searched, capturing branch correlation information during the path construction process.

[0062] S3: using the LLM, based on the structured input encapsulated by the path information of the shortest path and the branch association information, generating an SMT-LIB expression capable of triggering a branch corresponding to the shortest path;

[0063] S4: Use the SMT solver to solve the SMT-LIB expression output by the LLM, and generate new test cases based on the solution results.

[0064] In a preferred embodiment, step S1 specifically includes:

[0065] S11: In the process of generating test cases using SBST, the system collects and updates branch coverage information after each round of test case execution.

[0066] S12: Analyze the coverage growth trend through a sliding window to determine whether a trigger condition is met. If so, trigger coverage stagnation detection.

[0067] In a preferred embodiment, in step S12, the triggering condition specifically includes: in several consecutive generations of testing, the improvement of coverage is less than a preset threshold, or the target branch has never been covered within a set number of iterations.

[0068] In a preferred embodiment, in step S2, the shortest path from the program start node to the uncovered branch is searched by combining CFG and breadth-first algorithm, specifically including:

[0069] S21: Perform lexical analysis, syntax analysis, and semantic analysis on the program source code to divide it into basic blocks, create nodes for each basic block, add directed edges based on the control transfer relationship, build a CFG, and map uncovered branches to edges in the CFG;

[0070] S22: Add the starting point of the CFG to the queue and mark it as visited. Take out the head node of the queue and traverse all outgoing edges. Perform access judgment on the node pointed by each outgoing edge. If the node is not visited, add the node to the queue as the head node of the queue and re-execute the outgoing edge traversal and access judgment.

[0071] S23: When it is detected that the queue head node belongs to the target uncovered branch, the search is terminated, the shortest path is generated by backtracking the predecessor chain and converted into a statement sequence in the program source code.

[0072] In a preferred embodiment, in step S2, the branch association information specifically includes: a code snippet of the target branch and a code snippet that calls an external function; wherein, when the shortest path contains uncontrollable environment variables or external function dependencies, the system generates an alternative path suggestion or introduces a path instability indicator in the prompt word.

[0073] In a preferred embodiment, step S3 specifically includes:

[0074] S31: using LLM to identify and define all variables involved in the shortest path, and perform type declaration and initialization in a format that complies with SMT-LIB syntax;

[0075] S32: Based on the statement sequence corresponding to the shortest path, the LLM is driven to construct path constraints that describe the program execution conditions and convert them into standardized SMT-LIB code;

[0076] S33: Considering the external function calls contained in the shortest path, the LLM is driven to expand the function body logic, perform function inline expansion, and incorporate the function behavior into the overall path constraints to generate a complete SMT-LIB expression that can trigger the shortest path of the target branch.

[0077] In a preferred embodiment, step S4 specifically includes:

[0078] S41: Solve the SMT-LIB expression output by the LLM using the SMT solver to verify whether the SMT-LIB expression is satisfiable. If so, return the specific variable assignment in the SMT-LIB expression as test data, and generate a new test case using the test data.

[0079] S42: If not, return to step S3, use LLM to optimize the generated SMT-LIB expression, and re-use the SMT solver to solve the SMT-LIB expression output by LLM.

[0080] In a preferred embodiment, step S42 specifically includes:

[0081] S421: When an error occurs during the execution of an SMT-LIB expression in the SMT solver, the error information is returned to the LLM as feedback, allowing the LLM to optimize the generated SMT-LIB expression and fix format errors, undeclared variables, or logical conflicts in the SMT-LIB expression.

[0082] S422: Reuse the SMT solver to solve the SMT-LIB expression output by the LLM.

[0083] In a preferred embodiment, step S42 further includes:

[0084] S423: When the SMT runs normally in the SMT solver but the generated test case fails to reach the target branch, a structured input for guiding the LLM to optimize the SMT-LIB expression is generated based on the current SMT-LIB expression and path coverage information and the variable assignment result returned by the SMT solver;

[0085] S434: Utilize the LLM to optimize the generated SMT-LIB expression for the target branch that fails to be reached based on the structured input, and reuse the SMT solver to solve the SMT-LIB expression output by the LLM.

[0086] In a preferred embodiment, in step S423, based on the current SMT-LIB expression and path coverage information and the variable assignment results returned by the SMT solver, a structured input for guiding the LLM to optimize the SMT-LIB expression is generated, specifically including:

[0087] S4231: Based on the current path coverage information and the variable assignment results returned by the SMT solver, analyze the difference nodes between the current path coverage information and the untriggered target branches, and generate several first hint words for optimizing the SMT-LIB expression;

[0088] S4232: Based on the current SMT-LIB expression, all constraints in the current SMT-LIB expression are analyzed, a set of contradictory constraints in the SMT-LIB expression is extracted, and several second prompt words for the user to optimize the SMT-LIB expression are generated;

[0089] S4233: Based on several of the first prompt words and several of the second prompt words, generate a structured input for guiding LLM to optimize the SMT-LIB expression; wherein, the structured input is configured to drive the SMT-LIB expression to select a path that does not trigger the target branch at the difference node and correct the constraint contradictions in the previously generated SMT-LIB expression.

[0090] Therefore, the present invention proposes an automated testing method that is collaboratively assisted by LLM and SMT solvers, which effectively alleviates the problem of insufficient accuracy of large language models in complex path constraint reasoning. By having the language model be responsible for the extraction and expression of conditions at the semantic level, and then handing it over to the SMT solver for precise logical reasoning and solution, the system realizes the division of labor and cooperation between semantic understanding and logical solution, significantly reducing the computational and reasoning burden of the language model in path reachability analysis. This method not only improves the efficiency and accuracy of path coverage, but also has good adaptability and versatility, is applicable to a variety of languages ​​and complex program structures, and effectively improves the intelligence level and engineering practical value of automated testing.

[0091] In order to explain the present application more clearly, an application example of a test case generation method based on LLM and SMT solver of the present invention is provided below, which is as follows:

[0092] like Figure 2 As shown, the present invention proposes a test case generation method based on LLM and SMT solver, which aims to solve the problem that traditional search-based testing is difficult to effectively generate coverage paths when encountering complex logical branches. This method is driven by search-based testing (SBST). When coverage stagnation occurs, the execution path is semantically analyzed with the help of a large language model (LLM), and a logical expression describing the target path conditions is generated. Taking into account the limitations of LLM in logical reasoning, the system further introduces an SMT solver (SMT Solver) to verify and reason the logical expressions output by LLM, and finally generates feasible test inputs to improve test efficiency and coverage. The entire process forms a closed-loop optimization process that collaborates with semantic understanding and logical solving. It includes the following execution steps:

[0093] 1. Coverage monitoring and stagnation detection:

[0094] During the SBST test case generation process, the system collects and updates branch coverage information in real time after each round of test case execution, analyzing coverage growth trends using a sliding window approach. If coverage fails to improve significantly (below a preset threshold δ) over several consecutive generations of testing, or if a specific branch remains uncovered within a set number of iterations, a "coverage stagnation" determination is triggered.

[0095] 2. Generate paths and context for uncovered branches, and generate prompt information for use by the language model:

[0096] For uncovered target branches, the system performs static path analysis, combining the control flow graph (CFG) and a breadth-first strategy to search for the shortest path from the program start node to the currently uncovered branch. During path construction, the system captures the following information: the code snippet for the target branch; the code snippet for calling external functions, and so on. The system encapsulates this information as structured input, including the necessary SMT expression structure requirements, which serves as input for the subsequent large language model.

[0097] When the path contains uncontrollable environment variables or external API dependencies, the system can generate alternative path suggestions or introduce path instability indicators in the prompt word to improve the robustness of the prompt word.

[0098] 3.LLM outputs logical constraint expression and constraint solving:

[0099] The Large Language Model (LLM) generates logical expressions for describing path conditions based on path information and isostructured input in the form of SMT-LIB expressions (SMT-LIB code). The generation process consists of three steps. First, the LLM identifies and defines all variables involved in the path, and declares and initializes them in a format that conforms to the SMT-LIB syntax. Second, based on the extracted path information, the LLM constructs path constraints that describe the program execution conditions and converts them into standardized SMT-LIB code. Finally, for external function calls included in the path, the LLM needs to further expand its function body logic, perform inline expansion, and incorporate the processed function behavior into the overall path constraints to generate a complete SMT-LIB expression that can trigger the target branch.

[0100] It should be noted that when generating an SMT-LIB expression, the LLM is first used to identify and define all variables involved in the shortest path, and perform type declarations and initialization in a format that conforms to the SMT-LIB syntax; then, based on the statement sequence corresponding to the shortest path, the LLM is driven to construct path constraints that describe the program execution conditions and convert them into standardized SMT-LIB code; finally, considering the external function calls contained in the shortest path, the LLM is driven to expand the function body logic, perform function inline expansion, and incorporate the function behavior into the overall path constraints to generate a complete SMT-LIB expression that can trigger the shortest path of the target branch.

[0101] The system inputs the SMT-LIB code output by the LLM into the solver and performs a solve. The system first verifies whether the logical expression is satisfiable. If so, it returns specific variable assignments as test data and generates new test cases based on this data. If the solve fails, the system adjusts the prompts based on the run information and regenerates the SMT-LIB code. This is described in detail in Step 4.

[0102] 4. Generate test cases and complete closed-loop feedback:

[0103] SMT-LIB code generated by a large language model (LLM) can encounter two types of problems after being solved by an SMT solver: first, the code may fail to execute correctly in the SMT solver, resulting in syntactic or semantic errors; second, even if the solver successfully returns variable assignments, the test cases generated based on these results still fail to cover the target branches. To address these issues, this method designs a feedback-driven iterative optimization mechanism to continuously correct and optimize the generated SMT-LIB code.

[0104] The specific process is as follows: If the SMT-LIB code reports an error when running in the solver, the system returns the error information as feedback to the LLM, prompting it to fix format errors, undeclared variables or logical conflicts in the code, and regenerate SMT-LIB code that conforms to the semantic specifications. If the code can be solved normally but the generated test case fails to reach the target branch, the current path coverage information and the variable assignment results returned by the SMT solver are fed back to the LLM. The system also uses carefully designed prompts to guide the LLM to adjust the logical structure of the SMT-LIB code for the unreached branches to make it closer to the execution conditions of the target path.

[0105] It should be noted that when guiding the LLM to adjust the logical structure of the SMT-LIB code for unreached branches, the system first analyzes the difference nodes between the current path coverage information and the untriggered target branches based on the current path coverage information and the variable assignment results returned by the SMT solver, and generates several first prompt words for optimizing the SMT-LIB expression. Then, based on the current SMT-LIB expression, all constraints in the current SMT-LIB expression are analyzed, and the sets of contradictory constraints in the SMT-LIB expression are extracted, generating several second prompt words for user-optimized SMT-LIB expressions. Finally, based on the several first prompt words and the several second prompt words, a structured input is generated to guide the LLM to optimize the SMT-LIB expression. The structured input is configured to drive the SMT-LIB expression to select a path that does not trigger the target branch at the difference node and correct the constraint contradictions in the previously generated SMT-LIB expression.

[0106] Through the above feedback mechanism, the system forms a closed-loop process of "generation-solution-verification-feedback-regeneration", continuously iterating and optimizing the SMT-LIB code until the generated test cases can successfully cover the target branches or the set maximum number of iterations is reached.

[0107] Therefore, the present invention effectively combines a large language model (LLM) with an SMT solver to form a collaborative test case generation system that combines semantic analysis and logical reasoning. This system also employs a closed-loop feedback mechanism to monitor path coverage in real time and adjust the path condition generation strategy based on SMT solver feedback. Ultimately, this system improves software testing efficiency and coverage, alleviates the logical reasoning limitations of LMM-based test case generation, and enhances coverage of complex branching paths and test case effectiveness.

[0108] It should be understood that, in the description of this specification, reference to terms such as "one embodiment," "another embodiment," "other embodiments," or "first to Nth embodiments" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples.

[0109] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0110] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A test case generation method based on LLM and SMT solver, characterized in that: The method comprises the following steps: S1: When using SBST to generate test cases, the system performs coverage stagnation detection after each round of test case execution. S2: Based on the uncovered branches in the coverage stagnation detection results, static path analysis is performed. Combining CFG and breadth-first algorithm, the shortest path from the program start node to the uncovered branch is searched, capturing branch correlation information during the path construction process. S3: using the LLM, based on the structured input encapsulated by the path information of the shortest path and the branch association information, generating an SMT-LIB expression capable of triggering a branch corresponding to the shortest path; S4: Use the SMT solver to solve the SMT-LIB expression output by the LLM, and generate new test cases based on the solution results; Wherein, step S4 specifically includes: S41: Solve the SMT-LIB expression output by the LLM using the SMT solver to verify whether the SMT-LIB expression is satisfiable. If so, return the specific variable assignment in the SMT-LIB expression as test data, and generate a new test case using the test data. S42: If not, return to step S3, optimize the generated SMT-LIB expression using LLM, and use the SMT solver again to solve the SMT-LIB expression output by LLM; Wherein, step S42 specifically includes: S421: When an error occurs during the execution of an SMT-LIB expression in the SMT solver, the error information is returned to the LLM as feedback, allowing the LLM to optimize the generated SMT-LIB expression and fix format errors, undeclared variables, or logical conflicts in the SMT-LIB expression. S422: Reuse the SMT solver to solve the SMT-LIB expression output by LLM; S423: When the SMT runs normally in the SMT solver but the generated test case fails to reach the target branch, a structured input for guiding the LLM to optimize the SMT-LIB expression is generated based on the current SMT-LIB expression and path coverage information and the variable assignment result returned by the SMT solver; S434: Utilize the LLM to optimize the generated SMT-LIB expression for the target branch that fails to be reached based on the structured input, and reuse the SMT solver to solve the SMT-LIB expression output by the LLM.

2. The test case generation method based on LLM and SMT solver according to claim 1, characterized in that: Step S1 specifically includes: S11: In the process of generating test cases using SBST, the system collects and updates branch coverage information after each round of test case execution. S12: Analyze the coverage growth trend through a sliding window to determine whether a trigger condition is met. If so, trigger coverage stagnation detection.

3. The test case generation method based on LLM and SMT solver according to claim 2, characterized in that: In step S12, the triggering conditions specifically include: in several consecutive generations of testing, the improvement of coverage is less than a preset threshold, or the target branch has never been covered within a set number of iterations.

4. The test case generation method based on LLM and SMT solver according to claim 1, characterized in that: In step S2, the shortest path from the program start node to the uncovered branch is searched by combining CFG and the breadth-first algorithm, specifically including: S21: Perform lexical analysis, syntax analysis, and semantic analysis on the program source code to divide it into basic blocks, create nodes for each basic block, add directed edges based on the control transfer relationship, build a CFG, and map uncovered branches to edges in the CFG; S22: Add the starting point of the CFG to the queue and mark it as visited. Take out the head node of the queue and traverse all outgoing edges. Perform access judgment on the node pointed by each outgoing edge. If the node is not visited, add the node to the queue as the head node of the queue and re-execute the outgoing edge traversal and access judgment. S23: When it is detected that the queue head node belongs to the target uncovered branch, the search is terminated, the shortest path is generated by backtracking the predecessor chain and converted into a statement sequence in the program source code.

5. The test case generation method based on LLM and SMT solver according to claim 1, characterized in that: In step S2, the branch association information specifically includes: the code snippet of the target branch and the code snippet that calls the external function; wherein, when the shortest path contains uncontrollable environment variables or external function dependencies, the system generates an alternative path suggestion or introduces a path instability indicator in the prompt word.

6. The test case generation method based on LLM and SMT solver according to claim 1, characterized in that: Step S3 specifically includes: S31: using LLM to identify and define all variables involved in the shortest path, and perform type declaration and initialization in a format that complies with SMT-LIB syntax; S32: Based on the statement sequence corresponding to the shortest path, the LLM is driven to construct path constraints that describe the program execution conditions and convert them into standardized SMT-LIB code; S33: Considering the external function calls contained in the shortest path, the LLM is driven to expand the function body logic, perform function inline expansion, and incorporate the function behavior into the overall path constraints to generate a complete SMT-LIB expression that can trigger the shortest path of the target branch.

7. The test case generation method based on LLM and SMT solver according to claim 1, characterized in that: In step S423, based on the current SMT-LIB expression and path coverage information and the variable assignment results returned by the SMT solver, a structured input is generated to guide the LLM to optimize the SMT-LIB expression, specifically including: S4231: Based on the current path coverage information and the variable assignment results returned by the SMT solver, analyze the difference nodes between the current path coverage information and the untriggered target branches, and generate several first hint words for optimizing the SMT-LIB expression; S4232: Based on the current SMT-LIB expression, all constraints in the current SMT-LIB expression are analyzed, a set of contradictory constraints in the SMT-LIB expression is extracted, and several second prompt words for the user to optimize the SMT-LIB expression are generated; S4233: Based on several of the first prompt words and several of the second prompt words, generate a structured input for guiding LLM to optimize the SMT-LIB expression; wherein, the structured input is configured to drive the SMT-LIB expression to select a path that does not trigger the target branch at the difference node and correct the constraint contradictions in the previously generated SMT-LIB expression.

Images