Test case generation method and device and readable storage medium

Abstract

The invention discloses a test case generation method and device and a readable storage medium, and belongs to the technical field of automatic software testing. The method comprises the following steps: performing method call analysis on a to-be-tested program to generate a method call set; inputting the test case and a program code into a large language model to generate an initial test case; and evaluating the coverage rate, and if the saturation condition is not met, identifying an uncovered code line and feeding back the uncovered code line to the model to iteratively generate a new case until the coverage rate is saturated. According to the method, the high-coverage test case can be automatically and efficiently generated, and the method is particularly suitable for a scientific calculation program.

Description

Technical Field

[0001] This invention belongs to the field of automated software testing technology, specifically relating to a test case generation method, generation device, and readable storage medium based on method call analysis and large language models (such as ChatGPT). Background Technology

[0002] Software testing is a crucial step in ensuring software quality, and unit testing, as one of the core technologies in software testing, plays a vital role throughout the software development lifecycle. High-quality unit tests can effectively discover code defects and improve software reliability. However, traditional unit test writing typically relies on developers manually constructing test cases, which is not only time-consuming and labor-intensive but also prone to insufficient test coverage due to the influence of complex code logic. With the increasing complexity of software systems, automated unit testing technology has gradually become a research hotspot. Existing automated test case generation methods mainly include search-based test generation, constraint-solving-based test generation, and random strategy-based test generation. These methods improve testing efficiency to some extent, but still have limitations when handling complex program structures, cross-functional dependencies, and difficult-to-cover branches. For example, search-based software testing (SBST) typically relies on genetic algorithms or other heuristic search strategies to optimize test cases, but its test coverage is prone to getting stuck in local optima and struggles to cover complex logical branches. Constraint-based testing, while capable of analyzing program paths using techniques such as symbolic execution, suffers from high computational costs when handling high-dimensional data and complex mathematical calculations. Random testing, on the other hand, struggles to ensure sufficient coverage of generated test cases, especially for scientific computing programs, where the effectiveness of these methods is limited. In recent years, large language models (LLMs) have seen a rise in applications in code generation and automated testing. LLMs possess powerful code understanding capabilities and can generate logically sound test cases by combining contextual information, providing new ideas for automated testing. However, during research and practice, the inventors have discovered that existing LLM-based testing technologies still face the following pressing problems:

[0003] (1) Superficial understanding of program structure context: Existing solutions usually only take the program source code itself as input, lacking analysis of the complex function call relationships and data flow inside the code. This makes it difficult for LLM to generate test cases that can effectively trigger deep logic and cross-function interactions, especially for scientific computing programs with complex structures, where boundary conditions and abnormal branches are difficult to cover.

[0004] (2) Lack of a systematic iterative optimization mechanism: Existing technologies mostly rely on the single generation of LLM or simply make simple repeated requests, lacking a closed-loop optimization process based on coverage feedback. This makes it impossible for the test case set to improve itself after generation, and the test coverage is prone to bottlenecks, making it difficult to further improve through automation.

[0005] (3) Insufficient effectiveness for specific programming domains: For example, scientific computing programs typically involve high-precision mathematical operations and complex algorithmic logic, and existing automated testing tools (such as search-based Pynguin or randomized testing) struggle to generate effective test inputs that can reveal deep-seated defects. Furthermore, simple LLM generation also performs poorly in such domains due to the aforementioned two issues. Summary of the Invention

[0006] The purpose of this invention is to provide a test case generation method, generation device, and readable storage medium, which aims to solve at least one of the following technical problems existing in the prior art:

[0007] Insufficient test coverage: Traditional manual testing or existing automated testing tools (such as test generation methods based on search, constraint solving or randomization) are difficult to effectively cover complex program structures, especially boundary conditions, abnormal branches and cross-function call logic in scientific computing programs.

[0008] Low level of automation and intelligence: Existing methods rely too much on the manual design and experience of test engineers. When the code changes, a lot of manpower needs to be repeatedly invested in updating and maintaining test cases, which is inefficient.

[0009] Insufficient utilization of LLM: Existing test generation technologies based on large language models lack an understanding of the deep structure of the program (such as method call relationships) and lack an effective feedback optimization mechanism, resulting in weak targeting of generated test cases and difficulty in continuously improving coverage.

[0010] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0011] In a first aspect, embodiments of the present invention provide a test case generation method, the method comprising the following steps:

[0012] Step S1: Perform code preprocessing on the program under test. Identify all called methods and their calling order in the program under test through static code analysis, and generate a method call set.

[0013] Step S2: Input the program code and the method call set into the pre-trained large language model, and automatically generate initial test cases through the constructed prompt words;

[0014] Step S3: Execute the initial test cases and evaluate their code coverage to determine whether the code coverage has reached the preset saturation condition; if the saturation condition has not been reached, proceed to step S4.

[0015] Step S4: Identify the lines of code not covered by the initial test cases, input the information of the uncovered lines of code as feedback into the large language model to generate new test cases, and then return to step S3.

[0016] Secondly, embodiments of the present invention provide a test case generation apparatus, comprising:

[0017] processor;

[0018] Memory for storing program instructions executed by the processor;

[0019] When the program instructions are executed by the processor, the device implements the test case generation method.

[0020] Thirdly, embodiments of the present invention provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0021] Fourthly, embodiments of the present invention provide a step in which a program or instruction is stored on the readable storage medium, and the program or instruction is executed by a processor to implement the test case generation method.

[0022] Compared with the prior art, the advantages of this invention are as follows:

[0023] (1) Significantly improves the depth and breadth of test coverage: By providing key context for program structure through method call analysis, and combining iterative feedback mechanism to accurately target coverage blind spots, this method can systematically cover code branches that are difficult to reach by traditional methods and single LLM generators (such as boundary conditions, exception handling paths and complex function call chains in scientific computing programs). Experiments show that this method improves branch coverage from 50%-60% of traditional tools to over 85%-90% on multiple open-source scientific computing projects.

[0024] (2) Achieves a higher degree of automation and intelligence: This invention seamlessly integrates static analysis, LLM generation, and dynamic evaluation, forming a fully automated "generation-evaluation-optimization" closed loop. This greatly reduces the workload of test engineers in manually designing and maintaining test cases, and can quickly adapt to code changes. The test suite can be updated simply by re-executing this method, significantly reducing testing costs and improving software iteration efficiency.

[0025] (3) Highly targeted and efficient test cases: Based on precise guidance of method call relationships and specific uncovered lines of code, the test cases generated by LLM avoid blindness and redundancy. The generated test case set is smaller, but the coverage target is clearer, thereby shortening the test execution time and improving the running efficiency of the test suite.

[0026] (4) Possesses good versatility and scalability: The technical solution described in this invention does not depend on specific large language model products (such as ChatGPT) and can be applied to any pre-trained model with code understanding and generation capabilities. At the same time, the method is not only applicable to conventional software unit testing, but its combination of static analysis and dynamic feedback is particularly suitable for handling scientific computing programs involving complex mathematical models and large-scale data processing, demonstrating broad applicability. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a test case generation method provided by the present invention;

[0028] Figure 2 This is the iterative feedback flowchart provided by the present invention;

[0029] Figure 3 This is one of the hardware structure diagrams of the electronic device provided by the present invention;

[0030] Figure 4 This is the second schematic diagram of the hardware structure of the electronic device provided by the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The test case generation method provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0034] Please see Figure 1 The present invention provides a test case generation method, which mainly includes the following steps:

[0035] Step S101: Method call analysis, generating a set of method calls.

[0036] This step corresponds to step S1 in the claims. In specific implementation, the Python ast (abstract syntax tree) module is used to perform static analysis on the source code of the program under test. By defining a custom class that inherits from ast.NodeVisitor (such as MethodCollector in the embodiment), all function call nodes (ast.Call) in the code are traversed, and the names of all called methods are collected to form a method call set.

[0037] For example, the following is a specific Python code example:

[0038] Python

[0039] import ast

[0040] class MethodCollector(ast.NodeVisitor):

[0041] def __init__(self):

[0042] self.method_calls = set()

[0043] def visit_Call(self, node):

[0044] # Collect the names of the called functions

[0045] if isinstance(node.func, ast.Name):

[0046] self.method_calls.add(node.func.id)

[0047] # Handling object method calls (such as random.uniform)

[0048] elif isinstance(node.func, ast.Attribute):

[0049] self.method_calls.add(node.func.attr)

[0050] # Continue traversing child nodes

[0051] self.generic_visit(node)

[0052] def analyze_code(code_string):

[0053] tree = ast.parse(code_string)

[0054] collector = MethodCollector()

[0055] collector.visit(tree)

[0056] return list(collector.method_calls)

[0057] By executing the `analyze_code` function and passing in the source code string of the program under test, a set of method calls such as ['func', 'random', 'sqrt', 'uniform'] can be extracted. This set reveals the internal and external functions that the program under test depends on, providing crucial structured context for subsequent understanding of code interaction logic by a large language model.

[0058] Step S102: Generate initial test cases based on the large language model.

[0059] The large language models include, but are not limited to, pre-trained models with code understanding and generation capabilities such as ChatGPT, the GPT series, and BERT.

[0060] This step corresponds to step S2 in the claims. The program source code and method call set obtained in step S101 are input into a large language model such as ChatGPT through a structured prompt word template. The prompt word template aims to guide the model to generate high-quality, targeted test cases. A typical prompt word example is as follows:

[0061] text

[0062] You are a unit test generator for Python programs. Please use pytestto write unit tests for the given Python code that cover all branches. Thecomments on each line indicate line number. And the functions invocated by the above code are as follows \[ 'func', 'random', 'sqrt', ... \].

[0063] With this prompt, the model can fully understand the structure and dependencies of the code under test, thereby generating initial unit test cases that conform to the pytest framework specifications and are designed to cover all branch paths, including defining test functions, constructing input parameters, and asserting expected outputs.

[0064] Step S103: Coverage assessment and saturation judgment.

[0065] This step corresponds to step S3 in the claims. After generating the initial test cases, execute these test cases using the standard Python coverage evaluation tool Coverage.py and calculate code coverage. The specific commands are as follows:

[0066] text

[0067] coverage run -m pytest test_generated.py

[0068] coverage report

[0069] The "preset saturation condition" is crucial for determining whether the iterative optimization process should terminate. In a preferred embodiment, this condition is defined as follows: in three consecutive iterations, the improvement in code coverage (including statement coverage and branch coverage) is less than 1%. This threshold can be adjusted according to the stringency of the testing requirements, but this clear standard avoids premature termination or invalid iterations due to small fluctuations in coverage, ensuring the sufficiency and efficiency of the optimization process.

[0070] Step S104: Iterative feedback and test case optimization.

[0071] This step corresponds to step S4 in the claims. Using the analysis capabilities of tools such as Coverage.py, the code line numbers not covered by the current test case set are accurately identified. This information on uncovered code lines is used as core feedback to construct new prompt words, which are then input into ChatGPT to drive targeted optimization.

[0072] Examples of prompts used during the iteration phase are as follows:

[0073] text

[0074] The above test cases only give 86% coverage. The following lines from the code file are not being covered given by the line numbers - 22, 24, 26,28, 36-43, 45-49, 133. Please use pytest to provide more test cases to coverthese uncovered. You branches do not need to output the previous code.

[0075] This warning explicitly identifies a coverage vulnerability and guides the model to generate new, supplementary test cases designed to cover these specific lines of code. The newly generated test cases will then be added to the existing test suite.

[0076] Iterative Loop: After executing step S104, the process returns to step S103 to re-evaluate the optimized overall test coverage and perform saturation judgment again. This loop continues, forming a closed-loop optimization process of "generation-evaluation-feedback" until the test coverage no longer significantly improves (reaching saturation). A flowchart of this process can be found in the appendix. Figure 2 As shown.

[0077] In the iterative feedback phase of step S104, the prompts provided to the large language model, in addition to explicitly indicating the uncovered line numbers, can further include guiding information to improve the generation quality. For example:

[0078] text

[0079] The existing test cases do not provide complete code coverage, failing to cover the following critical lines: [22, 24, 36-43]. These lines involve boundary condition checks in the `calculate_critical_value` function. Please focus on these uncovered code areas and generate new pytest test cases. Special note:

[0080] 1. Design test inputs for boundary parameters (such as inputs of 0, maximum, and minimum values).

[0081] 2. Verify the function's expected output or exception-throwing behavior under boundary conditions.

[0082] All you need to do is output the code for the newly added test cases.

[0083] This structured feedback not only points out "where there is no coverage", but also guides the model on "how to cover" through contextual hints, thereby generating more targeted and efficient test cases.

[0084] This invention provides a test case generation device, comprising:

[0085] processor;

[0086] Memory for storing program instructions executed by the processor;

[0087] When the program instructions are executed by the processor, the device implements the test case generation method.

[0088] like Figure 3 As shown, this embodiment of the invention also provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described test case generation method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0089] It should be noted that the first electronic device in the embodiments of the present invention includes the mobile electronic device and the non-mobile electronic device described above.

[0090] Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present invention.

[0091] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0092] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0093] It should be understood that, in this embodiment of the invention, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understandable that the aforementioned modem processor may not be integrated into the processor 710.

[0094] This invention also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described test case generation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0095] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0096] This invention also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described test case generation method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0097] It should be understood that the chip mentioned in the embodiments of the present invention may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0100] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A test case generation method, characterized in that, Includes the following steps: Step S1: Perform code preprocessing on the program under test. Identify all called methods and their calling order in the program under test through static code analysis, and generate a method call set. Step S2: Input the program code and the method call set into the pre-trained large language model, and automatically generate initial test cases through the constructed prompt words; Step S3: Execute the initial test cases and evaluate their code coverage to determine whether the code coverage has reached the preset saturation condition; If the saturation condition is not met, proceed to step S4; Step S4: Identify the lines of code not covered by the initial test cases, input the information of the uncovered lines of code as feedback into the large language model to generate new test cases, and then return to step S3.

2. The test case generation method according to claim 1, characterized in that, In step S1, static code analysis includes: using Python's Abstract Syntax Tree module to parse the program under test, traversing the function call nodes therein, in order to extract the set of method calls.

3. The test case generation method according to claim 1, characterized in that, In step S2, the prompt words include at least the program code, the set of method calls, and instructions for guiding the generation of test cases that cover all branch paths.

4. The test case generation method according to claim 1, characterized in that, In step S3, the code coverage is calculated using a coverage analysis tool; The saturation condition is that in N consecutive iterations, the increase in code coverage is less than a threshold K, where N is an integer greater than or equal to 2 and K is a percentage greater than zero.

5. The test case generation method according to claim 4, characterized in that, N=3, K=1%.

6. The test case generation method according to claim 1, characterized in that, In step S4, the information of the uncovered code line includes the uncovered code line number.

7. The test case generation method according to claim 1, characterized in that, The large language model is a generative pre-trained model based on the Transformer architecture.

8. A test case generation device, characterized in that, include: processor; Memory for storing program instructions executed by the processor; When the program instructions are executed by the processor, the apparatus causes the device to implement the test case generation method as described in any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the test case generation method as described in any one of claims 1 to 7.

Images