Conversation-driven regression test generation method based on code big language model
Through a dialogue-driven method based on the code large language model and combined with static program analysis, high-quality C language regression test cases are generated, which solves the problems of inefficiency and insufficient coverage in the existing technology, and achieves more efficient and targeted test case generation.
Patent Information
- Application Number
- CN202410975322.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-29
AI Technical Summary
When generating regression test cases for C language programs, the prior art is inefficient, insufficient coverage, and it is difficult to fully capture the deep semantics and boundary conditions of the code, resulting in insufficient comprehensive and effective testing.
A dialogue-driven method based on a code-large language model is adopted, combined with static program analysis, high-quality regression test cases are generated through natural language processing and code generation capabilities, and test cases are optimized with the model interaction to enhance the expressiveness of the dialogue model.
Significantly improve the coverage and effectiveness of test cases, reduce the workload of manual writing, improve test efficiency, and the generated test cases are closer to the actual code execution environment, have higher targeting and accuracy, and reduce maintenance costs. .
Smart Images

Figure CN119046134A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer technology and relates to a dialogue-driven regression test generation method based on a large code language model. Background Art
[0002] Generating high-quality regression test cases is the key to ensuring the functional integrity and stability of the system after software changes. However, traditional methods usually require manual writing of test cases, which often relies on manual experience, is not only inefficient, but also prone to missing key situations. As the system changes and updates, manually written test cases need to be constantly updated and maintained, increasing the cost and workload of testing. At the same time, since arrays usually involve complex data dependencies and access patterns, commonly used test case automatic generation methods such as symbolic execution and fuzz testing are difficult to generate effective test cases covering all possible boundary conditions and abnormal situations. Moreover, for test scenarios that require precise control of array generation and access, the randomness of fuzz testing may not provide sufficient accuracy, and symbolic execution may have difficulty generating effective test cases due to the complexity of path constraints.
[0003] Regression testing is an important part of software testing. It ensures that the software can still run correctly after changes and no new defects are introduced. Regression testing is especially important for C language programs. C language is a low-level programming language with relatively simple syntax and semantics, but it is prone to subtle errors, such as pointer operations. The memory model of C language is relatively simple and does not provide rich memory management functions like other modern programming languages. This makes it more difficult to test memory-related issues, such as memory leaks, allocation errors, etc. C language is widely used in system programming, embedded development and other fields, and these scenarios have very high requirements for program robustness and performance. Through regression testing, the behavior of the program in different scenarios can be verified to ensure its compatibility and stability on various operating systems and hardware platforms. Programs developed in C language are usually more complex, especially large projects. Regression testing can help discover potential logical errors, boundary conditions and performance bottlenecks, so as to fix these problems in advance and reduce the cost of later maintenance and debugging. During the development process, it is often necessary to modify or refactor the code to adapt to new requirements or improve existing functions. A good regression test suite can ensure that the modified code does not break existing functions and maintain overall stability. In modern software development, continuous integration and deployment (CI / CD) has become a standard practice. Regression testing can be run before each code submission or deployment to detect potential problems in a timely manner and ensure software quality and reliability.
[0004] Traditionally, regression test cases are usually written manually by developers, which is time-consuming and error-prone. With the development of machine learning and artificial intelligence technologies, especially natural language processing (NLP) and code generation technologies, researchers have begun to explore the use of code large language models to generate regression test cases. Code large language models are advanced natural language processing models that can understand and generate code. These models are trained with a large amount of code data to learn the structure and semantics of the code, so that they can generate corresponding code snippets when given a certain description or condition. With the development of code large language models, researchers can use their powerful code understanding and generation capabilities to guide them to generate high-quality regression test cases for C language programs. These models can generate test input data with good coverage and effectiveness based on given descriptions or conditions, thereby helping to improve the comprehensiveness and effectiveness of tests.
[0005] Static program analysis technology can analyze code paths and data flows, identify whether test cases cover all possible execution paths, and detect whether test cases can effectively trigger abnormal conditions, boundary conditions or specific security vulnerabilities, thereby improving the comprehensiveness and depth of testing. In addition, static program analysis technology can evaluate the impact of test cases on code quality. By analyzing the complexity, structure, and dependencies of the code, the writing of test cases can be optimized to ensure that the tests cover key parts and improve the effectiveness and efficiency of the tests. The reports generated by static analysis tools can help large language models understand the quality issues of test cases and guide large models to make adjustments to test strategies to better ensure the stability and reliability of the software.
[0006] In software development, code often needs to be updated and modified. By generating test cases through language models, new test cases can be quickly generated to cover the changed parts of the code. This ensures that after each code update, the test coverage remains high and effective, reducing potential errors introduced by changes. The language model can understand the test requirements described in natural language and convert them into specific code test cases. This approach not only increases the test coverage, but also helps to generate more comprehensive and effective test cases to capture more potential errors and boundary conditions. In the development mode of continuous integration (CI) and continuous deployment (CD), frequent code submission and deployment require fast and effective testing. The test cases generated by the language model can be integrated with the automated testing framework, automatically executed and provide timely feedback, which helps to maintain the efficiency and stability of the development process. The use of advanced natural language processing and code generation technologies, such as language models, not only improves the level of test automation, but also brings new innovations and possibilities to test methods and tools. This innovation-driven approach can continuously optimize the test suite to adapt to changes and needs in the software development process.
[0007] At present, there are mainly the following test case generation methods for C language:
[0008] Manually designed by testers:
[0009] Automatic generation based on genetic algorithms, symbolic execution or fuzz testing:
[0010] Generate test cases using the code model:
[0011] Among them, the manual design method has certain limitations in terms of efficiency, coverage and continuous integration. In agile development and continuous integration environments, fast feedback and frequent test execution are required. Manual tests often cannot meet these requirements because they usually require a long execution cycle. Therefore, current test case generation methods for C language are generated through automated or semi-automated methods.
[0012] Test case automatic generation method based on symbolic execution:
[0013] Symbolic execution is a static program analysis technique that searches for the execution path of a program under different input conditions by symbolically calculating the program's symbolic variables instead of specific input values. In symbolic execution, program variables are treated as unknown symbols rather than specific numerical values, so multiple execution paths and conditional branches of the program can be searched. This method enables symbolic execution to automatically discover potential errors in the program, verify program features, and generate test cases. It is particularly suitable for program analysis with complex conditions and a large number of execution paths. The symbolic execution tool KLEE is a popular open source tool for automatically performing symbolic execution of programs to search for possible paths and input conditions of the program.
[0014] Automatic test case generation method based on fuzz testing:
[0015] Fuzz testing is a testing technique that discovers potential vulnerabilities and abnormal behaviors of software systems by generating a large number of random or semi-random input data. The core is to use automated tools (called fuzzers) to generate a variety of possible inputs, including edge cases and uncommon data sets, to test the robustness and security of the program. As a black-box testing method, fuzz testing focuses on the behavior of input and output without paying attention to internal implementation details. Through intelligent fuzz testing techniques, such as coverage-based and status monitoring, the efficiency and accuracy of vulnerability discovery can be improved, thereby helping developers improve and optimize software and enhance its reliability and security. AFL (American Fuzzy Lop) is a popular fuzz testing-based tool that is used to automatically generate test cases to discover vulnerabilities and security issues in software programs, and is used for general testing of C programs.
[0016] Automatic test case generation method based on genetic algorithm:
[0017] Genetic algorithm is a convergent search and optimization technology that simulates the genetic mechanism in the process of biological evolution. In the scenario of generating test cases, the genetic algorithm randomly generates test cases for the initial population and uses crossover, mutation and other operations to perform genetic operations in the population to gradually optimize the quality and coverage of the test cases. The fitness function is used to evaluate the effect of each test case. The algorithm selects and retains individuals with high fitness and discards individuals with poor performance. Genetic algorithms have advantages in generating complex test scenarios and optimizing coverage, which can help improve the efficiency and comprehensiveness of software testing, especially in scenarios that need to handle a large number of possible situations. It shows its powerful search ability and automatic optimization ability. EvoSuite is a well-known test case generation tool based on genetic algorithms, focusing on generating unit test cases with high coverage. It combines symbolic execution and genetic algorithms to optimize the test case generation process. However, EvoSuite is a tool mainly used to automatically generate test cases for Java programs, and needs to be used for C programs through some configuration and plug-ins.
[0018] Automatic test case generation method based on random testing
[0019] Random testing is a software testing method that generates random inputs to explore the execution path of a program to discover potential software defects or errors. This testing method does not rely on the specific internal logic of the program, but attempts to cover as many situations as possible in the input space, including boundary conditions and abnormal situations. Random testing is usually used for preliminary and quick checks of programs, especially for large programs or when testing needs are urgent.
[0020] Disadvantages of existing technology:
[0021] Disadvantages of the automatic test case generation method based on symbolic execution:
[0022] Symbolic execution may cause path explosion problems, especially when dealing with complex programs. Because it tries to explore all possible paths, this may lead to a sharp increase in execution time and memory consumption, even beyond acceptable ranges. KLEE's configuration and use may be complex for users. It is necessary to understand how symbolic execution works and how to set symbolic variables and constraints, which may have a certain learning curve for non-experts. For programs involving dynamic memory allocation (such as malloc, free), KLEE's symbolic execution capabilities may be limited. In these cases, additional manual annotations or special processing are required for symbolic execution. Symbolic execution relies on path constraints to generate test cases, but nonlinear constraints in array operations may cause symbolic solvers to be unable to effectively handle or solve these constraints.
[0023] Disadvantages of automatic test case generation method based on fuzz testing:
[0024] AFL has limited test coverage for complex programs, especially programs involving multiple conditions and complex control flows, which make it difficult to fully explore all possible execution paths. AFL may not perform well when dealing with highly optimized code because it inserts test code based on instrumentation technology, and the optimized code structure may affect the instrumentation effect, thereby reducing the test effect. For discovering deep vulnerabilities or specific security issues, AFL may not be as effective as some tools designed specifically for security testing because its main design purpose is to discover basic vulnerabilities in programs through fuzz testing.
[0025] Disadvantages of the automatic test case generation method based on genetic algorithm:
[0026] The selection of the initial test case set has a significant impact on the quality of the final generated test cases. Improper selection may lead to falling into a local optimal solution. Although the algorithm is highly automated, developers are usually required to manually adjust the generated test cases to improve coverage and effectiveness. In addition, EvoSuite is a tool mainly used to automatically generate test cases for Java programs, and requires some configuration and plug-ins for C programs. Due to the large differences between C and Java in data types, algorithm implementation, etc., the genetic algorithm needs to make corresponding adjustments to the characteristics of the C language when generating test cases.
[0027] Disadvantages of the automatic test case generation method based on random testing:
[0028] Due to the random nature, the generated test cases may not have sufficient quality and effectiveness to fully test all aspects of the software. Randomly generated test cases usually cannot effectively cover all possible paths and boundary conditions of the software, which may lead to failure to discover critical errors. Due to the lack of structure and goal orientation, random testing is difficult to focus on specific functions or critical paths, thus limiting the depth and breadth of the test. At the same time, the lack of understanding of the semantics and context of the code leads to a mismatch between the test cases and the actual usage scenarios. Randoop, a commonly used open source tool for random testing, mainly focuses on the Java language, and there are certain restrictions on the use of the C language. Summary of the invention
[0029] In view of the problems in the prior art, the present invention provides a dialogue-driven regression test generation method based on a large code language model.
[0030] A method for generating a dialogue-driven regression test based on a large code language model includes the following steps: training a large language model for a programming language or a code generation task; using a dialogue interaction method to drive the generation of test cases; and performing regression testing.
[0031] The present invention utilizes a large code language model to perform static program analysis in a human-in-loop semi-automatic manner to generate high-quality regression test cases for C language programs. Regression testing is a testing method in software testing, which aims to verify that after a software system is modified or updated, existing functions still operate normally and no new errors or problems are introduced.
[0032] The natural language processing and code understanding capabilities of the large language model enable it to extract key information from the natural language description of the test requirements and the tested functions to generate test cases. Static analysis tools then evaluate and analyze the test case effects, manually map the evaluation results to the corresponding natural language descriptions and feed them back to the large language model, guiding the large language model to optimize the test cases in a targeted manner. In this process, LangChain can be used to retain historical conversations with the large model, thereby enhancing the performance of the conversation model. By learning the corpus in historical conversations, the model can better understand the test intent, thereby generating targeted test cases with high coverage to ensure the effectiveness and comprehensiveness of the test. This method will significantly reduce the workload of manually writing test cases and improve test efficiency and coverage.
[0033] The advantages of the present invention are:
[0034] Generating test cases using large code models has several significant advantages over symbolic execution, fuzz testing, genetic algorithms, and random testing:
[0035] 1. Capability of context information collection: The code big model can better collect context information, so the generated test cases are closer to the actual code execution environment, avoiding some problems such as unrealistic paths or excessive randomness in symbolic execution and random testing;
[0036] 2. Natural language information fusion: Large models can integrate natural language information outside of code (such as developer descriptions or defect reports), which helps generate more targeted and realistic test cases. Symbolic execution and traditional automation methods usually lack this contextual awareness capability.
[0037] 3. Code semantic understanding: The big model can deeply understand the semantics of the code, which helps to generate more effective test cases, while traditional methods may not be able to capture the deep semantic relationship behind the code;
[0038] 4. With the help of static analysis tools, we can make full use of the relevant branch coverage information achieved by test cases, greatly improving the generation efficiency and quality of array type test cases, which is a problem that other testing methods are difficult to solve. At the same time, static analysis tools can deeply analyze the code structure and logic, identify potential boundary conditions and abnormal situations, and thus help generate more comprehensive and targeted array type test cases.
[0039] 5. The entity recognition technology of natural language processing can be used to extract test case identifiers, test parameter requirements and other information in the test requirement document.
[0040] 6. Improved test coverage: By better understanding the code context and semantics, the test cases generated by the large model can more comprehensively cover the various logical branches of the code and improve test coverage.
[0041] 7. Improved testing efficiency: Since large models can generate more targeted test cases, the number of invalid test cases is reduced, thereby speeding up the testing process and improving testing efficiency.
[0042] 8. Reduced maintenance costs: Automatically generated test cases are closer to actual scenarios, reducing the time and cost of manually writing and maintaining test cases.
[0043] 9. Greater adaptability: Large models can handle multiple programming languages and frameworks, and by learning different coding styles and patterns, the generated test cases are more adaptable.
[0044] 10. Due to the ability to combine code semantics and contextual information, the test cases generated by the large model are superior to traditional methods in terms of accuracy and reduce false positives and negatives.
[0045] 11. In particular, for array-related test case generation, the code big model can further exert its capabilities with the help of static analysis tools. For example, it can generate arrays that better meet the requirements and data structures containing different types and values based on the feedback information of the static analysis tool. In addition, the big model can also generate boundary value test cases for arrays, such as testing array out-of-bounds, subscript out-of-bounds and other potential problems. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. As shown in the figure:
[0047] Figure 1Generate high-quality regression test case flow charts for the large code model of the present invention. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] Embodiment 1: A method for generating a regression test based on a large code language model driven by a dialogue. Since the writing and maintenance of test cases require a lot of manpower, this will not only prolong the test cycle, delay the release time of the software, but also increase the development and maintenance costs. The existing manual testing method is not only inefficient, but also prone to errors. How to improve the test efficiency and shorten the test cycle through automated means is the key to improving the effectiveness of software regression testing. The core problem to be solved by the present invention is how to use the large code model in combination with static analysis tools to generate high-quality regression test cases through dialogue interaction. At present, the large code model has shown certain advantages of its own. For example, it can better capture contextual information, can integrate natural language information outside the code (such as developer descriptions or defect reports, etc.) to help generate test cases, and can better understand the semantics of the code. Using the natural language processing and code generation capabilities of the large model, test cases with high coverage and pertinence can be generated to ensure the effectiveness and comprehensiveness of the test. In particular, for array-related test case generation, the large code model can further exert its capabilities with the help of static analysis tools. For example, it can generate arrays that meet the requirements more according to the feedback information of the static analysis tool, as well as data structures containing different types and values. In addition, the large model can also generate boundary value test cases for arrays, such as testing array out-of-bounds, subscript out-of-bounds and other potential problems. At the same time, by comparing the newly generated test cases with the existing test cases, the large model can identify redundant test cases. These redundant test cases usually refer to those with low test coverage or repeated with other test cases. By deleting these redundant test cases, the test set can be significantly streamlined, and those efficient and effective test cases can be retained, thereby significantly improving the utilization efficiency of test resources.
[0050] A dialogue-driven regression test generation method based on a code large language model includes: an automatic test case generation and evaluation method for C language programs, a method for correcting and optimizing generated test cases by utilizing feedback information from static analysis tools to dialogue with the code large model, a customized prompt template construction method for C language program types, and a test case generation method for natural language description of test requirements.
[0051] The present invention solves the problem of searching for high-quality test cases in a huge test case space, and in particular, achieves the goal of generating higher-quality regression test cases for arrays. By combining human intelligence with the generation capability of a large code model, the accuracy, coverage, and effectiveness of test cases can be effectively improved, thereby accelerating the software development process and improving the quality assurance level of the software.
[0052] Example 2: Figure 1 As shown, a dialogue-driven regression test generation method based on a large code language model includes the following steps:
[0053] Step 1: Collect and prepare a data set for C language programs, including source code, corresponding natural language description of test requirements, test case data constraints, and official test case examples;
[0054] Using public data before the release of a large model is prone to data leakage. That is, if the data from the pre-training phase of a large language model is used to perform the current task, it may lead to overfitting problems. Because the model has been exposed to the same data content in the pre-training phase, it may remember part of the data instead of truly learning the general rules, thus affecting the performance in the current task. For example, an overfitted model may remember certain specific cases during the training phase instead of truly understanding the rules behind them. This causes the model to only match specific patterns in the training data during testing, and not generalize well to new data. Overfitting may make the model perform very accurately on a specific data set, but this does not mean that it truly understands the nature of the problem or can operate effectively in a wider range of situations. In view of the above reasons, data before the release date of the corresponding large language model should be collected.
[0055] Step 2: Determine the appropriate large language model;
[0056] The differences between large language models are mainly reflected in the following aspects:
[0057] 1. Size: The size of the language model is an important distinction, including the number of parameters, number of layers, size of training data, etc. Generally speaking, larger language models usually have better performance.
[0058] 2. Training data: Different large language models may use different training data sets, and the source and quality of these data sets will affect the performance of the model. Some large language models may be trained on data in a specific field, so that they can perform better in that field.
[0059] 3. Architecture design: The architecture design of large language models may also be different, such as different attention mechanisms, network structures or activation functions. These design choices will directly affect the learning ability and performance of the model.
[0060] 4. Training technology: There may be differences in the training technology of large language models, such as differences in training algorithms, optimizer selection, learning rate scheduling, etc. These technical choices will affect the convergence speed and performance of the model.
[0061] In general, the differences between large language models mainly lie in the size, training data, architecture design, and training technology, which together affect the performance of the model. Therefore, it is necessary to select the large language model with the best performance within the scope of capabilities based on the actual situation.
[0062] When selecting the large language model with the best performance, follow the following process:
[0063] Step 2.1. Select Model: Identify the large list of language models to be evaluated.
[0064] Step 2.2, sampling data set: use random sampling and other methods to sample the evaluation data in the data set.
[0065] Step 2.3, Evaluation Criteria: Determine the criteria for the evaluation model, including:
[0066] Formatting of generated tests: Ensure that the output format meets the expected standards.
[0067] Test case effectiveness: Manual inspection of the generated test cases is effective in detecting issues.
[0068] Cost of running large models: The computational resources and time consumed when evaluating the model's performance.
[0069] Step 2.4, calculate the overall score: Use the pre-defined criteria and use the following quantitative formula to calculate the score of each model: performance score = w1×format compliance + w2×test case effectiveness-w3×cost, w1, w2, w3 are weights reflecting the importance of the criteria, and w1+w2+w3=1.
[0070] Step 2.5, select the best model: select the language model that performs best on the sample set based on the comprehensive score.
[0071] Step 3: Select the function to be tested from the data set and start generating test cases.
[0072] Step 4: Design prompt templates for different types of C language programs to guide the model to generate test cases in a specific type of standardized format.
[0073] When designing prompt templates for different types of C language programs, you can follow the following process:
[0074] Step 4.1, Requirements Analysis: Analyze the operations that different types of C language programs may involve and the information formats that need to be input.
[0075] Step 4.2, Functional design: Design a general prompt template, considering including basic command prompts, parameter input type instances, error prompt words, etc.
[0076] Step 4.3, optimize the template: Test the designed prompt template, make the expression as unambiguous as possible, and guide the large language model to answer in line with expectations.
[0077] Through the above process, prompt templates suitable for different types of C language programs can be designed to improve the accuracy of large language model generation tests.
[0078] Step 5: Use the large language model to perform semantic analysis on the test requirements and feed back the analysis results to itself to deepen the large language model's understanding of the test requirements.
[0079] Specifically, the process of semantic analysis of test requirements by the large language model is as follows:
[0080] Step 5.1: Semantic analysis requirements: Use a large language model to perform semantic analysis in order to understand the nature of the test requirements and the complexity involved. The model needs to be able to identify errors and boundary conditions that may occur in different types of C language programs, such as memory leaks, null pointer references, etc.
[0081] Step 5.2, Analysis Method: When designing test cases, the model can analyze possible logical errors in the code, such as logical errors in loops or conditional judgment errors. When testing different types of C language programs, the model needs to understand the specific functions and implementation details of these programs, such as the correctness and efficiency of the algorithm.
[0082] Step 5.3, Result Feedback: Feedback the results of semantic analysis to the model itself to help it further deepen its understanding and identification of test requirements; the feedback information can include identified common programming errors, optimization suggestions, and improvement suggestions for test case design, which can help large models generate higher quality test cases.
[0083] Step 5.4: Deepen understanding: The model learns more about the context and semantics of the test requirements by analyzing the feedback results. The model can use the feedback to improve its accuracy and comprehensiveness of various test requirements, thereby better supporting the needs of users when designing and executing tests.
[0084] Step 6: The large language model generates a preliminary test case based on the dialogue information.
[0085] Step 7: Check whether the Oracle (expected output) type of the generated test case matches the function under test. If it does not match, give relevant information feedback to guide the large model to correct the test case data type.
[0086] Step 8: Check whether the test input parameters of the generated test case correspond one-to-one with the parameters of the function being tested. If non-function parameters or missing parameters appear, relevant information feedback is given to guide the large model to correct the test case data type.
[0087] If you want to detect whether the expected output of the generated test case and the corresponding test input are in line with expectations, you can first use a static analysis tool to parse the C program code under test into an Abstract Syntax Tree (AST). This tree structure can more clearly represent the grammatical structure of the code. In AST, each node represents a grammatical element in the code, such as an expression, statement, function definition, etc. By analyzing AST, you can extract various information in the code, including variable names, function names, control flow structures, etc. Then use the parameter type, return value type and other information obtained with the help of static analysis tools to detect the legality of the test cases generated by the large language model.
[0088] Step 9: Test the generated test cases and generate a test case quality assessment report. If the test cases meet the requirements, save the test cases to the corresponding location and go to step 3; otherwise, go to step 10.
[0089] Specifically, the generation of the use case quality assessment report is achieved through gcov and lcov, and the process is as follows:
[0090] Step 9.1. Compile the program and call coverage detection: Include the code coverage detection options -fprofile-arcs and -ftest-coverage in the gcc compilation command, and compile the main function file and the tested function file together. This will generate an executable file and generate .gcno and .gcda files of the tested function in the corresponding directory for coverage data collection.
[0091] Step 9.2, collect coverage data: Use lcov to collect coverage data and output it to the coverage.info file, which contains the overall code coverage data.
[0092] Step 9.3, extract the coverage data of a specific file: Use lcov to extract the coverage data of the tested function file from the coverage.info file to a temporary file (for example, test1.info).
[0093] Step 9.4: Repeat the above steps until all test cases are executed and the coverage information of all test cases is obtained.
[0094] Step 9.5, merge coverage data: Use lcov to merge the testX.info files containing coverage information of multiple test cases into a final coverage.info.
[0095] Step 10: Use static analysis tools to analyze the tested function code that has been set with corresponding test cases as input, detect the coverage of the code by the test cases, and extract important information based on the generated test case quality assessment report, including uncovered branches, and feed these more targeted feedback information back to the large language model to optimize the test cases.
[0096] When the line coverage information and conditional branch coverage information of the current test case set are obtained, if they do not meet expectations, such as the line coverage should exceed 90%, use static analysis tools to analyze the coverage of the tested function code by the set test cases, and use the feedback information from the analysis report to optimize the test cases.
[0097] The following is a detailed description of the steps:
[0098] Step 10.1. Select static analysis tool:
[0099] First, choose a static analysis tool that is suitable for your project, and make sure it supports the analysis of C language code and can detect the coverage of test cases, such as Cppcheck, Clang Static Analyzer, etc.
[0100] Step 10.2: Prepare the C language code to be tested and configure the static analysis tool:
[0101] Prepare the test cases to be tested and the C language code of the tested function and configure the selected static analysis tool into the development environment. Set the tool parameters and paths so that it can access and analyze the source code files containing the tested function.
[0102] Step 10.3: Run the static analysis tool.
[0103] Invoke static analysis tools to analyze the code of the function under test. The tool will automatically detect which parts of the code are covered by the test cases and which parts of the code are not covered and record them.
[0104] Step 10.4, generate analysis report:
[0105] Static analysis tools generate detailed reports about the current test case coverage information, including the coverage of each branch or path of the tested function. The report will show which code paths are covered by the test cases, which conditional branches or code paths under specific inputs are not covered.
[0106] Step 10.5: Extract information to optimize test cases.
[0107] Analyze the reports generated by static analysis tools, especially important information such as uncovered code paths, to obtain more targeted feedback information. Combine the feedback information with the corresponding prompt template to guide the optimization test cases of large language models.
[0108] Step 11. Repeat the above steps until the use case quality assessment report is generated and the task is completed.
[0109] Embodiment 3: A method for generating a dialog-driven regression test based on a large code language model, comprising the following steps: Algorithm 1 uses a large language model to generate a test case workflow for a C program:
[0110] Step 1: Collect and prepare a set of costumes.
[0111] Step 2: Collect C source code, natural language description of corresponding test requirements, numerical constraints of test cases, and official test case examples.
[0112] Step 3. Select a suitable large language model.
[0113] Step 4. Select a suitable large language model (e.g., Code Llama).
[0114] Step 5: Logarithmically play each function in the set.
[0115] Step 6: traverse the tested function f in the data set and execute it.
[0116] Step 7: Generate test cases for function f.
[0117] Step 8. Design a prompt template.
[0118] Step 9, prompt__template = design prompt template (f).
[0119] Step 10: semantic analysis.
[0120] Step 11, semantic_analysis = model, semantic analysis (test requirements, f).
[0121] Step 12, model.feedback (semantic_analysis).
[0122] Step 13: Generate initial test cases.
[0123] Step 14, initial_test_cases = model.Generate test cases (prompt_template).
[0124] Step 15. Check the Oracle type.
[0125] Step 16, oracle__check = check Oracle type (initial_test_cases, f).
[0126] Step 17: If the conditional judgment test fails (Oracle) type, it is executed.
[0127] Step 18, model.feedback("Oracle type does not match").
[0128] Step 19, test__cases = model, generate test cases (prompt_template).
[0129] Step 20: End condition judgment.
[0130] Step 21: Check the parameters.
[0131] Step 22, param_check = check test input parameters (test_cases, f).
[0132] Step 23: Execute if the conditional judgment test input parameter fails.
[0133] Step 24, model.feedback("parameter mismatch").
[0134] Step 25, test_cases = model.Generate test cases (prompt__template).
[0135] Step 26: End condition judgment.
[0136] Step 27. Evaluate the test cases.
[0137] Step 28, quality_report = evaluate test cases (initial_test_cases, f).
[0138] Step 29: Execute if the conditional judgment test case passes the evaluation.
[0139] Step 30, save the test case (test_cases, storage location).
[0140] Step 31, the rule is executed.
[0141] Step 32, static analysis.
[0142] Step 33, use a static analysis tool to analyze the function under test, and use the generated test cases as input to check the code coverage.
[0143] Step 34, static_analysis_report = static analysis(f, test_cases).
[0144] Step 35, feedback_info = extract important information(static_analysis_report).
[0145] Step 36, model.feedback(feedback_info).
[0146] Step 37, the conditional judgment ends.
[0147] Step 38, the traversal ends;
[0148] Step 39, the algorithm ends.
[0149] Example 4: As Figure 1 shown, a dialogue-driven regression test generation method based on a code large language model has the following steps:
[0150] Step 1, the algorithm starts.
[0151] Step 2, collect and prepare the data set.
[0152] Step 3, select a suitable large language model.
[0153] Step 4, for each function in the data set.
[0154] Step 5, generate test cases for function f.
[0155] Step 6, design a prompt template.
[0156] Step 7, semantic analysis.
[0157] Step 8, generate test cases.
[0158] Step 9, judge "Check Oracle type?".
[0159] If not, return to Step 8 "Generate test cases".
[0160] If yes, execute Step 10, judge "Check parameters?".
[0161] Step 10, judge "Check parameters?".
[0162] Yes, perform Step 11: Evaluate test cases.
[0163] Return to Step 8: Generate test cases.
[0164] Step 11: Evaluate test cases.
[0165] Step 12: Judge "Is it in compliance with the regulations?"
[0166] Return to Step 8: Generate test cases through static analysis and model feedback.
[0167] Yes, save the test cases and end the algorithm.
[0168] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention. It should be noted that there are some unclear or incorrect expressions in the original text, such as "匦", "匈规", "等匌替扢厌改进", "包匫", which may affect the accurate understanding. The translation is done based on the best possible interpretation.
Claims
1. A method for generating a dialogue-driven regression test based on a large code language model, characterized in that: The process includes the following steps: training a large language model for programming languages or code generation tasks; using conversational interaction to drive the generation of test cases; and performing regression testing.
2. According to claim 1, a method for generating a dialog-driven regression test based on a large code language model, characterized in that: It also contains the following steps: Step 1: Collect and prepare a data set for C language programs, including source code, corresponding natural language description of test requirements, test case data constraints, and official test case examples; Step 2: Determine the appropriate large language model; Step 3: Select the function to be tested from the data set and start generating test cases; Step 4: Design prompt templates for different types of C language programs to guide the model to generate test cases of specific types and standard formats; Step 5: Use the big language model to perform semantic analysis on the test requirements and feed back the analysis results to itself to deepen the big language model's understanding of the test requirements; Step 6: The large language model generates a preliminary version of test cases based on the conversation information; Step 7: Check whether the Oracle (expected output) type of the generated test case matches the function under test. If not, give relevant information feedback to guide the large model to correct the test case data type. Step 8: Check whether the test input parameters of the generated test case correspond one-to-one with the parameters of the function being tested. If non-function parameters or missing parameters appear, relevant information feedback is given to guide the large model to correct the test case data type; Step 9: Test the generated test cases and generate a test case quality assessment report. If the test cases meet the requirements, save the test cases to the corresponding location and go to step 3; otherwise, go to step 10. Step 10: Use static analysis tools to analyze the tested function code for which corresponding test cases have been set as input, detect the coverage of the code by the test cases, and extract important information, including uncovered branches, based on the test case quality assessment report generated by the static analysis tool, and feed back these more targeted feedback information to the large language model to optimize the test cases; Step 11: Repeat the above steps until qualified test cases are generated for all tested functions in the dataset, and the task is completed.
3. The method for generating a dialog-driven regression test based on a large code language model according to claim 1, characterized in that: Step 1 also includes the following steps: collecting data before the release date of the corresponding large language model.
4. The method for generating a dialog-driven regression test based on a large code language model according to claim 1, characterized in that: There are also the following steps in step 2: When selecting the large language model with the best performance, follow the following process: Step 2.
1. Select model: Determine the list of large language models to be evaluated; Step 2.2, sampling data set: using random sampling method to sample evaluation data in the data set; Step 2.3, Evaluation Criteria: Determine the criteria for the evaluation model, including: Format standardization of generated tests: Ensure that the output format meets the expected standards; Test case validity: Manual inspection of the generated test cases to see if they are effective in detecting problems; Cost of running large models: The computational resources and time consumed when evaluating the model's performance. Step 2.4, calculate the overall score: use the pre-defined criteria and use the following quantitative formula to calculate the score of each model: performance score = w1 × format compliance + w2 × test case effectiveness - w3 × cost, w1, w2, w3 are weights reflecting the importance of the criteria, and w1 + w2 + w3 = 1; Step 2.5: Select the best model: Select the large language model that performs best on the sampled dataset based on the comprehensive score.
5. The method for generating a dialog-driven regression test based on a large code language model according to claim 1, characterized in that: Step 4 also includes the following steps: When designing prompt templates for different types of C language programs, follow the following process: Step 4.1, Requirements Analysis: Analyze the operations that different types of C language programs may involve and the information formats that need to be input; Step 4.2, Functional design: Design a general prompt template, including basic command prompts, parameter input type instances, and error prompt words; Step 4.3, optimize the template: test the designed prompt template, make the expression as unambiguous as possible, and guide the large language model to answer as expected; Design prompt templates suitable for different types of C language programs to improve the accuracy of large language model generation tests.
6. The method for generating a dialog-driven regression test based on a large code language model according to claim 1, characterized in that: Step 5 also includes the following steps: Step 5.1: Semantic analysis requirements: Use a large language model to perform semantic analysis to understand the nature and complexity of the test requirements. The model needs to be able to identify errors, boundary conditions, memory leaks, and null pointer references that may occur in different types of C language programs. Step 5.2, Analysis Method: When designing test cases, the model analyzes possible logic errors in the code, logic errors in loops or conditional judgment errors. When testing different types of C language programs, the model understands the specific functions and implementation details of these programs as well as the correctness and efficiency of the algorithms. Step 5.3, Result Feedback: Feedback the results of semantic analysis to the model itself to help it further deepen its understanding and identification of test requirements; the feedback information includes common programming errors identified, optimization suggestions, and improvement suggestions for test case design; Step 5.4, Deepen understanding: The model learns more about the context and semantics of the test requirements by analyzing the feedback results. The model uses the feedback to improve its accuracy and comprehensiveness of various test requirements and support the needs of users when designing and executing tests.
7. The method for generating a dialog-driven regression test based on a large code language model according to claim 1, characterized in that: Step 9 also includes the following steps: Generating a use case quality assessment report is done through gcov and lcov. The process is as follows: Step 9.1, compile the program and enable coverage detection: Include the code coverage detection options -fprofile-arcs and -ftest-coverage in the gcc compilation command, compile the main function file and the function file under test together to generate an executable file, and generate the .gcno and .gcda files of the function under test in the corresponding directory for coverage data collection; Step 9.2, Collect coverage data: Use lcov to collect coverage data and output it to the coverage.info file, which contains the overall code coverage data; Step 9.3, extract the coverage data of a specific file: Use lcov to extract the coverage data of the tested function file from the coverage.info file to a temporary file (test1.info); Step 9.4: Repeat the above steps until all test cases are executed and coverage information of all test cases is obtained; Step 9.5, Merge coverage data: Use lcov to merge the testX.info files containing coverage information of multiple test cases into a final coverage.info.
8. The method for generating a dialog-driven regression test based on a large code language model according to claim 1, characterized in that: Step 10 also includes the following steps: After obtaining the line coverage information and conditional branch coverage information of the current test case set, if it does not meet expectations, the line coverage should exceed 90%. Use static analysis tools to analyze the coverage of the tested function code by the set test cases, and use the feedback information of the analysis report to optimize the test cases; Step 10.
1. Select static analysis tool: First, choose a static analysis tool that is suitable for the project, make sure it supports the analysis of C language code, and can detect the coverage of test cases. Step 10.2: Prepare the C language code to be tested and configure the static analysis tool: Prepare the C language code of the test cases and the tested functions and configure the selected static analysis tool into the development environment, set the tool parameters and paths, and be able to access and analyze the source code files containing the tested functions; Step 10.3, run the static analysis tool: Start the static analysis tool to analyze the code of the function under test. The tool will automatically detect which parts of the code are covered by the test cases and which parts of the code are not covered and make records; Step 10.4, generate analysis report: The static analysis tool generates a detailed report about the current test case coverage information, including the coverage of each branch or path of the tested function. The report indicates which code paths are covered by the test cases and which conditional branches or code paths under specific inputs are not covered. Step 10.5: Extract information to optimize test cases. Analyze the reports generated by static analysis tools, especially the important information of uncovered code paths, to obtain more targeted feedback information, and combine the feedback information with the corresponding prompt template to guide the large language model to optimize test cases.
Citation Information
Cited By
Software configuration item test case automatic generation method based on large language model
CN119336609A
Dynamic language test case automatic generation method based on large language model
CN119576799A
Automatic generation method of dynamic language test cases based on large language model
CN119576799B
Unit test generation method based on large language model
CN119782178A
A Unit Test Generation Method Based on a Large Language Model
CN119782178B
