Fuzzy test case generation method and system based on large language model

Abstract

The invention discloses a fuzz test case generation method and system based on a large language model, and relates to the technical field of fuzz testing, and the method comprises the steps: collecting first input data, and generating a preliminary prompt through a first prompt generation method; a test case is generated through prompt optimization; an optimal prompt is selected, a final test case is generated, and more test cases are generated through loop optimization. According to the fuzz test case generation method based on the large language model, multiple types of input data are collected and the preliminary prompt is automatically generated, so that the test cases with high adaptability can be automatically generated for different types of data, and the effectiveness of fuzz test is improved. By prompting the optimization algorithm, the quality of the test case is remarkably improved, and the generated test case is more accurate and effective. Through loop optimization of multi-round iteration and optimal prompt, the quality and coverage range of the fuzzy test case are greatly improved, and better effects are achieved in the aspects of effectiveness, accuracy and coverage range.

Description

Technical Field

[0001] The present invention relates to the field of fuzzy testing technology, and in particular to a fuzzy testing case generation method and system based on a large language model. Background Art

[0002] In the field of software testing, fuzz testing is an efficient automated technology that detects potential defects and vulnerabilities in the system by generating abnormal or random input data. This method has been proven to be very effective in identifying hidden problems in software systems. With the continuous development of fuzz testing tools, they have revealed numerous security vulnerabilities in real-world applications. Compilers, interpreters, and constraint solvers have also become the focus of fuzz testing. As key components in the software development process, these systems are particularly critical for their security testing. At present, fuzz testing tools are mainly divided into two categories: generation-based and mutation-based. Generation-based tools try to construct complete code snippets that conform to the syntax of the target language, while mutation-based tools apply variation operations on a set of high-quality input samples to explore a wider input space. However, these traditional methods have some inherent limitations and challenges:

[0003] (1) Specificity limitations: Traditional fuzz testing tools are often designed for specific programming languages ​​or systems, which limits their universality. Developing a fuzz testing tool requires a lot of time and expertise. For example, Csmith for the C / C++ compiler contains more than 80,000 lines of code, and Syzkaller requires tens of thousands of rules to generate and modify Linux system calls. Since each programming language has its own uniqueness, migrating a fuzz testing tool from one language environment to another is a complex process. In addition, an effective fuzz testing strategy may not be universal between different systems.

[0004] (2) Insufficient coverage of the input space: Both generation-based and mutation-based fuzz testing have limitations in exploring the entire input space. Generation-based fuzz testing tools rely heavily on the grammatical rules of the target language to construct code, which limits them to testing only a small part of the language's features. Mutation-based fuzz testing tools are limited by the diversity of their mutation operations and the quality of their seeds, which limits their ability to explore new input spaces. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is that the existing generation-based and mutation-based fuzz testing methods have poor adaptability and the problem of how to test the entirety of the target language.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a fuzzy test case generation method based on a large language model, comprising collecting first input data, generating preliminary prompts through a first prompt generation method; generating test cases through prompt optimization; selecting the optimal prompt and generating a final test case, and cyclically optimizing to generate more test cases.

[0008] As a preferred solution of the fuzzy test case generation method based on a large language model described in the present invention, the collecting of the first input data includes receiving user input describing the target object of the test and the test case type data to be generated.

[0009] As a preferred solution of the fuzzy test case generation method based on a large language model described in the present invention, wherein: the generating of preliminary prompts by the first prompt generation method includes extracting candidate fuzzy test prompts.

[0010] As a preferred solution of the fuzzy test case generation method based on a large language model described in the present invention, wherein: the test case generation through prompt optimization includes passing the candidate fuzzy test prompt prompt to the generated LLM, and generating a code segment according to the prompt.

[0011] As a preferred solution of the method for generating fuzzy test cases based on a large language model described in the present invention, wherein: the selecting the optimal prompt and generating the final test case includes scoring all generated code segments based on a scoring function;

[0012] The scoring function is based on the number of unique code snippets that are effectively generated as a metric.

[0013] As a preferred solution of the fuzzy test case generation method based on a large language model described in the present invention, wherein: the selection of the optimal prompt and generation of the final test case includes selecting the prompt corresponding to the code snippet with the highest score as the optimal prompt, and passing the optimal prompt again to generate the LLM to obtain the fuzzy test case.

[0014] As a preferred solution of the fuzzy test case generation method based on a large language model described in the present invention, the loop optimization to generate more test cases includes generating new prompts through continuous iterations and scoring and optimizing to improve the quality and coverage of the test cases.

[0015] Another object of the present invention is to provide a fuzzy test case generation system based on a large language model, which can gradually improve the effectiveness and coverage of test cases through intelligent selection and feedback loops, solving the problem that current generation-based and mutation-based fuzzy testing methods can only test a small part of the language.

[0016] As a preferred solution of the fuzzy test case generation system based on a large language model described in the present invention, it includes: a prompt generation module, a test case generation module, and a test case optimization module; the prompt generation module is used to collect first input data, and generate preliminary prompts through a first prompt generation method; the test case generation module is used to generate test cases through prompt optimization; the test case optimization module is used to select the optimal prompt and generate a final test case, and cyclically optimize to generate more test cases.

[0017] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a fuzzy test case generation method based on a large language model.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a fuzzy test case generation method based on a large language model.

[0019] Beneficial effects of the present invention: The fuzzy test case generation method based on a large language model provided by the present invention can automatically generate adaptable test cases for different types of data by collecting various types of input data and automatically generating preliminary prompts, thereby improving the effectiveness of fuzzy testing. The quality of test cases is significantly improved through the prompt optimization algorithm, and the generated test cases are more accurate and effective. The quality and coverage of fuzzy test cases are greatly improved through multiple rounds of iterations and cyclic optimization of optimal prompts. The present invention achieves better results in terms of effectiveness, accuracy and coverage. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 An overall flow chart of a fuzzy test case generation method based on a large language model provided for the first embodiment of the present invention.

[0022] Figure 2 A schematic diagram of a test case generation method based on a large language model for a fuzzy test case generation method based on a large language model provided in the first embodiment of the present invention.

[0023] Figure 3 A prompt optimization algorithm flow chart of a fuzzy test case generation method based on a large language model provided in the first embodiment of the present invention.

[0024] Figure 4 A partial example diagram of the C language standard specification of a fuzzy test case generation method based on a large language model provided in the first embodiment of the present invention.

[0025] Figure 5 An optimal prompt example diagram of a fuzzy test case generation method based on a large language model provided in the first embodiment of the present invention.

[0026] Figure 6 A graph showing the results of a C compiler test experiment for a fuzzy test case generation method based on a large language model provided in the second embodiment of the present invention.

[0027] Figure 7 A graph showing the results of a Java compiler test experiment for a fuzzy test case generation method based on a large language model provided in the second embodiment of the present invention.

[0028] Figure 8 A Go tool chain test experiment result diagram of a fuzzy test case generation method based on a large language model provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0030] Example 1, reference Figure 1-Figure 6 , as an embodiment of the present invention, provides a fuzzy test case generation method based on a large language model, comprising:

[0031] S1: Collect first input data and generate preliminary prompts through a first prompt generation method.

[0032] Furthermore, collecting the first input data includes receiving user input describing the target object of the test and the type of test case to be generated.

[0033] It should be noted that the first input data includes but is not limited to technical documentation, code examples and functional descriptions of the system.

[0034] Technical documents such as API documentation, protocol specifications, programming language standards (such as the standard library specification of C language), user manuals, etc. These documents describe the system's functions, interfaces, and behavioral expectations in detail.

[0035] Code samples can be source code snippets of the target system, definitions of functions or classes for testing, or even complete project code.

[0036] Functional description includes the functional requirements of the system or component, the expected behavior of the input and output, and the natural language description of the boundary conditions. Historical defect data: includes detailed information about previously reported vulnerabilities or errors (such as error descriptions, triggering conditions, and repair methods), which helps generate test cases that can detect similar defects.

[0037] In certain cases, it can be supplemented with system operation configuration, dependent external library information, runtime parameters, etc., to help generate test cases in specific environments.

[0038] Further, at the forefront of natural language processing (NLP), large language models (LLMs) have emerged as a powerful tool for processing natural language and code tasks. These models are based on the transformer architecture and can be divided into different types according to their design and functionality: decoder models, such as GPT-3 and StarCoder, focus on generating text. Encoder models, such as BERT and CodeBERT, excel at understanding text. And encoder-decoder models, such as BART and CodeT5, combine the capabilities of generation and understanding. Instruction-based LLMs, such as ChatGPT and GPT-4, and models fine-tuned from human feedback (RLHF) through reinforcement learning, have shown that they can understand and execute complex instructions. These models perform specific tasks in two main ways: fine-tuning and prompting. Fine-tuning is to further train the model on a dataset for a specific task to update the model's weights to better adapt it to the task requirements. However, obtaining a suitable dataset can be challenging, and the cost of fine-tuning increases as the model size grows. In contrast, hinting methods do not require explicit updating of model weights, but instead guide the model to complete a specific task by providing a task description and examples, i.e., the process of selecting inputs, where the user tries different input instructions to find the best solution.

[0039] In order to improve efficiency, studies have developed automatic prompting technology to select the most appropriate prompts by utilizing the model's own gradients. Currently, this method has evolved to use a calculated validity proxy score to replace the gradient-based selection method, further improving automation and efficiency. In the present invention, a large language model is used to solve key problems in the field of fuzz testing. Unlike previous automatic prompting and agent-based methods, the method of the present invention directly uses GPT-4 to synthesize prompts for fuzz testing targets and scores them to ensure that the generated test cases can effectively explore potential vulnerabilities and defects. This method not only improves the quality and coverage of test cases, but also reduces the complexity and cost of fuzz testing through automation and intelligent means.

[0040] Fuzz testing is an automated software testing technique that aims to trigger unexpected behaviors of the system under test by inputting abnormal or random data. It is essential for discovering software defects and security vulnerabilities. Traditional fuzz testing techniques can be divided into two main categories: generation-based methods and mutation-based methods. Generation-based fuzz testers rely on predefined grammar rules and a deep understanding of the target language to construct complete code snippets. For example, Csmith hardcodes language specifications to ensure the effectiveness of generated code snippets in C and C++ compiler testing. jsfunfuzz combines language grammar and historically discovered bug snippets to generate new inputs to test JavaScript engines. On the other hand, mutation-based fuzz testers generate new fuzz inputs by applying a series of iterative transformations on seed inputs. These transformations include not only basic mutation operations, but may also include complex transformations such as ensuring type consistency, adding historically triggered bug snippets, and using coverage feedback to guide the mutation process.

[0041] Many fuzz testers combine generation and mutation to improve the effectiveness and coverage of testing. However, these fuzz testers are usually language-specific or system-specific implementations rather than general-purpose. In contrast, general-purpose fuzz testers such as AFL and libFuzzer use the fitness function of a genetic algorithm to prioritize further mutations to achieve new code coverage. These general-purpose fuzz testers focus on byte-level transformations and do not rely on knowledge of the system under test. However, when applied to systems that accept programming languages ​​as input, they have difficulty generating valid inputs. In order to expand the scope of fuzz testing and apply it to emerging areas, some studies have proposed learning-based fuzz testers. These techniques usually focus on training neural networks to generate fuzz testing inputs. For example, TreeFuzz parses the training corpus into a tree structure and learns a probabilistic generation model to synthesize new inputs. Deep learning models have been applied to fuzz testing of PDF parsers, OpenCL, C language, network protocols, and JavaScript.

[0042] Recently, studies have begun to directly use large language models (LLMs) for fuzz testing, such as TitanFuzz, which uses Codex to generate seed programs and combines InCoder to perform template-based mutations to test deep learning libraries. The method proposed in the present invention is different from previous learning- and LLM-based fuzz testers in that it can generate test cases for different programming languages. By introducing an innovative automatic prompting stage, the method of the present invention can accept inputs in any format to achieve general and targeted fuzz testing, rather than being limited to specific types of inputs (such as function source code) or specific uses (such as unit testing). This method brings greater flexibility and applicability to the field of fuzz testing, enabling it to adapt to the ever-changing needs of software testing.

[0043] It should be noted that generating preliminary prompts by the first prompt generation method includes extracting candidate fuzzy test prompts.

[0044] like Figure 2 The construction of the present invention is based on the collaborative work of two models, a LLM responsible for extracting information and a LLM responsible for generating test cases. The purpose of this design is to optimize the cost-effectiveness ratio and ensure the efficiency and effectiveness of the method. In the extraction stage, the large-scale basic model selected has excellent natural language processing capabilities and can deeply understand and refine the complex information input by the user. In view of the fact that directly applying such large models to input generation may face efficiency bottlenecks (mainly due to the higher inference cost in the autoregressive generation process). Therefore, in the generation stage, a smaller-scale model is used as the generation LLM to improve the efficiency of fuzz testing.

[0045] So given that user input may contain redundant or irrelevant information, the extraction LLM is responsible for automatically extracting concise and information-rich fuzz testing prompts.

[0046] It should also be noted that in order to enhance the coverage of test cases, a rule-guided approach can be adopted to generate prompts, technical documentation, code examples, and functional descriptions in scenarios with clear input data structure and consistent format, scenarios with a large amount of repetitive or patterned input, or scenarios with clear domain knowledge and a lot of prior knowledge available.

[0047] For technical documents, keyword matching and text structure analysis are used to extract paragraphs containing functional descriptions, parameter information, or restrictions.

[0048] For code examples, function definitions, variable declarations, and comment contents are extracted through syntax analysis, and code parts that are not related to test case generation are filtered out.

[0049] For functional description, entity recognition technology is used to extract key noun phrases and verb phrases from natural language descriptions.

[0050] The rule-guided prompts can be used directly or optimized through extractive LLM. After the initial screening guided by rules, the screened content is further refined and optimized using extractive LLM (such as GPT-4).

[0051] The filtered content is converted into a structured prompt form through natural language generation technology, and different variants are generated according to the rules to increase diversity.

[0052] Example of instructions generated using the GPT-4 model: "Based on the filtered input, create a brief prompt containing a description of the target function and parameters." This step combines the results of rule guidance with the natural language processing capabilities of the model to generate high-quality prompts and reduce the pressure of the scoring step.

[0053] S2: Generate test cases through prompt optimization.

[0054] Furthermore, generating test cases through prompt optimization includes passing a candidate fuzz test prompt to a generated LLM, and generating a code segment according to the prompt. The given user input is converted into a prompt suitable for LLM to generate a test case through a prompt optimization algorithm. The user input may describe the instruction document of the system under test, the specific function to be tested, or a code example, etc. Unlike traditional fuzzers that require input to follow a specific format, the method of the present invention can directly understand the natural language description or code example in the user input. However, some information in the user input may be redundant or irrelevant, so directly using the user input as a prompt for generating LLM may be invalid. Therefore, the goal of prompt optimization is to generate a refined prompt so that LLM generates a valid test case. Selecting the optimal prompt and generating the final test case includes scoring all generated code segments based on a scoring function, and the scoring function is based on the number of unique code segments generated effectively as a metric.

[0055] It should be noted that if Figure 3 The algorithm shown describes the specific process of the prompt optimization algorithm. The input of the algorithm includes user input (userInput) and the number of sample prompts to be generated, and the output is the optimal prompt selected for generating test cases. First, in order to simplify the user's input, the method inputs an automatic prompt instruction to the extraction LLM: "Please summarize the above information in a concise manner to describe the usage and functionality of the target.", and sets the parameter temp = 0, so that the extraction LLM tends to select the output with the highest probability and generate a high-confidence prompt. The generated prompt can be formalized as a conditional probability P (prompt|userInput, instruction). The algorithm then sets the parameter temp to 1, so that the model will introduce more randomness when selecting the output, thereby producing more diverse results, and at the same time, each generated prompt is added to the candidate list until the required number of samples is reached.

[0056] After obtaining enough candidate prompts, each prompt is input into the generated LLM to generate a code segment. Finally, all generated code segments are scored based on the scoring function, and the prompt with the highest score is selected. Although the scoring function can be based on a variety of different indicators, such as code coverage, error detection ability, or input complexity, in order to ensure the lightweight and universality of the method, the scoring function selected by the present invention is based on the number of unique code segments that are effectively generated, that is, the number of test cases accepted by the target system test case (SUT). The choice of this metric is based on the emphasis on input validity in fuzz testing to ensure that the test input can touch the deep logic of the SUT. The scoring function is defined as:

[0057] Σc∈M G (p)[isVaild(c,SUT)]

[0058] Among them, MG is the generated LLM, p is the candidate prompt, and isValid is a function that returns 1 if the generated code c is valid, otherwise it returns 0. Finally, the input prompt with the highest score is selected as the initial input prompt for fuzz testing. The automatic prompt step proposed in the present invention integrates the prompt generation and scoring mechanism, realizes the automatic generation and selection of prompts suitable for the fuzz testing target, and improves the efficiency and effectiveness of fuzz testing.

[0059] S3: Select the best hint and generate the final test case, and then optimize the loop to generate more test cases.

[0060] Furthermore, selecting the optimal prompt and generating the final test case includes selecting the prompt corresponding to the code snippet with the highest score as the optimal prompt, and passing the optimal prompt again to generate the LLM to obtain the fuzzy test case.

[0061] It should be noted that in the embodiment, the C compiler is selected as the test object, aiming to explain in detail the generation process of the optimal prompt through specific examples. In order to ensure the comprehensiveness and accuracy of the test, the present invention adopts the standard specification of the C language as the user input, such as Figure 4 This specification describes in detail the core features of the C language, including but not limited to the composition of the standard library, the use of variables, the functions of common functions, and how they are used in actual programming.

[0062] Next, the standard specification and automatic prompt instructions are input into the extraction LLM (GPT-4, calling the interface provided by OpenAI, with max_token set to 500) to generate a candidate prompt set. After obtaining the prompt set, the set is input into the generation LLM (StarCoder, which has been trained on more than one trillion code tokens in more than 80 languages) to generate code snippets. Finally, all code snippets are scored using a scoring function, and the prompt corresponding to the code snippet with the highest score is the optimal prompt, such as Figure 5 Shown is the optimal prompt generated in the first iteration.

[0063] The optimal prompt consists of three parts: feature summary, detailed description, and supplementary explanation. The "Feature Summary" section provides a concise summary of the core features of the C language standard library, which is a collection of multiple header files that define various functions, macros, and types used in C programming. The "Feature Summary" section further explains the functions of each key header file in the library and their application in actual programming, providing a detailed explanation of the functions and macros contained in each header file, and how they interact to support complex programming tasks, such as:<stdio.h> Handle input and output operations,<stdlib.h> Provides general utility tools,<string.h> Provides string processing functions,<math.h> Contains commonly used mathematical functions,<time.h> Utilities that support time and date,<ctype.h> Determine the character data type,<limits.h> Define the range of integer types,<stdbool.h> The Boolean type and macros were introduced. The supplementary explanation section describes the evolution of the standard library, explaining that with the update of the C language standard, the new version of the standard library introduces new header files and functions to adapt to the ever-changing programming needs and technological developments.

[0064] Example 2, reference Figure 6-Figure 8 , is an embodiment of the present invention, which provides a fuzzy test case generation method based on a large language model. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0065] First, in order to verify the feasibility of the fuzzy test case generation method based on a large language model proposed in the present invention, and to comprehensively evaluate the optimization effect of the method proposed in the present invention, the experiment selected C compiler, Java compiler and Go tool chain as the objects under test. The experiment aims to compare the performance of the test cases generated by the method of the present invention and the dedicated fuzz testing tool of the object under test in terms of code coverage through 24 hours of continuous testing. Code coverage is one of the key indicators for measuring the quality of test cases, which reflects the scope of code that the test case can reach. Through this indicator, the potential of different methods in discovering potential defects and improving software quality can be objectively evaluated.

[0066] In the experiment of testing the C compiler, the present invention selected the GCC compiler widely recognized by the industry as the main object of the experiment. The GCC compiler is widely respected by developers for its excellent performance and stability, and is the cornerstone for building many key systems and applications. Given its core position in the software infrastructure, GCC's requirements for security and reliability are naturally very strict.

[0067] In the experimental preparation stage, the present invention presets the official document of the standard C library as user input, provides it to the method of the present invention, and introduces Csmith as a control tool. As a well-known generation-based fuzz testing tool, Csmith has been widely recognized by the industry for its ability to automatically generate test cases and identify potential compiler defects. By comparing with Csmith, the performance of the method of the present invention can be objectively measured. The experimental results are as follows: Figure 6 shown.

[0068] It can be seen that the method proposed in the present invention is significantly better than the optimized Csmith tool. After the experiment was carried out for 24 hours, the method of the present invention and Csmith achieved coverage of 69.8% and 38.2% respectively, and the coverage was increased by 31.6%. It can be seen that the fuzzy test case generation method based on the large language model proposed in the present invention shows good performance in the test process of the C compiler, especially over time, it can continuously generate effective test cases compared to the Csmith tool.

[0069] In the experiment of testing the Java compiler, the present invention selected the Java compiler javac in the OpenJDK project as the core target of the experiment. As a key tool for converting Java source code into bytecode, the performance and stability of the javac compiler are crucial to supporting the huge Java application ecosystem. Given Java's dominant position in cross-platform application development, there are extremely high standards for the security and robustness of javac. The experiment uses the latest version of the standard Java API documentation as user input, and Hephaestus is included in the experiment as a comparison tool. Hephaestus is an advanced fuzz testing tool that integrates generation and mutation techniques. It is designed specifically for JVM compilers and is particularly good at discovering errors related to type systems. The specific experimental results are shown below. Figure 7 shown.

[0070] In the early stage of the experiment, the code coverage of both methods was low, but the method of the present invention increased the number of valid use cases faster than the Hephaestus tool, which shows that the method of the present invention can generate more effective use cases in the early stage. As the experiment continued, the code coverage of both methods increased, but the growth trend of the method of the present invention seemed to be more significant, which may mean that the method of the present invention can generate valid use cases more stably in long-term testing. In addition, at the end of the experiment, the method of the present invention and Hephaestus achieved coverage of 36.1% and 22.5%, respectively, with a coverage increase of 13.6%.

[0071] In the test experiment for the Go tool chain, the present invention selected the official documentation of the Go standard library as the user's default input. The Go tool chain provides a wealth of built-in functions for the Go language, and its performance and reliability are crucial to the widespread application of the Go language. In view of the increasing popularity of the Go language in system programming and network service development, ensuring the security and robustness of the Go standard library has become an important task. In the experimental design, the present invention uses go-fuzz as a control tool. It is a fuzz testing tool specially designed for the Go language. It adopts a coverage-based mutation testing method and uses pre-defined templates to generate test inputs for the Go tool chain. The specific experimental results are as follows: Figure 8 shown.

[0072] It can be seen from the experimental results that before 9 hours, the effect of the method of the present invention is not as good as that of the go-fuzz tool. The reason is that the go-fuzz tool is designed based on the code coverage index, so the test cases generated in the early stage of the experiment can obtain good coverage. However, as the experiment progresses, the test cases generated by this mutation-based fuzz testing method significantly reduce the growth of code coverage. At the end of the experiment, the final code coverage is 52.4%, which is only 15.6% higher than the initial test case. Although the fuzz test case generation method based on the large language model proposed by the present invention is weaker than the go-fuzz tool in the early stage of the experiment, the coverage index grows rapidly. At the end of the experiment, the code coverage has reached 73.2%, which is 20.8% higher than the go-fuzz tool.

[0073] Embodiment 3 is an embodiment of the present invention, which provides a fuzzy test case generation system based on a large language model, including a prompt generation module, a test case generation module, and a case optimization module.

[0074] The prompt generation module is used to collect the first input data and generate preliminary prompts by the first prompt generation method. The test case generation module is used to generate test cases by prompt optimization. The test case optimization module is used to select the optimal prompt and generate the final test case, and cyclically optimize and generate more test cases.

[0075] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0076] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0077] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0078] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0079] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A fuzzy test case generation method based on a large language model, characterized in that: include: Collecting first input data, and generating preliminary prompts by a first prompt generation method; Optimize test case generation through prompts; The best hint is selected and the final test case is generated, and more test cases are generated through loop optimization.

2. The method for generating fuzzy test cases based on a large language model as claimed in claim 1, characterized in that: The collecting of the first input data includes receiving data input by a user describing a target object of the test and a type of test case to be generated.

3. The method for generating fuzzy test cases based on a large language model as claimed in claim 2, characterized in that: The generating of preliminary prompts by the first prompt generating method includes extracting candidate fuzzy test prompts.

4. The method for generating fuzzy test cases based on a large language model as claimed in claim 3, characterized in that: The test case generation through prompt optimization includes passing the candidate fuzz test prompt prompt to generate LLM, and generating a code segment according to the prompt.

5. The method for generating fuzzy test cases based on a large language model as claimed in claim 4, characterized in that: The selecting the optimal hint and generating the final test case comprises scoring all generated code segments based on a scoring function; The scoring function is based on the number of unique code snippets that are effectively generated as a metric.

6. The method for generating fuzzy test cases based on a large language model as claimed in claim 5, characterized in that: The selecting the optimal prompt and generating the final test case includes selecting the prompt corresponding to the code snippet with the highest score as the optimal prompt, and passing the optimal prompt again to generate the LLM to obtain the fuzzy test case.

7. The method for generating fuzzy test cases based on a large language model as claimed in claim 6, characterized in that: The loop optimization to generate more test cases includes generating new prompts through continuous iterations and scoring and optimizing to improve the quality and coverage of the test cases.

8. A system using the method for generating fuzzy test cases based on a large language model as claimed in any one of claims 1 to 7, characterized in that: Including prompt generation module, test case generation module, and test case optimization module; The prompt generation module is used to collect first input data and generate a preliminary prompt by a first prompt generation method; The test case generation module is used to generate test cases through prompt optimization; The test case optimization module is used to select the best prompt and generate the final test case, and cyclically optimize to generate more test cases.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for generating fuzzy test cases based on a large language model according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating fuzzy test cases based on a large language model according to any one of claims 1 to 7 are implemented.

Images