Fuzzy test case generation method and system based on large language model

Through the fuzzy test case generation method that works in a collaborative manner in large language models, the problems of low applicability and coverage of existing fuzzy test tools are solved, and the ability to generate test cases and discover potential defects is achieved, which improves the security of the software system. sex.

CN118964212APending Publication Date: 2024-11-15STATE GRID SHANDONG ELECTRIC POWER CO +4
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411242771.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing fuzz testing tools have limited applicability to specific programming languages ​​and systems and have limitations in exploring input spaces, resulting in complex development and migration, low effectiveness and coverage.

Method used

Using a fuzzy test case generation method based on large language models, through the collaborative work of extracting models and generating models, user input is automatically extracted and test cases that conform to the target language syntax and semantic constraints are generated, and the iterative optimization algorithm is used to improve the diversity of test cases and Coverage.

Benefits of technology

Improve the effectiveness and coverage of test cases, enhance support for a variety of programming languages ​​and systems, significantly improve the ability to detect defects and vulnerabilities, and improve the security and reliability of software systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964212A_ABST
    Figure CN118964212A_ABST
Patent Text Reader

Abstract

The invention discloses a fuzzy test case generation method and system based on a large language model. The method comprises the following steps: receiving user input information for describing a target test case; automatically extracting the user input information based on a preset extraction model to obtain a fuzzy test prompt; inputting the fuzzy test prompt into a generation model, generating a code segment by using the generation model, and performing scoring; using the code segment with the highest score as an optimal prompt, inputting the code segment into the generation model again, and generating a fuzzy test case by using the generation model; and iteratively executing the steps until a predefined number of fuzz test cases are generated, and forming a fuzz test case set by using all the fuzz test cases. According to the scheme, the diversity and the coverage rate of the test cases are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of software fuzz testing, and in particular relates to a fuzz test case generation method and system based on a large language model. Background Art

[0002] In the field of software testing, fuzz testing is an automated software testing technology that triggers unexpected behaviors of the system under test by inputting abnormal or random data. It is crucial for discovering software defects and security vulnerabilities. Fuzz testing detects potential defects and vulnerabilities in the system by generating abnormal or random input data. With the continuous development of fuzz testing tools, they have revealed numerous security vulnerabilities in real-world applications. Compilers, interpreters, and constraint solvers have also become the focus of fuzz testing. As key components in the software development process, these systems are particularly critical for their security testing. At present, fuzz testing tools are mainly divided into generation-based testing tools and mutation-based testing tools. Generation-based tools try to construct complete code snippets that conform to the syntax of the target language, while mutation-based tools apply variation operations on a set of high-quality input samples to explore a wider input space. However, these traditional methods have some inherent limitations and challenges:

[0003] (1) Traditional fuzz testing tools are designed for specific programming languages ​​or systems, which limits their universality. Developing a fuzz testing tool requires a lot of time and expertise. For example, Csmith for the C / C++ compiler contains more than 80,000 lines of code, and Syzkaller requires tens of thousands of rules to generate and modify Linux system calls. Since each programming language has its own uniqueness, migrating a fuzz testing tool from one language environment to another is a complex process. In addition, an effective fuzz testing strategy is not universal across different systems.

[0004] (2) Both generation-based and mutation-based fuzz testing have limitations in exploring the entire input space. Generation-based fuzz testing tools rely heavily on the grammatical rules of the target language to construct code, which limits them to testing only a small part of the language's features. Mutation-based fuzz testing tools are limited by the diversity of their mutation operations and the quality of their seeds, which limits their ability to explore new input spaces. Summary of the invention

[0005] In order to address the deficiencies in the prior art, the present invention provides a fuzzy test case generation method and system based on a large language model to solve the problems of specificity limitations and insufficient input space coverage faced by existing fuzzy testing tools.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0007] The present invention first discloses a fuzzy test case generation method based on a large language model, the method comprising the following steps:

[0008] Step 1: Receive user input information for describing a target test case;

[0009] Step 2: Automatically extract user input information based on a preset extraction model to obtain fuzzy test prompts;

[0010] Step 3: Input the fuzz test prompt into the generation model, use the generation model to generate code segments and perform scoring;

[0011] Step 4: taking the code segment with the highest score as the optimal hint, inputting it into the generation model again, and generating a fuzzy test case using the generation model;

[0012] Step 5: Iterate step 2 to step 4 until a predefined number of fuzzy test cases are generated, and all fuzzy test cases are combined into a fuzzy test case set.

[0013] The present invention further includes the following preferred embodiments:

[0014] The automatic extraction of user input information based on a preset extraction model to obtain a fuzzy test prompt further includes:

[0015] First, an automatic prompt instruction is input to the extraction model, and the parameter temp=0 is set, so that the extraction LLM tends to select the output with the highest probability, generates a high-confidence fuzzy test prompt, and formalizes the generated prompt into a conditional probability;

[0016] The parameter temp is then set to 1 to allow the model to introduce randomness when selecting outputs, and each generated fuzz test hint is added to the candidate list until the preset number of sample candidate fuzz test hints is reached.

[0017] The method of using the generative model to generate code segments and score them further includes:

[0018] After obtaining the candidate fuzz test hints for the sample number, all generated code segments are scored based on a scoring function, wherein the scoring function is based on the number of unique code segments generated effectively, that is, the number of test cases accepted by the target system test case.

[0019] The scoring function is defined as:

[0020] Σc∈M G (p)[isVaild(c, SUT)]

[0021] Among them, M Gis the generation model, p is the candidate prompt, isValid is the judgment function, and returns 1 if the generated code c is valid, otherwise returns 0.

[0022] The extraction model is GPT-4, and the generation model is StarCoder.

[0023] The present invention also discloses a fuzzy test case generation system based on a large language model using the aforementioned fuzzy test case generation method based on a large language model, comprising:

[0024] A user input module, used for receiving user input information for describing a target test case;

[0025] A prompt extraction module is used to automatically extract user input information based on a preset extraction model to obtain a fuzzy test prompt;

[0026] A code segment generation module, used for inputting the fuzzy test prompt into a generation model, generating code segments using the generation model and scoring them;

[0027] A case generation module, used to take the highest-scoring code segment as the optimal prompt, input it into the generation model again, and generate a fuzzy test case using the generation model;

[0028] The case set building module is used to generate a predefined number of fuzz test cases and compose all fuzz test cases into a fuzz test case set.

[0029] Accordingly, the present application also discloses a terminal, including a processor and a storage medium;

[0030] The storage medium is used to store instructions;

[0031] The processor is used to operate according to the instructions to execute the steps of the aforementioned fuzzy test case generation method based on the large language model.

[0032] Correspondingly, the present application also discloses a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the aforementioned fuzzy test case generation method based on a large language model are implemented.

[0033] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a fuzzy test case generation method and system based on a large language model, which understands the description documents or code specifications of the system under test by training a large language model, and then generates more effective test cases, thereby improving the diversity and coverage of test cases. Compared with traditional generation-based or mutation-based fuzz testers, the present invention has higher flexibility and adaptability, and can provide effective fuzz testing support for a variety of programming languages ​​and systems; through intelligent selection and feedback loops, the effectiveness and coverage of test cases are gradually improved, which not only improves the efficiency of the testing process, but also significantly enhances the ability to discover potential defects and vulnerabilities, and improves the security and reliability of software systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of the fuzzy test case generation method based on a large language model in the present invention.

[0035] Figure 2 It is a flowchart of the prompt optimization algorithm in the present invention.

[0036] Figure 3 This is an example diagram of the C language standard specification part in the present invention.

[0037] Figure 4 This is the optimal prompt example diagram in the present invention.

[0038] Figure 5 It is a schematic diagram of the test experimental results of the C compiler in the present invention.

[0039] Figure 6 It is a schematic diagram of the Java compiler test experimental results in the present invention.

[0040] Figure 7 It is a schematic diagram of the Go tool chain test experimental results in the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0042] The embodiments described in this application are only some embodiments of the present invention, not all embodiments. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the protection scope of the present invention.

[0043] Large language model (LLM) has become a powerful tool for processing natural language and code tasks. In view of the shortcomings of the prior art, the present invention proposes a fuzzy test case generation method and system based on large language model, using LLM as a powerful generation engine, according to the document information or sample code provided by the user, through the automated prompt optimization algorithm, to generate test cases that meet the grammatical and semantic constraints of the target language for the tested system implemented in different programming languages. LLM has been pre-trained on billions of code fragments, which enables it to have programming language knowledge and code pattern recognition capabilities. Therefore, LLM can generate a large number of test cases that follow the grammatical and semantic constraints of the target language, greatly improving the effectiveness and coverage of the test cases. The method based on LLM has higher flexibility and adaptability. It is not limited to a specific programming language or system implementation, and can be easily extended to multiple programming languages ​​and tested systems. The user only needs to provide relevant documents or sample codes, and LLM can generate test cases that meet the requirements based on these inputs.

[0044] The present invention is based on the collaborative work of two models, an LLM for extracting information and an LLM for generating test cases, with the aim of optimizing the cost-effectiveness ratio and ensuring the efficiency and effectiveness of the method. In the extraction stage, the selected basic large model has excellent natural language processing capabilities and can deeply understand and refine the complex information input by the user. In view of the fact that directly applying such large models to input generation may face efficiency bottlenecks (mainly due to the high inference cost in the autoregressive generation process), in the generation stage, a smaller-scale model is used as the generation LLM to improve the efficiency of fuzzy testing.

[0045] In a preferred embodiment, GPT-4 is selected as the extraction LLM, and StarCoder is selected as the generation LLM. The combination of GPT-4's advanced language understanding and StarCoder's expertise in code generation ensures that the generated fuzzy test cases not only follow the grammatical rules, but also can effectively detect potential defects in the target system.

[0046] See also Figure 1 As shown, the fuzzy test case generation method based on a large language model disclosed in the present invention comprises the following steps:

[0047] Step 1: Receive user input information for describing a target test case.

[0048] First, user input is received, including but not limited to technical documentation of the system under test, code examples, etc., which are used to describe the required test cases in detail.

[0049] Step 2: Automatically extract user input information based on the preset extraction model to obtain fuzzy test prompts.

[0050] In a further preferred embodiment, the extraction LLM converts the given user input into a prompt suitable for LLM to generate test cases through a prompt optimization algorithm. The user input may describe the instruction document of the system under test, the specific function to be tested, or the code example. Unlike the traditional fuzzer that requires the input to follow a specific format, the method of the present invention can directly understand the natural language description or code example in the user input.

[0051] However, some information in the user input may be redundant or irrelevant, so directly using the user input as the prompt for generating LLM may be ineffective. Therefore, the goal of prompt optimization is to generate a refined prompt so that LLM generates effective test cases.

[0052] like Figure 2 The algorithm shown describes the specific process of the prompt optimization algorithm. The algorithm input includes user input (userInput) and the number of sample prompts to be generated, and the output is the optimal prompt selected for generating test cases.

[0053] First, in order to simplify the user's input, the method inputs an automatic prompt instruction to the extraction LLM: "Please summarize the above information in a concise manner to describe the usage and functionality of the target.", and sets the parameter temp = 0, so that the extraction LLM tends to select the output with the highest probability and generate a high-confidence prompt. The generated prompt can be formalized as a conditional probability P (prompt|userInput, instruction). The temp parameter refers to a parameter used to control the degree of randomness in machine learning and artificial intelligence algorithms, especially in probabilistic models and generative models to adjust the randomness of the output of generative models (such as language models). The algorithm then sets the parameter temp to 1, so that the model will introduce more randomness when selecting outputs, thereby producing more diverse results, and each generated prompt is added to the candidate list until the required number of samples is reached.

[0054] Step 3: Input the fuzzy test prompt into the generation model, use the generation model to generate code segments and perform scoring.

[0055] Subsequently, each extracted prompt is passed to the generated LLM, which generates specific code snippets based on the prompts and performs scoring to evaluate the effectiveness of each prompt in generating test cases.

[0056] Step 4: The code segment with the highest score is used as the optimal prompt, input into the generation model again, and the generation model is used to generate a fuzzy test case.

[0057] Select the prompt with the highest score as the optimal prompt, and pass the optimal prompt to the generated LLM again to generate the final fuzzy test case;

[0058] After obtaining enough candidate prompts, each prompt is input into the generated LLM to generate a code segment. All generated code segments are scored based on the scoring function, and the prompt with the highest score is selected. Although the scoring function can be based on a variety of different indicators, such as code coverage, error detection ability, or input complexity, in order to ensure the lightweight and universality of the method, the scoring function selected by the present invention is based on the number of unique code segments that are effectively generated, that is, the number of test cases accepted by the target system test case (SUT). The selection of this metric is based on the emphasis on input validity in fuzz testing to ensure that the test input can touch the deep logic of the SUT.

[0059] In a preferred embodiment, the scoring function is defined as:

[0060] Σc∈M G (p)[isVaild(c, SUT)]

[0061] Among them, M G To generate LLM, p is a candidate prompt, isValid is a judgment function, and returns 1 if the generated code c is valid, otherwise it returns 0. Finally, the input prompt with the highest score is selected as the initial input prompt for fuzz testing.

[0062] Step 5: Iterate step 2 to step 4 until a predefined number of fuzzy test cases are generated, and all fuzzy test cases are combined into a fuzzy test case set.

[0063] Repeat steps 2 to 4 to iteratively generate fuzzy test cases until the number required by the user is reached and the final test case set is obtained.

[0064] The automatic prompting step proposed in the present invention integrates prompt generation and scoring mechanisms, realizes automatic generation and selection of prompts suitable for fuzz testing targets, and improves the efficiency and effectiveness of fuzz testing.

[0065] In the preferred embodiment, the C compiler is selected as the test object. The following is a detailed explanation of the optimal prompt generation process through specific examples. In order to ensure the comprehensiveness and accuracy of the test, the standard specification of the C language is used as the user input, such as Figure 3 This specification describes in detail the core features of the C language, including but not limited to the composition of the standard library, the use of variables, the functions of common functions, and how they are used in actual programming.

[0066] Next, the standard specification and automatic prompt instructions are input into the extraction LLM (GPT-4, calling the interface provided by OpenAI, with max_token set to 500) to generate a candidate prompt set. After obtaining the prompt set, the set is input into the generation LLM (StarCoder, which has been trained on more than one trillion code tokens in more than 80 languages) to generate code snippets. Finally, all code snippets are scored using a scoring function, and the prompt corresponding to the code snippet with the highest score is the optimal prompt, such as Figure 4 Shown is the optimal prompt generated in the first iteration.

[0067] The optimal prompt consists of three parts: feature summary, detailed description, and supplementary explanation. The "feature summary" section provides a concise summary of the core features of the C language standard library, which is a collection of multiple header files that define various functions, macros, and types used in C programming; the "feature summary" section further explains the functions of each key header file in the library and their application in actual programming, providing a detailed explanation of the functions and macros contained in each header file, and how they interact to support complex programming tasks, such as:<stdio.h> Handle input and output operations,<stdlib.h> Provides general utility tools,<string.h> Provides string processing functions,<math.h> Contains commonly used mathematical functions,

[0068] <time.h> Utilities that support time and date,<ctype.h> Determine the character data type,<limits.h> Define the range of integer types,<stdbool.h> The Boolean type and macros were introduced, etc. The supplementary explanation section explains the evolution of the standard library, explaining that with the update of the C language standard, the new version of the standard library introduced new header files and functions to adapt to the ever-changing programming needs and technological developments.

[0069] According to the method of the present invention, the optimal prompt is passed to the generated LLM again to obtain a fuzzy test case for the C compiler. By directly applying the multi-language processing capability of the LLM, a series of fuzzy test cases are generated for the system under test. In addition, the prompt is continuously optimized through an iterative process to further guide the LLM to generate new test cases. Through intelligent selection and feedback loops, the effectiveness and coverage of test cases are gradually improved. Each iteration fine-tunes the prompt based on the previous test results to guide the LLM to generate more accurate and diverse test inputs. This dynamic test case generation mechanism not only improves the efficiency of the testing process, but also significantly enhances the ability to discover potential defects and vulnerabilities.

[0070] In order to verify the feasibility of the fuzzy test case generation method based on a large language model proposed in the present invention, and to comprehensively evaluate the optimization effect of the method proposed in the present invention, C compiler, Java compiler and Go tool chain were selected as the objects under test. Through 24 hours of continuous testing, the performance of the test cases generated by the method of the present invention and the dedicated fuzzy testing tool of the object under test in terms of code coverage was compared. Code coverage is one of the key indicators for measuring the quality of test cases, which reflects the scope of code that the test case can reach. Through this indicator, the potential of different methods in discovering potential defects and improving software quality can be objectively evaluated.

[0071] In the experiment of testing the C compiler, the present invention selected the GCC compiler as the main object of the experiment. In the experimental preparation stage, the official document of the standard C library was preset as user input and provided to the method of the present invention, and Csmith was introduced as a control tool. By comparing with Csmith, the performance of the method of the present invention can be objectively measured. The test results are shown in Figure 2. Figure 5 shown.

[0072] It can be seen that the method proposed in the present invention is significantly better than the optimized Csmith tool. After the experiment was carried out for 24 hours, the method of the present invention and Csmith achieved coverage of 69.8% and 38.2% respectively, and the coverage was increased by 31.6%. It can be seen that the fuzzy test case generation method based on the large language model proposed in the present invention shows good performance in the test process of the C compiler, especially over time, it can continuously generate effective test cases compared to the Csmith tool.

[0073] In the experiment of testing the Java compiler, the present invention selected the Java compiler javac in the OpenJDK project as the core target of the experiment. The latest version of the standard Java API document was used as the user input, and Hephaestus was included in the experiment as a comparison tool. The specific experimental results are shown in the figure. Figure 6 shown.

[0074] In the early stage of the experiment, the code coverage of both methods was low, but the method of the present invention increased the number of valid use cases faster than the Hephaestus tool, which shows that the method of the present invention can generate more effective use cases in the early stage. As the experiment continued, the code coverage of both methods increased, but the growth trend of the method of the present invention seemed to be more significant, indicating that the method of the present invention can generate valid use cases more stably in long-term testing. In addition, at the end of the experiment, the method of the present invention and Hephaestus achieved coverage of 36.1% and 22.5%, respectively, with a coverage increase of 13.6%.

[0075] In the test of Go toolchain, the author selected the official document of Go standard library as the default input of the user. Go-fuzz was used as the control tool. The specific experimental results are as follows: Figure 7 shown.

[0076] As the experiment progressed, the test cases generated by this mutation-based fuzz testing method significantly reduced the growth of code coverage. At the end of the experiment, the final code coverage was 52.4%, which was only 15.6% higher than the initial test case. Although the fuzz test case generation method based on the large language model proposed in the present invention was weaker than the go-fuzz tool in the early stage of the experiment, the coverage index increased rapidly. At the end of the experiment, the code coverage reached 73.2%, which was 20.8% higher than the go-fuzz tool.

[0077] It can be seen that the beneficial effect of the present invention is that the present invention provides a fuzzy test case generation method and system based on a large language model, which understands the description documents or code specifications of the system under test by training a large language model, and then generates more effective test cases, thereby improving the diversity and coverage of test cases. Compared with traditional generation-based or mutation-based fuzz testers, the present invention has higher flexibility and adaptability, and can provide effective fuzz testing support for a variety of programming languages ​​and systems; through intelligent selection and feedback loops, the effectiveness and coverage of test cases are gradually improved, which not only improves the efficiency of the testing process, but also significantly enhances the ability to discover potential defects and vulnerabilities, and improves the security and reliability of software systems.

[0078] The present invention may be a system, a method and / or a computer program product. The present invention also discloses a fuzzy test case generation system based on a large language model based on the aforementioned fuzzy test case generation method based on a large language model, comprising:

[0079] A user input module, used for receiving user input information for describing a target test case;

[0080] A prompt extraction module is used to automatically extract user input information based on a preset extraction model to obtain a fuzzy test prompt;

[0081] A code segment generation module, used for inputting the fuzzy test prompt into a generation model, generating code segments using the generation model and scoring them;

[0082] A case generation module, used to take the highest-scoring code segment as the optimal prompt, input it into the generation model again, and generate a fuzzy test case using the generation model;

[0083] The case set building module is used to generate a predefined number of fuzz test cases and compose all fuzz test cases into a fuzz test case set.

[0084] Based on the spirit of the present invention, those skilled in the art can easily think that a computer program product can be obtained based on the aforementioned fuzzy test case generation method based on a large language model. The computer program product may include a computer-readable storage medium, which carries computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure. That is, the present application also includes a terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the aforementioned fuzzy test case generation method based on a large language model.

[0085] Computer readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device. Computer readable storage medium can be, for example, - but not limited to - electrical storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, punch card or groove protrusion structure with instructions stored thereon and any suitable combination of the above. Computer readable storage medium used here is not interpreted as instantaneous signal itself, such as radio wave or other free propagating electromagnetic wave, electromagnetic wave propagated by waveguide or other transmission medium (for example, light pulse by optical fiber cable) or electrical signal transmitted by wire.

[0086] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0087] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network-including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fuzzy test case generation method based on a large language model, characterized in that: The following steps are involved: Step 1: Receive user input information for describing a target test case; Step 2: Automatically extract user input information based on a preset extraction model to obtain fuzzy test prompts; Step 3: Input the fuzz test prompt into the generation model, use the generation model to generate code segments and perform scoring; Step 4: taking the code segment with the highest score as the optimal hint, inputting it into the generation model again, and generating a fuzzy test case using the generation model; Step 5: Iterate step 2 to step 4 until a predefined number of fuzzy test cases are generated, and all fuzzy test cases are combined into a fuzzy test case set.

2. The method for generating fuzzy test cases based on a large language model according to claim 1, characterized in that: The automatic extraction of user input information based on a preset extraction model to obtain a fuzzy test prompt further includes: First, an automatic prompt instruction is input to the extraction model, and the parameter temp=0 is set, so that the extraction LLM tends to select the output with the highest probability, generates a high-confidence fuzzy test prompt, and formalizes the generated prompt into a conditional probability; The parameter temp is then set to 1 to allow the model to introduce randomness when selecting outputs, and each generated fuzz test hint is added to the candidate list until the preset number of sample candidate fuzz test hints is reached.

3. The method for generating fuzzy test cases based on a large language model according to claim 2, characterized in that: The method of using the generative model to generate code segments and score them further includes: After obtaining the candidate fuzz test hints for the sample number, all generated code segments are scored based on a scoring function, wherein the scoring function is based on the number of unique code segments that are effectively generated, that is, the number of test cases accepted by the target system test case.

4. The method for generating fuzzy test cases based on a large language model according to claim 3, characterized in that: The scoring function is defined as: Σc∈M G (p)[isVaild(c,SUT)] Among them, M G is the generation model, p is the candidate prompt, isValid is the judgment function, and returns 1 if the generated code c is valid, otherwise returns 0.

5. The method for generating fuzzy test cases based on a large language model according to claim 4, characterized in that: The extraction model is GPT-4, and the generation model is StarCoder.

6. A fuzzy test case generation system based on a large language model, characterized in that: include: A user input module, used for receiving user input information for describing a target test case; A prompt extraction module is used to automatically extract user input information based on a preset extraction model to obtain a fuzzy test prompt; A code segment generation module, used for inputting the fuzzy test prompt into a generation model, generating code segments using the generation model and scoring them; A case generation module, used to take the highest-scoring code segment as the optimal prompt, input it into the generation model again, and generate a fuzzy test case using the generation model; The case set building module is used to generate a predefined number of fuzz test cases and compose all fuzz test cases into a fuzz test case set.

7. The fuzzy test case generation system based on a large language model according to claim 6 is characterized in that: The prompt extraction module is further used for: First, an automatic prompt instruction is input to the extraction model, and the parameter temp=0 is set, so that the extraction LLM tends to select the output with the highest probability, generates a high-confidence fuzzy test prompt, and formalizes the generated prompt into a conditional probability; The parameter temp is then set to 1 to allow the model to introduce randomness when selecting outputs, and each generated fuzz test hint is added to the candidate list until the preset number of sample candidate fuzz test hints is reached.

8. The fuzzy test case generation system based on a large language model according to claim 7, characterized in that: The code segment generation module is further used to: After obtaining the candidate fuzz test hints for the sample number, all generated code segments are scored based on a scoring function, wherein the scoring function is based on the number of unique code segments that are effectively generated, that is, the number of test cases accepted by the target system test case.

9. The fuzzy test case generation system based on a large language model according to claim 8, characterized in that: The scoring function is defined as: Σc∈M G (p)[isVaild(c,SUT)] Among them, M G is the generation model, p is the candidate prompt, isValid is the judgment function, and returns 1 if the generated code c is valid, otherwise returns 0.

10. The fuzzy test case generation system based on large language model according to claim 9, characterized in that: The extraction model is GPT-4, and the generation model is StarCoder.

11. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the fuzzy test case generation method based on a large language model according to any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for generating fuzzy test cases based on a large language model described in any one of claims 1 to 5 are implemented.

Citation Information

Cited By

  • Artificial intelligence guidance generation program testing method combined with symbolic execution

    CN119537233A

  • Large-model-assisted program function automatic sensing fuzzy testing method and large-model-assisted program function automatic sensing fuzzy testing system

    CN119690854A

  • Fuzzy test method and device for automatically generating business test case by large model

    CN120066961A

  • Kernel fuzzy test case generation method and device based on shuffling algorithm

    CN120803957A

  • Fuzzy test seed generation method and system based on anti-fact deduction and large language model cooperative driving

    CN121858431A