Fuzz testing method and system based on large model code generation and automatic program repair

Abstract

A fuzz testing method and system for code generation and automatic program repair based on a large model is used to solve the problems of high error rate and high testing cost of code generation during operation of existing large model fuzzers. The present invention includes four stages: prompt word screening, fuzzy loop, program repair and vulnerability detection. A large language model is used as the main generator, and the operator selection problem in the fuzzy loop stage is modeled as a multi-armed tiger problem. The Thompson sampling algorithm is used to improve the diversity and effectiveness of program generation. By performing loop repair on the generated program, program errors caused by the lack of API combination constraints are reduced. The present invention significantly improves the effectiveness of programs generated by fuzzy testing based on large models, solves the problem of high error rate of programs generated by existing fuzzers based on large models, and also has higher code coverage.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a fuzzy testing method and system for code generation and program automatic repair based on a large model. Background Art

[0002] Fuzz testing is an automated testing method used to generate inputs that can cause unexpected behavior in the system under test. Currently, many practical fuzzers have been developed for specific systems under test, such as compilers, constraint solvers, etc. These modules are fundamental components of software development. For example, identifying defects in compilers is crucial because their errors may affect all downstream applications. When testing such systems, a piece of code is usually generated to detect system vulnerabilities by analyzing program crash information.

[0003] Fuzz testing techniques based on code generation, such as MorphQ, first generate initial code using syntax templates and constraints provided by users, then perform semantic transformation on the initial code, and finally perform differential testing on the initial code and the transformed code to detect potential bugs in Qiskit. However, such fuzzers are usually limited by user templates and API constraints, and program generation is limited to a certain range, which significantly reduces the generation capability.

[0004] Recently, researchers have combined large models with fuzz testing to improve the code generation capabilities during fuzz testing. However, existing fuzz testing methods based on large models fail to fully consider the handling of generated erroneous programs, which may result in a large number of false positives and false negatives in the generated programs, thereby adding additional manual verification work. Fuzz4All is the first fuzzer that uses large models for multi-system integrated testing. It automatically generates prompt words through large models and cyclically generates new code to test different systems under test. However, in the experimental tests actually run by the inventors of this case, the error rate of the programs generated by Fuzz4All generally exceeded 50%, and even reached 80% on some systems under test, and the quality of the generated code was low.

[0005] In order to solve the problem of low effectiveness of generated programs, Beijing Yunqi Wuyin Technology Co., Ltd. proposed a method for vulnerability repair verification based on a large model. This method uses LLVM Sanitizer to capture the location of the error and applies the large model to repair it, but in the process of repairing the code, the errors of the system under test itself will be ignored. At the same time, the company proposed another vulnerability code repair method based on a pre-trained large model, which obtains a large model dedicated to repairing program errors by fine-tuning the large model. These two related patents only fix program errors and cover up errors within the system under test, which leads to a situation where the code coverage is high but the bugs of the system under test cannot actually be found, and does not meet the requirements of fuzz testing for the system under test. Summary of the invention

[0006] In view of the problem of high code error rate in fuzz testing of the system under test, the present invention proposes a program repair loop to solve the problem of high error rate and improve the effectiveness of program generation during the fuzz testing process.

[0007] This paper proposes a fuzz testing method based on code generation and automatic program repair based on a large model. The method consists of four stages, namely, the prompt word screening stage, the fuzzy loop stage, the program repair stage, and the vulnerability detection stage. The prompt word screening stage uses the best prompt word generated by the large model; the fuzzy loop stage selects the mutation operator to mutate the prompt word through the Thompson sampling algorithm; the program repair stage repairs the program based on the program generated by the prompt word combined with the runtime error; the vulnerability detection stage detects each generated code to obtain the bug information of the object under test itself.

[0008] In order to achieve the above object, the technical solution specifically adopted by the present invention is a fuzz testing method based on code generation and automatic program repair based on a large model, comprising the following steps:

[0009] (1) Prompt word screening stage: obtain several groups of prompt words provided by the user, generate codes corresponding to each group of prompt words based on the prompt words through the generative large model; score the running results of the code through the scoring function to obtain the score , and based on the ratings Choose the best prompt word;

[0010] (2) Fuzzy loop stage: Generate code based on the best prompt word; in each iteration, if the code generated by the generative large model is error-free, three mutation operators Generate, Mutate, and Semantics are applied to mutate the code, and the Thompson sampling algorithm is used to select the most appropriate operator from the three operators in the current state. Based on the most appropriate operator in the current state, the program code with the highest accuracy and the largest code coverage is generated;

[0011] (3) Program repair phase:

[0012] Determine whether the program code generated in step (2) has an error. If so, use the program code generated in step (2) as the output of step (3). If not, form a code error pair with the program code and the corresponding error, and use the error correction model to correct it. Obtain the corrected code as the output of step (3) to improve the effectiveness of the code.

[0013] (4) Vulnerability detection phase:

[0014] Determine whether the output of step (3) contains a running system vulnerability, and if so, generate a user prompt.

[0015] Preferably, the prompt word screening stage specifically includes the following steps:

[0016] (1-1) Obtain N prompt words provided by the user , the prompt word Includes the language used to generate the code, the relevant syntax or library functions used to generate the code, and a description of the system under test;

[0017] (1-2) Using a large generative model will generate N prompt words Generate programs separately , each prompt word will generate M programs , the number of codes that can run normally in these M codes is taken as the score of each prompt word, and the specific score The calculation formula is as follows:

[0018]

[0019] in, Indicates The score of the group code, represents the jth program, Indicates the return value of the program. ;

[0020] (1-3) By scoring N codes Sort and get the prompt word with the highest score If there are multiple clue words with the highest score and the same score, then according to the clue word number order Select from small to large.

[0021] Preferably, the fuzzy cycle stage is specifically implemented as follows:

[0022] (2-1) After selecting the best prompt word After that, the prompt word will be used as the initial prompt word Participate in the fuzzy loop stage and pass the large model Generate the corresponding code ,If an error occurs when the code is running, the program repair phase will be entered, and if the code does not report an error, the fuzzy loop phase will continue.

[0023] (2-2) Define the mutation operator so that each round of the fuzzy loop can generate inconsistent codes. In the fuzzy loop stage, three different mutation operators are set to mutate the prompt words in each round to prompt the large model to generate more diverse codes. The three operators defined in this scheme are specifically:

[0024] The Generate operator is used to prompt the large model to generate code; in the first round of fuzzy loop, the Generate operator is used to generate the initial code; in each subsequent round of fuzzy loop, a random selection is made from the three mutation operators according to the optimization algorithm to achieve further code mutation.

[0025] The Mutate operator is used to mutate the code generated by the previous round of fuzzy loop; this operator mainly fine-tunes the original code, such as inserting other functions into the code or modifying parameters.

[0026] The Semantics operator is mainly used to perform semantic transformation on the code, that is, to re-implement a section of code in another way while keeping the function and role of the code unchanged.

[0027] (2-3) Selecting mutation operators: by converting the problem of mutation operator selection into a multi-armed bandit problem, and using the Thompson sampling algorithm to select the best operator for each round of fuzzy loop; and mutating the prompt word according to the content of the operator;

[0028] The bonus value of each slot machine The distribution of , indicating that in action (Pull or not pull) case, get reward value But this possibility is unknown, so at each time step , the agent will pull an arm and observe the output derived from the Bernoulli distribution, thus obtaining a reward value distribution.

[0029] For a multi-armed bandit problem, we can use the standard distribution as the prior probability on each arm. In the case of standard The probability density function of the distribution is

[0030]

[0031] In formula (2), ,in is the gamma function, since the prior probability satisfies distribution, so the posterior probability also satisfies distribution, and or It increases by 1 with each success or failure of an observation. Each operator can be regarded as an arm of a multi-armed slot machine. After the operator is selected to generate a program, the execution of the program will be recorded, and the success or failure of the program will be fed back to the next cycle.

[0032] (2-4) Prompt word variation: The best prompt word obtained in the prompt word screening stage On this basis, the best operator selected by the Thompson sampling algorithm is added, and the code generated by the large model in this cycle is used as a new prompt word for the next cycle.

[0033] Preferably, the program repair phase is specifically implemented as follows:

[0034] (3-1) First, run the program code generated in the last iteration of step (2) on the system under test, and collect and save the error information obtained; the error information includes the undefined function used and the undefined type used; if there is no error, use the program code generated in step (2) as the output of the program repair phase,

[0035] (3-2) Then, the program code and the corresponding error are constructed into a code error pair, and a prompt word of the error correction model is constructed based on the code error pair, wherein the prompt word of the error correction model includes the program code, a separator for separating the code and the error information, and the corresponding error;

[0036] (3-3) Setting relevant parameters of the error correction model, wherein the relevant parameters include p, max_length, temperature and context length;

[0037] Input the prompt word obtained in (3-2) into the error correction model to obtain the repaired code output by the error correction model, and determine whether the maximum number of iterations in the program repair phase has been reached. If not, repeat steps (3-1) to (3-3).

[0038] Preferably, the vulnerability detection stage is specifically implemented as follows:

[0039] (4-1) The output of the program repair phase is put into the system under test for execution, and the information generated during the code execution is recorded on the hard disk in the form of a log file.

[0040] (4-2) Regularly scan the code running information in the log file generated by (4-1), match the text of the running information with regular expressions, and determine whether there are characters related to the bug report. If so, send the bug report to the user. The regular expression matching method is used because the system under test often has a unique reporting method for its own bugs, which generates special bug report text.

[0041] Preferably, the value of N is greater than 3, and the value of M is greater than 30.

[0042] Preferably, the value of kernel sampling p is 1.0, the maximum length of generated text max_length is 512, the parameter temperature used to control the randomness and diversity of generated text is 1.0, and the context length is less than 5000. The limitation of context length is to avoid the memory usage exceeding the upper limit of the graphics card due to the error message being too long, thereby interrupting the program operation;

[0043] The present invention also provides a protocol fuzz testing system based on multi-dimensional feedback information, the system is used to run the fuzz testing method for code generation and automatic program repair based on a large model, including:

[0044] The prompt word screening module is used to: obtain several groups of prompt words provided by the user, generate codes corresponding to each group of prompt words based on the prompt words through the generative large model; score the running results of the code through the scoring function to obtain the score , and based on the ratings Choose the best prompt word;

[0045] The fuzzy loop module is used to: generate code based on the best prompt word; in each iteration, if the code generated by the generative large model is error-free, three mutation operators Generate, Mutate and Semantics are applied to mutate the code, and the Thompson sampling algorithm is used to select the most appropriate operator from the three operators in the current state. Based on the most appropriate operator in the current state, the program code with the highest accuracy and the largest code coverage is generated;

[0046] The program repair module is used to: determine whether the program code generated in step (2) has an error. If so, use the program code generated in step (2) as the output of step (3). If not, use the program code and the corresponding error to form a code error pair, and use the error correction model to correct it, and obtain the corrected code as the output of step (3) to improve the effectiveness of the code;

[0047] (4) Vulnerability detection phase module, used to determine whether the output of step (3) contains a system vulnerability in operation, and if so, generate a user prompt.

[0048] In order to ensure the generation capability of the fuzzer, the present invention adopts a large language model as the main generator, models the operator selection problem in the fuzzy loop stage as a multi-armed bandit problem, and uses the Thompson sampling algorithm to improve the diversity and effectiveness of program generation. In order to improve the effectiveness of code generation, the present invention adds a program repair stage to perform loop repair on the generated program, thereby reducing program errors caused by the lack of API combination constraints. The present invention significantly improves the effectiveness of programs generated by fuzz testing based on large models, solves the problem of high error rate of existing fuzzer-generated programs based on large models, and also has higher code coverage.

[0049] The present invention proposes a general fuzz testing tool for code generation and program repair based on a large model, which has the following characteristics and beneficial effects compared with other fuzz testing tools:

[0050] 1) The present invention adopts the Thompson sampling algorithm in the fuzzy loop stage to achieve diversified program generation, thereby enhancing the testing capability of the object under test.

[0051] 2) In the error correction loop, the present invention improves the effectiveness of the generated code by providing the code and its error information and using LLM for accurate correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is the overall flow chart of the present invention;

[0053] Figure 2 This is a graph of gcc coverage changing over time; this graph is the result of taking the average of the code line coverage numbers obtained by performing five fuzz tests (each time for 24 hours) on LMFuzz (the present invention), Fuzz4All (the latest large model fuzzer), GrayC, and CSmith (traditional fuzzer).

[0054] Figure 3This is a graph of g++ coverage changing over time; this graph is the result of taking the average of the code line coverage numbers obtained by performing five fuzz tests (each time for 24 hours) on LMFuzz (the present invention), Fuzz4All (the latest large model fuzzer), and Yarpgen (the traditional method fuzzer).

[0055] Figure 4 This is a graph of go coverage changing over time; this graph is the result of taking the average of the code line coverage numbers obtained by performing five fuzz tests (each time for 24 hours) using LMFuzz (the present invention), Fuzz4All (the latest large model fuzzer), and go-fuzz (a traditional method fuzzer).

[0056] Figure 5 This is a graph of cvc5 coverage changing over time; this graph is the result of taking the average of the code line coverage numbers obtained by performing five fuzz tests (each time for 24 hours) on LMFuzz (the present invention), Fuzz4All (the latest large model fuzzer), and TypeFuzz (the traditional method fuzzer).

[0057] Figure 6 This is a graph of qiskit coverage changing over time; this graph is the result of taking the average of the code line coverage numbers obtained by performing five fuzz tests (each time for 24 hours) on LMFuzz (the present invention), Fuzz4All (the latest large model fuzzer), and MorphQ (the traditional method fuzzer).

[0058] Figure 7 This is a comparison chart of the generation capabilities of the Thompson sampling algorithm TS and the random algorithm Random. This chart shows the corresponding code line coverage numbers obtained by LMFuzz (the present invention) using five languages ​​for generation when using the Thompson sampling algorithm and the random algorithm. DETAILED DESCRIPTION

[0059] The present invention will be described systematically and completely below in conjunction with the accompanying drawings of the present invention.

[0060] 1. Prompt word screening stage

[0061] The prompt word screening stage of the present invention specifically includes the following steps:

[0062] (1-1) The user prepares N (N can be 3) prompt words, which need to provide clear generation information, including the language of the generated code, the relevant syntax or library functions that can be used to generate the code, and the relevant overview of the system under test.

[0063] (1-2) The generative model will generate programs for the three prompts given by the user. Each prompt will generate M (the value of M can be 30) programs. The number of codes that can run normally in these M codes is used as the score of each prompt. The specific scoring formula is as follows:

[0064]

[0065] in, Indicates The score of the group code, Indicates the jth ) programs, Indicates the return value of the program.

[0066] (1-3) Sort the scores of the N codes and select the prompt word with the highest score as the initial prompt word for the next stage. If there are multiple prompt words with the same highest score, select them in ascending order according to the prompt word number.

[0067] 2. Fuzzy cycle stage

[0068] The fuzzy cycle stage of the present invention specifically includes the following steps:

[0069] (2-1) After selecting the best prompt word, the corresponding code is generated through the large model. If an error occurs when the code is running, the program repair phase is entered. If the code does not report an error, the fuzzy loop phase continues.

[0070] (2-2) Define the mutation operator. In order to enable each round of the fuzzy loop to generate inconsistent codes, a mutation operator is set in the fuzzy loop stage to mutate the prompt words in each round, which is used to prompt the large model to generate more diverse codes. The three operators defined in this scheme include several specific ones. The specific implementation of each mutation operator is as follows:

[0071] Generate: The Generate operator is used to prompt the large model to generate code. In the first round of fuzzy loop, this operator is selected to generate the initial code. In each subsequent round of fuzzy loop, it is randomly selected from the three mutation operators according to the optimization algorithm to achieve further code mutation.

[0072] Mutate: The Mutate operator is used to mutate the code generated by the previous fuzzy cycle. The code generated after the previous fuzzy cycle is used as the original code, and the Mutate operator is used to fine-tune the original code, such as inserting other functions into the code or modifying parameters.

[0073] Semantics: Semantics operators are mainly used to perform semantic transformation on code. Semantic transformation means that while transforming the entire code, the functionality of the code remains unchanged. For example, there is a code that first customizes two complex number classes, then initializes them and adds them together. Now the result of the transformation can be to use the complex library to initialize two complex number variables, and then add them together. In this process, although the entire code has changed, the implemented functionality remains unchanged.

[0074] (2-3) Select the mutation operator by converting the mutation operator selection problem into a multi-armed bandit problem and using the Thompson sampling algorithm to select the best operator for each round of fuzzy loop. And mutate the prompt word according to the content of the operator.

[0075] (2-4) Prompt word variation. Based on the best prompt word obtained in the prompt word screening stage, the best operator selected by the Thompson sampling algorithm is added, and the code generated by the large model in this cycle is added to form a new prompt word for the next cycle.

[0076] The pseudo code of the entire fuzzy loop is as follows:

[0077]

[0078] Algorithm 1 describes the main fuzzy loop algorithm of the present invention. First, initialize the program execution limit time LimitTime to TimeX (TimeX can be 86400) seconds (line 1), the current running time Time to 0 (line 2), the number of loops LoopCount to -1 (line 3), and assign an initial value to the prompt word used in the loop (line 4). The exit condition of the loop is whether Time exceeds LimitTime, so the fuzzy loop is continuously executed (line 5). During the fuzzy loop process, first record the current timestamp (line 7) and the current number of loops (line 8), and then generate the code for the large model output through the prompt word (Line 10). Next, check whether there are any errors in the code (Line 12) and correct them. The specific correction process will be described in detail in Section 3.3 (Line 14). Then, take the generated code and prompt words as input, use the mutation operator and Thompson sampling algorithm to mutate the prompt words, so as to generate new prompt words for the next cycle (Line 17). Finally, update the total time spent (Line 18) and save the generated code (Line 19).

[0079] 3. Program repair phase

[0080] The program repair phase of the present invention specifically includes the following steps:

[0081] (3-1) First, run the erroneous code in the system under test, and collect and save the error information. The specific steps of this section are as follows:

[0082] (3-1-1) Get the program generated during the fuzzy loop process and run the program in the specified system under test. Since the programs that enter this stage are only those that generate errors, it is certain that the bugs generated when the program is running can be captured. The specific example is as follows:

[0083] Suppose code a is a program for outputting two random numbers, but the time library is used but not imported, resulting in an error.

[0084] Assume that code b is a program for adding two complex numbers, in which the complex class is used but the complex library is not imported, resulting in an error.

[0085] (3-1-2) At this time, the present invention will capture the error information that appears in the program. In code a, the present invention will capture two undefined function errors. In code b, the present invention will capture two errors caused by the inability to recognize the complex class.

[0086] (3-2) Then, the error message and the corresponding code are combined into a complete prompt word through the prompt word template. The specific steps of this section are as follows:

[0087] (3-2-1) Give a prompt word to make the large model more clear about its task. The prompt word used in this invention is "Your task now is to fix the bug in the code. Code and its error are separated

[0088] by token <code2err>. ". Using this prompt can not only make the large model clear about its own repair tasks, but also clearly point out the separator text. <code2err>, avoiding large models confusing codes, delimiters, and error messages.

[0089] (3-2-2) Next, the present invention adds the code generated in the previous stage after the prompt word. The code will serve as the fundamental basis for the error correction stage. The error correction model will modify the code based on the error information to make the code execute correctly as much as possible.

[0090] (3-2-3) Then, the present invention adds a separator after the code, and the separator is the text ` <code2err>`, of course, different users can also change the setting when using the invention, as long as the purpose of segmenting the code and the error message can be achieved with less text.

[0091] (3-2-4) Finally, the present invention adds the error information captured in (3-1) after the separator. The error information will point out the error in the corresponding code, improving the error correction model's ability to locate the error in the code.

[0092] (3-2-5) At this point, the prompt word construction of the error correction model is completed, and its structure is prompt word, code, separator and error message in order.

[0093] (3-3) Input the complete error code and error message into the large model to obtain the repaired code output by the large model. The specific steps of this section are as follows:

[0094] (3-3-1) First, use the prompt word output in (3-2) as the input of this stage

[0095] (3-3-2) Then, set the parameters related to the large model, including p, max_length, temperature and context length. (In the present invention, p can be 1.0, max_length can be 512, temperature can be 1.0, and context length can be 5000 characters). The limitation of context length is to avoid the error message being too long, causing the video memory to exceed the upper limit of the graphics card, thereby interrupting the program operation.

[0096] (3-3-3) Thus, if the generated code after the correction is still wrong and the number of loops does not exceed the maximum number of loops, this code will enter the next program repair phase.

[0097] Taking code a and code b in (3-1) as an example, after the automatic program repair of the large model, the large model adds the missing header file to the wrong code a.<stdio.h> , added the missing header file for the wrong code b<time.h> To solve the problems in these two sample codes.

[0098] In this way, LMFuzz provides an opportunity for code with simple errors to re-enter the fuzz loop, thereby improving the effectiveness and diversity of the entire fuzz test. It is worth noting that LMFuzz can not only fix errors caused by missing header files, but also handle problems such as undefined variables, syntax errors, and mismatched function parameters.

[0099] The pseudo code of the entire program repair is as follows:

[0100]

[0101] Algorithm 2 shows the pseudocode implementation of program repair, which mainly involves the specific implementation of the CorrectError function in Algorithm 1. First, initialize the counter Count to 0 (line 2). If there is an error in the current program and the number of corrections has not reached the set upper limit (line 3), the error information when the program is running will be obtained (line 4) and formed into a Code-Error pair together with the program code (line 5). Subsequently, the counter Count is increased by 1 (line 6), and the error correction prompt word is combined with the Code-Error pair and passed to the large model for code correction (line 7). If the program is still incorrect and the number of loops has not reached the upper limit, the process will continue.

[0102] 4. Vulnerability detection stage

[0103] (4-1) After the program is successfully repaired or the number of repairs reaches the upper limit, the vulnerability detection phase begins. The code output in the vulnerability repair phase is received, put into the system under test for execution, and the information generated during the code execution is recorded on the hard disk in the form of a log file.

[0104] (4-2) Regularly scan the code running information in the log file generated by (4-1). Here, the text of the running information is matched by regular expressions to observe whether specified characters are generated. The regular expression matching method is adopted because the system under test often has a unique reporting method for its own bugs, which generates special bug report text. For example, the bug report information of GCC often contains the keyword "internal compiler error".

[0105] (4-3) After successfully matching the regular expression, the present invention will send a bug report to the user's mailbox (which needs to be set in advance) to remind the user to provide feedback and fix the bug.

[0106] The above specific implementation modes are only used to further illustrate the present invention, and are not the only implementation modes of the present invention, and should not be understood as limiting the present invention.

[0107] Experimental results:

[0108] Table 1: Comparison between LMFuzz and other fuzzers

[0109]

[0110] Experiment 1: comparative experiment. Figure 7 The line coverage of LMFuzz, Fuzz4All, and different traditional fuzzers corresponding to each system under test changes over time within 24 hours on 5 systems under test using different languages. In order to reduce the randomness of fuzz testing and large model code generation, 5 24-hour experiments are generally conducted, and the average of the results is taken for display.

[0111] Figure 2 In the figure, ① represents the curve of LMFuzz, ② represents the curve of Fuzz4All, ③ represents the curve of GrayC, and ④ represents the curve of Csmith.

[0112] Figure 3 In the figure, ① represents the curve of LMFuzz, ② represents the curve of Fuzz4All, and ③ represents the curve of YarpGen.

[0113] Figure 4 In the figure, ① represents the curve of LMFuzz, ② represents the curve of TypeFuzz, and ③ represents the curve of Fuzz4All.

[0114] Figure 5 In the figure, ① represents the curve of LMFuzz, ② represents the curve of go-fuzz, and ③ represents the curve of Fuzz4All.

[0115] Figure 6 In the figure, ① represents the curve of LMFuzz, ② represents the curve of Fuzz4All, and ③ represents the curve of MorphQ.

[0116] Table 1 shows the comparison of the effectiveness and line coverage of the code generated by LMFuzz and other fuzzers within 24 hours. It can be found that the program efficiency of LMFuzz is about twice that of Fuzz4All in these five projects, which shows that the method can indeed generate more efficient code. Although the efficiency is not as good as traditional generative fuzzers and mutation fuzzers, the line coverage of the program generated by the method exceeds that of traditional fuzzers in all projects, and far exceeds that of traditional fuzzers in several individual projects (GCC, G++, Qiskit).

[0117] Experiment 2: Ablation experiment of Thompson sampling algorithm

[0118] Table 2 Comparison of generation capabilities (TS algorithm and random algorithm)

[0119]

[0120] Here, we mainly compare the generation capabilities provided by the random algorithm and the Thompson sampling algorithm in the fuzzy loop. As shown in Table 2, TS is the abbreviation of the Thompson sampling algorithm (Thompson sampling) used to solve the multi-armed bandit problem, and Random represents the random selection algorithm. Through observation, it is found that on all the tested objects, LMFuzz using the TS algorithm can obtain more coverage lines than LMFuzz using the random algorithm, which proves that using the multi-armed bandit algorithm in the fuzzy loop stage can generate more diverse programs.

[0121] Experiment 3: Ablation of program repair loop

[0122] Table 3 Ablation of error correction cycles

[0123]

[0124] We observed that without program repair, the programs generated by LMFuzz not only have low efficiency but also insufficient code coverage. However, after two cycles, although the efficiency of the generated code is significantly improved, the code coverage is reduced compared to when only one error correction is performed. This is because increasing the number of repair rounds will extend the time of each fuzz cycle, thereby reducing the number of programs generated in a limited time, which in turn leads to a decrease in code coverage and program diversity. Therefore, there is no linear relationship between the efficiency and diversity of the code. It is necessary to control the number of repairs to maintain the diversity of the program while improving the efficiency. As shown in Table 3, for most of the tested objects, after one repair, the generated program of LMFuzz has sufficient diversity and maintains a high efficiency.

[0125] Experiment 4: Bug finding capability. Table 4 summarizes the bugs found by LMFuzz on several systems we studied. These results not only illustrate the actual ability of LMFuzz in finding bugs, but also show that our LMFuzz has a certain versatility and can be widely applied to most systems. Among them, Total refers to the total number of bugs found by LMFuzz for each tested system, Unknown indicates the number of bugs that the author has confirmed as previously undiscovered, Known indicates the number of bugs that have been fixed and discovered, and Pending indicates the number of bug reports to which the author has not yet responded.

[0126] Table 4 Bugs found by LMFuzz

[0127]

[0128] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A fuzz testing method based on large model code generation and automatic program repair, characterized in that: The following steps are involved: (1) Prompt word screening stage: obtain several groups of prompt words provided by the user, generate codes corresponding to each group of prompt words based on the prompt words through the generative large model; score the running results of the code through the scoring function to obtain the score , and based on the ratings Choose the best prompt word; (2) Fuzzy loop stage: Generate code based on the best prompt word; In each iteration, if the code generated by the generative large model is error-free, three mutation operators Generate, Mutate and Semantics are applied to mutate the code, and the Thompson sampling algorithm is used to select the most appropriate operator in the current state from the three operators. Based on the most appropriate operator in the current state, the program code with the highest accuracy and the largest code coverage is generated; (3) Program repair phase: Determine whether the program code generated in step (2) has an error. If so, use the program code generated in step (2) as the output of step (3). If not, form a code error pair with the program code and the corresponding error, and use the error correction model to correct it. Obtain the corrected code as the output of step (3) to improve the effectiveness of the code. (4) Vulnerability detection phase: Determine whether the output of step (3) contains a running system vulnerability, and if so, generate a user prompt.

2. A fuzz testing method for code generation and automatic program repair based on a large model according to claim 1, characterized in that: The prompt word screening stage specifically includes the following steps: (1-1) Obtain N prompt words provided by the user , the prompt word Includes the language used to generate the code, the relevant syntax or library functions used to generate the code, and a description of the system under test; (1-2) Using a large generative model will generate N prompt words Generate programs separately , each prompt word will generate M programs , the number of codes that can run normally in these M codes is taken as the score of each prompt word, and the specific score The calculation formula is as follows: ； in, Indicates The score of the group code, represents the jth program, Indicates the return value of the program. ; (1-3) By scoring N codes Sort and get the prompt word with the highest score If there are multiple clue words with the highest score and the same score, then according to the clue word number order Select from small to large.

3. A fuzz testing method for code generation and automatic program repair based on a large model according to claim 1, characterized in that: The fuzzy loop stage is specifically implemented as follows: (2-1) After selecting the best prompt word After that, the prompt word will be used as the initial prompt word Participate in the fuzzy loop stage and pass the large model Generate the corresponding code ,If an error occurs when the code is running, the program repair phase will be entered, and if the code does not report an error, the fuzzy loop phase will continue; (2-2) Define the mutation operator. The Generate operator is used to prompt the large model to generate code. In the first round of fuzzy loop, the Generate operator is used to generate the initial code. In each subsequent round of fuzzy loop, the optimization algorithm randomly selects from the three mutation operators to achieve further code mutation. The Mutate operator is used to mutate the code generated by the previous fuzzy cycle; The Semantics operator is mainly used to perform semantic transformation on the code, keeping the functions and effects implemented by the code unchanged; (2-3) Selecting mutation operators: by converting the problem of mutation operator selection into a multi-armed bandit problem, and using the Thompson sampling algorithm to select the best operator for each round of fuzzy loop; and mutating the prompt word according to the content of the operator; (2-4) Prompt word variation: The best prompt word obtained in the prompt word screening stage On this basis, the best operator selected by the Thompson sampling algorithm is added, and the code generated by the large model in this cycle is used as a new prompt word for the next cycle.

4. A fuzz testing method for code generation and automatic program repair based on a large model according to claim 1, characterized in that: The specific implementation of the program repair phase is as follows: (3-1) First, run the program code generated in the last iteration of step (2) on the system under test, and collect and save the error information obtained; the error information includes the undefined function used and the undefined type used; if there is no error, use the program code generated in step (2) as the output of the program repair phase, (3-2) Then, the program code and the corresponding error are constructed into a code error pair, and a prompt word of the error correction model is constructed based on the code error pair, wherein the prompt word of the error correction model includes the program code, a separator for separating the code and the error information, and the corresponding error; (3-3) Setting relevant parameters of the error correction model, wherein the relevant parameters include p, max_length, temperature and context length; Input the prompt word obtained in (3-2) into the error correction model to obtain the repaired code output by the error correction model, and determine whether the maximum number of iterations in the program repair phase has been reached. If not, repeat steps (3-1) to (3-3).

5. A fuzz testing method for code generation and automatic program repair based on a large model according to claim 1, characterized in that: The vulnerability detection phase is specifically implemented as follows: (4-1) Put the output of the program repair phase into the system under test and run it, and record the information generated during the code running in the form of a log file on the hard disk; (4-2) Regularly scan the code execution information in the log file generated by (4-1), match the text of the execution information with regular expressions, and determine whether there are characters related to the bug report. If so, send a bug report to the user.

6. A fuzz testing method for code generation and automatic program repair based on a large model according to claim 2, characterized in that: The value of N is greater than 3, and the value of M is greater than 30.

7. A fuzz testing method for code generation and automatic program repair based on a large model according to claim 3, characterized in that: The value of kernel sampling p is 1.0, the maximum length of generated text max_length is 512, the parameter temperature used to control the randomness and diversity of generated text is 1.0, and the context length is less than 5000.

8. A protocol fuzzy testing system based on multi-dimensional feedback information, characterized in that: The system is used to run the fuzz testing method for code generation and automatic program repair based on a large model as described in claim 1, comprising: The prompt word screening module is used to: obtain several groups of prompt words provided by the user, generate codes corresponding to each group of prompt words based on the prompt words through the generative large model; score the running results of the code through the scoring function to obtain the score , and based on the ratings Choose the best prompt word; The fuzzy loop module is used to: generate code based on the best prompt word; in each iteration, if the code generated by the generative large model is error-free, three mutation operators Generate, Mutate and Semantics are applied to mutate the code, and the Thompson sampling algorithm is used to select the most appropriate operator from the three operators in the current state. Based on the most appropriate operator in the current state, the program code with the highest accuracy and the largest code coverage is generated; The program repair module is used to: determine whether the program code generated in step (2) has an error. If so, use the program code generated in step (2) as the output of step (3). If not, use the program code and the corresponding error to form a code error pair, and use the error correction model to correct it, and obtain the corrected code as the output of step (3) to improve the effectiveness of the code; (4) Vulnerability detection phase module, used to determine whether the output of step (3) contains a system vulnerability in operation, and if so, generate a user prompt.

Images