An integrated fuzz testing method and system for automatically selecting fuzzer combinations

Abstract

The present invention proposes an integrated fuzz testing method and system for automatically selecting a fuzzer combination. The present invention is divided into a training phase and a testing phase. The training phase includes: first, on each selected program to be tested, the performance of each optional fuzzifier on the program to be tested is tested in turn; then, based on the performance of each optional fuzzifier used alone on each program to be tested, the performance expectations of various fuzzer combinations on each fuzzer are calculated respectively; finally, a code classification network including a feature extraction part and a classification part is constructed. The testing phase includes: inputting the program to be tested into the code classification network to obtain a suitable fuzzer combination; then, using this fuzzer combination to perform parallel testing on the program to be tested, and after the test is completed, all test results are collected and deduplicated. The present invention can automatically select a suitable fuzzer combination for the program to be tested, realize integrated testing of heterogeneous fuzzers in a parallel manner, and improve fuzz testing performance.

Description

Technical Field

[0001] The present invention relates to the field of program testing technology, and is an integrated fuzz testing method and system for automatically selecting a fuzzer combination, and in particular to an integrated fuzz testing method and system for using machine learning to assist in selecting a suitable fuzzer. Background Art

[0002] Fuzz testing is a testing technique that automatically discovers vulnerabilities by generating randomly mutated inputs and feeding them into the program under test, then observing the program's behavior. It has become a crucial technique for finding software errors and vulnerabilities. In recent years, open-source fuzz testing tools such as AFL and HonggFuzz have been successfully used in practice.

[0003] Currently, commonly used software fuzzers are primarily coverage-guided, generating input sets that maximize code coverage. This requires generating a large number of test cases and inputting them into the target program to collect feedback, a task that consumes significant CPU resources. Furthermore, the ever-increasing size of project software code also increases the computing resources and testing time required for fuzz testing. Therefore, it is necessary to improve testing efficiency through fuzzer parallelism, that is, running multiple fuzzer instances simultaneously.

[0004] To this end, popular fuzzers (such as AFL) all come with a built-in parallel mode. In this mode, multiple fuzzer instances share a seed corpus to synchronize their work. However, while using simple parallel methods can indeed increase code coverage, it cannot solve the inherent limitations of specific fuzzers. For example, a single instance of AFL has difficulty solving the problem of matching magic bytes (a fixed address, fixed number of bits of checksum that can be used to identify key information such as file type), and this problem still exists even when using multiple AFL instances in parallel.

[0005] By observing various test results on the fuzz testing platform benchfuzz and our test results (such as Figure 1 The performance of different fuzzers on different test programs is affected by the characteristics of the target programs, indicating that different fuzzers have their own strengths and weaknesses. Therefore, selecting a suitable combination of heterogeneous fuzzers and employing a parallelized integrated fuzz testing approach can achieve better overall performance than a single parallelized fuzzer.

[0006] <<EnFuzz:Ensemble Fuzzing with Seed Synchronization among DiverseFuzzersEnFuzz> EnFuzz, an integrated fuzz testing solution for heterogeneous fuzzers, was proposed. By manually selecting a group of well-performing heterogeneous fuzzers for parallelized integrated testing, code coverage was increased and testing efficiency was improved. The integrated fuzz testing method proposed in CN110633201A builds on EnFuzz by adding test input file synchronization and result integration mechanisms.

[0007] However, manually selecting the right fuzzer combination requires a significant amount of prior knowledge. Testers need a deep understanding of the internal characteristics of each fuzzer, as well as the characteristics of the program under test. Therefore, manually selecting a fuzzer combination is difficult and time-consuming, reducing the overall efficiency of the testing process.

[0008] <<CUPID:Automatic Fuzzer Selection for Collaborative Fuzzing> The proposed integrated fuzz testing framework CUPID performs a single evaluation on a specific set of target programs (hereinafter referred to as "test program," "target program," and "tested target"), selects a set of fuzzers with good average performance, and then consistently uses this set of fuzzers for all subsequent tests of all other target programs. However, this approach cannot adaptively select fuzzer combinations based on the characteristics of the target program under test, and cannot guarantee good performance on all target programs under test.

[0009] In summary, integrated fuzz testing currently suffers from the inability to automatically select the appropriate fuzzer combination based on the characteristics of the target program being tested. This paper proposes a method that uses a neural network to classify the program under test and automatically selects a fuzzer combination that adapts to the characteristics of the program under test, thereby achieving better fuzz testing performance and improving testing efficiency. Summary of the Invention

[0010] The present invention proposes an integrated fuzz testing method and system for automatically selecting fuzzer combinations. The present invention is divided into a training phase and a testing phase. The training phase is divided into three steps: first, on each selected program to be tested, the performance of each optional fuzzer on the program to be tested is tested in turn, such as Figure 1As shown in the figure, the performance of each optional fuzzer used individually on each program under test is calculated based on the performance of each fuzzer combination on each fuzzer. Finally, a code classification network is constructed, which includes a feature extraction part and a classification part. The network is first trained using a public code classification dataset, and then a transfer learning method is used to enable it to provide a suitable fuzzer combination based on the characteristics of the program under test. The testing phase is divided into two steps: first, the program under test is input into the code classification network to obtain a suitable fuzzer combination; then, this fuzzer combination is used to parallelize the test program. After the test is completed, all test results are collected and deduplicated.

[0011] Specifically, the method steps of the training phase and the testing phase of the present invention are as follows:

[0012] 1. Training phase.

[0013] First, we collect performance data from each candidate fuzzer run individually on each program in a specific test set. Then, based on this performance data, we calculate the mathematical expectation of the number of branches covered by the parallel fuzzer combinations, thereby predicting the performance of various fuzzer combinations on each test program. Finally, we construct and train a code classification network and use transfer learning to enable it to predict the fuzzer combination that is suitable for the test program.

[0014] Specifically, the training phase can be divided into three steps: evaluating the performance of a single fuzzer, predicting the performance of a fuzzer combination, and building and training a classification network for the program under test.

[0015] 1-1 Evaluating the performance of a single fuzzer

[0016] For the optional fuzzer set, each optional fuzzer is tested one by one on the selected typical fuzz test program set, and then the test performance data of each fuzzer (including the number of covered branches and covered paths) is collected.

[0017] The input of this step is the list of optional fuzzers and the list of programs to be tested, and the output is the test performance data such as the coverage bitmap and the number of coverage paths obtained by using each optional fuzzer to test each program to be tested separately.

[0018] Furthermore, the workflow for evaluating the performance of a single fuzzer is as follows:

[0019] 1-1-1 selects the programs to be tested in the given list of programs to be tested in turn. If all the programs to be tested are tested, the 1-1 operation ends;

[0020] 1-1-2 Perform a fuzz test on the target program using the basic fuzzer AFL to prepare the initial seed for subsequent fuzzer performance evaluation;

[0021] 1-1-3 selects fuzzy fuzzers from the given optional fuzzy fuzzer list in turn. If all fuzzy fuzzers have been selected, return to step 1-1-1;

[0022] 1-1-4 Use the seed obtained in step 1-1-2 as the initial seed and perform a fuzz test on the target program using the selected fuzzer. To reduce random errors, this step needs to be repeated multiple times (the specific number of repetitions can be determined according to the actual test conditions);

[0023] 1-1-5 Run all the seeds obtained from each evaluation in step 1-1-4 on the corresponding target program once to obtain a branch coverage bitmap and a path coverage count. Each combination of branch coverage bitmap and path coverage count will be saved as the performance data of a single fuzzer on a specific target program for subsequent use in predicting the performance of various fuzzer combinations.

[0024] 1-2 Predicting the test performance of fuzzer combinations on various programs under test

[0025] Based on the coverage bitmaps obtained by testing each optional fuzzifier on each target program, the probability that each branch of the target program is covered by at least one fuzzifier is calculated when various fuzzer combinations are tested in parallel. Then, the performance score of the fuzzer combination on the target program is estimated based on the mathematical expectation of the number of branches covered on the target program.

[0026] The input of this step is the test performance data of each optional fuzzer on each program to be tested, the coverage bitmap and the number of coverage paths obtained in step 1-1, and the number of fuzzers required to be run in parallel during the test; the output is a table of predicted performance scores for each fuzzer combination on each program to be tested.

[0027] Furthermore, the workflow for predicting the performance of the fuzzer combination on each program under test is as follows:

[0028] 1-2-1 selects each program to be tested in the given fuzz test program list in turn. If all programs to be tested have been selected, step 1-2 ends.

[0029] 1-2-2 Read all branch coverage bitmaps and the number of covered paths obtained by each optional fuzzer on the program under test in step 1-1 (evaluating the performance of a single fuzzer) one by one;

[0030] 1-2-3 Select the fuzzer with the largest average number of covered paths in each test on the program to be tested;

[0031] 1-2-4 Use the branch coverage bitmap to calculate the probability that various fuzzifier combinations (the number of fuzzifiers in the fuzzifier combination is the input value) including the fuzzifier selected in step 1-2-3 cover each branch, and then calculate the mathematical expectation of the number of branches covered on the program under test;

[0032] 1-2-5 takes the mathematical expectation value of the number of covered branches calculated in step 1-2-4 as the performance score of the fuzzifier combination on the program to be tested, and gives a performance score table of all fuzzifier combinations on the program to be tested.

[0033] 1-3 Build and train the test program classification network

[0034] The main task of this step is to build and train a neural network that can classify the program under test according to its characteristics, and thus automatically give a fuzzifier combination suitable for the program under test.

[0035] Build a classification network: Use the backbone part (feature extraction part) and two fully connected layers (classification part) of the commonly used code classification network to build a classification network.

[0036] Training the classification network: First, the network is trained using a publicly available code classification task dataset. Then, using transfer learning, a network is trained to automatically generate the most suitable fuzzer combination based on the characteristics of the target being tested. The input for this step is each fuzz test program (sample) and its most suitable fuzzer combination (label). The output is a classification neural network that automatically generates the most suitable fuzzer combination for the program being tested.

[0037] Furthermore, the workflow for building and training the classification network for the program under test is as follows:

[0038] 1-3-1 Use the backbone part (feature extraction part) of the code classification network (various common code classification networks can be used) and two layers of fully connected layers (classification part) to form a basic code classification network;

[0039] 1-3-2 Use the public program classification task dataset to train the randomly initialized program classification network in step 1-3-1;

[0040] 1-3-3 Reconstruct a new code classification network with the same network structure as the neural network in step 1-3-1, use the feature extraction part trained in step 1-3-2 to initialize the feature extraction part, and randomly initialize the final classification network part;

[0041] 1-3-4 Test performance steps using predictive fuzzer combination on each program under test (step 1-2)

[0042] Each fuzz test program (sample) and its most suitable fuzzer combination (label) are used to train a new classification network;

[0043] 1-3-5 Save the trained classification network obtained in step 1-3-4 for use in the testing phase.

[0044] 2. Testing phase

[0045] First, after obtaining the program to be tested, the code classification network trained in 1-3 is used to classify the program to be tested and determine its suitable fuzzer combination. Finally, the fuzzers in the fuzzer combination are called separately to perform parallel fuzz testing on the program to be tested.

[0046] Specifically, it is divided into two steps: predicting the fuzzer combination suitable for the program to be tested, and using heterogeneous fuzzers to perform parallel testing on the program to be tested.

[0047] 2-1 Predicting the fuzzer combination suitable for the program to be tested

[0048] The program to be tested is input into the code classification network trained in stages 1-3 to obtain its classification, and then a fuzzifier combination suitable for the classification is obtained. The input of this step is the program to be tested, and the output is a fuzzifier combination suitable for the program to be tested.

[0049] 2-2 Using heterogeneous fuzzers to perform parallel testing on the program under test

[0050] Perform parallel fuzz testing on the program under test using heterogeneous fuzzers. After the test is complete, collect all test results, remove duplicates, and output a test report. The input to this step is the program under test and the fuzzer combination suitable for the program under test given in step 2-1. The output is the fuzz test result report.

[0051] Furthermore, the workflow for parallel testing of the program under test using heterogeneous fuzzers is as follows:

[0052] 2-2-1 Create an instance for all fuzzers in the fuzzer combination output by step 2-1;

[0053] 2-2-2 Use all fuzzer instances created in step 2-2-1 to fuzz the program under test separately, and synchronize newly discovered seeds regularly;

[0054] 2-2-3 After the fuzz testing time set by the user according to the test requirements has expired, the seeds generated by each fuzzer and the test cases that caused the target program to crash are collected;

[0055] 2-2-4 Run all the seeds collected in step 2-2-3 and the test cases that cause the target program to crash once on the target program, and collect statistics such as the global branch coverage bitmap, path coverage number, and unique crash number;

[0056] 2-2-5 Write all the statistical information obtained in step 2-2-4 into the test report.

[0057] Based on the above method, the present invention implements an integrated fuzz testing system. This system is divided into two parts, corresponding to the training phase and the testing phase. The training phase comprises three modules: a single fuzzer performance evaluation module, a fuzzer combination performance prediction module, and a test program classification network construction and training module, respectively implementing the functions of the three steps in the training phase of the above method. The testing phase comprises two modules: a fuzzer combination prediction module suitable for the test program and a parallel testing module for heterogeneous fuzzers, respectively implementing the functions of the two steps in the testing phase of the above method.

[0058] The main advantages of the present invention are:

[0059] 1) It can automatically select the appropriate fuzzer combination for test programs with different characteristics. Using a neural network to classify the test programs according to their characteristics, it can provide a targeted and appropriate fuzzer combination, thus avoiding the problem that a general fuzzer combination performs poorly on certain test programs.

[0060] 2) It can automatically collect the results of parallel tests and output the summarized test results. After the parallel test is completed, the present invention implements a test result summary program to summarize and remove duplicate test results from each fuzzer, and then output a complete test report, saving the time of manual data summary and analysis after fuzz testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The performance difference diagram of each fuzzer on different test programs.

[0062] Figure 2 It is the overall framework diagram of the present invention.

[0063] Figure 3 for Figure 2 Diagram of the structure of the section on evaluating the performance of a single fuzzer.

[0064] Figure 4 for Figure 2 Structural diagram of the predictive fuzzer combination performance section.

[0065] Figure 5 for Figure 2 The structural diagram of the classification network part of the training program under test.

[0066] Figure 6 for Figure 2 Result graph of the parallel test part. DETAILED DESCRIPTION

[0067] The following embodiments of the present invention will be described in detail with reference to the accompanying drawings of the present invention. The following embodiments are only used to further illustrate the present invention and are not the only implementation of the present invention.

[0068] like Figure 2 As shown, an integrated fuzz testing method and system for automatically selecting fuzzer combinations is divided into two major stages: training stage and testing stage.

[0069] 1. Training phase

[0070] The training phase is divided into three steps: performance evaluation of a single fuzzer, performance prediction of fuzzer combinations, and training of the classification network for the program under test.

[0071] 1-1 Performance Evaluation of a Single Fuzzer

[0072] The specific implementation method is as follows Figure 3 The specific steps are as follows:

[0073] 1-1-1 Select the target program set for testing and training this system. You can use any commonly used fuzz test program set. Select m software to be tested (for example, you can select 20 software to be tested in Google Fuzz Test Suite: 'boringssl', 'c-ares', 'freetype2',

[0074] 'guetzli','harfbuzz','json','lcms','libarchive',

[0075] 'libjpeg','libpng','libxml2','llvm-

[0076] libcxxabi','openthread','pcre2','proj4','re2','sqlite',

[0077] 'vorbis', 'woff2', and 'wpantund');

[0078] 1-1-2 Select the fuzzers available for this system. Generally speaking, we select n commonly used fuzzers with a certain diversity (for example, we can select 9 commonly used fuzzers in the industry: 'afl',

[0079] 'aflfast','fairfuzz','mopt','radamsa','lafintel',

[0080] 'qsym', 'honggfuzz' and 'libfuzzer');

[0081] 1-1-3 Use all the fuzzers selected in step 1-1-2 to perform instrumentation compilation on each target program in step 1-1-1, and generate binary files that can be used for each fuzzer test

[0082] (a total of m*n binary files);

[0083] 1-1-4 Use a basic fuzzer (such as AFL) to test all target programs for T hours (24 hours is recommended). Select x seeds (5 seeds are recommended) from the queue folder of the test results of each target program as the initial seeds for subsequent evaluation (the selection strategy is to select a seed every T / (x-1) hours or so, and give priority to seeds that have generated multiple seeds in a short period of time. The purpose of this is to cover all parts of the target program code space in a shorter time in subsequent evaluation tests, which not only saves time but also makes the evaluation more accurate).

[0084] 1-1-5 Use the seed of each target program obtained in step 1-1-4 as the initial seed, and use each available fuzzer to test each target program. The test duration is t hours (recommended 1 hour), and repeated R times (recommended 30 times);

[0085] 1-1-6 Run all the seeds generated by each test in step 1-1-5 on the corresponding test program to collect the number of paths covered by the fuzz test and obtain its branch coverage bitmap. These branch coverage bitmaps, path coverage counts, and their corresponding test program and test fuzzer are recorded in groups (a total of m*n*R groups) as test performance evaluation data for the fuzzer.

[0086] In the above process, steps 1-1-1 to 1-1-4 are the pre-processing stage of the entire system and need to be completed manually. Then steps 1-1-5 to 1-1-6 are automatically performed by the program. The pseudo code is as follows:

[0087]

[0088]

[0089] The first parameter targetsList in the first line is an array, each element of which is the name of a program under test; the parameter target is a string representing the currently selected program under test. The first parameter fuzzersList in the second line is an array, each element of which is the name of a fuzzer; the parameter fuzzer is a string representing the currently selected fuzzer. The first parameter repetitions in the third line is an unsigned integer representing the number of times each fuzzer repeats on the same program under test during the evaluation process; the parameter epoch is also an unsigned integer representing the number of repetitions currently reached. The parameter seeds, which first appears on line 4, is a dictionary whose key is the name of the program under test and whose value is the address of the seed folder of the program under test. The parameter binaryTarget is a two-level dictionary whose first-level key is the name of the program under test, whose second-level key is the name of the selected fuzzer, and whose value is the address of the binary file obtained after instrumenting and compiling the program under test using the selected fuzzer. The function writeFuzzingCommand returns the command line instructions required for fuzz testing as a string based on the selected fuzzer. The parameter fuzzingCommand is a string representing the command line instructions required for this fuzz test. The function execInScreen, which first appears on line 5, executes the command line instructions for the fuzz test in a new screen (executing in the screen makes it easier to view the current status during the test). The return value is the address of the test result folder. The parameter resultDir is a string representing the address of the returned test result folder. The collectResult function, first appearing on line 6, essentially collects the branches covered (collected in bitmap file format) and the number of covered paths (collected in unsigned integer format) from the test results of various fuzzers in a unified format. The parameter bitmap is a string representing the address of the collected bitmap file containing the covered branches; the parameter stats is an unsigned integer representing the number of covered paths collected. The save function, first appearing on line 7, saves the name of the program under test, the name of the fuzzer used, the number of test repetitions, the address of the collected bitmap file containing the covered branches, and the number of covered paths in grouped form.

[0090] 1-2 Fuzzer combination performance prediction

[0091] The specific implementation method is as follows Figure 4 The specific steps are as follows:

[0092] 1-2-1 Select each target program in the target program set to be tested for training this system in turn,

[0093] After the traversal is completed, the test is completed;

[0094] 1-2-2 respectively reads all branch coverage bitmaps tested by each optional fuzzer, and calculates the coverage probability of each fuzzer for each branch of the target program (in actual implementation, the branch coverage bitmaps obtained by repeated testing R times can be accumulated bit by bit (each accumulated result corresponds to a floating point number), and then the number of times each branch is covered is divided by R to replace the branch coverage probability);

[0095] 1-2-3 Read the number of paths covered by each fuzzifier in each test on the target program, calculate the average number of paths covered by each fuzzifier, and obtain the fuzzifier with the largest average number of covered paths;

[0096] 1-2-4 traverses all possible combinations (full permutation) of the remaining optional fuzzers, where the number of fuzzers in each combination is the number of fuzzers to be tested in parallel (recommended to be 4) minus 1. If the traversal is complete, go to step 1-2-7; otherwise, go to step 1-2-5.

[0097] 1-2-5 Combine the fuzzifier combination selected in step 1-2-4 and the fuzzifier with the largest average number of coverage paths obtained in step 1-2-3 to form a complete fuzzifier combination;

[0098] 1-2-6 superimposes the branch coverage probabilities obtained by these fuzzers in step 1-2-2 to obtain the coverage probability of each branch of the program to be tested selected by the fuzzifier combination selected in step 1-2-5, and then calculates the mathematical expectation of the number of covered branches, and returns to step 1-2-4;

[0099] 1-2-7 ranks each fuzzifier combination according to the mathematical expectation of the number of branches covered on the program under test, and outputs the fuzzifier combination that ranks first as the fuzzifier combination that best suits the program under test. Then, return to step 1-2-1.

[0100] The pseudo code of the above process is as follows:

[0101]

[0102]

[0103] Among them, the parameter targetsList that first appears in the first line is an array, each element of which is the name of a program to be tested; the parameter target is a string, which represents the currently selected program to be tested. The parameter fuzzersList that first appears in the second line is an array, each element of which is the name of a fuzzer; the parameter fuzzer is a string, which represents the currently selected fuzzer. The parameter bitmap that first appears in the third line is a high-order array, each element of which is an array of the same size as the bitmap in AFL (64K*8 bits), and each element stores the bitmap collected after a test. The parameter coveredPaths that first appears in the fourth line is a one-dimensional array, each element is an unsigned integer, and each is used to store the number of covered paths collected after a test. The parameter repetitions that first appears in the fifth line is an unsigned integer, which represents the number of times each fuzzer repeats on the same program to be tested during the evaluation process; the parameter epoch is also an unsigned integer, which represents the current number of repetitions. The first appearance of bitmaps on line 6 is a three-level dictionary. The first-level keyword is the name of the program under test, the second-level keyword is the name of the selected fuzzer, and the third-level keyword is the number of repeated tests. Its value is the bitmap (64K*8 bits) representing the branches covered after the program under test is tested with the fuzzer. The first appearance of stats on line 7 is a three-level dictionary. The first-level keyword is the name of the program under test, the second-level keyword is the name of the selected fuzzer, and the third-level keyword is the number of repeated tests. Its value is the number of covered paths collected after the program under test is tested with the fuzzer. The function calculationProbability that first appears in line 9 is used to calculate the probability of covering each branch using a given fuzzifier on a given program to be tested. Its specific implementation is to reverse each bit of each element of the bitmap (representing the result of a test) (1 becomes 0, 0 becomes 1) and then accumulate them (representing the number of times this branch is covered), and then map it to a floating-point number. Finally, each floating-point number is divided by the number of elements of the bitmap (first-level elements), representing the frequency of each branch being covered, and these frequencies (64K*8) are returned as the probability of covering each branch; the parameter p_bitmap is an array of three tuples, where the first element of each tuple is a high-dimensional array, and each element of this array (first-level element) is the probability (64K*8 floating-point numbers) that a fuzzifier covers each branch on the program to be tested. The second element of the tuple is a string, which represents the name of the program to be tested, and the third element of the tuple is also a string, which represents the name of the selected fuzzifier.The parameter averageCoveredPaths, which first appears on line 10, is an array of three tuples. The first element of each tuple is a floating-point number, representing the number of paths covered by a fuzzer on the program under test. The second element of the tuple is a string, representing the name of the program under test. The third element of the tuple is also a string, representing the name of the selected fuzzer. The function coveredMost, which first appears on line 12, gives the name of the fuzzer with the largest average number of paths covered on the given program under test. The parameter selectedFuzzer is a string, which represents the name of the fuzzer with the largest average number of paths covered on the given program under test. This fuzzer will be included in each subsequent fuzzer combination. The function combine, which first appears on line 13, returns the number of fuzzers to be tested in parallel, minus one permutation of different fuzzers, in addition to the selected fuzzer (i.e., the fuzzer with the largest average number of covered paths). The return value is a two-dimensional array, with each element (first-level element) being a fuzzer combination (an array of fuzzer names). The parameter fuzzersCombinations is a two-dimensional array, with each element (first-level element) being a fuzzer combination (an array of fuzzer names). The combination, which first appears on line 14, is a string array representing a fuzzer combination. The parameter trueCombination, which first appears on line 15, is a string array representing the actual fuzzer combination used for testing, which contains the fuzzer with the largest average number of covered paths. The function calculate, which first appears on line 16, calculates the mathematical expectation of the number of branches covered by a given fuzzer combination on a given program under test. It does this by calculating the probability that at least one fuzzer covers each branch based on the probability that each fuzzer in the combination covers that branch. These probabilities are then multiplied by 1 and added up. The return value is the mathematical expectation. The parameter coveredBenches is an array of triplets. The first element of each triple is a floating-point number representing the mathematical expectation of the number of covered branches. The second element of the tuple is a string representing the name of the program under test. The third element of the tuple is a string array representing the fuzzer combination used (listing the names of all fuzzers used). The function fingBest, which first appears on line 18, finds the fuzzer combination with the highest mathematical expectation of the number of branches covered on a given program under test. Its return value is a string array consisting of the names of each fuzzer in this fuzzer combination. The parameter bestCombination is a dictionary whose key is the name of the program under test and whose value is the fuzzer combination (string array) that best fits the program under test.

[0104] 1-3 Classification network training of the program to be tested

[0105] Use Figure 5 The neural network model shown (using CVRNN to extract features is used as an example to illustrate this implementation scheme, and the backbone network of other code classification networks can also be used to extract features), and the specific training steps are as follows:

[0106] 1-3-1 Use CVRNN feature extraction part and two layers of fully connected layers to form the basic code classification network (see the specific network structure Figure 5 );

[0107] 1-3-2 Use the basic code classification training set provided by CVRNN to train the basic code classification network (CVRNN recommends 30 rounds of training);

[0108] 1-3-3 Reconstruct a new neural network (hereinafter referred to as the actual classification network) with the same structure as the basic code classification network. The feature extraction part is initialized with the parameter values of the feature extraction part of the trained basic code classification network, and the last two fully connected layers used for classification are randomly initialized.

[0109] 1-3-4 freezes the parameters of the feature extraction part of the actual classification network, uses the abstract syntax tree of the test target program and 1-2 to obtain the most suitable fuzzer combination (there are actually only a few combinations) as training samples and labels, and trains this network (20 rounds of training are recommended).

[0110] Then, unfreeze the feature extraction part and train again (10 rounds are recommended) to achieve the purpose of fine-tuning;

[0111] 1-3-5 Save the actual classification network after training for use in the testing phase.

[0112] 2. Testing phase

[0113] It is divided into two parts: fuzzer combination prediction suitable for the program to be tested and parallel testing of heterogeneous fuzzers.

[0114] 2-1 Fuzzer combination prediction suitable for the program to be tested

[0115] The specific implementation steps are as follows:

[0116] 2-1-1 Use llvm to extract the abstract syntax tree of the program under test;

[0117] 2-1-2 Divide the abstract syntax tree of the program to be tested into segments (cut at if, else, switch, for, while, etc.);

[0118] 2-1-3 All the abstract syntax tree fragments obtained in step 2-1-2 are combined into a tensor and input into the actual code classification network obtained in the training phase to predict the category of the program to be tested this time;

[0119] 2-1-4 Based on the predicted category of the program to be tested, find the most suitable fuzzer combination for that category.

[0120] 2-2 Parallel Testing of Heterogeneous Fuzzers

[0121] The specific implementation method is as follows Figure 6 The specific steps are as follows:

[0122] 2-2-1 Select all fuzzifiers in the best fuzzifier combination from all available fuzzifiers;

[0123] 2-2-2 starts a process for each selected fuzzer;

[0124] 2-2-3 Use the fuzzer process in step 2-2-2 to perform fuzz testing based on the given initial seed and test parameters;

[0125] 2-2-4 After the test is completed, all the seeds obtained from the test and the test cases that will cause the target program to crash are run on the target program once, and statistical information such as the global branch coverage bitmap, path coverage number, and unique crash number are collected;

[0126] 2-2-5 Write the test results into a report and output it.

[0127] The pseudo code of the above process is as follows:

[0128]

[0129] Among them, the parameter fuzzersCombination that first appears in the first line is a string array, and each element of the array is the name of a fuzzer; the parameter fuzzer is a string, which represents the currently selected fuzzer. The parameter seeds that first appears in the second line is a string, which represents the initial seed used; the parameter target is a string, which represents the name of the program to be tested; the parameter parameters is also a string, which represents the customized parameters during fuzz testing; the parameter binaryTarget is a two-level dictionary, the first-level keyword is the name of the program to be tested, the second-level keyword is the name of the selected fuzzer, and the value is the address of the binary file obtained after the program to be tested is instrumented and compiled using the selected fuzzer; the function writeFuzzingCommand returns the command line instructions required for fuzz testing in the form of a string based on the selected fuzzer; the parameter fuzzingCommand is a string, which represents the command line instructions required for this fuzz test. The execInScreen function, first appearing on line 3, executes the fuzz test command line instructions in a new screen (executing in a screen makes it easier to view the current status during testing). The return value is the address of the test results folder. The resultDir parameter is a string array, each element of which represents the address of the test result folder returned by one of the fuzzers in the parallel test. The collectRsult function, first appearing on line 4, is actually a program that runs all the test seeds and test cases that cause the target program to crash on the target program, collecting statistics such as the global branch coverage bitmap, path coverage count, and unique crash count. The return value is a dictionary containing all the statistics. The result parameter is a dictionary whose key is the name of the statistic and whose value is the specific value of the statistic. The writeReport function, first appearing on line 5, outputs the test results to a formatted file for easy reading. Its return value is the address of the file. The reportAddr parameter is a string representing the address of the result file.

[0130] It should be noted that the above embodiments are intended only to illustrate the present invention and are not intended to limit the present invention. The various specific numerical values used in the embodiments are merely illustrative. Furthermore, those skilled in the art may devise alternative embodiments to those described above without departing from the scope of the appended claims, and these should not be construed as new methods. The classification neural network used in the examples is only one type of similar network, and similar effects can be achieved using other classification networks.

Claims

1. An integrated fuzz testing method for automatically selecting a fuzzer combination, characterized by: It is divided into training phase and testing phase; The training phase is divided into three steps: 1-1. Evaluate the performance of a single fuzzer: On each selected test program, test the performance of all optional fuzzers on the test program in turn; 1-2. Predict the test performance of fuzzer combinations on each program under test: Based on the performance of each optional fuzzer used alone on each program under test obtained in the previous step, calculate the performance expectations of various fuzzer combinations on each fuzzer; 1-3. Build and train a code classification network for the program under test: Build a code classification network that includes both feature extraction and classification. First, train the network using a publicly available code classification dataset. Then, use transfer learning to enable it to generate a suitable fuzzer combination based on the characteristics of the program under test. A classification network is constructed using the feature extraction part of a commonly used code classification network and the classification part consisting of two fully connected layers. When training this network, it is first trained using a public code classification task dataset. Then, using transfer learning methods, a network is trained that can automatically generate the most suitable fuzzer combination based on the characteristics of the target to be tested. The process of building and training the classification network for the program under test meets the following steps: 1-3-1. Use the feature extraction part of the code classification network and the classification part composed of two fully connected layers to form a basic code classification network; 1-3-2. Use the public program classification task dataset to train the randomly initialized code classification network in step 1-3-1; 1-3-3 Reconstruct a new code classification network I with the same network structure as the neural network in step 1-3-1, use the feature extraction part trained in step 1-3-2 to initialize the feature extraction part, and randomly initialize the final classification network part; 1-3-4 Use each fuzz test program and its best fuzzer combination to train a new code classification network I; 1-3-5 Save the trained code classification network I obtained in step 1-3-4 for use in the testing phase; During testing, the program to be tested is input into the trained code classification network I to obtain its classification, and then the fuzzifier combination suitable for the classification is obtained; The testing phase is divided into two steps: 2-1. Predicting the fuzzer combination suitable for the program under test: Input the program under test into the code classification network to obtain the appropriate fuzzer combination; 2-2. Use heterogeneous fuzzers to perform parallel testing on the program under test: Use this fuzzer combination to test the program under test in parallel. After the test, collect all test results and remove duplicates.

2. The integrated fuzz testing method for automatically selecting a fuzzifier combination according to claim 1, characterized in that: Predict the performance of various fuzzer combinations on each object in a given list of fuzz test programs without actual parallel testing; Its workflow follows these steps: 1-2-1. Select each program to be tested in the given fuzz test program list in turn. If all programs to be tested have been selected, this step ends. 1-2-2. Read one by one all branch coverage bitmaps and the number of covered paths obtained by each optional fuzzer in the step of evaluating the performance of a single fuzzer on the program to be tested; 1-2-3. Select the fuzzer with the largest average number of covered paths in each test on the program to be tested; 1-2-4. Use the branch coverage bitmap to calculate the probability that various fuzzer combinations, including the fuzzer selected in step 1-2-3, cover each branch, and then calculate the mathematical expectation of the number of branches covered on the program under test; where the number of fuzzers in the fuzzer combination is the input value; 1-2-5. Use the mathematical expectation value of the number of covered branches calculated in step 1-2-4 as the performance score of the fuzzifier combination on the program to be tested, give a performance score table of all fuzzifier combinations on the program to be tested, and filter out the fuzzifier combination with the highest performance score from the performance score table.

3. An integrated fuzz testing system that automatically selects fuzzer combinations, characterized by It consists of two parts: training part and testing part; The training part is divided into three modules: a single fuzzer performance evaluation module, a fuzzer combination performance prediction module, and a test program classification network construction and training module; while the testing part is divided into two modules: a fuzzer combination prediction module suitable for the test program and a parallel testing module for heterogeneous fuzzers. The test program classification network construction and training module is used to construct and train a neural network that can classify the target program according to its characteristics, and thus automatically give a fuzzer combination suitable for the target program; the input of this module is the most suitable fuzzer combination for each fuzz test object, and the output is a classification neural network that can automatically give the most suitable fuzzer combination for the test program.

4. The integrated fuzz testing system for automatically selecting a fuzzer combination according to claim 3, characterized in that: The performance evaluation module of a single fuzzer is used to run all optional fuzzers in this system separately on each software to be tested in the commonly used fuzz test object set, and then collect the performance of these fuzzers when running separately, including coverage bitmaps and the number of coverage paths; the input of this module is the list of optional fuzzers and the list of fuzz test objects, and the output is the coverage bitmap and the number of coverage paths of each optional fuzzer on each fuzz test object.

5. The integrated fuzz testing system for automatically selecting a fuzzer combination according to claim 4, characterized in that: The fuzzer combination performance prediction module is used to predict the performance of each fuzzer combination on each object in a given fuzz test object list without actual parallel testing. The input of this module is the coverage bitmap and the number of coverage paths of each optional fuzzer on each fuzz test object, as well as the number of fuzzers required for parallel testing. The output is a prediction score table for each fuzzer combination on each fuzz test object.

6. The integrated fuzz testing system for automatically selecting a fuzzer combination according to claim 5, characterized in that: The fuzzer combination prediction module suitable for the program to be tested is used to use the trained code classification network to predict the classification of the program to be tested when fuzz testing is required; the input of this module is the target program to be tested, and the output is the category to which the program to be tested belongs.

7. The integrated fuzz testing system for automatically selecting a fuzzifier combination according to claim 6, characterized in that: The parallel testing module of heterogeneous fuzzers is used to perform fuzz testing on the target program to be tested, trying to find possible vulnerabilities in it, and collect all test results after the test is completed, and output a test report after deduplication; the input of this module is the target program to be tested and the fuzzer combination suitable for the program to be tested given by the fuzzer combination prediction module, and the output is a fuzz test report.

Images