A parallel fuzz testing method based on complementarity

Abstract

The present invention discloses a parallel fuzz testing method based on complementarity, and the steps include: firstly, selecting a benchmark program with the highest similarity to a target program according to functionality; then, performing a FuzzBench benchmark test on an existing test tool based on the benchmark program to obtain a coverage ranking; using the tool with the highest coverage to perform a fuzz test on the target program and observing the effect, if the tool coverage is low, selecting a tool with the highest complementarity to the tool and assigning an initial value to the coverage record to make it cover the area that could not be covered before as much as possible; thereafter, repeatedly performing the above steps during operation, thereby improving the coverage of the fuzz testing tool by parallelization and the complementarity of the test tools. The present invention selects a benchmark program by similarity, and is more in line with the actual target program in the selection of the test tool. At the same time, the present invention dynamically selects a parallelization tool combination during operation to avoid the problem of low parallelization efficiency as much as possible.

Description

Technical Field

[0001] The present invention relates to the field of software engineering, and in particular to a parallel fuzzy testing method based on complementarity. Background Art

[0002] With the rapid development of computer science and technology, modern software systems have become incredibly complex and expansive, encompassing a wide range of functions and services. This complexity is reflected not only in the software architecture and functional integration, but also in the volume and complexity of the code. Against this backdrop, traditional single-use fuzz testing tools face significant challenges, as they often struggle to cover all possible paths and states within a software system, hindering the effective detection of potential vulnerabilities and errors.

[0003] To address this challenge, researchers have begun exploring the concept of parallel computing in fuzz testing to improve testing efficiency and effectiveness. The core idea of ​​parallel fuzz testing is to use multiple fuzz testing tools simultaneously to execute tests on the target program. This allows different testing tools to explore different parts of the program, complementing each other and collectively improving coverage and the probability of discovering vulnerabilities.

[0004] For parallel fuzz testing, the selection of fuzz testing tools is often based on the programmer's experience. The uncertainty caused by subjective judgment makes it difficult to guarantee the actual effectiveness of the test. Furthermore, although complementary testing is performed before testing to select tools that work well together, the implementation strategy is to run two test tools simultaneously and individually in the test environment, observing the coverage performance under different operation modes. If the coverage of the collaborative operation is greater than that of the individual runs, it is assumed that the two tools will achieve better results in actual parallel testing. However, there are differences between the test environment and the actual environment, and static selection methods are difficult to adapt to these differences and make adjustments to meet the needs.

[0005] Appropriate tool selection and dynamic adjustment of parallelization tool selection during parallel testing will solve the above problems. Summary of the Invention

[0006] The present invention proposes a parallel fuzzy testing method based on complementarity to solve the problem that the current parallel testing technology has never involved dynamic combination in tool selection and execution process, resulting in the difficulty in achieving ideal parallelization effect. The present invention is achieved through the following technical solutions.

[0007] A parallel fuzz testing method based on complementarity, characterized by comprising the following steps:

[0008] Step 1) predict and rank the pairwise parallel complementary execution effects of existing fuzz testing tools on the target program, and select the tool with the best single execution effect to execute on the target program first;

[0009] Step 2) After executing N times, if the current coverage is lower than x, select the next tool to join the parallel testing;

[0010] Step 3) Modify the coverage edge of the newly added tool to guide it to the uncovered area as much as possible, and repeat step 2).

[0011] The above-mentioned parallel fuzz testing method based on complementarity is characterized in that the step 1) specifically includes the following steps:

[0012] Step 11) Obtaining the benchmark program and the execution effect of a single fuzz testing tool on the benchmark program through FuzzBench, wherein the execution effect includes the execution result of each benchmark program, the number of vulnerabilities found, and the test coverage, and comprehensively evaluating the execution effect of the fuzz testing tool on the benchmark program by combining key indicators in the execution effect;

[0013] Step 12) Using the DeepSim tool, obtain a benchmark program with the highest functional similarity to the target program;

[0014] Step 13) Based on the benchmark program obtained in step 12), using the method provided by Cupid, the execution results in step 11) are used to calculate the complementary effect C_ij between each tool F_i and F_j, and the execution effect of each tool's parallelization tool is ranked according to C_ij;

[0015] Step 14) Select the tool that performs best individually and execute it on the target program.

[0016] The above-mentioned parallel fuzz testing method based on complementarity is characterized in that the step 2) specifically includes the following steps:

[0017] Step 21) After N time periods, if the coverage is lower than f(n), the execution effect of the current fuzz testing tool is considered unqualified, and then step 22) is performed; otherwise, step 21) is continued;

[0018] Step 22) Select the tool with the best parallel test effect among the tools selected last time. If the best tool has been selected, select the second best one and so on.

[0019] Step 23) Add the selected tools to the parallelized test.

[0020] The above parallelized fuzz testing method based on complementarity degree is characterized in that step 3) specifically includes the following steps:

[0021] Step 31): Select all edges with a coverage count greater than z through the covered edges and the coverage count, and consider that these edges have been effectively covered.

[0022] Step 32): Combine with the control flow graph of the target program, and consider that the edges with a coverage count less than z and the edges adjacent to these edges have not been effectively covered.

[0023] Step 33): Modify the coverage edge statistics table maintained locally by the newly added tool, manually increase the coverage count of the effectively covered edges, and guide the newly added tool to generate new test cases according to its own strategy to preferentially test the uncovered edges.

[0024] Step 34): Repeat step 2).

[0025] The above parallelized fuzz testing method based on complementarity degree is characterized in that in the parallelization process, dynamic allocation of different tool resources is involved; taking time N in step 2) as a cycle, assuming the total computing resource is x, and the number of tools added to run at the beginning of the cycle is num, then the default resource holding amount r of each tool is r = x / num. In this case, run for N / 2 time and observe the private statistics table of each tool. If the coverage rate Ci of tool i < base, it is considered that tool i has poor performance and is added to the poor performance queue for recording; if Ci > base, tool i is added to the good performance queue; in the latter half of N / 2 time, the tools in the poor performance queue will be reduced by r / 10 and evenly distributed to the tools in the good performance queue.

[0026] The above parallelized fuzz testing method based on complementarity degree is characterized in that the time N and the coverage rate f(n) in step 21) need to be finally determined by statistically analyzing the changes in the code coverage rate of existing fuzz testing tools over time, so as to ensure the correct evaluation of the execution effect of the fuzz testing tools.

[0027] The above parallelized fuzz testing method based on complementarity degree is characterized in that the edge coverage count z in steps 31) and 32) needs to be comprehensively considered and set according to the edge coverage count of the fuzz testing tool with the best execution effect on the target program and the coverage count of each edge during the running process; the program control flow graph in step 32) is obtained through static analysis.

[0028] The above-mentioned parallel fuzzy testing method based on complementarity is characterized in that there are two coverage edge statistics tables during the operation of step 2) and step 3), namely a public statistics table and a private statistics table. The private statistics table maintains a copy for each tool and records the number of edge coverage times during the operation of the tool. At the same time, the new data obtained during the maintenance tool testing process is incremental data and is updated in an incremental manner; the public statistics table is filled with the incremental data held by each private table.

[0029] The above-mentioned parallel fuzz testing method based on complementarity is characterized in that the different fuzz testing tools maintain their own seed pools and only obtain a global seed pool during initialization, wherein the seeds should carry edge coverage information; the seed pool and the coverage edge statistics table are updated in the same way.

[0030] The present invention adopts the above technical solution and has the following beneficial effects:

[0031] (1) The benchmark program is selected by similarity, which makes the selection of test tools more consistent with the actual target program.

[0032] (2) Only one fuzz testing tool is used in the early stage of fuzz testing, which reduces the cost of repeated testing.

[0033] (3) Dynamically select the combination of parallelization tools during operation to avoid the problem of low parallelization efficiency as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an overall flow chart of the parallel fuzzy testing method based on complementarity according to an embodiment of the present invention.

[0035] Figure 2 Schematic diagram of the implementation of step 1) of the parallel fuzz testing method based on complementarity in an embodiment of the present invention.

[0036] Figure 3 A specific flow chart of adding parallel testing to the new fuzz testing tool according to an embodiment of the present invention.

[0037] Figure 4 It is a schematic diagram of data transmission between the coverage edge statistics table and the seed pool between the fuzzy testing tools in an embodiment of the present invention.

[0038] Figure 5 It is a schematic diagram of the process of dynamic resource allocation during the parallelization process of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] The present invention provides a parallel fuzzy testing method based on complementarity. Figure 1 As shown, Figure 1 The overall flow chart of the parallel fuzz testing method based on complementarity according to an embodiment of the present invention includes the following specific steps:

[0041] Step 1) Predict and sort the parallel complementary execution effects of the existing fuzz testing tools on the target program, and first select the tool with the best single execution effect to execute on the target program first. Figure 2 As shown, Figure 2 Schematic diagram of the implementation of step 1) of the parallel fuzzy testing method based on complementarity in an embodiment of the present invention. Figure 2 An implementation of step 1) is provided, describing the performance of different fuzz testing tools on benchmark programs, the performance of complementarity, and the selection of fuzz testing tools. The specific steps include:

[0042] Step 11) Obtaining the benchmark program and the execution effect of a single fuzz testing tool on the benchmark program through fuzzbench, including key indicators such as the execution result of each benchmark program, the number of vulnerabilities found, and test coverage, and comprehensively evaluating the execution effect of the fuzz testing tool on the benchmark program based on the obtained key indicators;

[0043] Step 12) Using the DeepSim tool, obtain a benchmark program with the highest functional similarity to the target program;

[0044] Step 13) Based on the benchmark program obtained in step 12), using the method provided by Cupid, the complementary effect C_ij between each tool F_i and F_j is calculated using the execution effect in step 11), and the execution effect of each tool's parallelization tool is ranked according to C_ij;

[0045] Step 14) Select the tool that performs best individually and execute it on the target program.

[0046] Step 2) After executing N times, if the current coverage is lower than x, select the next tool to join the parallel test. Figure 3 As shown, Figure 3 This is a specific flow chart for adding parallel testing to the new fuzz testing tool according to an embodiment of the present invention, which describes the specific processes of steps 2) and 3). The specific steps of step 2) include:

[0047] After N time periods in step 21), if the coverage rate is lower than f(n), the execution performance of the current fuzz testing tool is considered unsatisfactory and the process proceeds to step 22. Otherwise, the process continues with step 21. The time period N and the coverage rate f(n) are determined by statistically analyzing the changes in code coverage rates of existing fuzz testing tools over time to ensure that the execution performance of the fuzz testing tool can be accurately evaluated.

[0048] Step 22) Select the tool with the best parallel test effect among the tools selected last time. If the tool has been selected, select the second best one and so on.

[0049] Step 23) Add the selected tools to the parallelized test.

[0050] Step 3) Modify the coverage edge of the newly added tool to guide it to the uncovered area as much as possible. Figure 4 As shown, Figure 4 This is a schematic diagram of data transmission between the coverage edge statistics table and the seed pool between fuzz testing tools according to an embodiment of the present invention, describing the interaction process of the coverage edge statistics table. The specific steps of step 3) include:

[0051] Step 31) Select all edges with a coverage count greater than z through the covered edges and their coverage counts, and consider these edges to have been effectively covered; wherein, the coverage count z needs to be comprehensively considered and set based on the edge coverage count of the fuzz testing tool that performs best on the target program and the coverage count of each edge during the operation process.

[0052] Step 32) In combination with the target program control flow graph, it is considered that the edge with a coverage number less than z and the edge adjacent to the edge are not effectively covered.

[0053] Step 33) Modify the coverage edge statistics table maintained locally by the newly added tool, manually increase the coverage times of the effectively covered edges, and guide the newly added tool to generate new test cases according to its own strategy to prioritize testing uncovered edges.

[0054] Step 34) Repeat step 2).

[0055] The above description is merely a preferred embodiment of the present invention. The specific embodiments described herein are intended only to explain the present invention and are not intended to limit the present invention. Any simple modifications, equivalent variations, and modifications made to the above embodiments by any person skilled in the art, without departing from the principles and spirit of the present invention, in accordance with the technical essence of the present invention, shall be included within the scope of protection of the present invention.

Claims

1. A parallel fuzz testing method based on complementarity, characterized in that: Including the following steps: Step 1) Predict the pairwise parallel complementary execution effects of existing fuzz testing tools on the target program and sort them, and first select the tool with the best single execution effect to execute on the target program first; specifically, Step 1) includes: Step 11) Obtain the benchmark program through FuzzBench and the execution effects of a single fuzz testing tool on the benchmark program. The execution effects include the execution results of each benchmark program, the number of vulnerabilities discovered, and the test coverage rate. Combine the key indicators in the execution effects to comprehensively evaluate the execution effects of the fuzz testing tools on the benchmark program; Step 12) Use the DeepSim tool to obtain the benchmark program with the highest functional similarity to the target program; Step 13) Based on the benchmark program obtained in Step 12), use the method provided by Cupid, use the execution effects in Step 11) to calculate the complementary effect C_ij between two tools F_i and F_j, and rank the execution effects of the parallelization tools for each tool according to C_ij; Step 14) Select the tool with the best single execution effect to execute on the target program; Step 2) After executing N times, if the current coverage rate is lower than x, then select the next tool to join the parallelization test; different tool resources are dynamically allocated in Step 2); taking the time N in Step 2) as a cycle, assuming the total computing resource is x, and the number of tools added to the running at the start of the cycle is num, then the default resource holding amount r of each tool is x / num. In this case, run for N / 2 times and observe the private statistical tables of each tool. If the coverage rate Ci of tool i < base, then it is considered that the effect of tool i is not good and it is added to the ineffective queue for recording; if Ci > base, tool i is added to the effective queue; in the latter half of N / 2 times, the tools in the ineffective queue will have their resources reduced by r / 10 and evenly distributed to the tools in the effective queue; Step 3) Modify the covered edges of the newly added tool to guide it to go to the uncovered area as much as possible, and repeat Step 2); 2. The parallel fuzzy testing method based on complementarity according to claim 1 is characterized in that: Specifically, Step 2) includes the following steps: Step 21) After N times, if the coverage rate is lower than f(n) at this time, then it is considered that the execution effect of the current fuzz testing tool is unqualified, and then go to Step 22), otherwise continue to execute Step 21); Step 22) Select the tool with the best parallelization test effect among the tools selected last time. If the best tool has been selected, then select the second best and so on; Step 23) Add the selected tool to the parallelization test; 3. The parallel fuzzy testing method based on complementarity according to claim 1 is characterized in that: Specifically, Step 3) includes the following steps: Step 31) Select all edges with a coverage count greater than z through the covered edges and the coverage count, and consider that these edges have been effectively covered; Step 32) Combine the control flow graph of the target program and consider that the edges with a coverage count less than z and the edges adjacent to these edges have not been effectively covered; Step 33) Modify the coverage edge statistics table maintained locally by the newly added tool, manually increase the coverage count of the effectively covered edges, and guide the newly added tool to generate new test cases according to its own strategy to prioritize testing the uncovered edges; Step 34) Repeat step 2).

4. The parallel fuzzy testing method based on complementarity according to claim 2 is characterized in that: The time N and coverage f(n) in step 21) need to be finally determined by statistically analyzing the changes in code coverage of existing fuzz testing tools over time, to ensure that the execution effect of the fuzz testing tool can be correctly evaluated.

5. The parallel fuzzy testing method based on complementarity according to claim 3 is characterized in that: The edge coverage times z in steps 31) and 32) need to be comprehensively considered and set based on the edge coverage times of the fuzz testing tool that performs best on the target program and the coverage times of each edge during the running process; the program control flow graph of step 32) is obtained through static analysis.

6. The parallel fuzzy testing method based on complementarity according to claim 1, 2 or 3, characterized in that: During the operation of step 2) and step 3), there are two types of coverage edge statistics tables, namely the public statistics table and the private statistics table. The private statistics table is maintained for each tool and records the number of edge coverage times during the operation of the tool. At the same time, the new data obtained during the tool testing process is maintained as incremental data and updated in an incremental manner; the public statistics table is filled with the incremental data held by each private table.

7. The parallel fuzzy testing method based on complementarity according to claim 1, characterized in that: The fuzz testing tool maintains its own seed pool and obtains the global seed pool only during initialization, wherein the seed should carry edge coverage information; the seed pool is updated in the same way as the coverage edge statistics table.

Images