A fuzz testing system for binary programs

Abstract

The present invention provides a fuzzy testing system for binary programs, which mainly includes three modules, namely a program tracking module, a test case selection module and a test case mutation module; the program tracking module uses hardware program tracking technology to obtain program execution characteristics and instruction characteristics for test case selection and mutation; the test case selection module maintains a priority queue according to the test case characteristics, and performs probabilistic selection when selecting a test case based on whether it is in the priority queue; the test case characteristics include the test case's own characteristics, fuzzy test execution characteristics and instruction characteristics obtained by hardware program tracking; the test case mutation module determines the mutation energy when the test case mutates randomly according to the test case characteristics. The present invention solves the problems that most current fuzzy tests do not support binary program testing or have poor testing effects, and insufficient use of effective information generated during the testing process.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a fuzzy testing system for binary programs. Background Art

[0002] Fuzz testing: Fuzz testing is a method of discovering software vulnerabilities by providing unexpected inputs to the target system or software and monitoring abnormal results. Since fuzz testing technology can convert a large number of manual tests into highly automated tests, which is between black box testing and white box testing, it is widely used in the field of vulnerability mining at home and abroad.

[0003] Some existing fuzz testing tools, such as AFL, select test cases in the order in which they are added, and determine whether the current test case is eligible for priority selection based on factors such as execution time, test case length, and whether it has been mutated. In addition, in order to support the testing of binary programs, some fuzz testing tools have introduced the hardware program tracing technology Intel Process Trace (abbreviated as Intel PT) to record the execution path of the binary program in the CPU.

[0004] For the test cases in fuzz testing, no matter they are initially provided or generated during the mutation process, the quality of these test cases will be ranked high or low, so giving priority to mutation and executing high-quality test cases will help improve the efficiency of fuzz testing in discovering program paths and vulnerabilities. However, the existing test case selection method uses factors such as execution time, test case length, and whether it has been mutated to determine the effectiveness of the test case, and does not consider more fine-grained information of each test case execution (for example, the number of conditional jump instructions and other instructions executed by the test case, the number of basic blocks triggered, etc.), which will lead to resource waste. At the same time, it may cause the fuzzer to execute a large number of test cases, but not to discover new paths faster and more (for example, in coverage-oriented fuzz testing, the execution of a large number of test cases cannot generate more new program paths to achieve higher path coverage), thus affecting the efficiency of fuzz testing.

[0005] Basic block: A combination of one or more instructions in a program. When the program reaches a basic block, it starts from the first instruction of the basic block, and then executes other instructions in the basic block in sequence until the last instruction is executed. A path in a program consists of one or more basic blocks.

[0006] Test case: The input data or file required for the target binary program to execute. Its size is the storage resource size occupied by the test case.

[0007] Fuzz testing is an important method in the field of vulnerability discovery. Although most of the popular fuzz testing tools have good vulnerability discovery capabilities, they still have the following two limitations:

[0008] (1) Currently, most fuzz testing tools use source code instrumentation to test programs that have access to source code. During the instrumentation process, they divide basic blocks and perform unique identification. Although this method is more efficient for source code testing, it has the possibility of conflict and collision in the unique identification of basic blocks, and also lacks support for binary programs.

[0009] (2) Although some fuzz testing tools can support testing of binary programs, such as AFL (America FuzzyLop)'s QEMU mode (abbreviated as QAFL) and PTfuzz, QAFL requires simulation and tracing of instructions, and it is difficult for the simulator to fully and correctly simulate the target instructions, which may cause errors when the target binary program is simulated and executed; even though PTfuzz uses hardware program tracing technology, it only uses the executed basic block sequence to build the program path, and does not make sufficient use of other fine-grained effective information obtained from tracing, such as the basic block jump type, the number of instructions executed, etc., resulting in a waste of resources.

[0010] Therefore, how to efficiently utilize the fine-grained information in the fuzzy test execution process to guide the selection of test cases and the determination of mutation energy while supporting the testing of binary programs is the key point of the present invention. Summary of the invention

[0011] In view of this, the present invention provides a fuzzy testing system for binary programs, which can improve the efficiency of fuzzy testing.

[0012] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0013] In the present invention, the test cases include initial test cases, test cases to be mutated and test cases to be verified; the initial test cases represent test cases provided manually for starting fuzz testing; the test cases to be mutated represent test cases selected from a test case queue for mutation; the test cases to be verified represent test cases generated by changing part of the content of the test cases to be mutated through different mutation methods.

[0014] A fuzz testing system for binary programs mainly includes three modules, namely a program tracing module, a test case selection module and a test case mutation module, wherein the program tracing module first loads a target binary program and an initial test case for actual execution to obtain the characteristics of the initial test case; then the test case selection module selects a test case for subsequent mutation operations according to the characteristics of the test case; then the test case mutation module changes the content of the test case to be mutated through different mutation methods to generate a new test case to be verified; finally, the test case to be verified is handed over to the program tracing module for actual execution and verification; the entire fuzz testing system repeatedly executes the above three modules until it is manually stopped or the time runs out.

[0015] The input of the program tracking module is the target binary program and test cases, and the output is the execution characteristics and instruction characteristics of the target binary program. When the target binary program is executed, this module uses the CPU program tracking mechanism to track the program execution behavior and obtain the execution characteristics and instruction characteristics of the target binary program;

[0016] The input of the test case selection module is the test case queue, and the output is the test case to be mutated selected according to the test case characteristics. When selecting a test case, if the current test case in the test case queue is in the priority queue, it will be directly selected as the test case to be mutated, otherwise a random value will be given for probabilistic selection;

[0017] The input of the test case mutation module is the test case to be mutated, and the output is the test case obtained by mutation to be actually executed for verification. When mutating the test case, this module is mainly divided into deterministic mutation and random mutation. The number of mutations of deterministic mutation is determined according to the size of the test case, and the number of mutations of random mutation is determined according to the mutation energy.

[0018] Furthermore, the test case features of the present invention include the test case features themselves, fuzzy test execution features and instruction features obtained by hardware program tracing. The test case features themselves are the test case size; the fuzzy test execution features are the test case execution time, mutation depth and path frequency; the instruction features obtained by hardware program tracing include the number of triggered cmp instructions, basic blocks and all instructions obtained after decoding the hardware program tracing data;

[0019] The mutation depth (i) of the test case: the generation to which the test case belongs after the mutation of the initial test case, that is, if the initial test case is recorded as the first generation, then all test cases mutated from the initial test case are the second generation, and so on;

[0020] The path frequency freq(p) of the test case: the average number of times the basic blocks contained in the execution path of the test case are executed;

[0021] The number of triggered cmp instructions cmp(i): the number of cmp instructions contained in the execution path of the test case;

[0022] The number of basic blocks triggered block(i): the number of basic blocks contained in the execution path of the test case;

[0023] The number of all triggered instructions inst(i): the number of all instructions contained in the execution path of the test case.

[0024] Furthermore, the program tracing module of the present invention is used to trace the actual execution of the target binary program, decode the traced data, and monitor abnormal behavior of program execution;

[0025] Tracking the actual execution of the target binary program: When the target binary program actually executes the test case, the hardware program tracking technology is used to record the actual execution of the target binary program in the CPU, and the tracked information is compressed and stored in a fixed area of ​​the memory, waiting for subsequent use;

[0026] Decode the traced data: decode the trace data compressed and stored in the memory into instruction information and basic block information as required, and generate the path information corresponding to the target binary program in this execution according to the basic block information; if the path is a new path, convert the instruction information into instruction features and save them together with the test case;

[0027] Monitor abnormal behavior of program execution: When test cases trigger program vulnerabilities, crashes, assertions, etc., the program will end abnormally. Fuzz testing needs to detect these abnormal behaviors to determine and save test cases that can cause the program to end abnormally.

[0028] Furthermore, the test case selection module of the present invention is used to select a high-quality test case for subsequent mutation; the high-quality test case is determined based on whether the test case currently accessed in the test case queue is in the priority queue.

[0029] Furthermore, the priority queue described in the present invention refers to a test case queue composed of test cases with the highest execution quality for each basic block, and the test cases with the highest execution quality for each basic block are determined based on five features of the test case size, execution time, number of triggered cmp instructions, number of triggered basic blocks and total number of triggered basic blocks for executing each basic block; the determination method is to calculate the test case quality score based on the five features for judging the quality of the test case; among the five features, the test cases with smaller length, shorter execution time, more triggered cmp instructions, more triggered basic blocks and more triggered basic blocks will be updated to the corresponding positions of the priority queue.

[0030] Furthermore, the test case quality scoring method of the present invention is:

[0031]

[0032] Where score(i) is the quality score of test case i, len(i) and time(i) are the size and execution time of test case i, respectively.

[0033] Furthermore, the test case mutation module of the present invention is used to generate a new test case to be verified according to different mutation methods; the mutation methods include deterministic mutation and random mutation, wherein the number of mutations of the deterministic mutation is determined according to the size of the test case, and the number of mutations of the random mutation is determined according to the mutation energy.

[0034] Furthermore, the mutation energy of the random mutation described in the present invention is determined based on four features, namely, the number of cmp instructions triggered by the test case, the frequency of the executed path, the execution time and the mutation depth; among the four features, the test case with more triggered cmp instructions, lower execution path frequency, shorter execution time and deeper mutation depth will be allocated more mutation energy.

[0035] Furthermore, the variation energy of the random variation described in the present invention is determined as follows:

[0036]

[0037] Where energy(i) is the mutation energy of test case i, and δ is the initial energy of test case i. If the calculated energy energy(i) is greater than the limit value M, M is used as the calculation result of energy(i).

[0038] Beneficial effects:

[0039] (1) The present invention proposes a fuzz testing system framework for binary programs, which enables testing of binary programs without plugging in; the present invention improves the recording of target binary program execution information, and can adopt a feedback mechanism to construct more accurate path information and fine-grained program execution information, thereby solving the problem that most fuzz testing tools do not support testing of binary programs or have poor testing effects, and improving the efficiency of fuzz testing and vulnerability discovery capabilities.

[0040] (2) The test case characteristics, fuzzy test execution characteristics, and instruction characteristics obtained by hardware program tracing are used to select test cases and determine mutation energy, thereby solving the blindness problem in test case selection and mutation.

[0041] (3) The core of the test case selection in the present invention is to give priority to high-quality test cases, and high-quality test cases are determined based on five features: test case size, execution time, number of triggered cmp instructions, number of triggered basic blocks, and total number of triggered basic blocks. Using these test case features, test cases with faster execution and more discovered paths can be given priority, thereby improving the efficiency of fuzz testing. In addition, the present invention also provides a test case quality score calculation formula, which is simple to calculate and easy to implement.

[0042] (4) The test case mutation energy in the present invention is determined based on the four features of the number of cmp instructions triggered by the test case, the frequency of the executed path, the execution time and the mutation depth, and then the number of random mutations is determined, thereby improving the efficiency and coverage of the fuzzy test. In addition, the calculation method of energy (i) provided by the present invention uses a proportional and inverse proportional method to construct the relationship between the number of cmp instructions and the path frequency and energy, which is simple, accurate and easy to calculate, and uses a limit value M to prevent the generation of excessive energy allocation, ensuring that it does not fall into another vicious cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the fuzz testing system framework for binary programs.

[0044] Figure 2 Schematic diagram of test case characteristics.

[0045] Figure 3 Schematic diagram of test case mutation depth. DETAILED DESCRIPTION

[0046] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0047] In the present invention, test cases include initial test cases, test cases to be mutated and test cases to be verified; the initial test cases represent test cases provided manually for starting fuzz testing; the test cases to be mutated represent test cases selected from a test case queue for mutation; the test cases to be verified represent test cases generated by changing part of the content of the test cases to be mutated through different mutation methods.

[0048] Aiming at the problem that most fuzz testing tools do not support testing of binary programs or have poor testing effects, and insufficient use of effective information obtained from program tracing leads to low fuzz testing efficiency, this paper proposes a fuzz testing system for binary programs. Figure 1 As shown in the figure, the system mainly consists of three modules, namely, program tracking module, test case selection module and test case mutation module. First, the program tracking module loads the target binary program and the initial test case for actual execution to obtain the characteristics of the initial test case; then the test case selection module selects a test case for subsequent mutation operations based on the characteristics of the test case; then the test case mutation module changes the content of the test case to be mutated through different mutation methods to generate a new test case to be verified; finally, the test case to be verified is handed over to the program tracking module for actual execution and verification; the entire fuzz testing system repeats the above three modules until it is manually stopped or the time runs out.

[0049] Specifically, the program tracing module uses hardware program tracing technology to obtain program execution characteristics and instruction characteristics for test case selection and mutation; the test case selection module maintains a priority queue based on the test case characteristics, and makes probabilistic selections when selecting test cases based on whether they are in the priority queue; the test case characteristics include the test case's own characteristics, the fuzzy test execution characteristics, and the instruction characteristics obtained by hardware program tracing; the test case mutation module determines the mutation energy during random mutation based on the test case characteristics.

[0050] In the present invention, the test case features include the test case features, the fuzzy test execution features, and the instruction features obtained by hardware program tracing. Figure 2 As shown, the test case characteristics are the test case size; the fuzzy test execution characteristics are the test case execution time, mutation depth and path frequency; the instruction characteristics obtained by hardware program tracing include the number of triggered cmp instructions, basic blocks and all instructions obtained after decoding the hardware program tracing data.

[0051] To facilitate the subsequent description, the definitions of the corresponding concepts are given here:

[0052] Definition 1 Test case: Let symbol i be a test case.

[0053] Definition 2 Test case queue: Let symbol Q represent the test case queue with partial order in fuzzy testing. That is, any test case i in the test case queue may be used for subsequent mutations.

[0054] Definition 3 Basic block: Let symbol b represent the smallest unit of program execution - basic block. b is composed of one or several instructions in the program. When the program reaches a basic block, it starts from the first instruction of the basic block, and then executes other instructions in the basic block in sequence until the last instruction is executed.

[0055] Definition 4 Program Path: Let symbol p represent a path from the beginning to the end of program execution, that is, when the program is executed, multiple basic blocks are executed according to program branch, jump, return and other conditions. The path p executed for test case i is expressed as And p is a sequence of basic blocks that is actually executed by test case i, p = {b1, b2, ..., b n}, n represents the number of basic blocks executed, and also represents the length of path p.

[0056] Definition 5 Mutation depth: Set depth(i) to represent the mutation depth of test case i, that is, to which generation the test case i belongs after the mutation of the initial test case, e.g. Figure 3 As shown in FIG, if the initial test case is recorded as the first generation, then all test cases mutated from the initial test case are the second generation, and so on.

[0057] Definition 6 Path frequency: Set F(b j ) represents basic block b j The number of times it is executed by all valid test cases is referred to as the basic block frequency, and its calculation formula is:

[0058]

[0059] Where P is the set of all program paths discovered by fuzz testing.

[0060] Let freq(p) represent the path frequency. freq(p) refers to the average frequency of basic blocks in path p, which is calculated as:

[0061]

[0062] Definition 7: Number of cmp instructions: Set the symbol cmp(i) to represent the number of cmp instructions triggered by test case i, that is, the number of cmp instructions contained in path p executed by test case i. Since a basic block contains at most one cmp instruction, the number of cmp instructions of test case i can be obtained by counting how many basic blocks in a path contain cmp instructions. The calculation method is:

[0063] cmp(i)=∑T(b j ),

[0064] Definition 8: Number of basic blocks: Let block(i) represent the number of basic blocks triggered by test case i, i.e., the number of basic blocks contained in a path p executed by test case i. The calculation method is:

[0065]

[0066] Definition 9: Total number of instructions: Let inst(i) represent the number of all instructions triggered by test case i, that is, the number of all instructions contained in the execution path of the test case. The total number of instructions of test case i can be obtained by adding the number of instructions contained in each basic block in a path. The calculation method is:

[0067]

[0068] Specifically, for each module in the fuzz testing system for binary programs, the detailed description is as follows:

[0069] (1) Program tracking module

[0070] The input of this module is the target binary program and test cases, and the output is the execution characteristics and instruction characteristics of the target binary program. When the target binary program is executed, this module is used to track the actual execution of the target binary program in the CPU, decode the traced data, and monitor the abnormal behavior of the program execution.

[0071] Track the actual execution of the target binary program: When the target binary program actually executes the test case, the hardware program tracking technology is used to record the actual execution of the target binary program in the CPU, and the tracked information is compressed and stored in a fixed area of ​​​​the memory for subsequent use.

[0072] Decode the traced data: Decode the compressed trace data stored in the memory into instruction information and basic block information as required, and generate the path information corresponding to the target binary program in this execution based on the basic block information; if the path is a new path, convert the instruction information into instruction features and save them together with the test case.

[0073] Monitor abnormal behavior of program execution: When test cases trigger program vulnerabilities, crashes, assertions, etc., the program will end abnormally. Fuzz testing needs to detect these abnormal behaviors to determine and save test cases that can cause the program to end abnormally.

[0074] (2) Test case selection module

[0075] The input of this module is the test case queue, and the output is the test case to be mutated selected according to the test case characteristics; when selecting a test case, if the currently accessed test case in the test case queue is in the priority queue, it is directly selected as the test case to be mutated, otherwise a random value is given for probabilistic selection.

[0076] The priority queue in the present invention refers to a test case queue composed of the test cases with the highest quality for executing each basic block, and the test cases with the highest quality for executing each basic block are determined based on the five features of the test case size, execution time, number of triggered cmp instructions, number of triggered basic blocks, and total number of triggered basic blocks for executing each basic block. When specifically determined, the test case quality score is calculated based on the above five features to judge the quality of the test case, and among these five features, the test cases with smaller length, shorter execution time, more triggered cmp instructions, more triggered basic blocks, and more total number of triggered basic blocks will be updated to the corresponding position of the priority queue.

[0077] Furthermore, the test case quality scoring method of the present invention is:

[0078]

[0079] Where score(i) is the quality score of test case i, len(i) and time(i) are the size and execution time of test case i, respectively.

[0080] (3) Test case mutation module

[0081] The input of this module is the test case to be mutated, and the output is the test case obtained by mutation to be actually executed for verification. When this module mutates the test case, its mutation methods include deterministic mutation and random mutation. The number of mutations of deterministic mutation is determined according to the size of the test case, and the number of mutations of random mutation is determined according to the mutation energy.

[0082] The mutation energy of random mutation is determined based on the four features of the number of cmp instructions triggered by the test case, the frequency of the executed path, the execution time and the mutation depth. Among the above four features, the test cases with more cmp instructions triggered, lower execution path frequency, shorter execution time and deeper mutation depth will be allocated more mutation energy. The specific method for determining the mutation energy of random mutation is as follows:

[0083]

[0084] Where energy(i) is the mutation energy of test case i, and δ is the initial energy of test case i. If the calculated energy energy(i) is greater than the limit value M, M is used as the calculation result of energy(i).

[0085] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A fuzz testing system for binary programs, characterized in that: The system mainly includes three modules, namely program tracking module, test case selection module and test case mutation module, among which: the program tracking module first loads the target binary program and the initial test case for actual execution to obtain the characteristics of the initial test case; then the test case selection module selects a test case for subsequent mutation operations according to the characteristics of the test case; then the test case mutation module changes the content of the test case to be mutated through different mutation methods to generate a new test case to be verified; finally, the test case to be verified is handed over to the program tracking module for actual execution and verification; the entire fuzz testing system repeatedly executes the above three modules until it is manually stopped or the time runs out; The input of the program tracking module is the target binary program and test cases, and the output is the execution characteristics and instruction characteristics of the target binary program. When the target binary program is executed, the module uses the CPU program tracking mechanism to track the program execution behavior and obtain the execution characteristics and instruction characteristics of the target binary program. The input of the test case selection module is the test case queue, and the output is the test case to be mutated selected according to the test case characteristics; when selecting a test case, if the current test case in the test case queue is in the priority queue, it will be directly selected as the test case to be mutated, otherwise a random value will be given for probabilistic selection; The input of the test case mutation module is the test case to be mutated, and the output is the test case obtained by mutation to be actually executed for verification; when mutating the test case, this module is mainly divided into deterministic mutation and random mutation, where the number of mutations of deterministic mutation is determined according to the size of the test case, and the number of mutations of random mutation is determined according to the mutation energy; The test case features include the test case features, fuzzy test execution features and instruction features obtained by hardware program tracing; the test case features are the test case size; the fuzzy test execution features are the test case execution time, mutation depth and path frequency; the instruction features obtained by hardware program tracing include the number of triggered cmp instructions, basic blocks and all instructions obtained after decoding the hardware program tracing data; The mutation depth (i) of the test case: the generation to which the test case belongs after the mutation of the initial test case, that is, if the initial test case is recorded as the first generation, then all test cases mutated from the initial test case are the second generation, and so on; The path frequency freq(p) of the test case: the average number of times the basic blocks contained in the execution path of the test case are executed; The number of triggered cmp instructions cmp(i): the number of cmp instructions contained in the execution path of the test case; The number of basic blocks triggered block(i): the number of basic blocks contained in the execution path of the test case; The number of all triggered instructions inst(i): the number of all instructions contained in the execution path of the test case; The test case selection module is used to select a high-quality test case for subsequent mutation; the high-quality test case is determined based on whether the test case currently accessed in the test case queue is in the priority queue; The priority queue refers to a test case queue composed of the test cases with the highest quality of executing each basic block, and the test cases with the highest quality of executing each basic block are determined based on five features of the test case size, execution time, number of triggered cmp instructions, number of triggered basic blocks, and total number of triggered basic blocks for executing each basic block; the determination method is to calculate the test case quality score based on the five features to judge the quality of the test case; among the five features, the test cases with smaller length, shorter execution time, more triggered cmp instructions, more triggered basic blocks, and more total number of triggered basic blocks will be updated to the corresponding positions of the priority queue; The test case quality scoring method is as follows: Where score(i) is the quality score of test case i, len(i) and time(i) are the size and execution time of test case i respectively; The test case mutation module is used to generate a new test case to be verified according to different mutation methods; the mutation methods include deterministic mutation and random mutation, wherein the number of mutations of the deterministic mutation is determined according to the size of the test case, and the number of mutations of the random mutation is determined according to the mutation energy; The mutation energy of the random mutation is determined based on four features: the number of cmp instructions triggered by the test case, the frequency of the executed path, the execution time and the mutation depth; among the four features, the test case with more cmp instructions triggered, the lower the execution path frequency, the shorter the execution time and the deeper the mutation depth will be allocated more mutation energy; The variation energy of the random variation is determined as follows: Where energy(i) is the mutation energy of test case i, and δ is the initial energy of test case i. If the calculated energy energy(i) is greater than the limit value M, M is used as the calculation result of energy(i).

2. The fuzz testing system for binary programs as claimed in claim 1, characterized in that: The program tracking module is used to track the actual execution of the target binary program, decode the data obtained by tracking, and monitor abnormal behavior of program execution; Tracking the actual execution of the target binary program: When the target binary program actually executes the test case, the hardware program tracking technology is used to record the actual execution of the target binary program in the CPU, and the tracked information is compressed and stored in a fixed area of ​​the memory, waiting for subsequent use; Decode the traced data: decode the trace data compressed and stored in the memory into instruction information and basic block information as required, and generate the path information corresponding to the target binary program in this execution according to the basic block information; if the path is a new path, convert the instruction information into instruction features and save them together with the test case; Monitor abnormal behavior of program execution: When test cases trigger program vulnerabilities, crashes, assertions, etc., the program will end abnormally. Fuzz testing needs to detect these abnormal behaviors to determine and save test cases that can cause the program to end abnormally.

Images