Fuzzing method and apparatus

Abstract

A method and device for fuzz testing are provided. The method comprises: performing fuzz testing on a target program by using a first test case set, and determining characteristic values of a plurality of characteristics of test cases in the first test case set; determining a first target characteristic according to influences of the plurality of characteristics on discovery of interesting test cases; selecting, from the first test case set, test cases whose characteristic values of the first target characteristic satisfy a first preset condition, to obtain a second test case set; and performing fuzz testing on the target program based on the second test case set.

Description

Technical Field

[0001] This disclosure relates to the field of software testing technology, and more specifically, to a fuzz testing method and apparatus. Background Technology

[0002] In related technologies, fuzz detection tools typically utilize manually designed energy allocation methods to allocate computational resources to test cases involved in mutations, hoping that the mutated input file will trigger abnormal behavior in the software under test. However, manually designed standards cannot be applied to all target programs, resulting in varying vulnerability discovery efficiency when testing different target programs. Summary of the Invention

[0003] This disclosure provides a fuzz testing method and apparatus to solve the above-mentioned problems.

[0004] In a first aspect, a fuzzing method is provided, comprising: performing fuzzing on a target program using a first test case set to determine feature values ​​of multiple features of test cases in the first test case set; determining a first target feature based on the influence of the multiple features on discovering interesting test cases; selecting test cases from the first test case set whose feature values ​​of the first target feature satisfy a first preset condition to obtain a second test case set; and performing fuzzing on the target program based on the second test case set.

[0005] Optionally, the method further includes: performing fuzz testing on the target program using a third set of test cases to determine the feature values ​​of multiple features of the test cases in the third set of test cases; determining a second target feature based on the impact of the multiple features on discovering interesting test cases; selecting test cases from the third set of test cases whose feature values ​​of the second target feature satisfy a second preset condition to obtain a fourth set of test cases; and performing mutation operations on the test cases in the fourth set of test cases to obtain the first set of test cases.

[0006] Optionally, the step of performing fuzz testing on the target program based on the second test case set includes: performing mutation operations on the test cases in the second test case set to obtain a fifth test case set; and using the fifth test case set to perform fuzz testing on the target program.

[0007] Optionally, the feature values ​​of the first target features of the test cases in the second test case set are all greater than or less than the feature values ​​of the first target features of the test cases in the first test case set other than the test cases in the second test case set.

[0008] Optionally, the ratio of the number of test cases in the second test case set to the number of test cases in the first test case set is less than or equal to 0.4.

[0009] Optionally, the feature values ​​of the second target features of the test cases in the fourth test case set are all greater than or less than the feature values ​​of the second target features of the test cases in the third test case set other than the test cases in the fourth test case set.

[0010] Optionally, the ratio of the number of test cases in the fourth test case set to the number of test cases in the third test case set is less than or equal to 0.4.

[0011] Optionally, the plurality of features includes one or more of the following: basic features of the test case, features of the basic blocks of the target program triggered by the test case, and execution path features triggered by the test case.

[0012] Optionally, the interesting test cases include one or more of the following: test cases that can trigger new execution paths, and test cases that can trigger abnormal situations in the target program.

[0013] Secondly, a fuzzing apparatus is provided, comprising: a first determining module, configured to perform fuzzing on a target program using a first test case set, and determine feature values ​​of multiple features of test cases in the first test case set; a second determining module, configured to determine a first target feature based on the influence of the multiple features on discovering interesting test cases; a first selecting module, configured to select test cases from the first test case set whose feature values ​​of the first target feature satisfy a first preset condition, thereby obtaining a second test case set; and a testing module, configured to perform fuzzing on the target program based on the second test case set.

[0014] Optionally, the apparatus further includes: a third determining module, configured to perform fuzz testing on the target program using a third test case set, and determine the feature values ​​of multiple features of the test cases in the third test case set; a fourth determining module, configured to determine a second target feature based on the influence of the multiple features on discovering interesting test cases; a second selecting module, configured to select test cases from the third test case set whose feature values ​​of the second target feature satisfy a second preset condition, to obtain a fourth test case set; and a mutation module, configured to perform mutation operations on the test cases in the fourth test case set to obtain the first test case set.

[0015] Optionally, the testing module is further configured to: perform mutation operations on the test cases in the second test case set to obtain a fifth test case set; and use the fifth test case set to perform fuzz testing on the target program.

[0016] Optionally, the feature values ​​of the first target features of the test cases in the second test case set are all greater than or less than the feature values ​​of the first target features of the test cases in the first test case set other than the test cases in the second test case set.

[0017] Optionally, the ratio of the number of test cases in the second test case set to the number of test cases in the first test case set is less than or equal to 0.4.

[0018] Optionally, the feature values ​​of the second target features of the test cases in the fourth test case set are all greater than or less than the feature values ​​of the second target features of the test cases in the third test case set other than the test cases in the fourth test case set.

[0019] Optionally, the ratio of the number of test cases in the fourth test case set to the number of test cases in the third test case set is less than or equal to 0.4.

[0020] Optionally, the plurality of features includes one or more of the following: basic features of the test case, features of the basic blocks of the target program triggered by the test case, and execution path features triggered by the test case.

[0021] Optionally, the interesting test cases include one or more of the following: test cases that can trigger new execution paths, and test cases that can trigger abnormal situations in the target program.

[0022] Thirdly, a fuzz testing apparatus is provided, comprising a memory and a processor, wherein the memory stores executable code and the processor is configured to execute the executable code to implement the method as described in the first aspect or any possible implementation thereof.

[0023] Fourthly, a computer-readable storage medium is provided having executable code stored thereon, which, when executed, enables the implementation of the method as described in the first aspect or any possible implementation thereof.

[0024] Fifthly, a computer program product is provided, including executable code, which, when executed, enables the implementation of the method as described in the first aspect or any possible implementation thereof.

[0025] The fuzzing method provided in this disclosure evaluates the efficiency of each feature in discovering valuable input files by using the feature values ​​of multiple features of test cases during the testing process to assess the efficiency of each feature in triggering paths and crashes. Based on the most effective feature in the current test round, the test case with the highest feature value of that feature is selected as the seed for the next test round. This enables adaptive efficiency evaluation on different target programs and determines the energy allocation method, thereby improving the vulnerability discovery efficiency of the fuzzing tool. Attached Figure Description

[0026] Figure 1 This is a schematic flowchart of the fuzz testing method provided in the embodiments of this disclosure.

[0027] Figure 2 This is a schematic diagram of the structure of a two-dimensional linked list for storing test cases provided in an embodiment of this disclosure.

[0028] Figure 3 This is a schematic structural diagram of a fuzzy testing device provided in an embodiment of the present disclosure.

[0029] Figure 4 This is a schematic structural diagram of a fuzzy testing device provided in another embodiment of the present disclosure.

[0030] Figure 5 This is a schematic structural diagram of a fuzzy testing device provided in another embodiment of this disclosure. Detailed Implementation

[0031] To facilitate understanding of this disclosure, it will be described in more detail below based on exemplary embodiments and in conjunction with the accompanying drawings. The same or similar reference numerals are used in the drawings to denote the same or similar modules. It should be understood that the drawings are merely illustrative, and the scope of protection of this disclosure is not limited thereto. Before introducing the embodiments of this disclosure, the concepts and terms involved in the technical solutions of this disclosure are defined and explained as follows.

[0032] Gray box fuzzy testing

[0033] Gray-box fuzzing (GF) is the process of discovering vulnerabilities in a program by providing it with a large number of test cases. Gray-box testing is an improvement on traditional randomized fuzzing. It can guide test case generation based on information about the target program's internal structure, thus improving testing efficiency.

[0034] Test cases

[0035] A test case is a description of a specific software product testing task, embodying the test plan, methods, techniques, and strategies. Its content includes test objectives, test environment, input data, test steps, expected results, test scripts, etc., ultimately forming a document. Simply put, a test case is a set of test inputs, execution conditions, and expected results compiled for a specific goal to verify whether a particular software requirement is met.

[0036] Basic blocks

[0037] A basic block (BBL) is a sequence of statements in a program that is executed as sequentially as possible. It has only one entry point and one exit point. The entry point is the first statement, and the exit point is the last statement. During execution, a basic block is entered only from its entry point and exited only from its exit point.

[0038] Specifically, there is only one entry point, meaning that no other code in the program can enter the code of the basic block other than the entry point through jumps or other means. There is only one exit point, meaning that the program will only execute instructions in other basic blocks after executing the last instruction.

[0039] Therefore, a typical characteristic of basic blocks is that once the first instruction in a basic block is executed, all instructions within the basic block will be executed sequentially and only once.

[0040] Control Flow Graph

[0041] A control flow graph (CFG) is a concept in program analysis. A CFG is a directed graph that describes the execution flow of a program. A CFG consists of code block nodes and edges connecting these blocks. A code block can be a function or a basic block. The CFG contains information about all possible branches of execution within the program. A graph describing the control flow within a single function is typically called an "intra-process" CFG, while a graph describing the control flow of the entire program is called an "inter-process" CFG.

[0042] program

[0043] A program, also known as a computer program, is a set of instructions that a computer can recognize and execute. A program runs on a computer to fulfill a user's specific needs. A program can consist of multiple subroutines. Each subroutine can also be called a computer function. A function can refer to a piece of code or a set of instructions that can be directly referenced by another function.

[0044] A program can be implemented using only one function (or subroutine) or multiple functions. This disclosure does not limit the number of functions a program can contain. A program can be written in various programming languages, and this disclosure does not limit the specific programming language used. For example, a program can be written in C. Another example is a program written in Java. A program can be implemented using a single programming language or multiple different programming languages; this disclosure does not limit the number of programming languages ​​used in a program.

[0045] Computer software (or simply software) can be a general term for programs and documentation within a computer system. In some contexts, software can also refer to a program. This disclosure does not distinguish between software and programs; that is, the program mentioned in this disclosure can also refer to computer software.

[0046] With the rapid development and popularization of the internet, software is being comprehensively and widely applied to personal life, enterprise production, business operations, and government management and services. However, the application of internet technology is constantly blurring the boundaries between the physical and virtual worlds, and its integration, penetration, and driving effect on the overall economic and social development are becoming increasingly apparent, while the risks and challenges it brings are also constantly increasing.

[0047] In related technologies, testing is necessary to ensure the robustness of computer programs during operation. The main purpose of testing is to uncover vulnerabilities in the program. A vulnerability in a program refers to an abnormal operating state, or in other words, a program that does not operate as expected. For example, a vulnerability might be due to the lack of certain checks in the program's code (e.g., the lack of checks for non-zero values ​​in the dividend of a division operation), leading to program crashes. Another example is the unintentional modification of certain variables in the program during execution (e.g., the unintentional modification of a global variable), affecting the normal operation of the program. Vulnerabilities in programs can have serious consequences. For instance, a vulnerability might cause the program or even the entire computer system to crash.

[0048] To facilitate understanding, we'll use an operating system as an example to illustrate the impact of program vulnerabilities on practical use. An operating system can be viewed as a large program. It is the core component of the entire computer system. The operating system abstracts low-level hardware resources and provides services to the application layer. The kernel (or operating system kernel) is the core of an operating system, representing the first layer of software extension based on the hardware. The kernel provides the most basic functions for the operating system and is fundamental to its operation. It manages system processes, memory, device drivers, files, and network systems. The performance and stability of the kernel determine the performance and stability of the entire operating system.

[0049] With the continuous development of internet technology, the number of various computing devices, mobile terminals, and Internet of Things (IoT) devices has exploded. To adapt to this, the number and types of operating systems are also constantly increasing. Correspondingly, the number and types of operating system kernels are also increasing. Because operating systems are widely used, operating system kernel vulnerabilities are characterized by high risk and wide-ranging impact. These characteristics make the discovery of vulnerabilities in operating system kernels a significant challenge.

[0050] Operating system kernel vulnerabilities have caused very serious damage. For example, the Dirty Cow vulnerability (CVE-2016–5195), which was discovered in 2016, could be exploited by attackers to gain kernel privilege escalation and thus control of the entire system. The Dirty Cow vulnerability affected multiple Linux kernel-based operating systems, including mobile, PC, and server operating systems.

[0051] Therefore, how to quickly and efficiently discover vulnerabilities has become an urgent problem to be solved.

[0052] In related technologies, code review can be used to address vulnerability localization. This relies on the experience of human experts, who summarize and determine vulnerability rules based on existing knowledge. Lexical analysis, syntactic analysis, and semantic analysis are then performed on the software source code at different compilation levels to review and identify code errors. However, due to the limitations of human experience, static code review suffers from high false positive and false negative rates in vulnerability detection, resulting in poor vulnerability localization accuracy.

[0053] In recent years, with the development of artificial intelligence technology, intelligent static vulnerability detection systems based on deep learning technology have been proposed, which has alleviated the predicament of static analysis technology to some extent. However, due to the scarcity of vulnerability datasets, the inability to fundamentally solve the problems of false positives and false negatives, and the security issues of the model itself, there is still considerable room for improvement.

[0054] To efficiently identify potential software vulnerabilities, automated vulnerability discovery technologies have garnered significant attention in both academia and industry. One widely used method is "grey-box fuzzing based on mutation algorithms." Gray-box fuzzing tools generate a large number of test cases using specific mutation algorithms. These test cases are used as input to the program under test, executing the target program and recording its execution status in real time. Test cases that trigger anomalous behavior in the program under test are saved locally for subsequent manual analysis; test cases that trigger new execution paths are added to the saved test case set. Based on a pre-defined energy allocation method, mutation counts are assigned to the test cases in the saved set, and the mutated test cases are used for the next round of testing, thus achieving automated vulnerability detection and reporting. Although this method has a certain degree of randomness, gray-box fuzzing tools remain one of the most effective vulnerability detection solutions currently available and are widely used in the field of vulnerability detection.

[0055] In the fuzzing described above, energy refers to the number of mutations a test case can undergo, while energy allocation refers to the number of mutations the fuzzing tool assigns to each test case. It's understandable that the energy allocation algorithm of a fuzzing tool has a significant impact on key parameters such as the direction of test case mutations, vulnerability triggering potential, and execution speed.

[0056] In related technologies, most fuzzing tools cannot adaptively explore which type of test cases are more suitable for vulnerability discovery in the current target program, and mostly rely on manually designed standards to allocate energy. For example, the most commonly used gray-box fuzzing tool, AFL, only focuses on short and fast-execution test cases, and has designed a customized energy allocation algorithm to allocate more energy to these short and fast-execution test cases. Emerging fuzzing tools such as MOpt, AFL++, and EcoFuzz ​​also mostly use manually designed standards to allocate more computing power to test cases that meet the standards.

[0057] However, manual annotation methods lack adaptability. Experiments show that the efficiency of energy allocation algorithms determined by different selection criteria varies in vulnerability discovery on different target programs. There is no absolutely optimal manual standard suitable for all target programs.

[0058] In view of this, the present disclosure provides a fuzzing testing method and apparatus to solve the above problems. The fuzzing testing method provided in this disclosure will be described in detail below with reference to the accompanying drawings.

[0059] Figure 1 This is a schematic flowchart of the fuzz testing method provided in the embodiments of this disclosure. Figure 1As shown, the fuzz testing method provided in this embodiment may include steps S110-S140.

[0060] In step S110, the target program is fuzz-tested using the first test case set to determine the feature values ​​of multiple features of the test cases in the first test case set.

[0061] The first test case set may include multiple test cases, which are sometimes referred to as seed files. These multiple test cases can be randomly generated by the testing tool, or they can be test cases specifically designed for the relevant characteristics of the target program.

[0062] The first test case set can be any set of test cases used by the user to perform fuzz testing on the target program. This disclosure does not impose specific limitations on the first test case set. For example, the first test case set can be the original seed file set used for the first round of testing, or it can be a set of test cases obtained by performing mutation operations on the seed files based on the previous round of testing.

[0063] When performing fuzz testing using the first set of test cases, recording the feature values ​​of multiple features of each test case can provide selection criteria for subsequent seed replacement.

[0064] In some implementations, the above-mentioned features may include one or more of the following: basic features of the test case, features of the basic blocks of the target program triggered by the test case, and features of the execution path triggered by the test case.

[0065] The basic characteristics of test cases may include, for example, execution speed, slow execution speed, large seed file, small seed file, seed file depth, number of interesting test cases found, and edge change efficiency; the characteristics of the basic blocks of the target program triggered by test cases may include, for example, the number of untriggered neighbor basic blocks, the number of basic blocks containing constant comparison instructions, the number of memory read / write operations, the number of memory function calls, the number of global variable assignment operations, the number of interesting test cases found, and the number of high-risk basic blocks; the execution path characteristics triggered by test cases may include, for example, rare paths, the number of triggered edges, and the number of triggered loops.

[0066] The basic characteristics of test cases can be obtained by monitoring the execution of the target program for each test case. For example, the characteristic value of "fast execution speed" can be determined by monitoring the execution time of the target program during testing.

[0067] The characteristics of the basic blocks that trigger the test case in the target program can be determined by identifying the basic blocks traversed by the execution path of the test case and calculating the sum of the characteristic values ​​of the basic blocks. Therefore, in some embodiments, the method provided by this disclosure further includes: performing static analysis on the target program before testing the target program using the first test case to obtain the basic block information of the target program.

[0068] Static analysis is a method of analyzing a program without running it. Static analysis can be performed manually or implemented using computer software or hardware. This disclosure does not limit the specific implementation method of static analysis.

[0069] Static analysis of a target program can be performed in various ways. For example, the program's source code can be analyzed directly. Alternatively, the program can be compiled, and the resulting files analyzed. The compilation process typically generates intermediate files and binary files. In some embodiments, the intermediate files generated after compilation can be analyzed. Generally, different programming languages ​​using the same compiler (e.g., gcc or clang) produce intermediate files with similar or identical forms. Therefore, analyzing the intermediate files generated after compilation is applicable to multiple programming languages, offering strong versatility. As an example, a program can be compiled into LLVM intermediate code. Intermediate code can also be called intermediate representation (IR). LLVM intermediate code has mature static analysis tools. Therefore, compiling a program into LLVM intermediate code can reuse existing tools, simplifying the process. A program can be compiled into LLVM intermediate code in various ways. For example, the clang compiler can be used to compile the program into LLVM intermediate form.

[0070] By analyzing the source code of the target program, the characteristic values ​​of the basic blocks in the target program can be determined.

[0071] Static analysis of a program can generate its control flow graph. The control flow graph represents all possible branches that can be executed in the program. By analyzing the control flow graph, the runtime context of basic blocks and information about the successor basic blocks can be obtained.

[0072] After obtaining the basic block characteristics of the target program using the above method, the basic block information can be obtained by analyzing the path used for testing. By summing the characteristic values ​​of all basic blocks passed through, the sum of the characteristic values ​​of all basic blocks passed through in the test case can be determined.

[0073] The execution path characteristics triggered by test cases can be determined based on the path coverage of the target program executed by the test cases.

[0074] In step S120, a first target feature value is determined based on the impact of the multiple features on the discovery of interesting test cases.

[0075] In this embodiment of the disclosure, the interesting test case refers to a test case that can trigger a new execution path or a test case that can trigger an abnormal situation in the target program.

[0076] The following example briefly illustrates the method for determining the first target feature value. Taking a first test case set including four test cases: test case A, test case B, test case C, and test case D, and the multiple features including fast execution speed, small seed file, and a small number of memory read / write operations as an example.

[0077] The execution speed characteristics of the four test cases are 1ms, 2ms, 1.5ms, and 1.7ms, respectively; the seed file sizes are 2KB, 1.7KB, 1.3KB, and 2.1KB, respectively; and the number of memory read / write operations are 5, 7, 7, and 10, respectively.

[0078] It is understandable that the three features mentioned above belong to different dimensions and are difficult to compare with each other. Therefore, in order to facilitate the determination of the impact of different features on the discovery of interesting test cases, this embodiment of the disclosure defines temporary scores for test cases with respect to different features, and compares the impact of each feature based on these temporary scores. Taking the feature of fast execution speed as an example, the feature value of each test case with respect to the feature of fast execution speed is compared with the best performance among all test cases. In this example, the best performance of the feature of "fast execution speed" among the four test cases is 1ms for test case A. Therefore, the feature values ​​of test cases B, C, and D with respect to the feature of "fast execution speed" can be compared with test case A, and the temporary scores of the four test cases with respect to the feature of fast execution speed are determined to be 1, 0.5, 0.66, and 0.59, respectively. Based on the same method, the temporary scores of the four test cases with respect to the features of "small seed file" and "memory read / write operands" are determined to be 0.65, 0.76, 1, 0.62 and 1, 0.71, 0.625, 0.5, respectively.

[0079] Therefore, the temporary scores of the four test cases on the three different characteristics of "fast execution speed", "small seed file" and "number of memory read and write operations" are as follows: Test case A: 1, 0.65, 1; Test case B: 0.5, 0.76, 0.71; Test case C: 0.66, 1, 0.625; Test case D: 0.59, 0.62, 0.5.

[0080] Assuming that test case B triggers a new execution path during test execution, then based on the temporary scores of test case B on the above three features, the feature "small seed file" is determined as the first target feature.

[0081] It should be understood that the above method for determining the first target feature based on temporary scores is only an example. There are many other ways to determine the first target feature, such as through artificial intelligence or based on experience.

[0082] In this embodiment, the fuzzing method can be gray-box fuzzing based on a mutation algorithm. That is, during testing, the seed file (i.e., test cases) needs to be mutated to generate mutated test cases, and a new round of testing is performed. Based on the results of the new round of testing, all or some of the test cases are mutated for the next round of testing, and this process is repeated until the test results converge or the test coverage reaches a certain requirement, at which point the test ends. As mentioned earlier, if all test cases are mutated in each round of testing, it will lead to slow execution speed, thus reducing testing efficiency, and also reduce the efficiency of vulnerability discovery.

[0083] Therefore, in the fuzz testing method of the embodiments of this disclosure, after determining the first target feature, the method further includes, in step S130, selecting a second set of test cases from the first set of test cases whose feature values ​​of the first target feature satisfy the first preset condition.

[0084] The first preset condition could be, for example, that the feature value of the first target feature is greater than a preset threshold. However, the test cases used when testing different target programs may be different, so the value of the preset threshold is difficult to determine.

[0085] Therefore, in some implementations, the first preset condition can be configured such that the feature values ​​of the first target feature of all test cases in the second test case set are greater than (or less than) the feature values ​​of the first target feature of all test cases in the first test case set excluding the test cases in the second test case set. Taking the example above, if the first target feature is "small seed file", then two test cases with smaller feature values ​​can be selected from the four test cases in the first test case set as the second test case set. That is, test case B and test case C are selected into the second test set. In this case, the feature values ​​of test case B and test case C in the second test case set regarding the feature "small seed file" are 1.7KB and 1.3KB respectively, both less than the 2KB and 2.1KB values ​​of the remaining test cases A and B (excluding test cases B and C) in the first test case set.

[0086] It should be understood that the number of test cases in the second test case set should be less than the number of test cases in the first test case set in order to achieve the effect of filtering test cases.

[0087] As a preferred embodiment, the ratio of the number of test cases in the second test case set to the number of test cases in the first test case set is less than or equal to 0.4. This allows the test cases selected from the first test case set to have better performance and is more conducive to improving testing efficiency.

[0088] In step S140, fuzz testing is performed on the target program based on the second set of test cases.

[0089] In some implementations, the fuzzing of the target program based on the second test cases may involve performing a mutation operation on all test cases in the combination of the second test cases to obtain a fifth test case set, and then using the test cases in the fifth test case set to perform fuzzing on the target program.

[0090] In the above method, the mutation operation targets selected test cases, as shown earlier. These test cases are more effective at discovering new execution paths and triggering exceptions in the target program. The new test cases obtained by mutating these test cases are more targeted at the target program, thus making fuzz testing more efficient.

[0091] The mutation operation can also be referred to as a mutation strategy. In this disclosure, the mutation operation can refer to the operation of mutating a seed file, that is, modifying the seed file. The mutation operation mentioned in this disclosure can also refer to determining a strategy for modifying the seed file. Applying this strategy to the seed file yields the mutated output file. This disclosure does not limit the specific meaning of the mutation operation.

[0092] As an example, a mutation operation can refer to modifying specific bytes in a seed file. For instance, a mutation operation could refer to modifying the third byte of a seed file.

[0093] As another example, a mutation operation can also refer to modifying a specific input into a specific output. For example, modifying the value of a single byte can also be called a mutation operation. In some embodiments, when a byte value is 8, replacing 8 with 9 and rewriting the original byte in the seed file can be called a mutation operation.

[0094] During the mutation process of a seed file, fuzzing may simultaneously modify multiple parts of the seed file. As an example, a single mutation of a seed file might modify 3 bytes of data at the same time.

[0095] This disclosure does not limit the specific modifications to the seed file that a mutation operation refers to. For example, when a fuzz test modifies 3 bytes of data in the seed file, the modification of those 3 bytes can be considered as a mutation operation. That is, a mutation operation can include three modifications to the seed file.

[0096] For example, when a fuzz test modifies 3 bytes of data in a seed file at the same time, the modification of 2 bytes of data can be considered as a mutation operation. That is, a mutation operation can include any 2 of the 3 operations that modify the seed file.

[0097] For example, when a fuzz test modifies 3 bytes of data in a seed file at the same time, the modification of one byte can be treated as a mutation operation. That is, when a fuzz test modifies 3 bytes of data in a seed file at the same time, the 3 modifications can be treated as 3 mutation operations.

[0098] The use of bytes here is for descriptive purposes only. This disclosure does not limit the length of the target unit in the seed file where the mutation operation actually takes effect. The byte mentioned in this disclosure can refer to any unit. For example, one byte can represent one bit. Similarly, one byte can represent one word. This disclosure does not limit the actual type of fuzzing operation on bytes. Operations on bytes can include, for example, bitflip, arithmetic, interest, dictionary, etc.

[0099] The fuzzing method provided in this disclosure evaluates the efficiency of each feature in discovering valuable input files by using the feature values ​​of multiple features of test cases during the testing process to assess the efficiency of each feature in triggering paths and crashes. Based on the most effective feature in the current test round, the test case with the highest feature value is selected as the seed for the next test round. This enables adaptive efficiency evaluation on different target programs and determines the energy allocation method, thereby improving the vulnerability discovery efficiency of the fuzzing tool.

[0100] In some implementations, the fuzzing method disclosed herein can be performed cyclically. That is, the first test case set for the current test round can be obtained by filtering from multiple test cases in the previous round based on the test results and target features. In this case, the first test case set can be determined using the following method: fuzz testing the target program using a third test case set to determine the feature values ​​of multiple features of the test cases in the third test case set; determining a second target feature based on the impact of these multiple features on discovering interesting test cases; selecting a fourth test case set from the third test case set whose feature values ​​of the second target feature satisfy a second preset condition; and performing mutation operations on the test cases in the fourth test case set to obtain the first test case set.

[0101] The third test case set can also be called the initial seed queue, which includes multiple test cases; or, the third test case set can be obtained using a method similar to that used for the first test case set.

[0102] It should be understood that in the embodiments of this disclosure, the first target feature and the second target feature are only for ease of description. The first target feature and the second target feature may be the same or different, and this disclosure does not specifically limit them.

[0103] Based on the second target characteristic, a fourth test case set can be selected from the third test case set, wherein the fourth test case set satisfies the second preset condition. It is understood that the number of test cases in the fourth test case set should be less than the number of test cases in the third test case set to achieve the desired test case selection effect.

[0104] Therefore, as a preferred implementation, similar to the second test case set mentioned above, the ratio of the number of test cases in the fourth test case set to the number of test cases in the third test case set is less than or equal to 0.4. This allows the test cases selected from the third test case set to have better performance and is more conducive to improving testing efficiency.

[0105] As mentioned above, based on the impact of each feature on the discovery of interesting test cases, a provisional score for each test case with respect to the target feature can be determined. This provisional score is the ratio of the feature value of a certain feature in each test case to the best performance of that feature value among multiple test cases.

[0106] Based on this, in some embodiments, the method of this disclosure further includes: accumulating feature values ​​of test cases with respect to multiple features to determine the potential score of the test case. Based on the potential score, the test cases in the test case set are stored in a two-dimensional linked list structure based on a hash table, such as... Figure 2As shown, each test case stores the potential score and feature values ​​of each feature. This disclosure defines a path checksum, which is the value obtained by hashing the seed execution path. It can be considered the ID of the execution path. Two seeds with the same path checksum mean they executed the same path and can be stored on the same checksum node.

[0107] In this embodiment of the disclosure, the number of high-risk basic blocks and the rare path feature among the multiple features change during runtime. Therefore, it is necessary to dynamically update the determination of whether one or more basic blocks of the target program are high-risk blocks, and whether the execution path of the test case is a rare path.

[0108] In some implementations, the method further includes: updating the number of high-risk basic blocks; when a test case triggers a rare crash on the original executable, using the test case to execute a binary program that records the last basic block; reading the identifier of the last basic block from the newly added shared memory; and marking the basic block and its adjacent basic blocks as high-risk blocks.

[0109] In some implementations, the fuzzing method of this disclosure further includes updating rare path features. For example... Figure 2 As shown, the capacity values ​​for rare paths and normal paths are set differently. First, the rarest execution path is determined. When the rarest execution path changes, the capacity of the original rare path is restored to the minimum value of 1, and the capacity of the new rare path is adjusted to 1 / 16 of the seed queue length.

[0110] In the method of this embodiment, determining a first target feature based on the influence of the multiple features on discovering interesting test cases, and selecting test cases from the first test case set whose feature values ​​satisfy a first preset condition to obtain a second test case set may involve: maintaining multiple feature queues, wherein each feature queue stores test cases with higher feature values ​​on the feature corresponding to that queue. The selection probability of each feature queue is determined by statistically analyzing the path triggering and crash detection efficiency of the test cases in each feature queue, and the feature queue with the highest selection probability is used as the second test case set.

[0111] The number of test cases contained in each feature queue is configured such that the ratio of the number of test cases in the first test case set is less than or equal to 0.4.

[0112] Based on the selection probabilities of the above multiple feature queues, during multi-round testing, the selection probabilities can be updated after each round of testing, and the selection probability distribution of each feature queue can be standardized so that the sum of the selection probabilities of all feature queues is 1. Feature queues are selected according to the probability distribution, and the test cases contained in them are tested in a concentrated manner. The selection probability of each feature queue is iteratively optimized.

[0113] Combined with the preceding text Figures 1-2 The method embodiments of this disclosure have been described in detail below, in conjunction with... Figures 3-5 This section introduces embodiments of the apparatus described herein. It should be understood that the apparatus embodiments and the method embodiments correspond to each other. Where the apparatus embodiments are not described in detail, please refer to the description in the method embodiments.

[0114] Figure 3 This is a schematic diagram of the structure of a fuzzy testing device provided in an embodiment of this disclosure. Figure 3 As shown, the device 300 may include: a first determining module 310, a second determining module 320, a first selecting module 330, and a testing module 340.

[0115] The first determining module 310 is used to determine the feature values ​​of multiple features of test cases in the first test case set during the process of performing fuzz testing on the target program using the first test case set.

[0116] The second determining module 320 is used to determine the first target feature based on the impact of the multiple features on the discovery of interesting test cases.

[0117] The first selection module 330 is used to select test cases from the first test case set whose feature values ​​of the first target feature satisfy the first preset condition, so as to obtain a second test case set.

[0118] Test module 340 is used to perform fuzz testing on the program based on the second set of test cases.

[0119] Optionally, in some embodiments, the device 300 further includes:

[0120] The third determination module is used to perform fuzz testing using a third set of test cases to determine the feature values ​​of multiple features of the test cases in the third set of test cases.

[0121] The fourth determination module is used to determine the second target feature based on the impact of the multiple features on the discovery of interesting test cases.

[0122] The second selection module is used to select test cases from the third test case set where the feature value of the second target feature satisfies the second preset condition, thereby obtaining the fourth test case set.

[0123] The mutation module is used to perform mutation operations on the test cases in the fourth test case set to obtain the first test case set.

[0124] Optionally, the testing module is further configured to: perform mutation operations on the test cases in the second test case set to obtain a fifth test case set; and use the fifth test case set to perform fuzz testing on the target program.

[0125] Optionally, the feature values ​​of the first target features of all test cases in the second test case set are greater than the feature values ​​of the first target features of all test cases in the first test case set other than those in the second test case set.

[0126] Optionally, the ratio of the number of test cases in the second test case set to the number of test cases in the first test case set is less than or equal to 0.4.

[0127] Optionally, the feature values ​​of the second target features of the test cases in the fourth test case set are all greater than or less than the feature values ​​of the second target features of the test cases in the third test case set other than the test cases in the fourth test case set.

[0128] Optionally, the ratio of the number of test cases in the fourth test case set to the number of test cases in the third test case set is less than or equal to 0.4.

[0129] Optionally, the plurality of features includes one or more of the following: basic features of the test case, features of the basic blocks of the target program triggered by the test case, and execution path features triggered by the test case.

[0130] Optionally, the interesting test cases include one or more of the following: test cases that can trigger new execution paths and / or trigger abnormal conditions in the target program.

[0131] Figure 4 This is a schematic structural diagram of a fuzzy testing device provided in another embodiment of this disclosure. Figure 4 The device 400 can be any device capable of performing fuzz testing. For example, the device 400 can be a computing device with computational capabilities. For instance, the device 400 can be an elastic cloud host or a server. The device 400 can include a memory 410 and a processor 420. The memory 410 can be used to store executable code. The processor 420 can be used to execute the executable code stored in the memory 410 to implement the steps in the various methods described above. In some embodiments, the device 400 may further include a network interface 430 through which the processor 420 exchanges data with external devices.

[0132] Figure 5 This is a schematic structural diagram of a fuzzy testing device provided in another embodiment of this disclosure. Figure 5 The device 500 includes a preprocessing module 510, an exploration and collection module 520, an evaluation and update module 530, and a centralized utilization module 540.

[0133] The preprocessing module 510 is configured to acquire basic block information and compile the executable program for the target program. This includes: during the compilation of the target program source code, analyzing the program control flow graph to obtain the context relationship of the basic blocks during runtime, and recording the subsequent basic block information of each basic block; performing specific static analysis on the basic block source code according to set feature requirements, and recording the feature values ​​of each basic block; based on the original coverage instrumented binary program, using new instrumentation to record the last basic block executed by the test case, and compiling the program to generate a binary executable program. The initial random seed setting uses the name of the source code file when executing the target program to ensure the consistency of the same basic block identifier in the two executable programs.

[0134] The exploration collection module 520 is used to perform fuzz testing on test cases in the test case set, record the characteristic information of test cases, and calculate the temporary score and potential score of test cases to provide selection criteria for seed replacement.

[0135] By monitoring the execution of the target program by the test cases, the basic characteristics of the test cases are obtained; the basic blocks traversed by the execution path of the test cases are located, the sum of the characteristic values ​​of the basic blocks is calculated, and the characteristics of the basic blocks that trigger the target program by the test cases are determined; the execution path characteristics triggered by the test cases are determined based on the path coverage of the target program by the test cases.

[0136] Based on the performance of each feature in discovering interesting test cases, the temporary scores of each test case on multiple features and the potential scores of each test case are calculated. The methods for determining the temporary scores and potential scores have been described in detail in the previous method embodiments and will not be repeated here.

[0137] The evaluation update module 530 is used to construct and maintain multiple feature queues mentioned above based on the information accumulated by the preprocessing module 510 and the exploration collection module 520, and to iteratively optimize the selection probability of the feature queues based on the performance of each feature in discovering interesting test cases during the testing process.

[0138] The centralized utilization module 540 is used to standardize the selection probability distribution of the feature queues after the evaluation and update module 530 ends, so that the sum of the selection probabilities of all feature queues is equal to 1. The feature queues are selected according to the probability distribution, and the test management contained therein is used for centralized testing. The selection probability of each feature queue is iteratively optimized and then input into the exploration and collection module 520 to start the next round of fuzzy testing.

[0139] Compared with existing technologies, the beneficial effects of this solution are as follows: First, this solution can reasonably allocate the selection probability of each feature queue based on its performance on the current software under test, so that efficient feature queues are used more often and inefficient feature queues are used less often. This achieves a program-sensitive, test case-oriented, multi-dimensional adaptive efficiency evaluation technique, thereby improving the vulnerability discovery efficiency of fuzzing tools. Second, this solution can evaluate the efficiency of different feature queues for each software under test and adaptively generate a suitable selection probability distribution, solving the problem that different features have different vulnerability discovery efficiencies on different software under test. Third, since energy allocation is an essential functional module of most fuzzing tools, and the input file features of interest differ in specific application scenarios, this solution can integrate the energy allocation schemes of most fuzzing tools, and the coupling between queues is low, making it flexible and convenient to add or delete feature queues, and it has scalability. Finally, the computational overhead required by this invention is minimal, and the proposed fuzzing adaptive energy allocation method and system can effectively improve the vulnerability discovery efficiency of fuzzing tools.

[0140] It should be understood that in the various embodiments of this disclosure, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0141] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0144] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can read or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs, DVDs) or semiconductor media (e.g., solid-state disks, SSDs), etc.

[0145] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A fuzz testing method, the method comprising: The target program is fuzz-tested using a first set of test cases to determine the feature values ​​of multiple features of the test cases in the first set of test cases. Based on the impact of the multiple features on the discovery of interesting test cases, a first target feature is determined, wherein the interesting test cases include one or more of the following: test cases that can trigger new execution paths, test cases that can trigger abnormal situations in the target program; From the first test case set, test cases whose feature values ​​of the first target feature satisfy the first preset condition are selected to obtain the second test case set; Fuzz testing is performed on the target program based on the second set of test cases. The step of determining the first target feature based on the impact of the multiple features on discovering interesting test cases includes: Define temporary scores for test cases in the first test case set regarding different features, compare the impact of each feature based on the temporary scores, and determine the first target feature.

2. The method according to claim 1, further comprising: The target program is subjected to fuzz testing using a third set of test cases to determine the feature values ​​of multiple features of the test cases in the third set of test cases; Based on the impact of the aforementioned features on the discovery of interesting test cases, a second target feature is determined; From the third set of test cases, test cases whose feature values ​​of the second target feature satisfy the second preset condition are selected to obtain a fourth set of test cases; The test cases in the fourth test case set are mutated to obtain the first test case set.

3. The method according to claim 1, wherein performing fuzz testing on the target program based on the second test case set includes: The test cases in the second test case set are mutated to obtain the fifth test case set. The target program is fuzz-tested using the fifth set of test cases.

4. According to the method of claim 1, the feature values ​​of the first target features of the test cases in the second test case set are all greater than or less than the feature values ​​of the first target features of the test cases in the first test case set other than the test cases in the second test case set.

5. The method according to claim 4, wherein the ratio of the number of test cases in the second test case set to the number of test cases in the first test case set is less than or equal to 0.

4.

6. The method according to claim 2, wherein the feature values ​​of the second target features of the test cases in the fourth test case set are all greater than or less than the feature values ​​of the second target features of the test cases in the third test case set other than the test cases in the fourth test case set.

7. The method according to claim 6, wherein the ratio of the number of test cases in the fourth test case set to the number of test cases in the third test case set is less than or equal to 0.

4.

8. The method according to claim 1 or 2, wherein the plurality of features includes one or more of the following: basic features of the test case, features of the basic blocks of the target program triggered by the test case, and execution path features triggered by the test case.

9. A fuzz testing apparatus, the apparatus comprising: The first determining module is used to perform fuzz testing on the target program using a first test case set, and to determine the feature values ​​of multiple features of the test cases in the first test case set; The second determining module is used to determine the first target feature based on the impact of the multiple features on the discovery of interesting test cases, wherein the interesting test cases include one or more of the following: test cases that can trigger new execution paths, test cases that can trigger abnormal situations in the target program; The first selection module is used to select test cases from the first test case set where the feature value of the first target feature satisfies the first preset condition, so as to obtain a second test case set. The testing module is used to perform fuzz testing on the target program based on the second set of test cases; The second determining module is further configured to define temporary scores for test cases in the first test case set regarding different features, compare the impact of each feature based on the temporary scores, and determine the first target feature.

10. The apparatus according to claim 9, further comprising: The third determination module is used to perform fuzz testing on the target program using a third set of test cases to determine the feature values ​​of multiple features of the test cases in the third set of test cases. The fourth determining module is used to determine the second target feature based on the impact of the multiple features on the discovery of interesting test cases; The second selection module is used to select test cases from the third test case set in which the feature value of the second target feature satisfies the second preset condition, thereby obtaining a fourth test case set. The mutation module is used to perform mutation operations on the test cases in the fourth test case set to obtain the first test case set.

11. The apparatus according to claim 9, wherein the testing module is further configured to: The test cases in the second test case set are mutated to obtain the fifth test case set. The target program is fuzz-tested using the fifth set of test cases.

12. The apparatus according to claim 9, wherein the feature values ​​of the first target features of the test cases in the second test case set are all greater than or less than the feature values ​​of the first target features of the test cases in the first test case set other than the test cases in the second test case set.

13. The apparatus of claim 12, wherein the ratio of the number of test cases in the second test case set to the number of test cases in the first test case set is less than or equal to 0.

4.

14. The apparatus according to claim 10, wherein the feature values ​​of the second target features of the test cases in the fourth test case set are all greater than or less than the feature values ​​of the second target features of the test cases in the third test case set other than the test cases in the fourth test case set.

15. The apparatus of claim 14, wherein the ratio of the number of test cases in the fourth test case set to the number of test cases in the third test case set is less than or equal to 0.

4.

16. The apparatus according to claim 9 or 10, wherein the plurality of features includes one or more of the following: basic features of the test case, features of the basic block of the target program triggered by the test case, and execution path features triggered by the test case.

17. A fuzz testing apparatus, comprising a memory and a processor, the memory storing executable code, the processor being configured to execute the executable code to implement the method as described in any one of claims 1-8.

Images