Program testing method and device based on fuzz testing

Abstract

This disclosure discloses a program testing method and apparatus based on fuzz testing. The method comprises: generating a function call graph and a control flow graph containing indirect function call relationships through static analysis; generating a reachable set using the function call graph and the control flow graph, wherein the code in the reachable set can reach the target code to be tested through jumps; calculating the distance between the code in the reachable set and the target code; and performing fuzz testing on the program using the distance.

Description

Technical Field

[0001] The present disclosure relates to the field of testing technology, and in particular to a program testing method and device based on fuzz testing. Background Art

[0002] Fuzz testing can greatly save manpower by testing programs in an automated way. However, existing fuzz testing generates test cases randomly according to certain rules, which is highly blind and has low testing efficiency. Summary of the Invention

[0003] In view of this, the present disclosure provides a program testing method and apparatus based on fuzz testing to improve the testing efficiency of fuzz testing.

[0004] In a first aspect, a program testing method based on fuzz testing is provided, comprising: using static analysis to generate a function call graph and a control flow graph containing function indirect call relationships; using the function call graph and the control flow graph to generate a reachable set, wherein the code in the reachable set reaches the target code to be tested by jumping; calculating the distance from the code in the reachable set to the target code; and using the distance to perform fuzz testing on the program.

[0005] Optionally, in some embodiments, the function call graph and the control flow graph include indirect calling relationships of functions.

[0006] Optionally, in some embodiments, the fuzz testing the program using the distance includes: alternately using coverage-based fuzz testing and guided fuzz testing to fuzz test the program, wherein the coverage-based fuzz testing uses coverage information generated during the fuzz testing process to update the seed set, and the guided fuzz testing uses distance information generated during the fuzz testing process to update the seed set.

[0007] Optionally, in some embodiments, the alternating use of coverage-based fuzz testing and directed fuzz testing to fuzz test the program includes: when using the coverage-based fuzz testing to fuzz test the program, after a first preset condition is met, switching to using directed fuzz testing to fuzz test the program; when using the directed fuzz testing to fuzz test the program, after a second preset condition is met, switching to using coverage-based fuzz testing to fuzz test the program.

[0008] Optionally, in some embodiments, the first preset condition includes that no new coverage information is generated within a first preset time, and the second preset condition includes that no new coverage information is generated within a second preset time.

[0009] Optionally, in some embodiments, calculating the distance between the codes in the reachable set and the target code includes: calculating the distance between the codes in the reachable set and the target code using a breadth-first search method.

[0010] Optionally, in some embodiments, generating a function call graph and a control flow graph through static analysis includes: generating the function call graph using fast type analysis; and generating the control flow graph based on the function call graph.

[0011] Optionally, in some embodiments, generating a function call graph and a control flow graph through static analysis includes: compiling the program into an intermediate code form of LLVM; and statically analyzing the intermediate code form to generate the function call graph and the control flow graph.

[0012] Optionally, in some embodiments, the program is an operating system kernel program.

[0013] Optionally, in some embodiments, the codes in the reachable set are basic block codes.

[0014] In a second aspect, a device for program testing based on fuzz testing is provided, comprising: an analysis module configured to generate a function call graph and a control flow graph containing function indirect call relationships using static analysis; a generation module configured to generate a reachable set using the function call graph and the control flow graph, wherein the code in the reachable set reaches the target code to be tested by jumping; a calculation module configured to calculate the distance from the code in the reachable set to the target code; and a testing module configured to perform fuzz testing on the program using the distance.

[0015] Optionally, in some embodiments, the function call graph and the control flow graph include indirect calling relationships of functions.

[0016] Optionally, in some embodiments, the fuzz testing the program using the distance includes: alternately using coverage-based fuzz testing and guided fuzz testing to fuzz test the program, wherein the coverage-based fuzz testing uses coverage information generated during the fuzz testing process to update the seed set, and the guided fuzz testing uses distance information generated during the fuzz testing process to update the seed set.

[0017] Optionally, in some embodiments, the alternating use of coverage-based fuzz testing and directed fuzz testing to fuzz test the program includes: when using the coverage-based fuzz testing to fuzz test the program, after a first preset condition is met, switching to using directed fuzz testing to fuzz test the program; when using the directed fuzz testing to fuzz test the program, after a second preset condition is met, switching to using coverage-based fuzz testing to fuzz test the program.

[0018] Optionally, in some embodiments, the first preset condition includes that no new coverage information is generated within a first preset time, and the second preset condition includes that no new coverage information is generated within a second preset time.

[0019] Optionally, in some embodiments, calculating the distance between the codes in the reachable set and the target code includes: calculating the distance between the codes in the reachable set and the target code using a breadth-first search method.

[0020] Optionally, in some embodiments, generating a function call graph and a control flow graph through static analysis includes: generating the function call graph using fast type analysis; and generating the control flow graph based on the function call graph.

[0021] Optionally, in some embodiments, generating a function call graph and a control flow graph through static analysis includes: compiling the program into an intermediate code form of LLVM; and statically analyzing the intermediate code form to generate the function call graph and the control flow graph.

[0022] Optionally, in some embodiments, the program is an operating system kernel program.

[0023] Optionally, in some embodiments, the codes in the reachable set are basic block codes.

[0024] According to a third aspect, a computer-readable storage medium is provided, on which executable code is stored. When the executable code is executed, the method according to the first aspect can be implemented.

[0025] According to a fourth aspect, a computer program product is provided, comprising an executable code, which, when executed, can implement the method described in the first aspect.

[0026] The program testing method provided by the embodiment of the present disclosure first uses the function call graph and control flow graph to perform reachability analysis on the target code and generate a reachable set before calculating the distance between the code in the program and the target code. When calculating the distance between the code in the program and the target code, only the distance between the code in the reachable set and the target code is calculated. Compared with the related art that calculates the distance between the full amount of code and the target code, only calculating the distance between the code in the reachable set and the target code can effectively improve the efficiency of calculating the distance between codes, thereby improving testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of a program testing method based on fuzz testing provided by an embodiment of the present disclosure.

[0028] Figure 2 This is a flowchart of a program testing method based on fuzz testing provided by another embodiment of the present disclosure.

[0029] Figure 3 It is a structural diagram of a device for program testing based on fuzz testing provided in one embodiment of the present disclosure.

[0030] Figure 4 It is a structural diagram of a device for program testing based on fuzz testing provided in another embodiment of the present disclosure.

[0031] Figure 5 It is a structural diagram of a device for program testing based on fuzz testing provided in another embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present disclosure are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments.

[0033] Basic Blocks

[0034] A basic block (BBL) is a sequence of statements executed sequentially within a program. A basic block has only one entry point and one exit point: the entry point is the first statement in the block, and the exit point is the last statement. A basic block is executed only at its entry point and exit point.

[0035] There is only one entry point, which means that no other code in the program can jump to other code in the basic block except the entry point. There is only one exit point, which means that the program will not execute instructions in other basic blocks until it executes the last instruction.

[0036] A typical feature of a basic block is that once the first instruction in the basic block is executed, all instructions in the basic block will be executed only once in sequence.

[0037] Function call graph

[0038] A call graph (CG) is a concept used in program analysis. It is a directed graph that describes the call relationships between all functions in a program. Each node in the graph represents a function, and the directed edges between nodes represent the call relationships between functions.

[0039] Control Flow Graph

[0040] A control flow graph (CFG) is a concept used in program analysis. A control flow graph is a directed graph that describes the execution flow of a program. It consists of code block nodes and edges connecting them. A code block can be a function or a basic block. The control flow graph contains information about all possible branches within the program. A graph that describes control flow within a function is typically called an "intraprocedural" control flow graph, while a graph that describes control flow for the entire program is called an "interprocedural" control flow graph.

[0041] Gray-box fuzz testing

[0042] Graybox fuzzing (GF) is the process of discovering vulnerabilities in a program by providing it with a large number of test cases. Graybox testing is an improvement over traditional random fuzz testing. It can guide test case generation based on the target program's internal structure, thereby improving testing efficiency.

[0043] Coverage-based grey-box fuzz testing

[0044] Coverage-based graybox fuzzing (CGF) aims to maximize code coverage of the target program in order to discover vulnerabilities. Coverage-based graybox fuzzing, also known as coverage-based fuzz testing, is currently the most widely used fuzz testing method.

[0045] Guided gray-box fuzz testing

[0046] The core concept of Directed Greybox Fuzzing (DGF) is to focus testing efforts on target code areas within a program, rather than spending a significant amount of time exploring random paths like traditional coverage-based greybox fuzzing. Directed greybox fuzzing, also known as guided fuzzing or targeted fuzzing, is highly practical and can better utilize computing resources. It is more effective than coverage-based greybox fuzzing when testing large-scale programs.

[0047] A program, also known as a computer program, is a set of instructions that a computer can recognize and execute. A program can be run on a computer to fulfill a user's needs. A program can consist of multiple subroutines. Each subroutine is also called a computer function. A function is a section of code or set of instructions that can be directly referenced by another function.

[0048] A program can be implemented by only one function (or subroutine), or it can be implemented by multiple functions. This disclosure does not limit the number of functions contained in a program. A program can be written in a variety of different types of programming languages, and this disclosure does not limit the specific type of programming language used by the program. For example, a program can be written in C language. For another example, a program can be written in JAVA language. A program can be implemented in the same programming language, or it can be implemented in multiple different types of programming languages. This disclosure does not limit the number of programming languages ​​used in a program.

[0049] Computer software (or software) can be a general term for programs and documents in a computer system. In some scenarios, software can also refer to programs. This disclosure does not distinguish between software and programs. That is, the program mentioned in this disclosure can also refer to computer software.

[0050] To ensure the robustness of a program during operation, it is usually necessary to test the program. The main purpose of testing is to uncover vulnerabilities in the program. A vulnerability in a program refers to an abnormal operating state of the program, or in other words, a state where the program does not conform to the expected operating state. For example, a program vulnerability can refer to a lack of certain judgments in program writing (for example, a lack of judgment on the non-zero dividend of a division operation), causing the program to crash. For another example, a program vulnerability can refer to the unexpected modification of certain variables in the program during operation (for example, the unexpected modification of a global variable), affecting the normal operation of the program. Vulnerabilities in a program can have serious consequences. For example, a program vulnerability can cause the program or even the computer system to crash.

[0051] To facilitate understanding, we'll use the operating system as an example to illustrate the impact of program vulnerabilities on practical use. An operating system can be thought of as a large program. It's the core component of the entire computer system. It abstracts underlying hardware resources and provides services to the application layer. The kernel (or operating system core) is the core of an operating system and the first layer of software extensions based on the hardware. The kernel provides the most basic functions for the operating system and is the foundation of its operation. The kernel manages system processes, memory, device drivers, files, and network systems. The kernel's performance and stability determine the performance and stability of the operating system.

[0052] With the continuous development of internet technology, the number of various computing devices, mobile terminals, IoT devices, and other devices has exploded. To adapt to this growth, the number and variety of operating systems has also continued to increase. Consequently, the number and variety of operating system kernels have also increased. Due to the widespread use of operating systems, operating system kernel vulnerabilities are characterized by high risk and wide-ranging impact. These characteristics make vulnerability discovery in operating system kernels a significant challenge.

[0053] Operating system kernel vulnerabilities have caused significant damage. For example, the Dirty COW vulnerability (CVE-2016–5195) that broke out in 2016 could be exploited by attackers to escalate kernel privileges and thereby control the entire system. The Dirty COW vulnerability affected multiple Linux kernel-based operating systems, including those for mobile devices, PCs, and servers. Therefore, program vulnerability detection is an urgent issue in the field of software and computer system security.

[0054] Currently, the main methods used for program vulnerability detection include manual analysis, static analysis, and fuzz testing. Traditional vulnerability detection methods rely heavily on manual analysis. Manual analysis relies heavily on the tester's knowledge and experience, requiring significant manpower. Furthermore, manual analysis can only detect a limited number of vulnerabilities, making it difficult to meet the testing needs of programs, especially large ones.

[0055] Program static analysis refers to a code analysis technique that scans program code without executing it, using techniques such as lexical analysis, syntax analysis, control flow, and data flow analysis to verify whether the code meets standards for compliance, security, reliability, and maintainability. Program static analysis can also be referred to as static analysis. In some embodiments, static analysis can also be used to discover vulnerabilities based on manually summarized vulnerability patterns.

[0056] With the development of static analysis technology, static analysis can detect more defects that traditionally only dynamic testing can find. For example, static analysis can detect defects such as symbolic execution, abstract interpretation, and value dependency analysis.

[0057] Compared with manual analysis methods, static analysis methods have a high degree of automation. However, static analysis methods in related technologies have problems such as high false positive rate and poor scalability.

[0058] Fuzz testing modifies the program's runtime state by feeding it a large number of test cases and monitoring it for unusual behavior to detect vulnerabilities. Compared to manual and static analysis, fuzz testing offers a high degree of automation and a low false positive rate, making it a popular vulnerability detection method.

[0059] A test case can be a set of test inputs, execution conditions, and expected results written for a specific purpose, used to verify whether a program produces the expected output after executing the set of inputs. The test cases mentioned in this disclosure can refer to a complete set of test cases or just the test inputs of a program.

[0060] Fuzz testing input can also be called a seed or seed file. A seed file can be used as input for testing a target program. Therefore, it can also be called an input file. A seed file can also refer to the output file generated by a fuzz testing tool for testing a target program. Therefore, a seed file can also be called an output file. A seed file can be used as input for fuzz testing, and through fuzz testing mutations, a new seed file for testing can be generated.

[0061] Traditional fuzz testing relies on random generation of seed files. Testing a program using this randomly generated input file (or seed file) is a highly blind process, wastes significant computing resources, and is inefficient. Therefore, gray-box fuzz testing is currently the most common approach. Gray-box fuzz testing primarily includes coverage-based gray-box fuzz testing and guided gray-box fuzz testing.

[0062] Coverage-based fuzz testing focuses on improving program coverage to provide more comprehensive testing. However, this coverage-based fuzz testing technique still suffers from high blindness and low efficiency. For example, when testing large programs (such as operating system kernels), it is difficult to achieve high coverage in a short period of time, which seriously affects the efficiency of vulnerability discovery.

[0063] To improve vulnerability discovery efficiency, directed fuzz testing can be used. Unlike coverage-based fuzz testing, directed fuzz testing focuses solely on coverage of specific code locations within the program (for ease of description, these locations are referred to as target code). The goal of directed fuzz testing is to generate test cases that trigger the target code, thereby enabling vulnerability discovery within the target code.

[0064] Existing guided fuzz testing techniques primarily use distance information as feedback, continuously guiding the fuzz test to generate test cases closer to the target code location (e.g., program input) until the target location is triggered. Distance information can refer to the distance between other code in the program and the target code. Compared to coverage-based fuzz testing, guided fuzz testing is more efficient.

[0065] However, the guided fuzz testing provided by related technologies has a large time overhead when calculating static distances, which seriously affects the efficiency of vulnerability mining.

[0066] In view of this, the present disclosure provides a program testing method based on fuzz testing to improve the testing efficiency of fuzz testing.

[0067] Figure 1 This is a flowchart of a program testing method based on fuzz testing provided by an embodiment of the present disclosure. Figure 1 As shown, the testing method provided by the present disclosure may include steps S110 to S140.

[0068] In step S110, static analysis is used to generate a function call graph and a control flow graph. Static analysis is a method for analyzing a program without running it. Static analysis can be performed manually or implemented using computer software or hardware. This disclosure does not limit the specific implementation method of static analysis.

[0069] A program can be statically analyzed in a variety of ways. For example, the source code of the program can be directly analyzed. For another example, the program can be compiled and the files generated after compilation can be analyzed. The compilation process usually generates intermediate files and binary files. In some embodiments, the intermediate files generated after compilation can be analyzed. Generally speaking, different programming languages ​​​​are compiled using the same compiler (such as gcc or clang) to obtain intermediate files of the same or similar form. Therefore, analyzing the intermediate files generated after compilation can be applied to a variety of programming languages ​​​​and has strong versatility.

[0070] As an example, a program can be compiled into LLVM's intermediate code form. This intermediate code form is also called an intermediate representation (IR). LLVM's intermediate code has mature static analysis tools. Therefore, compiling a program into LLVM's intermediate code form allows for reuse of existing tools, simplifying implementation. There are many ways to compile a program into LLVM's intermediate code form. For example, the clang compiler can be used to compile a program into LLVM's intermediate code form.

[0071] By performing static analysis on a program, we can generate a function call graph and a control flow graph. The function call graph shows the calling relationships between functions in the program. The control flow graph shows all possible branches in the program.

[0072] As previously explained, a program typically contains multiple functions. By generating a function call graph, we can identify the calling relationships between functions, thereby facilitating the calculation of the distance between the program's code and the target code. The method for generating a function call graph and control flow graph through static analysis will be described later in conjunction with specific examples.

[0073] In step S120, a reachable set is generated using the function call graph and the control flow graph, wherein the code in the reachable set can reach the target code to be tested through a jump.

[0074] The code in the reachable set can refer to function-level code or basic block-level code. This disclosure does not limit the specific scope of the code in the reachable set. As described above, a basic block is the smallest code block that can be executed continuously in a program. Therefore, storing basic block-level code in the reachable set makes the distance between the code and the target code to be tested more accurate and reduces the false positive rate.

[0075] The target code to be tested refers to the code block that serves as the target during directed fuzz testing. This target code can refer to either function-level code or basic-block-level code. Using basic-block granularity for target code improves the accuracy and specificity of the fuzz testing process, reducing false positives.

[0076] The code in the reachable set can reach the target code to be tested by jumping, which means that the code in the reachable set can be executed to the target code to be tested by jumping. Jumping can refer to jumping between different code blocks. For example, a program jumps from one code block to another during execution. Jumping can also refer to the execution of one code block to another during the sequential execution of a program. This disclosure does not limit the specific reference to the jump behavior between code blocks.

[0077] This disclosure does not limit the specific references to different code blocks. For example, different code blocks can refer to different functions in a program. In another example, different code blocks can refer to different basic blocks in different functions. In another example, different code blocks can refer to different basic blocks in the same function.

[0078] In step S130, the distance between the codes in the reachable set and the target code is calculated. The present disclosure does not limit the specific type of the distance between the codes in the reachable set and the target code. For example, the distance between the codes in the reachable set and the target code may refer to the static distance between the codes.

[0079] The distance between the code in the reachable set and the target code can be calculated in many ways. As an example, the distance between the code in the reachable set and the target code can be obtained using a breadth-first algorithm (such as a breadth-first search algorithm). Take a control flow graph containing V points and E edges as an example. Each point can refer to a code block, and each edge can refer to the pointing relationship between different code blocks. Using the Dijkstra method, the complexity of the algorithm is O(V 2 ), using a breadth-first search algorithm, the algorithm complexity is O(V + E). Therefore, compared to the Dijkstra method used in the prior art, using breadth-first search to calculate distance can effectively reduce algorithm complexity and improve computational efficiency without affecting the accuracy of the calculation results.

[0080] In some embodiments, the distance between two code blocks with adjacent jump relationships (e.g., direct call relationships or indirect call relationships) can be set to a fixed value. By setting the distance between code blocks with adjacent jump relationships to a fixed value, the complexity of distance calculation can be further reduced and the computational efficiency of the distance calculation process can be improved.

[0081] In other embodiments, weights may be set for the distances between different adjacent jump relationships. For example, jump relationships with higher jump probabilities may be weighted higher or lower. By setting weights for distances, the accuracy of distance calculations can be improved, making fuzz testing more efficient when performed based on the distances.

[0082] In step S140, the program is fuzz tested using the distance. The program can be fuzz tested using the distance in a variety of different ways, and the present disclosure does not limit the specific way of fuzz testing the program using the distance.

[0083] In some embodiments, performing fuzz testing on a program using distance may include: selecting one or more seed files from a seed set to mutate to obtain new seed files; performing fuzz testing on the program using the new seed files; and updating the fuzzy test seed set using distance information.

[0084] The present disclosure does not limit the source of the test seed set. For example, instead of setting a seed set, a predetermined number of seed files can be randomly generated using traditional fuzz testing methods as the test seed set. In another example, an existing seed file can be obtained as the test seed set.

[0085] There are many ways to fuzz test a program using a new seed file. For example, you can use the new seed file as input to execute the program and monitor whether the output or execution result meets expectations to determine whether the seed file triggers program vulnerabilities or generates new code coverage.

[0086] The seed set can be updated in a variety of ways, and the present disclosure does not limit the specific way of updating the seed set. As an example, the distances of the seed files can be sorted after each execution of the fuzz test, and the N seed files with the closest distances can be retained as the input files for the fuzz test. The distance of the seed file can, for example, refer to the distance from the code covered during the execution of the seed file to the target code. The present disclosure does not limit the specific method of calculating the distance of the seed file. For example, the closest distance from the code covered during the execution of the seed file to the target code can be selected as the distance of the seed file. For another example, the average value of the distance from all covered codes to the seed file during the execution of the seed file can be selected as the distance of the seed file. N can refer to any natural number. The actual value of N can be set according to the test requirements, and the present disclosure does not limit the specific number of seed files retained during the fuzz test.

[0087] The fuzz testing method provided by the embodiments of the present disclosure first performs a reachability analysis on the code in the program before calculating the distance from the code in the program to the target code. The distance is calculated only for the code in the reachable set, reducing the distance calculation overhead for other code unrelated to the target code. Therefore, compared with the targeted fuzz testing of related technologies, the fuzz testing provided by the embodiments of the present disclosure has low computational overhead and high testing efficiency. The fuzz testing provided by the embodiments of the present disclosure can also be called a lightweight fuzz testing method.

[0088] The previous article mentioned the use of static analysis to generate function call graphs and control flow graphs. The following will introduce how to use static analysis to generate function call graphs and control flow graphs.

[0089] Static analysis can be used to generate function call graphs and control flow graphs in a variety of ways. The present disclosure does not limit the specific static analysis method. In practice, function call graphs or control flow graphs can be generated using only static analysis. When calculating the distance between codes, the distance between the code in the program and the target code can be calculated using only the function call graph, or the distance between the code in the program and the target code can be calculated using only the control flow graph. It should be understood that calculating the distance between codes using only the function call graph or the control flow graph is simple. Calculating the distance based on both the function call graph and the control flow graph at the same time makes the calculation more accurate and can reduce the probability of false positives during fuzz testing.

[0090] In some embodiments, the function call graph and control flow graph generated by static analysis may include indirect call relationships of functions. Function call graphs and control flow graphs including indirect call relationships of functions may be generated in a variety of ways. For example, a function call graph may be generated using Rapid Type Analysis, and a control flow graph may be generated based on the function call graph.

[0091] The fast type analysis method can generate not only direct calling relationships of functions, but also indirect calling relationships. Therefore, the function call graph generated by the fast type analysis method is more comprehensive, which can avoid false positives and false negatives caused by not considering the indirect calling relationships of functions in related technologies, thereby improving test accuracy.

[0092] The control flow graph generated based on the function call graph containing the indirect function call relationship also contains the indirect function call relationship. The reachable set is generated by using the function call graph and control flow graph containing the indirect function call relationship. The code in the reachable set is more comprehensive. Therefore, generating the function call graph and control flow graph containing the indirect function call relationship can improve the reliability of fuzz testing.

[0093] To facilitate understanding, this section briefly describes indirect call relationships between functions. Functions in a program may have indirect call relationships. These relationships can also be referred to as indirect calls, indirect jumps, or indirect branches. Indirect calls can refer to control flow jumps within a program. Indirect call addressing is not achieved through direct addressing, but rather through indirect addressing calculations.

[0094] Taking C language (such as C++) as an example, any function call implemented using function pointers or virtual functions in a program may generate an indirect jump. For example, function Func1 calls function pointer pfunc. Function pointer pfunc can represent a pointer to a function. Under different conditions, function pointer pfunc may correspond to different function implementations. For example, function pointer pfunc may point to Func2 or Func3. The function that function pointer pfunc actually points to in function Func1 (for example, Func2 or Func3) is determined by the actual program execution. This type of function call, which requires determining the specific function based on the actual program execution, is called an indirect function call.

[0095] In some embodiments, fuzz testing a program using distance may include alternately using coverage-based fuzz testing and directed fuzz testing to fuzz the program, wherein the coverage-based fuzz testing updates a seed set using coverage information generated during the fuzz testing process, and the directed fuzz testing updates the seed set using distance information generated during the fuzz testing process.

[0096] Both coverage-based and guided fuzz testing can create bottlenecks during individual testing. For example, a bottleneck can occur when new seeds generated from a seed set, regardless of how they are mutated, fail to cover new code. In other words, a bottleneck can occur when newly generated seeds fail to increase the program's code coverage during fuzz testing. It should be understood that the higher the coverage of a program's code during testing, the more comprehensive the testing of the program is.

[0097] By alternately using coverage-based fuzz testing and guided fuzz testing to fuzz a program, we can avoid the testing bottlenecks caused by using only one fuzz testing method. At the same time, we can also make up for the limitations of static analysis and make rational use of distance information.

[0098] Coverage-based fuzz testing and guided fuzz testing can be used alternately in a variety of ways, and this disclosure does not limit the specific alternation method.

[0099] In some embodiments, a program may be fuzz tested using coverage-based fuzz testing first, and then switched to guided fuzz testing to fuzz test the program after a first condition is met. The first condition may be selected based on actual testing needs. For example, the first condition may refer to the coverage generated by the test meeting a preset coverage index. For another example, the first condition may refer to the test process no longer generating new code coverage. For another example, the first condition may refer to meeting a preset test duration (e.g., 5 minutes or 1 hour).

[0100] In other embodiments, a program may be fuzz tested using directed fuzz testing first. When the second condition is met, the program may be fuzz tested using directed fuzz testing. It should be understood that the first and second conditions are merely for ease of description and are not intended to limit specific conditions. The description of the second condition can refer to the description of the first condition.

[0101] In some other embodiments, a fuzz testing method may be randomly selected for initial use, and when certain conditions are met, a switch is made to another fuzz testing method. For example, coverage-based fuzz testing may be used for testing first, and when a first preset condition is met, the program may be switched to guided fuzz testing for testing. While using guided fuzz testing, when a second preset condition is met, the program may be switched to coverage-based fuzz testing for fuzz testing.

[0102] It should be understood that the first preset condition and the second preset condition are only for convenience of description. The first preset condition and the second preset condition can be the same condition or different conditions. The present disclosure does not limit the specific setting method of the first preset condition and the second preset condition.

[0103] As an example, the first preset condition may include that no new coverage information is generated within a first preset time, and the second preset condition may include that no new coverage information is generated within a second preset time. The first preset time and the second preset time may be the same. For example, the first preset time and the second preset time may both be set to 5 minutes. The first preset time and the second preset time may also be different. For example, the first preset time may be set to 3 minutes, and the second preset time may be set to 6 minutes.

[0104] As another example, the first preset condition may include executing a fuzz test for a first preset time period, and the second preset condition may include not generating new coverage information within a second preset time period.

[0105] By alternating between coverage-based fuzz testing and guided fuzz testing, testing efficiency can be improved and testing bottlenecks can be avoided.

[0106] It should be understood that as technology develops, other gray-box fuzz testing methods may emerge. This disclosure does not limit the specific types of fuzz testing that can be used interchangeably.

[0107] As described above, operating system kernel code is characterized by its diverse nature, large code size, and complex call relationships. The method provided in this disclosure can be applied to testing operating system kernel code. Compared to existing operating system kernel code testing methods, the method provided in this disclosure can improve test accuracy and efficiency and reduce false positives.

[0108] In order to make the scheme clearer, the following will be combined Figure 2 The program testing method provided by the present disclosure is introduced with a specific embodiment.

[0109] Figure 2 FIG. 1 is a flow chart of a program testing method based on fuzz testing provided by another embodiment of the present disclosure. Figure 2 As shown, this embodiment may include steps S201 to S207, and each step is introduced below.

[0110] Step S201: Compile program code into LLVM intermediate code. A program may refer to any program, for example, a kernel program of an operating system. Program code may be compiled into LLVM intermediate code using a variety of methods, for example, using the clang compiler.

[0111] Step S202: Generate a function call graph and a control flow graph. The function call graph and the control flow graph can be generated in a variety of ways. For example, a breadth-first algorithm can be used to generate a function call graph, and a control flow graph can be generated using the function call graph. Preferably, the function call graph and the control flow graph can include indirect call relationships between functions. By generating a function call graph and a control flow graph that include indirect call relationships, the function call graph and the control flow graph can be made more accurate and comprehensive, reducing omissions due to a lack of indirect call relationships.

[0112] In some embodiments, a breadth-first algorithm can be used to directly generate a function call graph that includes indirect function call relationships. In other embodiments, a traditional static analysis method can be used to first generate a function call graph that does not include indirect function call relationships, and then the breadth-first algorithm can be used to supplement the missing indirect function call relationships. By combining these two methods, the accuracy of the generated function call graph can be improved.

[0113] Step S203: Generate a reachable set. The reachable set can be generated using a function call graph and a control flow graph. The code in the reachable set can reach the target code to be tested through jumps and other methods. The code in the reachable set and the target code can both be function-level code. Preferably, the code in the reachable set and the target code can both be basic block-level code. A basic block is the smallest logical unit of code. By performing reachable set analysis at the granularity of a basic block, the test precision of the fuzz test can be improved, the test accuracy can be increased, and false positives can be reduced.

[0114] Step S204 calculates the distance between the codes in the reachable set and the target code. The distance between the codes in the reachable set and the target code can be calculated using a variety of methods. For example, the Dijkstra method can be used to calculate the distance between the codes in the reachable set and the target code. Alternatively, a breadth-first algorithm or a depth-first algorithm can be used to calculate the distance between the codes in the reachable set and the target code. The breadth-first algorithm has a simple calculation process, low complexity, and can improve computational efficiency.

[0115] As an example, when the code in the reachable set and the target code are both basic block-level codes, the distance of each code block can be calculated using the following formula:

[0116]

[0117] where d i is the distance from basic block i to the target basic block. Basic block j can directly call basic block i. Direct call, for example, can mean that basic block i can be directly executed from basic block j without going through other basic blocks. It can be understood that a basic block is the smallest logical unit of code. Therefore, the edges of a control flow graph composed of basic blocks may not have weight information. That is, in a control flow graph composed of basic blocks, the distance between adjacent basic blocks can take a fixed value (for example, 1). When there is no weight information for the distance between basic blocks, the distance calculation process can be further simplified and the efficiency of distance calculation can be improved.

[0118] In some embodiments, steps S201 to S204 may be performed in the initial stage of fuzz testing, and steps S201 to S204 may no longer be performed during the dynamic testing process of fuzz testing. That is, the distance between the reachable set and the target code may not be updated during the fuzz testing process. It should be understood that the actual calling relationship of the program usually does not change during the testing process. Therefore, the function call graph and the control flow graph usually do not change during the testing process. After obtaining the distance between the reachable set and the target code through static analysis, the distance is no longer updated, which can reduce the calculation process and effectively save computing resources. At this time, steps S201 to S204 can be called the static analysis stage.

[0119] In other embodiments, the function call graph and control flow graph can be dynamically updated during the fuzz testing process. It should be understood that although the actual function call relationship usually does not change during the test process, the function call graph and control flow graph generated by static analysis may not be comprehensive. For example, the function call graph and control flow graph generated by static analysis may not include the function call relationship and control relationship between code blocks that need to be actually run to confirm the call relationship. By updating the function call graph and control flow graph during the fuzz testing process, and updating the distance between the code in the reachable set and the target code based on the updated function call graph and control flow graph, the accuracy of the test can be improved and the missed reports can be reduced.

[0120] The function call graph and control flow graph can be updated in various ways. For example, they can be updated in real time. That is, the function call graph and control flow graph are updated every time a test case is executed. In another example, the function call graph and control flow graph can be updated after a certain period of time (e.g., 5 minutes) or after a certain number of test cases (e.g., 100).

[0121] Step S205: Perform fuzz testing using predefined seed files. The predefined seed files can be any number of seed files. For example, a conventional fuzz testing method can be used to randomly generate seed files for fuzz testing without setting a starting seed file. In another example, existing seed files can be used as the initial seed set for fuzz testing mutations.

[0122] Taking the operating system kernel as an example, a typical operating system kernel will provide a certain number of test files. You can use the test files provided by the program as the starting seed set for fuzz testing to improve testing efficiency.

[0123] In some embodiments, during the execution of step S205, seed files that can trigger program exceptions may be added to a seed set until the number of seed files in the seed set reaches or exceeds a predetermined number. Program exceptions may include causing the program to produce unexpected output, unexpectedly modifying certain variables in the program, generating new code coverage (or increasing coverage) during the execution of the seed file, etc.

[0124] The predetermined number can be set according to the test requirements. As an example, the predetermined number can be set to 100. That is, when the number of seed files in the seed set reaches or exceeds 100, step S205 can be stopped and the next step can be continued.

[0125] Step S206: Update the seed set based on distance. Updating the seed set based on distance can also be called the guided fuzz testing phase or the close-range seed utilization phase. The seed set can be updated based on distance in a variety of ways. For example, during the fuzz testing process, the distance of each seed can be calculated, and the top M seed files with the closest distance can be retained as the mutated seed set. M can be any natural number. The specific value of M can be set according to the test requirements.

[0126] As an example, M can be set to 30. That is, each time the fuzz test selects a seed file for mutation, it can select a seed file from the first 30 closest seed files as the starting point for mutation. The seed file at the starting point of mutation can be, for example, the input of the fuzz test tool. The fuzz test tool can make random changes to the input seed file to generate a new seed file. For example, the fuzz test can delete several characters from the input seed file and output the file with the above characters deleted as a new seed file. This new seed file can be used as the input of the program to test the program.

[0127] It's understandable that the closer the seed files are, the higher the probability of triggering the target code. Therefore, we sort the seed files by distance and perform mutation operations on the first M closest seed files to generate new seed files for testing. These new seed files have a higher probability of triggering the target code. This improves the efficiency of fuzz testing.

[0128] In some embodiments, only the first M seed files with the closest distance may be retained after the update, and other seed files may be deleted to reduce the storage space occupied during the test process.

[0129] As described above, one type of fuzz testing (e.g., guided fuzz testing) can create a test bottleneck during the testing process. Therefore, an exit condition can be set for step S206. For example, step S206 can be exited after a certain number of test cases are generated. Another example is that step S206 can be exited after no new code coverage is generated for T1 minutes.

[0130] After exiting step S206, step S207 can be continued. In some embodiments, when it is determined that the current fuzz test execution meets the preset exit conditions, the current fuzz test can also be exited. The preset exit conditions can be set according to the test needs. For example, it can be set to exit the current fuzz test after executing 100,000 test cases. For another example, it can be set to exit the fuzz test after executing for a certain period of time (for example, 20 hours, 2 days, etc.).

[0131] Step S207: updating the seed set according to the coverage rate. Updating the seed set according to the coverage rate can also be called the coverage-based fuzz testing phase or the new path exploration phase.

[0132] In step S207, a seed file with high coverage is selected from the seed set as an input file for mutation to generate a new seed file; the program is tested using the new seed file; and the seed that triggers the new path is added to the seed set.

[0133] When selecting seed files based on coverage, the seed files can be sorted using coverage as a weight, and the top K seed files with the highest coverage are selected for mutation. K can be any natural number, and this disclosure does not limit the actual value of K. It should be understood that the method of selecting seed files based on coverage is not limited to the above method.

[0134] Similar to step S206, the stage corresponding to step S207 may also produce a test bottleneck. Therefore, an exit condition can also be set for step S207. The specific setting of the exit condition can be found in the corresponding description in step S206. After exiting step S207, step S206 can be re-executed. That is, in the actual fuzz testing process, guided fuzz testing and coverage-based fuzz testing can be used alternately to improve test efficiency and make full use of distance information. After exiting step S207, when it is determined that the current fuzz test meets the exit condition, the current fuzz test can also be exited.

[0135] It should be understood that all of the above steps can be used, or some of the steps can be selected for fuzz testing as needed. For example, step S205 can be omitted, and dynamic fuzz testing can be started directly from step S206 or step S207.

[0136] The fuzz testing method provided by the disclosed embodiments can fully utilize the distance information generated by static analysis to improve the efficiency and accuracy of fuzz testing. By calculating only the distance from the reachable code to the target code, it can reduce the computing resources used and alleviate the high computational overhead of the distance calculation stage in guided fuzz testing. By analyzing and utilizing indirect function call relationships, it can compensate for the false positives caused by existing tools that do not consider indirect calls.

[0137] Combined with the previous article Figure 1 and Figure 2 , describes the method embodiment of the present disclosure in detail, and the following will be combined with Figures 3 to 5 , the device embodiment of the present disclosure is described in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for parts not described in detail in the device embodiment, reference can be made to the corresponding description of the method embodiment above.

[0138] Figure 3 FIG. 1 is a schematic diagram of a program testing device based on fuzzy testing provided by an embodiment of the present disclosure. Figure 3 As shown, the apparatus 300 may include an analysis module 310 , a generation module 320 , a calculation module 330 , and a testing module 340 .

[0139] The analysis module 310 may be configured to generate a function call graph and a control flow graph including function indirect call relationships using static analysis.

[0140] The generation module 320 may be configured to generate a reachable set using the function call graph and the control flow graph, wherein the codes in the reachable set reach the target code to be tested through jumps.

[0141] The calculation module 330 may be configured to calculate the distance between the codes in the reachable set and the target code.

[0142] The testing module 340 may be configured to perform fuzz testing on the program using the distance.

[0143] Optionally, in some embodiments, the function call graph and the control flow graph include indirect calling relationships of functions.

[0144] Optionally, in some embodiments, the fuzz testing the program using the distance includes: alternately using coverage-based fuzz testing and guided fuzz testing to fuzz test the program, wherein the coverage-based fuzz testing uses coverage information generated during the fuzz testing process to update the seed set, and the guided fuzz testing uses distance information generated during the fuzz testing process to update the seed set.

[0145] Optionally, in some embodiments, the alternating use of coverage-based fuzz testing and directed fuzz testing to fuzz test the program includes: when using the coverage-based fuzz testing to fuzz test the program, after a first preset condition is met, switching to using directed fuzz testing to fuzz test the program; when using the directed fuzz testing to fuzz test the program, after a second preset condition is met, switching to using coverage-based fuzz testing to fuzz test the program.

[0146] Optionally, in some embodiments, the first preset condition includes that no new coverage information is generated within a first preset time, and the second preset condition includes that no new coverage information is generated within a second preset time.

[0147] Optionally, in some embodiments, calculating the distance between the codes in the reachable set and the target code includes: calculating the distance between the codes in the reachable set and the target code using a breadth-first search method.

[0148] Optionally, in some embodiments, generating a function call graph and a control flow graph through static analysis includes: generating the function call graph using fast type analysis; and generating the control flow graph based on the function call graph.

[0149] Optionally, in some embodiments, generating a function call graph and a control flow graph through static analysis includes: compiling the program into an intermediate code form of LLVM; and statically analyzing the intermediate code form to generate the function call graph and the control flow graph.

[0150] Optionally, in some embodiments, the program is an operating system kernel program.

[0151] Optionally, in some embodiments, the codes in the reachable set are basic block codes.

[0152] Figure 4 FIG. 1 is a schematic diagram of a program testing device based on fuzzy testing provided by another embodiment of the present disclosure. Figure 4 As shown, the apparatus 400 may include a program analysis module 410 , a static distance calculation module 420 , and an adaptive fuzzy testing module 430 .

[0153] Program analysis module 410 may include a data preprocessing module 411, a call analysis module 412, and a control analysis module 413. Data preprocessing module 411 may compile the program into LLVM intermediate code. Call analysis module 412 may be used to generate a function call graph containing indirect function call relationships. Control analysis module 413 may be used to generate a control flow graph containing indirect function call relationships.

[0154] Static distance calculation module 420 may include reachability analysis module 421 and distance calculation module 422. Reachability analysis module 421 may analyze the program using the function call graph and control flow graph generated by program analysis module 410 to obtain a reachable set of code that can reach the target code. Reaching the target code may, for example, mean that the code can be executed from the code to the target code via a jump between codes. Distance calculation module 422 may be used to calculate the distance between the code in the reachable set and the target code. For example, a breadth-first algorithm may be used to calculate the distance between the code in the reachable set and the target code.

[0155] The adaptive fuzz testing module 430 may include an initialization module 431, a close-range seed utilization module 432, and a new path exploration module 433. The initialization module 431 may initialize the fuzz testing and generate a predetermined number of seed files to form a seed set. The seed files may be generated, for example, randomly. The close-range seed utilization module 432 may use distance information to update the seed files in the seed set. The new path exploration module 433 may use coverage information to update the seed files in the seed set. The close-range seed utilization module 432 and the new path exploration module 433 may be executed alternately.

[0156] Figure 55 is a schematic diagram of the structure of a device for program testing based on fuzz testing provided by another embodiment of the present disclosure. The device 500 can be, for example, a device or server that can be used for testing. The device 500 may include a memory 510 and a processor 520. The memory 510 can be used to store executable code. The processor 520 can be used to execute the executable code stored in the memory 510 to implement the steps in the various methods described above. In some embodiments, the device 500 may also include a network interface 530, and data exchange between the processor 520 and an external device can be implemented through the network interface 530.

[0157] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0158] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with the embodiments of the present disclosure can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0159] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0161] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0162] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A program testing method based on fuzz testing, comprising: Generate function call graphs and control flow graphs using static analysis; Generate a reachable set using the function call graph and the control flow graph, wherein the code in the reachable set reaches the target code to be tested through jumping; Calculating the distance between the codes in the reachable set and the target code; Performing fuzz testing on the program using the distance, The performing fuzz testing on the program using the distance includes: The program is fuzz tested alternately using coverage-based fuzz testing and guided fuzz testing, wherein the coverage-based fuzz testing uses coverage information generated during the fuzz testing process to update the seed set, and the guided fuzz testing uses distance information generated during the fuzz testing process to update the seed set. 2 . The method according to claim 1 , wherein the function call graph and the control flow graph include indirect calling relationships of functions.

3. The method according to claim 1, wherein the alternately using coverage-based fuzz testing and guided fuzz testing to perform fuzz testing on the program comprises: When the coverage-based fuzz test is used to perform fuzz testing on the program, after a first preset condition is met, switching to using the guided fuzz test to perform fuzz testing on the program; When the directed fuzz test is used to perform fuzz testing on the program, after a second preset condition is met, the program is switched to be fuzz tested using coverage-based fuzz testing. 4 . The method according to claim 3 , wherein the first preset condition comprises that no new coverage information is generated within a first preset time, and the second preset condition comprises that no new coverage information is generated within a second preset time.

5. The method according to claim 1, wherein calculating the distance between the codes in the reachable set and the target code comprises: The distance between the codes in the reachable set and the target code is calculated using a breadth-first search method.

6. The method according to claim 1, wherein generating a function call graph and a control flow graph by using static analysis comprises: generating the function call graph using fast type analysis; The control flow graph is generated based on the function call graph.

7. The method according to claim 1, wherein generating a function call graph and a control flow graph by using static analysis comprises: Compile the program into LLVM intermediate code form; The intermediate code form is statically analyzed to generate the function call graph and control flow graph. The method according to claim 1 , wherein the program is an operating system kernel program.

9. The method according to any one of claims 1 to 8, wherein the codes in the reachable set are basic block codes.

10. A device for program testing based on fuzz testing, comprising: an analysis module configured to generate a function call graph and a control flow graph including function indirect call relationships using static analysis; a generating module configured to generate a reachable set using the function call graph and the control flow graph, wherein the code in the reachable set reaches the target code to be tested by jumping; A calculation module, configured to calculate the distance between the codes in the reachable set and the target code; A testing module is configured to perform fuzz testing on the program using the distance, The performing fuzz testing on the program using the distance includes: The program is fuzz tested alternately using coverage-based fuzz testing and guided fuzz testing, wherein the coverage-based fuzz testing uses coverage information generated during the fuzz testing process to update the seed set, and the guided fuzz testing uses distance information generated during the fuzz testing process to update the seed set. The apparatus according to claim 10 , wherein the function call graph and the control flow graph include indirect calling relationships between functions.

12. The apparatus according to claim 10, wherein the alternately using coverage-based fuzz testing and directed fuzz testing to perform fuzz testing on the program comprises: When the coverage-based fuzz test is used to perform fuzz testing on the program, after a first preset condition is met, switching to using the guided fuzz test to perform fuzz testing on the program; When the directed fuzz test is used to perform fuzz testing on the program, after a second preset condition is met, the program is switched to be fuzz tested using coverage-based fuzz testing. 13 . The device according to claim 12 , wherein the first preset condition comprises that no new coverage information is generated within a first preset time, and the second preset condition comprises that no new coverage information is generated within a second preset time.

14. The apparatus according to claim 10, wherein the calculating the distance between the codes in the reachable set and the target code comprises: The distance between the codes in the reachable set and the target code is calculated using a breadth-first search method.

15. The apparatus according to claim 10, wherein generating a function call graph and a control flow graph by using static analysis comprises: generating the function call graph using fast type analysis; The control flow graph is generated based on the function call graph.

16. The apparatus according to claim 10, wherein generating a function call graph and a control flow graph by using static analysis comprises: Compile the program into LLVM intermediate code form; The intermediate code form is statically analyzed to generate the function call graph and control flow graph. The device according to claim 10 , wherein the program is an operating system kernel program.

18. The apparatus according to any one of claims 10 to 17, wherein the codes in the reachable set are basic block codes.

19. A device for program testing based on fuzz testing, comprising a memory and a processor, wherein the memory stores executable code, and the processor is configured to execute the executable code to implement the method according to any one of claims 1 to 9.

Images