An Optimization Method for Industrial Control Protocol Fuzz Testing Based on Coverage Guidance

Abstract

The present invention discloses an optimization method for industrial control protocol fuzz testing based on coverage guidance. This method combines mutation testing on the basis of traditional coverage-guided fuzz testing. By comparing the output results of the program under test and the mutated PUT, high-quality test cases are screened. At the same time, the concept of coverage-guided tracking is introduced. By encoding the current coverage boundary in the binary file of the program under test, self-reporting can be performed when new coverage is generated in the test case, eliminating unnecessary tracking of the coverage-guided fuzzer, improving the fuzz testing efficiency, and greatly enhancing the vulnerability detection ability and testing efficiency of coverage-guided fuzz testing.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial control security, and particularly relates to an optimization method for industrial control protocol fuzz testing based on coverage guidance. Background Art

[0002] With the rapid development of industrial informatization and intelligence, industrial control systems are transitioning from closed networks, proprietary protocols, and operating systems to open networks, general TCP / IP protocols, and operating systems. While emerging technologies improve the production and management efficiency of industrial enterprises, industrial control security also faces new challenges such as the continuous threat of security vulnerabilities accelerating penetration and complex and diverse attack methods. Fuzz testing is an effective technical means for discovering potential vulnerabilities in industrial control systems. Its basic idea is to input a large number of randomized malformed data into the target to be tested, detect the running state of the target to be tested, and promptly feedback anomalies, thereby discovering security vulnerabilities existing in the target to be tested.

[0003] The traditional fuzz testing methods mainly have the following deficiencies: (1) Fuzz testing adopts a completely random mutation strategy, blindly generating test cases without considering the execution process of industrial control protocols, generating a large number of invalid test cases, and resulting in low fuzz testing efficiency; (2) Low code coverage rate, existing fuzz testing methods tend to inefficiently traverse the search space, generating a large number of repeated execution paths; (3) High failure rate of test cases, most of the malformed data packets generated by existing simulation test tools may not be recognized by the target to be tested, resulting in the target to be tested not responding to these requests. Summary of the Invention

[0004] Therefore, the key technical problem to be solved by the present invention is that although the existing technology can perform adaptive fuzz testing on industrial control devices with network protocols when applied to industrial control protocol fuzz testing methods, there are problems such as a single core algorithm for the mutation strategy, high failure rate of test cases, and high redundancy. Coverage-guided fuzz testing aims to explore all the binary codes of the target by maximizing the code coverage rate of the generated test cases to improve the vulnerability detection ability.

[0005] Based on traditional coverage-guided fuzz testing, this invention combines mutation testing. By comparing the output results of the program under test (PUT) and the mutated PUT, high-quality test cases are screened. At the same time, the concept of coverage-guided tracing is introduced. By encoding the current coverage boundary in the binary file of the PUT (program under test), self-reporting can be achieved when new coverage is generated in the test case, eliminating unnecessary tracing of the coverage-guided fuzzer and improving the efficiency of fuzz testing. Another object of this invention is to provide a coverage-guided fuzz testing framework based on fault detection awareness, using mutation testing to improve the ability of traditional coverage-guided fuzz testing technology in detecting errors. This framework consists of input, mutation engine, test engine, fuzz engine, and seed queue output.

[0006] To achieve the above object, the technical solution of this invention is as follows: An optimization method for industrial control protocol fuzz testing based on coverage guidance, which mainly includes the following steps:

[0007] S1: Select the PUT and the system configuration file, and set the mutation testing configuration and fuzz testing configuration;

[0008] The mutation testing settings include a set of mutators for creating mutants before fuzz testing and a selection strategy for mutation during fuzz testing. s And the fuzz testing setting refers to the selection of the initial seed I.

[0009] S2: Design a mutation engine, generate mutants of the PUT according to the mutation strategy specified in the system configuration file, collect the mutated versions of the PUT and the corresponding mutation information, and construct a mutation pool M for subsequent use;

[0010] The mutation information includes the location where the mutation occurs and the mutants used, etc. The mutation engine provides two modes for mutant generation: an offline mode for constructing the mutation pool and an online mode for dynamic selection. The offline mode is used to prepare for fault detection-aware fuzz testing and is only executed once to avoid unnecessary overhead. The online mode works continuously during execution and selects mutants as the input of the test engine according to the selection strategy.

[0011] The mutant mentioned in S2 is the PUT after executing the mutation strategy, that is, the mutant is the mutated program under test. Executing the PUT and the mutated PUT means inputting the test input into the PUT and the mutated PUT respectively and observing the execution results of both.

[0012] S3: Design a test engine to select n mutants from the mutant pool M as test inputs, execute the instrumentation PUT and the mutant program respectively, collect the execution result information of each, compare them, generate code coverage and mutation statistics information, and use the code coverage and mutation statistics information as feedback for subsequent use;

[0013] The test engine adopts a coverage analysis method based on fault detection awareness. Compared with the traditional coverage analysis method, a factor of state is introduced on the basis of code coverage and test results. The state is used to measure whether the test input i causes a program execution vulnerability. The introduction of the state is beneficial to improving the code path coverage. Specifically, if the mutant value of an i in the last execution is non-zero, it means that i has the ability to detect errors, then i will be saved in the seed queue Q, while the i that cannot effectively detect errors will be saved in the seed queue F.

[0014] S4: Design a fuzzing engine to calculate the mutant value P of the mutant compared to PUT according to the feedback information mut ∈[0,1]. The mutant value will be used as the input for calculating the mutation probability and affect the generation of test case seeds, and the generated test case seeds will be used as the input of the test engine;

[0015] The fuzzing engine adopts an improved mutation probability evaluation algorithm. For the above test inputs i with error detection capabilities, on the basis of the existing error display marks, this algorithm will amplify the mutation probability of i to reward these test inputs that discover errors, and punish the test inputs that do not display errors in the opposite way (mutation represents the process of generating sub-inputs from parent seeds). Compared with the existing mutation probability calculation methods, this algorithm retains the fault detection ability of test inputs and is beneficial to generating test cases with higher coverage.

[0016] S5: Design a fuzz testing framework as Figure 2 shown. The fuzz testing framework (the fuzz testing framework mentioned in the present invention is modified based on the existing mainstream framework AFL, so the fuzz testing framework mentioned here is proposed on the basis of AFL) will maintain two seed queues as outputs during the test execution, including the valid seed queue Q and the failed input seed queue F. The queue Q adjusts the position of the seed individuals in the queue according to the increase and decrease of the mutant value, and the queue length is set by (the double-line symbol represents a natural number, P limit represents the queue length, that is, it means the queue length is a natural number). When the number of seeds in the queue overflows, the individuals at the tail of the queue will be discarded; the queue F will not be completely discarded either, but the probability of generating sub-inputs for the individuals in it will become smaller.

[0017] The fuzzing engine takes the seed individual at the head of the queue as a fuzzing test case and inputs it into the parser module of AFL++ (American Fuzzy Loop Plus Plus, which is a variant of AFL). Finally, it is input into the test engine through the packet sender. The execution status of the PUT is monitored through the execution engine (the execution engine is part of the fuzzing test framework), and the execution program polls the status of the child process in each fuzzing iteration. When the PUT crashes, the fuzzing test stops; when the PUT enters an error state, the return code is obtained and the exception is recorded, and a new instance is forked to resume the fuzzing process, so as to realize the real-time adjustment of the test case generation process according to the error feedback of the PUT.

[0018] Further, the mutation engine in step S2 is a component in the fuzzing test framework. The mutation engine provides two modes: an offline mode for mutation creation and an online mode for dynamic mutation selection. The offline mode is used to prepare for fault-detection-aware fuzzing tests. Mutants are created according to the mutants specified in the configuration. Each mutant retains the mutated version of the PUT and the corresponding mutation information (such as the mutation location and the mutant used), and a mutant pool is constructed for subsequent use. The offline mode is only executed once to avoid unnecessary overhead caused by repeated creation. In contrast, the online mode works continuously during test execution. The mutation engine periodically selects a subset of mutants from the mutant pool according to the selection strategy specified in the configuration. The selected mutants are sent to the test engine for mutation testing. Through the selection strategy, usually only a few mutants are selected to ensure the efficiency of the fuzzing test.

[0019] Further, the operation of the test engine in step S3 to execute the PUT and the mutants and compare the test results includes the following sub-steps:

[0020] S301: Define the selected PUT as p, and the functionality of the PUT as B = {b1, b2,..., b n}. Then, given a test input i, a functional output of the PUT can be expressed as o = b[p, i]. Then, the output set O = {o1, o2,..., o n}. In addition, the mutant pool is denoted as M, and the mutant subset is defined as M′ = {mut1, mut2,..., mut n}. The output set is O′, and mut i′ represents the i′-th mutant, where i′ = 1, 2,..., n, and n is the number of mutants (the output set O represents the set of output results of the PUT after executing the test input, and the output set O′ represents the set of output results of the mutants after executing the test input).

[0021] S302: Execute the PUT and mutants according to the test input. To improve the execution efficiency of mutant testing, we reduce the number of mutants to be executed by strategically selecting a subset of mutants, and execute the selected mutants and the PUT simultaneously in multiple threads, and collect the test results O and O' respectively. If a crash occurs during the test, the execution fails and the output result is not recorded. In addition, the time for mutant testing is also limited. Assuming the test time for the PUT is t, the upper limit of the test time for the mutant is t' = t × (1 + r), where 0 ≤ r ≤ 1 and r represents the additional execution time multiplier, which is a constant used to control the threshold of the mutant test time.

[0022] S303: Compare the test results of the PUT and the mutants. If the output result of a mutant mut i′ is inconsistent with that of the PUT, then mut i′ will be eliminated by the test input i. Only the mutants with completely known output results can survive. Calculate the number k of eliminated mutants and the number s of surviving mutants, and record k and s as mutant statistics.

[0023] Furthermore, the process of calculating the mutant values and generating test case seeds in step S4 mainly includes the following sub-steps:

[0024] S401: The calculation formula for the mutant value is as follows:

[0025]

[0026] The mutant value can be used to reflect the ability of the test input to detect vulnerabilities. The more mutants eliminated by the test input, the larger P mut is, indicating that the test input has a stronger ability to detect vulnerabilities.

[0027] S402: Store the executed test input in the seed queue. According to the reward and punishment mechanism, increase the mutation probability of the seeds with non-zero mutant values and decrease the mutation probability of the seeds that have not detected vulnerabilities. Here, the mutation probability represents the probability of generating a child input from the parent seed. Let ( represents a positive integer) be the parameter for adjusting the mutation probability. Multiply the mutation probability by factor k to increase the probability, and divide the mutation probability by factor k otherwise.

[0028] The present invention combines mutation testing to guide fuzz testing for vulnerability detection, solves the misconception in traditional coverage-guided schemes that only higher coverage is used as the evaluation benchmark, and proposes a new standard that combines mutation values and coverage to screen test cases that can detect more vulnerabilities during iterative updates. At the same time, the coverage-guided tracking tracks fewer test cases by restricting the tracking to only test cases with increased coverage, improving the efficiency of fuzz testing. The fuzz testing optimization method designed by the present invention greatly improves the vulnerability detection ability and testing efficiency of coverage-guided fuzz testing. Description of the Drawings

[0029] Figure 1 It is a flowchart of the optimization method for industrial control protocol fuzz testing based on coverage guidance of the present invention;

[0030] Figure 2 It is a framework diagram of the fuzz testing proposed by the present invention;

[0031] Figure 3 It is a schematic diagram of the update of the seed queue. Detailed Embodiments

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer and more definite, the technical solutions of the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] As Figure 1 shown, the implementation of the optimization method for industrial control protocol fuzz testing based on coverage guidance of the present invention mainly has the following five steps:

[0034] S1: Figure 2 Describes the entire fuzz testing framework proposed by the present invention. First, select PUT and the system configuration file as inputs, and retain the input seeds as outputs. The system settings include mutation testing settings and fuzz testing settings. The mutation testing settings refer to a set of mutators for creating mutants before fuzz testing and a mutation selection strategy during fuzz testing. The fuzz testing settings refer to the settings of the initial seeds and the allocated resources.

[0035] S2: The mutation engine generates mutants of PUT according to the mutation strategy specified in the configuration. We use the mutation testing tool Pitest to create mutants of PUT, then retain the mutated versions of PUT and the corresponding mutation information for each mutant, and construct a mutation pool for subsequent use. To ensure the efficiency of fuzz testing, finally, only select 10 mutants from the mutation pool as the input of the test engine;

[0036] The mutation engine is a component in the fuzzing framework. The mutation engine provides two modes: an offline mode for mutation creation and an online mode for dynamic mutation selection. The offline mode is used to prepare fault detection-aware fuzz testing. Mutants are created according to the mutants specified in the configuration. Each mutant retains the mutated version of the PUT and the corresponding mutation information (such as the mutation location and the mutant used), and a mutant pool is constructed for subsequent use. The offline mode is executed only once to avoid unnecessary overhead caused by repeated creation. In contrast, the online mode works continuously during test execution. The mutation engine periodically selects a subset of mutants from the mutant pool according to the selection strategy specified in the configuration. The selected mutants are sent to the test engine for mutated testing.

[0037] S3: The test engine executes the instrumented PUT and mutants respectively with the initial seed as the test input, collects the execution result information, and makes a comparison to generate mutation statistics; The operations of the test engine executing the PUT and mutants and comparing the test results include the following sub-steps:

[0038] S301: Define the selected PUT as p, and the functionality of the PUT as B = {b1, b2,..., b n}(n represents the number of mutants). Then, given the test input i, a functional output of the PUT can be expressed as o = b[p, i], and the output set O = {o1, o2,..., o n}. In addition, the mutant pool is denoted as M, and the subset of mutants is defined as M' = {mut1, mut2,..., mut n}. The output result is O'. n is the number of mutants. The execution result exec is represented as a triple <s, O, e>, where s is the execution status of the PUT, s ∈ (success, failure), success indicates successful execution, failure indicates failed execution, and e is the crash that occurred during the execution.

[0039] S302: Execute the PUT and mutants according to the test input. To improve the execution efficiency of mutated testing, we select a subset of mutants to reduce the number of mutants to be executed, and execute the selected mutants and the PUT simultaneously in multiple threads, and collect the test results O and O' respectively. If a crash occurs during the test, the execution fails and the output result will not be recorded; In addition, the testing time of mutated testing is also restricted. Assume the testing time of the PUT is t, then the upper limit of the testing time of the mutant is t' = t × (1 + r), 0 ≤ r ≤ 1. r represents the additional execution time multiplier, which is a constant used to control the threshold of the mutant testing time.

[0040] S303: Compare the test results of the PUT and mutants. If the output result of a mutant mut is inconsistent with that of the PUT, then mut will be eliminated by the test input i. Only mutants with completely known output results can survive. Calculate the number of eliminated mutants mut k and the number of surviving mutants mut s , and record them as mutation statistics information.

[0041] As the core of the fuzz testing framework, the test engine extends the test execution part of the traditional coverage-guided fuzzer. The test engine accepts inputs from the fuzzing engine. After each test input execution, the test engine collects the achieved code coverage and mutation statistics to build feedback information and sends it to the fuzzing engine to guide the generation of inputs. In addition to executing the instrumented PUT with the received test inputs, the test engine also needs to execute the same test inputs for the mutants selected by the fuzzing engine.

[0042] S304: During the test execution, the optimized lightweight instrumentation tool will be limited to tracking test cases that increase code coverage, generating code coverage information, and finally the execution engine will send the code coverage information and mutation statistics information to the fuzzing engine as feedback information;

[0043] Coverage-guided tracing only tracks test cases that increase code coverage by inserting preselected software interrupts at the beginning of each uncovered basic block. The first test case that triggers the interrupt will be marked as having increased code coverage, and the complete tracing code coverage will be executed, and the interrupt information will be recorded; for each newly accessed basic block within the coverage range of the test case, the relevant interrupts will be removed from the record. As this process repeats, only test cases that execute newly covered code will trigger interrupts, thus marking them as having increased coverage.

[0044] S4: The fuzzing engine calculates the mutation value P mut ∈[0,1]. The mutation value affects the generation of test case seeds as an evaluation criterion; the process of calculating the mutation value and generating test case seeds mainly includes the following sub-steps:

[0045] S401: The calculation formula of the mutation value is as follows:

[0046]

[0047] The mutation value can be used to reflect the ability of a test input to detect vulnerabilities. The more mutants eliminated by a test input, the larger P mut is, indicating that the test input has a stronger ability to detect vulnerabilities.

[0048] S402: Store the executed test inputs into the seed queues Q and F respectively. According to the reward and punishment mechanism, increase the mutation probability of the seeds with non-zero mutation values and decrease the mutation probability of the seeds that have not detected vulnerabilities. Here, the mutation probability represents the probability of generating child inputs from the parent seeds. Let be the parameter for adjusting the mutation probability. To increase the probability, multiply the mutation probability by factor k , and vice versa, divide the mutation probability by factor k . The specific algorithm is as follows:

[0049]

[0050] where BASE, factor, and factor k are all positive constants (all three represent constants. BASE is used to measure the baseline of n, which is equivalent to initializing n to a value, and factor and factor k are both constants used to expand n).

[0051] The fuzzing engine can use the feedback information returned from the test engine to realize the fuzzification of fault detection perception, and update queues Q and F according to the above reward and punishment mechanism. This can ensure that the test inputs with vulnerability detection capabilities will be mutated many times to generate more child inputs.

[0052] S5: The fuzz testing framework maintains two seed queues, namely the valid seed queue Q and the failed input seed queue F. As Figure 3 shown, queue Q adjusts the position of the seed individuals in the queue according to the increase or decrease of the mutation value. The queue length is set by . When the quantity overflows, the individuals at the tail of the queue will be discarded; queue F will not be completely abandoned either, but the probability of generating child inputs from the individuals in it will become smaller; repeat this process until both seed queues reach the maximum threshold, and then continue with the next step.

[0053] The fuzzing engine takes the seed individual at the head of the queue as the fuzz test case and inputs it into the parser module (the parser module is part of the AFL++ framework and is a common part of mainstream fuzz testing frameworks), and finally inputs it into the test engine through the packet sender; the execution engine monitors the execution status of the PUT. The execution program polls the status of the child process (the execution process of each test input belongs to a child process) in each fuzz iteration. When the PUT crashes, the fuzz testing stops; when the PUT enters an error state, obtain the return code and record the exception, and fork a new instance to resume the fuzz process, so as to realize the real-time adjustment of the test case seed generation process according to the error feedback of the PUT.

[0054] Obviously, the above embodiments are merely examples for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or alterations can be made based on the above description. It is not necessary and impossible to enumerate all implementation manners here. And the obvious changes or alterations derived therefrom still fall within the protection scope of the present invention.

Claims

1. An optimization method for industrial control protocol fuzz testing based on coverage guidance, characterized in that, The method includes the following steps: S1: Select the program under test PUT and the system configuration file, and set the mutation test configuration and the fuzz test configuration; Among them, the mutation test setting includes a set of mutators for creating mutants before fuzz testing and a selection strategy for mutation during fuzz testing ; and the fuzz testing setting refers to the selection of initial seeds ; S2: Design a mutation engine to generate mutants of PUTs according to the mutation strategies specified in the system configuration file, collect the mutated versions of PUTs and the corresponding mutation information, and build a mutation pool for subsequent use; for subsequent use; The mutation information includes the location where the mutation occurs and the mutants used. The mutation engine provides two modes for mutant generation: an offline mode for constructing a mutation pool and an online mode for dynamic selection. The offline mode is used to prepare fault detection-aware fuzz testing and is only executed once to avoid unnecessary overhead. The online mode works continuously during execution and selects mutants as the input to the test engine according to the selection strategy. Among them, the mutant is the PUT after executing the mutation strategy; S3: Design a test engine to select from the mutant pool and pick mutants as test inputs. Use the test engine to execute the PUT and the mutants respectively, collect the execution result information of each, compare them, generate code coverage and mutation statistics information, and use the two pieces of information representing code coverage and mutation statistics as feedback information for subsequent use; The said test engine introduces the factor of state based on code coverage and test results, where the state is used to measure whether the test input triggers program execution vulnerabilities. The introduction of the state helps to improve the code path coverage. If the mutant value in the last execution is non-zero, it indicates the ability to detect errors, then it will be saved in the seed queue while those that cannot effectively detect errors will be saved in the seed queue as well; S4: Design a fuzzy engine to calculate the mutation value of the mutant compared to the PUT based on the feedback information , and the mutation value will be used as the input for calculating the mutation probability and affect the generation of test case seeds, and the generated test case seeds will in turn be used as the input for the test engine; The fuzzy engine, for test inputs with error detection capabilities , on the basis of the existing error display markers, will amplify the mutation probability to reward these test inputs that discover errors, and punish the test inputs that do not display errors in the opposite way; S5: Design a fuzz testing framework which will maintain two seed queues as output during the test execution, including a valid seed queue and a seed queue for failed inputs ; The queue adjusts the position of seed individuals in the queue according to the increase or decrease of the mutation value, and the queue length is set by where , and when the number of seeds in the queue overflows, the individuals at the tail of the queue will be discarded; The queue will not be completely discarded, but the probability of generating sub-inputs by the individuals in it will become smaller; The fuzz engine uses the seed individual at the head of the queue as a fuzz test case, inputs it into the parser module of AFL++, and finally inputs it into the test engine through the packet sender; monitors the execution status of PUT through the execution engine in the fuzz test framework. The execution program polls the status of the subprocess in each fuzz iteration. When PUT crashes, the fuzz test stops; when PUT enters an error state, obtains the return code and records the exception, and forks a new instance to resume the fuzz process, so as to realize real-time adjustment of the test case generation process according to the error feedback of PUT; In step S3, the specific implementation of using the test engine to execute PUT and mutants respectively includes: S301: Define the selected PUT as , define the functionality of the PUT as , then, given a test input , a functional output of the PUT is represented as , then the output set . Additionally, define the mutant subset as , whose output set is . represents the th mutant, . is the number of mutants; S302: Execute PUTs and mutants according to the test inputs. To improve the execution efficiency of mutation testing, reduce the number of mutants to be executed by strategically selecting a subset of mutants, and execute the selected PUTs and mutants simultaneously in multiple threads, and collect the test results separately and , if a crash occurs during the test, the execution fails and the output result is not recorded; in addition, the time for mutation testing is also limited. Assuming the test time for PUT is , then the upper limit of the test time for the mutant is , represents the additional execution time magnification factor, which is a constant used to control the threshold of the mutant test time; S303: Compare the test results of the PUT and mutants. If the output result of a mutant is inconsistent with that of the PUT, then it will be eliminated by the test input . Only mutants with completely known output results can survive. Calculate the number of eliminated mutants and the number of surviving mutants , and record and as mutation statistical information; In step S4, the mutation value of the mutant compared to the PUT is calculated according to the feedback information , and the mutation value will be used as the input for calculating the mutation probability and affect the generation of test case seeds, specifically including: S401: The calculation formula of the mutation value is as follows: ； The mutation value is used to reflect the ability of the test input to detect vulnerabilities. The more mutants eliminated by the test input, the larger it is, indicating that the test input has a stronger ability to detect vulnerabilities; S402: Store the executed test input into the seed queue. Increase the mutation probability of seeds with non-zero mutation values according to the reward and punishment mechanism, and decrease the mutation probability of those seeds that have not detected vulnerabilities. Here, the mutation probability represents the probability of generating a child input from a parent seed. Let be the parameter for adjusting the mutation probability, where represents a positive integer. To increase the probability, multiply the mutation probability by , and vice versa, divide the mutation probability by .

Images