An Adaptive Fuzz Testing Method and Device for Perception Coverage Metric Feedback

Abstract

The present application provides a fuzz testing method and device for adaptively perceiving coverage metric feedback, which relates to the field of information security and can also be used in the financial field. The method includes: selecting a corresponding coverage metric according to the pre-extracted feature information of the program under test; during fuzz testing, using the selected coverage metric to guide the selection of test cases and measure the space exploration efficiency of the program under test; adjusting the coverage metric according to a preset exploration efficiency threshold and the space exploration efficiency, and determining whether to continue or terminate the fuzz testing. The present application can perform static analysis on the characteristics of the program under test and dynamically adjust the coverage metric during the fuzz testing process, so as to make full use of the advantages of different coverage metrics, improve the program path coverage rate and vulnerability discovery rate.

Description

Technical Field

[0001] The present application relates to the field of information security and can be used in the financial field. Specifically, it is a fuzz testing method and device with adaptive perception of coverage metric feedback. Background Art

[0002] Fuzzing is a method of discovering software vulnerabilities by providing unexpected inputs to the target system and detecting abnormal results. Currently, fuzz testing techniques based on the covering principle can heuristically explore the program space through coverage feedback to alleviate the blindness of pure black-box testing.

[0003] However, in the existing fuzz testing methods based on the covering principle, the test path coverage is not comprehensive enough, and often only a small part of the test paths of the program can be explored, thus missing many opportunities to discover potential program vulnerabilities. In addition, existing fuzz testing tools often use a single coverage metric to test the program, without considering the different characteristics of the program, nor considering the different effects of different coverage metrics at different stages of fuzz testing, which affects the effect of fuzz testing. Summary of the Invention

[0004] Aiming at the problems in the prior art, the present application provides a fuzz testing method and device with adaptive perception of coverage metric feedback, which can statically analyze the characteristics of the program to be tested and dynamically adjust the coverage metric during the fuzz testing process, so as to make full use of the advantages of different coverage metrics and improve the program path coverage rate and vulnerability discovery rate.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a fuzz testing method with adaptive perception of coverage metric feedback, including:

[0007] Selecting a corresponding coverage metric according to the pre-extracted characteristic information of the program to be tested;

[0008] When performing fuzz testing, using the selected coverage metric to guide the selection of test cases and measure the space exploration efficiency of the program to be tested;

[0009] Adjusting the coverage metric according to a preset exploration efficiency threshold and the space exploration efficiency, and determining to continue or terminate the fuzz testing.

[0010] Further, the characteristic information includes: the number of conditional jump statements, the number of memory read and write statements, and the number of library function call statements and system call statements; the steps of extracting the characteristic information of the program to be tested include:

[0011] Extract the number of the conditional jump statements, the number of the memory read / write statements, and the number of the library function call statements and system call statements by using an interactive disassembler.

[0012] Further, the coverage metric includes: a branch coverage metric, a memory coverage metric, and a context-sensitive coverage metric; the selecting the corresponding coverage metric according to the pre-extracted characteristic information of the program to be tested includes:

[0013] Compare the number of the conditional jump statements, the number of the memory read / write statements, and the number of the library function call statements and system call statements;

[0014] Determine the coverage metric according to the comparison result; wherein, if the number of the conditional jump statements is the largest, the coverage metric is the branch coverage metric; if the number of the memory read / write statements is the largest, the coverage metric is the memory coverage metric; if the number of the library function call statements and system call statements is the largest, the coverage metric is the context-sensitive coverage metric.

[0015] Further, the fuzz testing method with adaptive perception coverage metric feedback further includes:

[0016] Generate a same-path test case set according to the program execution path of the test case;

[0017] Delete the test cases in the same-path test case set to make the test cases in the same-path test case set unique.

[0018] Further, the guiding the test case selection by using the selected coverage metric and measuring the space exploration efficiency of the program to be tested includes:

[0019] Determine a test execution benefit calculation factor according to the selected coverage metric;

[0020] Determine a first execution benefit of performing fuzz testing on the program to be tested in the current test cycle according to the test execution benefit calculation factor and the corresponding weight;

[0021] Determine the relative execution benefit of the current test cycle according to a second execution benefit of performing fuzz testing on the program to be tested in the previous test cycle calculated in advance and the first execution benefit;

[0022] Determine the space exploration efficiency according to the relative execution benefit.

[0023] Further, after determining the space exploration efficiency according to the relative execution benefit, it further includes:

[0024] If the relative execution benefit is less than a preset benefit reference threshold, adjust the coverage metric.

[0025] Further, the adjusting the coverage metric includes:

[0026] Determine the selected probability corresponding to each coverage metric using the beta distribution function;

[0027] Adjust the coverage metric according to the selected probability corresponding to each coverage metric.

[0028] Further, the adjusting the coverage metric according to the selected probability corresponding to each coverage metric includes:

[0029] Determine the maximum selected probability according to the selected probability corresponding to each coverage metric;

[0030] Set the coverage metric corresponding to the maximum selected probability as the coverage metric, and update the parameters of the beta distribution function.

[0031] In a second aspect, the present application provides a fuzz testing device for adaptively sensing coverage metric feedback, including:

[0032] A coverage metric selection unit for selecting a corresponding coverage metric according to the pre-extracted feature information of the program under test;

[0033] A space exploration efficiency determination unit for determining the space exploration efficiency of the program under test according to the selected coverage metric during fuzz testing;

[0034] A fuzz testing termination unit for determining whether to continue or terminate the fuzz testing according to a preset exploration efficiency threshold and the space exploration efficiency.

[0035] Further, the feature information includes: the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements; the device further includes:

[0036] A feature information determination unit for extracting the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements using an interactive disassembler.

[0037] Further, the coverage metrics include: branch coverage metrics, memory coverage metrics, and context-sensitive coverage metrics; the coverage metric selection unit includes:

[0038] A quantity comparison module for comparing the quantity of the conditional jump statements, the quantity of the memory read / write statements, and the quantity of the library function call statements and system call statements;

[0039] A coverage metric determination module for determining the coverage metric according to the comparison result; wherein, if the quantity of the conditional jump statements is the largest, the coverage metric is the branch coverage metric; if the quantity of the memory read / write statements is the largest, the coverage metric is the memory coverage metric; if the quantity of the library function call statements and system call statements is the largest, the coverage metric is the context-sensitive coverage metric.

[0040] Further, the fuzz testing device for adaptive perception coverage metric feedback further includes:

[0041] A path set generation unit for generating a same-path test case set according to the program execution path of the test case;

[0042] A test case deletion unit for deleting the test cases in the same-path test case set to make the test cases in the same-path test case set unique.

[0043] Further, the space exploration efficiency determination unit includes:

[0044] A benefit calculation factor determination module for determining a test execution benefit calculation factor according to the selected coverage metric;

[0045] A first execution benefit determination module for determining a first execution benefit of performing fuzz testing on the program under test in the current test cycle according to the test execution benefit calculation factor and the corresponding weight;

[0046] A relative execution benefit determination module for determining the relative execution benefit of the current test cycle according to a second execution benefit of performing fuzz testing on the program under test in the previous test cycle calculated in advance and the first execution benefit;

[0047] A space exploration efficiency determination module for determining the space exploration efficiency according to the relative execution benefit.

[0048] Further, the space exploration efficiency determination unit is further specifically configured to:

[0049] If the relative execution benefit is less than a preset benefit reference threshold, adjust the coverage metric.

[0050] Further, the space exploration efficiency determination unit includes:

[0051] A selected probability determination module, configured to determine the selected probability corresponding to each coverage metric by using a beta distribution function;

[0052] A coverage metric adjustment module, configured to adjust the coverage metrics according to the selected probabilities corresponding to the respective coverage metrics.

[0053] Further, the coverage metric adjustment module includes:

[0054] A maximum selected probability determination sub-module, configured to determine the maximum selected probability according to the selected probabilities corresponding to the respective coverage metrics;

[0055] A coverage metric setting sub-module, configured to set the coverage metric corresponding to the maximum selected probability as the coverage metric, and update the parameters of the beta distribution function.

[0056] In a third aspect, the present application provides an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the fuzzy testing method for adaptive perception coverage metric feedback are implemented.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the fuzzy testing method for adaptive perception coverage metric feedback are implemented.

[0058] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the fuzzy testing method for adaptive perception coverage metric feedback are implemented.

[0059] Aiming at the problems in the prior art, the fuzzy testing method and device for adaptive perception coverage metric feedback provided by the present application can introduce multiple coverage metrics with different granularities, avoid the limitations brought by a single metric, statically analyze the characteristics of the program to be tested, select appropriate coverage metrics for programs with different characteristics, and dynamically adjust the coverage metrics by using the multi-armed bandit algorithm during the fuzzy testing process, so as to make full use of the advantages of different coverage metrics and improve the program path coverage rate and vulnerability discovery rate. Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0061] Figure 1 One of the flowcharts of the fuzz testing method for adaptive perception coverage metric feedback in the embodiments of the present application;

[0062] Figure 2 The flowchart for selecting the corresponding coverage metric index in the embodiments of the present application;

[0063] Figure 3 One of the flowcharts of the fuzz testing method for adaptive perception coverage metric feedback in the embodiments of the present application;

[0064] Figure 4 The flowchart for determining the spatial exploration efficiency in the embodiments of the present application;

[0065] Figure 5 One of the flowcharts for adjusting the coverage metric index in the embodiments of the present application;

[0066] Figure 6 One of the flowcharts for adjusting the coverage metric index in the embodiments of the present application;

[0067] Figure 7 One of the structural diagrams of the fuzz testing device in the embodiments of the present application;

[0068] Figure 8 The structural diagram of the coverage metric index selection unit in the embodiments of the present application;

[0069] Figure 9 One of the structural diagrams of the fuzz testing device in the embodiments of the present application;

[0070] Figure 10 The structural diagram of the spatial exploration efficiency determination unit in the embodiments of the present application;

[0071] Figure 11 The structural diagram of the spatial exploration efficiency determination unit in the embodiments of the present application;

[0072] Figure 12 The structural diagram of the coverage metric index adjustment module in the embodiments of the present application;

[0073] Figure 13 The structural schematic diagram of the electronic device in the embodiments of the present application;

[0074] Figure 14 The functional schematic diagram of the fuzz testing method for adaptive perception coverage metric feedback in the embodiments of the present application;

[0075] Figure 15 The schematic diagram of the Thompson sampling process in the embodiments of the present application. Detailed implementation manners

[0076] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0077] It should be noted that the fuzz testing method and device with adaptive perception coverage metric feedback provided by the present application can be used in the financial field and can also be used in any field other than the financial field. The application fields of the fuzz testing method and device with adaptive perception coverage metric feedback provided by the present application are not limited.

[0078] The present application provides a fuzz testing method and device based on adaptive perception coverage metric feedback. The method and device are designed to flexibly schedule different coverage metric indicators to guide the efficient execution of fuzz testing and avoid the limitations brought by analyzing using a single metric. The embodiments of the present application select a variety of coverage metric indicators with different granularities, analyze the characteristics of the program under test, select appropriate coverage metric indicators for the programs under test with different characteristics, and use the multi-armed bandit algorithm (Bandit) to dynamically adjust the coverage metric at different stages of program testing, so that the coverage metric of fuzz testing has self-adaptability, thereby making full use of the advantages of different coverage metric indicators to improve the program path coverage rate and vulnerability discovery rate.

[0079] In one embodiment, referring to Figure 1 , in order to be able to perform static analysis on the characteristics of the program under test and dynamically adjust the coverage metric indicators during the process of fuzz testing, so as to make full use of the advantages of different coverage metric indicators and improve the program path coverage rate and vulnerability discovery rate, the present application provides a fuzz testing coverage metric scheduling method, including:

[0080] S101: Select the corresponding coverage metric indicator according to the pre-extracted characteristic information of the program under test;

[0081] S102: When performing fuzz testing, use the selected coverage metric indicator to guide the selection of test cases and measure the space exploration efficiency of the program under test;

[0082] S103: Adjust the coverage metric indicator according to the preset exploration efficiency threshold and the space exploration efficiency, and determine whether to continue or terminate the fuzz testing.

[0083] It can be understood that, referring to Figure 14 shown, the schematic diagram includes four links: test preparation 1401, program feature extraction 1402, fuzz testing execution 1403, and coverage metric adjustment 1404. Specifically, the functions and relationships of each link are as follows:

[0084] (1) Test preparation

[0085] The test preparation phase is used to prepare the program to be tested and input the initial test cases. The initial test cases can be one or more. Preferably, the initial cases can be streamlined (e.g., using the afl-cmin tool) to reduce the useless cases that execute duplicate paths, and minimize the streamlined cases (e.g., using the afl-tmin tool), that is, reduce the size of a single case, thereby reducing the execution overhead of the cases and improving the test efficiency.

[0086] (2) Program feature extraction

[0087] The program feature extraction phase is used to extract the static features of the program to be tested, so as to be able to select more appropriate coverage metrics for different types of programs to be tested at the initial stage of fuzz testing and alleviate the cold start problem. For example, for a program that frequently processes file reading and writing, using memory-sensitive coverage is more likely to trigger vulnerabilities in the program than branch coverage.

[0088] This phase mainly obtains the number of program size, jump statements, memory read and write statements, system call statements, etc. in the program through static analysis methods, and uses these as the features of the program to be tested to determine the coverage metrics used in the initial stage of fuzz testing (during the fuzz testing process, this metric is likely to be dynamically adjusted later, for details see the following description).

[0089] In one embodiment, the feature information includes: the number of conditional jump statements, the number of memory read and write statements, and the number of library function call statements and system call statements; the steps of extracting the feature information of the program to be tested include: using an interactive disassembler to extract the number of conditional jump statements, the number of memory read and write statements, and the number of library function call statements and system call statements.

[0090] (3) Fuzz testing execution

[0091] The fuzz testing execution phase is the process of performing fuzz testing. It runs the program to be tested with the test cases in the test case queue as input, and records the changes in the program state during the running process, that is, records the program path coverage found during the execution with the coverage metrics selected in the program feature extraction phase, and uses this metric result as feedback to retain the test cases that discover new paths or program crashes as output, and add them to the test case queue as candidates for the next fuzz testing input.

[0092] Among them, the test case queue is initially composed of the initial cases in the test case preparation phase, and new cases will be continuously added as the fuzz testing progresses.

[0093] In addition, after each test iteration cycle is executed, the test execution benefit of this test iteration cycle is calculated; when the relative execution benefit is less than the set threshold τ, the coverage metric is adjusted.

[0094] (4) Coverage Metric Adjustment

[0095] The coverage metric adjustment process is used to dynamically adjust the coverage metric method according to the program space exploration status at different stages of fuzz testing, aiming to explore the program execution path more comprehensively from different perspectives and improve the test depth and breadth of fuzz testing.

[0096] Specifically, the embodiments of the present application can select 7 program coverage metrics, including: basic block coverage (hereinafter referred to as bb for short), branch coverage (hereinafter referred to as bc for short), context-sensitive coverage (hereinafter referred to as ctx for short), memory-sensitive coverage (hereinafter referred to as mc for short), 4-gram coverage (hereinafter referred to as n4 for short), 8-gram coverage (hereinafter referred to as n8 for short), 16-gram (hereinafter referred to as n16 for short). The above 7 program coverage metrics together constitute the benchmark coverage metric set C over . When the test execution benefit enters the inefficient growth period, the current coverage metric can be adjusted through the multi-armed bandit algorithm, aiming to retain different test cases using different coverage feedbacks and increase the diversity of test cases. In addition, the embodiments of the present application also fully consider the situation of combining several different coverage metrics at the same time, that is, by combining multi-armed bandits to simultaneously select several different coverages for measurement, to achieve complementary advantages.

[0097] The specific definitions of the 7 coverage metrics are as follows:

[0098] Basic block coverage: Basic block coverage means taking basic blocks as the path unit. Every time a new basic block appears in the program under test, it is recorded as a new path, and the test case that discovers the new basic block is added to the use case queue.

[0099] Branch coverage: Branch coverage means recording the jump from the previous basic block to the current basic block as a path, and adding the test case that discovers the new jump to the use case queue.

[0100] Context-sensitive coverage: Context-sensitive coverage is based on branch coverage. By considering the context information of the jump (such as the function call stack) to distinguish different paths, that is, each branch coverage path with different contexts is recorded as a new path, and the corresponding test case is saved.

[0101] Memory-sensitive coverage: Memory-sensitive coverage means aiming at discovering new memory access locations, that is, each path reaching a new memory access location is a new path, and the corresponding test case is added to the use case queue.

[0102] 4-gram coverage, 8-gram coverage, 16-gram coverage: These three coverage metrics all belong to a variable-parameter N-gram coverage metric, where N (N is a non-negative integer) is a flexibly configurable parameter, representing the jump from the previous N basic blocks to the current basic block. Therefore, 4-gram coverage, 8-gram coverage, and 16-gram coverage correspond to the cases where N = 4, N = 8, and N = 16 respectively.

[0103] As can be seen from the above description, the fuzz testing coverage metric scheduling method provided by this application can introduce multiple coverage metric indicators with different granularities, avoid the limitations brought by a single metric indicator, statically analyze the characteristics of the program under test, select appropriate coverage metric indicators for programs with different characteristics, and utilize the multi-armed bandit algorithm to dynamically adjust the coverage metric indicators during the fuzz testing process, so as to make full use of the advantages of different coverage metric indicators and improve the program path coverage rate and vulnerability discovery rate.

[0104] In one embodiment, referring to Figure 2 , the coverage metric indicators include: branch coverage metric indicator, memory coverage metric indicator, and context-sensitive coverage metric indicator; select the corresponding coverage metric indicator according to the pre-extracted characteristic information of the program under test, including:

[0105] S201: Compare the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements;

[0106] S202: Determine the coverage metric indicator according to the comparison result; among them, if the number of conditional jump statements is the largest, the coverage metric indicator is the branch coverage metric indicator; if the number of memory read / write statements is the largest, the coverage metric indicator is the memory coverage metric indicator; if the number of library function call statements and system call statements is the largest, the coverage metric indicator is the context-sensitive coverage metric indicator.

[0107] It can be understood that for a given program under test P, extract the program characteristic information through a static analysis method (such as using the IDA tool) to guide the selection of the initial coverage metric indicator, such as extracting the program type T ype and the program size S ize , analyze the number f of conditional jump statements in the program a , the number f of memory read / write statements b and the number f of library function call and system call statements cIf the program scale is large and the proportion of jump statements is large, branch coverage is used as the measurement metric in the initial stage; if the proportion of memory read / write statements in the program is large, memory coverage is used as the measurement metric in the initial stage; if the proportion of the number of library function call and system call statements in the program is large, context-sensitive coverage is used as the measurement metric in the initial stage. The specific implementation method can also refer to the description in steps S201 to S202, that is, take the maximum value among the "number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements", and select the coverage measurement metric.

[0108] As can be seen from the above description, the fuzz testing coverage measurement scheduling method provided by this application can select the corresponding coverage measurement metric according to the pre-extracted characteristic information of the program to be tested.

[0109] In one embodiment, referring to Figure 3 , the fuzz testing coverage guidance method further includes:

[0110] S301: Generate a same-path test case set according to the program execution path of the test case;

[0111] S302: Prune the test cases in the same-path test case set so that the test cases in the same-path test case set are unique.

[0112] It can be understood that the preparatory work before testing includes: preparing the program to be tested and inputting initial test cases, and the initial test cases can be one or more. The initial test cases can be obtained by web crawling or from existing fuzz testing case sets, and then form the initial case set S in , and then preprocess the test cases in S in , streamline the number of cases and reduce the size of the cases to save the program execution overhead. The specific method can refer to the description in steps S301 to S302.

[0113] As can be seen from the above description, the fuzz testing coverage measurement scheduling method provided by this application can prune the test cases.

[0114] In one embodiment, referring to Figure 4 , when performing fuzz testing, use the selected coverage measurement metric to guide the test case selection and measure the space exploration efficiency of the program to be tested, including:

[0115] S401: Determine the test execution benefit calculation factor according to the selected coverage measurement metric;

[0116] S402: Determine the first execution benefit of performing fuzz testing on the program to be tested in the current test cycle according to the test execution benefit calculation factor and the corresponding weight;

[0117] S403: Determine the relative execution benefit of the current test cycle according to the second execution benefit and the first execution benefit of fuzz testing the program to be tested in the previous test cycle, which are pre-calculated.

[0118] S404: Determine the space exploration efficiency according to the relative execution benefit.

[0119] It can be understood that steps S401 to S404 are to determine the space exploration efficiency of the program to be tested during the execution of fuzz testing. It can also be understood as, during the execution of fuzz testing, using the currently selected coverage metric as feedback, recording the coverage of program paths, perceiving the current program space exploration efficiency by calculating the relative execution benefit, and then adjusting the coverage metric in a timely manner (for the specific adjustment method, refer to the description of steps S501 to S502).

[0120] Specifically, for the program P to be tested, set t as the iteration cycle (also called the test cycle in steps S401 to S404), that is, after every M test cases are executed, calculate the test execution benefit of the current iteration cycle. The so-called iteration cycle refers to the time taken to execute M test cases. The so-called test execution benefit refers to the program state change that is beneficial for fuzz testing to discover new paths or discover crashes. Preferably, select 6 program states including the number of crash triggers, the crash trigger time, the number of new paths discovered, the path depth, the path complexity, and the number of rare paths discovered as the evaluation criteria for benefit calculation (also called the test execution benefit calculation factors in steps S401 to S404). The calculation formula is as shown in formula (1), where R j represents the benefit of the j-th iteration cycle, and C i corresponds to the values of the above 6 evaluation criteria, and W i is the weight corresponding to the evaluation criterion i.

[0121] It should be noted that the crash trigger time is inversely proportional to the execution benefit. Therefore, during specific calculation, the value of the crash trigger time needs to be inverted. Then, calculate the ratio of the benefit of the current cycle j (also called the first execution benefit in steps S401 to S404) to the benefit of the previous cycle j - 1 (j > 1) (also called the second execution benefit in steps S401 to S404) as the relative execution benefit R w , as shown in formula (2). If the relative execution benefit R w is less than the set threshold τ, it proves that it belongs to the low-efficiency operation stage and the coverage metric needs to be adjusted.

[0122]

[0123]

[0124] It should also be noted that the specific values of the test execution benefit calculation factors corresponding to different coverage metrics may be different. For example, assume that b1, b2, b3, and b4 represent basic blocks. If the paths b1->b2 and b3->b4 have been executed respectively, for the path b1->b3->b2->b4, statement coverage will consider that it has executed a duplicate path (because the triggered basic blocks are the same as the executed paths, all being b1, b2, b3, and b4), while branch coverage will consider that a new path has been triggered (because branch coverage considers basic block jumps, and the path b1->b3->b2->b4 contains new jumps b1->b3, b3->b2, and b2->b4).

[0125] Therefore, it is necessary to determine the test execution benefit calculation factor according to the selected coverage metric.

[0126] As can be seen from the above description, the fuzz testing method with adaptive perception coverage metric feedback provided by this application can determine the space exploration efficiency of test cases for the program under test according to the selected coverage metric.

[0127] In one embodiment, after determining the space exploration efficiency according to the relative execution benefit, it further includes:

[0128] If the relative execution benefit is less than the preset benefit reference threshold, adjust the coverage metric. Among them, the benefit reference threshold can be set to the same value as the exploration efficiency threshold.

[0129] Specifically, referring to Figure 5 , adjusting the coverage metric includes:

[0130] S501: Use the beta distribution function to determine the selection probability corresponding to each coverage metric;

[0131] S502: Adjust the coverage metric according to the selection probability corresponding to each coverage metric.

[0132] It should be noted that steps S501 to S502 are implemented using the multi-armed bandit algorithm, and specific details are described below.

[0133] Furthermore, referring to Figure 6 , adjusting the coverage metric according to the selection probability corresponding to each coverage metric includes:

[0134] S601: Determine the maximum selection probability according to the selection probability corresponding to each coverage metric;

[0135] S602: Set the coverage metric corresponding to the maximum selection probability as the coverage metric.

[0136] It can be understood that the above steps are implemented using the multi-armed bandit algorithm (Bandit), which can dynamically adjust the current coverage metric, aiming to leverage the advantages of different coverage metrics, increase the diversity of test cases, and thus explore more program paths to alleviate the bottleneck of the current inefficient operation. Specifically, the embodiments of the present application model the problem of selecting coverage metrics in fuzz testing as a multi-armed bandit problem, that is, each coverage metric corresponds to an arm in the multi-armed bandit. The reward obtained after selecting an arm is used as the reward for this selection, and the exploration state where the current program is located is used as the environment.

[0137] Preferably, this step uses the Thompson sampling algorithm in the multi-armed bandit algorithm to perform the selection of coverage metrics. The Thompson sampling algorithm is a natural Bayesian algorithm, and its basic idea is to select the arm to be used in each round of decision-making according to the probability that each arm in the multi-armed bandit becomes the best arm. Its specific process in the embodiments of the present application is as follows:

[0138] ① Each coverage metric is used as a candidate, that is, the above 7 coverage metrics: bb, bc, ctx, mc, 4-gram, 8-gram, 16-gram respectively correspond to different arms in the multi-armed bandit, and they are numbered from 1 to 7. Therefore, the multi-armed bandit has 7 arms.

[0139] ② According to the Thompson sampling algorithm, the selection probability of each arm corresponds to a beta distribution function with parameters α and β (α and β are positive real numbers), that is, the selection probability of each coverage metric follows the Beta(α i ,β i )(i = 1, 2, 3, …, 7) prior distribution. Each time a selection is made, let the beta distribution function corresponding to each arm independently generate a random number, sort according to this random number, and select the coverage metric that generates the largest random number as the coverage metric for this time. The process is shown in Figure 15 as shown.

[0140] ③ According to the execution reward, update the beta distribution function. In the embodiments of the present application, the reward of the multi-armed bandit problem is simplified to only two cases of 0 and 1, that is, the Bernoulli multi-armed bandit problem. Specifically, if the relative execution benefit of the coverage metric selected this time is greater than the threshold τ (pre-set by those skilled in the art as needed), the reward is 1, and its corresponding beta function is updated to Beta(α + 1, β); otherwise, the reward is 0, and its corresponding beta function is updated to Beta(α, β + 1).

[0141] It should be noted that at the start of the Thompson sampling algorithm, the parameters α and β need to be initialized. When initializing, the parameter α corresponding to the coverage metric k (k is a number from 1 to 7) selected in the initial stage of fuzz testingk Initialize to 2, β k Initialize to 1. For example, if the branch coverage bc is selected in the initial stage (i.e., the first iteration cycle), then assign the α corresponding to the branch coverage to 2 and β to 1, and the parameters α and β corresponding to the other coverage metrics are all initialized to 1, which conforms to the uniform distribution.

[0142] ④ In addition, the embodiments of the present application also consider the case of combining different coverage metrics. That is, if the relative execution benefit corresponding to a single coverage metric cannot reach the threshold τ, then the combinatorial multi-armed bandit algorithm is used, and multiple coverage metrics are combined. Specifically, multi-action Thompson sampling is adopted, that is, a Beta distribution Beta(α i , β i )(i = 1, 2,..., 7) is maintained for each basic arm. In each round, a random variable θ i , β i ) is sampled from its distribution Beta(α i ), and all basic arms are sorted from high to low according to this random variable, and the top two basic arms are selected to form the super arm to be selected in this round. The coverage metric corresponding to this super arm is the coverage metric standard to be combined and used in this stage.

[0143] Preferably, the combined use of different coverage metrics here is achieved by means of input case intersection, that is, the test cases retained as feedback using two coverage metrics share a case queue.

[0144] Repeat steps ③ and ④ until the set time threshold is exhausted or manual stop.

[0145] As can be seen from the above description, the fuzz testing method with adaptive perception coverage metric feedback provided by the present application can adjust the coverage metric.

[0146] To sum up, the beneficial effects of the fuzz testing method and device based on adaptive perception coverage metric feedback proposed in the embodiments of the present application are at least as follows:

[0147] First, considering the characteristics of different programs, more appropriate coverage metrics are flexibly selected for programs with different characteristics, avoiding the limitations of fixed single metrics, making the testing more targeted, and improving the testing efficiency;

[0148] Second, considering the different stages of program testing and the different coverage gain efficiencies, the multi-armed bandit algorithm is used to dynamically adjust the coverage metrics, making the coverage metric selection adaptive, and improving the path exploration efficiency of fuzz testing;

[0149] Third, the combined use of different coverage metric indicators is considered, and the advantages of each coverage metric indicator are fully utilized by using the combinatorial multi-armed bandit algorithm.

[0150] Based on the same inventive concept, an embodiment of the present application further provides a fuzz testing device, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of the fuzz testing device for solving problems is similar to the fuzz testing method with adaptive perception coverage metric feedback, the implementation of the fuzz testing device can refer to the implementation of the method for determining software performance benchmarks, and the repeated parts will not be described again. As used hereinafter, the term "unit" or "module" may be a combination of software and / or hardware that can implement a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0151] In one embodiment, referring to Figure 7 , in order to be able to perform static analysis on the characteristics of the program under test and dynamically adjust the coverage metric indicators during the fuzz testing process, so as to make full use of the advantages of different coverage metric indicators and improve the program path coverage rate and vulnerability discovery rate, the present application provides a fuzz testing device with adaptive perception coverage metric feedback, including: a coverage metric indicator selection unit 701, a space exploration efficiency determination unit 702, and a fuzz testing termination unit 703.

[0152] The coverage metric indicator selection unit 701 is configured to select a corresponding coverage metric indicator according to the pre-extracted characteristic information of the program under test;

[0153] The space exploration efficiency determination unit 702 is configured to determine the space exploration efficiency of the program under test according to the selected coverage metric indicator during the fuzz testing;

[0154] The fuzz testing termination unit 703 is configured to determine whether to continue or terminate the fuzz testing according to a preset exploration efficiency threshold and the space exploration efficiency.

[0155] In one embodiment, the characteristic information includes: the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements; the device further includes:

[0156] The characteristic information determination unit is configured to extract the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements by using an interactive disassembler.

[0157] In one embodiment, referring to Figure 8, the coverage metric includes: branch coverage metric, memory coverage metric, and context-sensitive coverage metric; the coverage metric selection unit 701 includes: a quantity comparison module 801 and a coverage metric determination module 802.

[0158] The quantity comparison module 801 is configured to compare the quantity of the conditional jump statements, the quantity of the memory read / write statements, and the quantity of the library function call statements and system call statements.

[0159] The coverage metric determination module 802 is configured to determine the coverage metric according to the comparison result; wherein, if the quantity of the conditional jump statements is the largest, the coverage metric is the branch coverage metric; if the quantity of the memory read / write statements is the largest, the coverage metric is the memory coverage metric; if the quantity of the library function call statements and system call statements is the largest, the coverage metric is the context-sensitive coverage metric.

[0160] In one embodiment, refer to Figure 9 , the fuzz testing device for adaptive perception coverage metric feedback further includes: a path set generation unit 901 and a test case deletion unit 902.

[0161] The path set generation unit 901 is configured to generate a same-path test case set according to the program execution path of the test case.

[0162] The test case deletion unit 902 is configured to delete the test cases in the same-path test case set so that the test cases in the same-path test case set are unique.

[0163] In one embodiment, refer to Figure 10 , the space exploration efficiency determination unit 702 includes: a benefit calculation factor determination module 1001, a first execution benefit determination module 1002, a relative execution benefit determination module 1003, and a space exploration efficiency determination module 1004.

[0164] The benefit calculation factor determination module 1001 is configured to determine a test execution benefit calculation factor according to the selected coverage metric.

[0165] The first execution benefit determination module 1002 is configured to determine a first execution benefit of fuzz testing the program to be tested in the current test cycle according to the test execution benefit calculation factor and the corresponding weight.

[0166] The relative execution benefit determination module 1003 is configured to determine the relative execution benefit of the current test cycle according to the second execution benefit of fuzz testing the program to be tested in the previous test cycle calculated in advance and the first execution benefit.

[0167] A space exploration efficiency determination module 1004 is configured to determine the space exploration efficiency according to the relative execution benefit.

[0168] In one embodiment, the space exploration efficiency determination unit 702 is further specifically configured to:

[0169] If the relative execution benefit is less than a preset benefit reference threshold, adjust the coverage metric.

[0170] In one embodiment, referring to Figure 11 , the space exploration efficiency determination unit 702 includes: a selected probability determination module 1101 and a coverage metric adjustment module 1102.

[0171] The selected probability determination module 1101 is configured to determine the selected probability corresponding to each coverage metric by using a beta distribution function;

[0172] The coverage metric adjustment module 1102 is configured to adjust the coverage metric according to the selected probability corresponding to each coverage metric.

[0173] In one embodiment, referring to Figure 12 , the coverage metric adjustment module 1102 includes:

[0174] A maximum selected probability determination sub-module 1201 is configured to determine the maximum selected probability according to the selected probability corresponding to each coverage metric;

[0175] A coverage metric setting sub-module 1202 is configured to set the coverage metric corresponding to the maximum selected probability as the coverage metric and update the parameters of the beta distribution function.

[0176] From a hardware level, in order to be able to perform static analysis on the characteristics of the program to be tested and dynamically adjust the coverage metric during the process of fuzz testing, so as to make full use of the advantages of different coverage metrics and improve the program path coverage rate and vulnerability discovery rate, this application provides an embodiment of an electronic device for implementing all or part of the content in the fuzz testing method for adaptive perception coverage metric feedback. The electronic device specifically includes the following content:

[0177] A processor, a memory, a communications interface, and a bus; wherein, the processor, the memory, and the communications interface communicate with each other through the bus; the communications interface is used to implement information transmission between the fuzz testing device and related devices such as a core business system, a user terminal, and a related database, etc.; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., and this embodiment is not limited thereto. In this embodiment, the logic controller can be implemented with reference to the embodiments of the fuzz testing method with adaptive perception coverage metric feedback in the embodiments, as well as the embodiments of the fuzz testing device, the content of which is incorporated herein, and the repeated parts will not be elaborated.

[0178] It can be understood that the user terminal can include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device can include smart glasses, a smart watch, a smart bracelet, etc.

[0179] In practical applications, part of the fuzz testing method with adaptive perception coverage metric feedback can be executed on the side of the electronic device as described above, or all operations can be completed in the client device. Specifically, it can be selected according to the processing capacity of the client device and the limitations of the user usage scenario, etc. This application does not make a limitation in this regard. If all operations are completed in the client device, the client device may further include a processor.

[0180] The above-mentioned client device can have a communication module (i.e., a communication unit), and can be communicatively connected to a remote server to achieve data transmission with the server. The server can include a server on the side of the task scheduling center, and in other implementation scenarios, it can also include a server of an intermediate platform, such as a server of a third-party server platform communicatively linked to the task scheduling center server. The server can include a single computer device, or can include a server cluster composed of multiple servers, or a server structure of a distributed device.

[0181] Figure 13 This is a schematic block diagram of the system composition of the electronic device 9600 according to an embodiment of the present application. As Figure 13 shown, the electronic device 9600 can include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It should be noted that this Figure 13 is exemplary; other types of structures can also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0182] In one embodiment, the function of the fuzz testing method with adaptive perception coverage metric feedback can be integrated into the central processing unit 9100. Among them, the central processing unit 9100 can be configured to perform the following controls:

[0183] S101: Select a corresponding coverage metric according to the pre-extracted feature information of the program under test;

[0184] S102: When performing fuzz testing, use the selected coverage metric to guide test case selection and measure the space exploration efficiency of the program under test;

[0185] S103: Adjust the coverage metric according to the preset exploration efficiency threshold and the space exploration efficiency, and determine whether to continue or terminate the fuzz testing.

[0186] As can be seen from the above description, the fuzz testing method with adaptive perception coverage metric feedback provided by this application can introduce multiple coverage metrics with different granularities, avoid the limitations brought by a single metric, statically analyze the characteristics of the program under test, select appropriate coverage metrics for programs with different characteristics, and use the multi-armed bandit algorithm to dynamically adjust the coverage metrics during the fuzz testing process, so as to make full use of the advantages of different coverage metrics and improve the program path coverage rate and vulnerability discovery rate.

[0187] In another embodiment, the fuzz testing device can be separately configured from the central processing unit 9100. For example, the data composite transmission device fuzz testing device can be configured as a chip connected to the central processing unit 9100, and the function of the fuzz testing method with adaptive perception coverage metric feedback can be realized through the control of the central processing unit.

[0188] As Figure 13 shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily have to include Figure 13 all the components shown in Figure 13 ; in addition, the electronic device 9600 may further include

[0189] components not shown in Figure 13 ; reference can be made to the prior art.

[0190] Among them, the memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. It can store the above-mentioned failure-related information, and can also store programs for executing relevant information. And the central processing unit 9100 can execute the programs stored in the memory 9140 to achieve information storage or processing, etc.

[0191] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to supply power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display can be, for example, an LCD display, but is not limited thereto.

[0192] The memory 9140 can be a solid-state memory. For example, a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It can also be a memory that stores information even when powered off, can be selectively erased and has more data. An example of this memory is sometimes called an EPROM, etc. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes called a buffer). The memory 9140 can include an application / function storage unit 9142, which is used to store application programs and function programs or the processes for operating the electronic device 9600 through the central processing unit 9100.

[0193] The memory 9140 can also include a data storage unit 9143, which is used to store data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 can include various drivers of the electronic device for communication functions and / or for executing other functions of the electronic device (such as a messaging application, an address book application, etc.).

[0194] The communication module 9110 is a transmitter / receiver 9110 that transmits and receives signals via the antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0195] Based on different communication technologies, in the same electronic device, multiple communication modules 9110 can be provided, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, so as to implement normal telecommunication functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers, etc. Additionally, the audio processor 9130 is also coupled to a central processor 9100, enabling recording on the device through the microphone 9132 and playing the sounds stored on the device through the speaker 9131.

[0196] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all steps of the fuzz testing method for adaptive perception coverage metric feedback with the execution subject being a server or a client in the above embodiment. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, all steps of the fuzz testing method for adaptive perception coverage metric feedback with the execution subject being a server or a client in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0197] S101: Select a corresponding coverage metric index according to the pre-extracted feature information of the program under test;

[0198] S102: When performing fuzz testing, use the selected coverage metric index to guide the selection of test cases and measure the space exploration efficiency of the program under test;

[0199] S103: Adjust the coverage metric index according to a preset exploration efficiency threshold and the space exploration efficiency, and determine whether to continue or terminate the fuzz testing.

[0200] As can be seen from the above description, the fuzz testing method for adaptive perception coverage metric feedback provided by the present application can introduce multiple coverage metric indexes with different granularities, avoid the limitations brought by a single metric index, statically analyze the characteristics of the program under test, select a suitable coverage metric index for programs with different characteristics, and dynamically adjust the coverage metric index using the multi-armed bandit algorithm during the fuzz testing process, thereby making full use of the advantages of different coverage metric indexes and improving the program path coverage rate and vulnerability discovery rate.

[0201] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, apparatus, or computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0202] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce a device for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0203] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device that implements the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0205] Specific embodiments are applied in the present invention to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An adaptive fuzz testing method for perceptual coverage metric feedback, characterized in that Including: Selecting a corresponding coverage metric according to the pre-extracted characteristic information of the program to be tested; The characteristic information is the static characteristic of the program to be tested, and the characteristic information includes: the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements; the coverage metrics include: branch coverage metric, memory coverage metric, and context-sensitive coverage metric; When performing fuzz testing, using the selected coverage metric to guide test case selection and measure the space exploration efficiency of the program to be tested; the selection method of the coverage metric is the Thompson sampling algorithm, and the selected coverage metric includes: cross-combining different coverage metrics through input test cases; Adjusting the coverage metric according to a preset exploration efficiency threshold and the space exploration efficiency, and determining to continue or terminate the fuzz testing.

2. The fuzz testing method for adaptive perception coverage metric feedback according to claim 1, wherein The steps of extracting the characteristic information of the program to be tested include: Using an interactive disassembler to extract the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements.

3. The fuzz testing method for adaptive perception coverage metric feedback according to claim 2, wherein The selecting a corresponding coverage metric according to the pre-extracted characteristic information of the program to be tested includes: Comparing the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements; Determining the coverage metric according to the comparison result; wherein, if the number of conditional jump statements is the largest, the coverage metric is the branch coverage metric; if the number of memory read / write statements is the largest, the coverage metric is the memory coverage metric; if the number of library function call statements and system call statements is the largest, the coverage metric is the context-sensitive coverage metric.

4. The fuzz testing method for adaptive perception coverage metric feedback according to claim 1, characterized in that The using the selected coverage metric to guide test case selection and measure the space exploration efficiency of the program to be tested includes: Determining a test execution benefit calculation factor according to the selected coverage metric; Determining a first execution benefit of performing fuzz testing on the program to be tested in the current test cycle according to the test execution benefit calculation factor and the corresponding weight; Determining the relative execution benefit of the current test cycle according to the pre-calculated second execution benefit of performing fuzz testing on the program to be tested in the previous test cycle and the first execution benefit; Determining the space exploration efficiency according to the relative execution benefit.

5. The fuzz testing method for adaptively perceiving coverage metric feedback according to claim 4, characterized in that After determining the space exploration efficiency according to the relative execution benefit, it further includes: If the relative execution benefit is less than a preset benefit reference threshold, adjusting the coverage metric.

6. The fuzz testing method for adaptive perception coverage metric feedback according to claim 5, characterized in that, The adjusting the coverage metric includes: Using the beta distribution function to determine the selection probability corresponding to each coverage metric; Adjusting the coverage metric according to the selection probability corresponding to each coverage metric.

7. The fuzz testing method for adaptive perception coverage metric feedback according to claim 6, characterized in that The adjusting the coverage metric according to the selection probability corresponding to each coverage metric includes: Determining the maximum selection probability according to the selection probability corresponding to each coverage metric; Set the coverage metric corresponding to the maximum selected probability as the coverage metric, and update the parameters of the beta distribution function.

8. An adaptive perception coverage metric feedback fuzz testing device, characterized in that, It includes: A coverage metric selection unit, configured to select a corresponding coverage metric according to the feature information of the program under test extracted in advance; The feature information is the static feature of the program under test, and the feature information includes: the number of conditional jump statements, the number of memory read / write statements, and the number of library function call statements and system call statements; the coverage metrics include: branch coverage metric, memory coverage metric, and context-sensitive coverage metric; A space exploration efficiency determination unit, configured to, when performing fuzz testing, use the selected coverage metric to guide the test case selection and measure the space exploration efficiency of the program under test; the selection method of the coverage metric is the Thompson sampling algorithm, and the selected coverage metrics include: cross-combining different coverage metrics through input test cases; A fuzz testing termination unit, configured to adjust the coverage metric according to a preset exploration efficiency threshold and the space exploration efficiency, and determine whether to continue or terminate the fuzz testing.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the fuzz testing method for adaptive perception coverage metric feedback according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the fuzz testing method for adaptive perception coverage metric feedback according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the fuzz testing method for adaptive perception coverage metric feedback according to any one of claims 1 to 7 are implemented.

Images