A method and system for scheduling fuzzing mutations based on an actor-critic algorithm

Abstract

The application relates to the technical field of computer software security analysis, and particularly discloses a fuzzy test mutation scheduling method and system based on an Actor-Critic algorithm, wherein the method comprises the following steps: based on a program to be tested and a test sample, performing a taint inference test to obtain the corresponding relationship between input sample byte changes and program execution paths; taking the corresponding relationship as a reinforcement learning environment state, improving a mutation scheduling strategy by using an Actor-Critic algorithm model, generating a mutation action space, setting a reward function, and defining strategy model network parameters and value function model network parameters; performing seed fuzzy testing by using the mutation scheduling strategy, taking the program execution path and the coverage rate as reward evaluation indexes, and updating the strategy model network parameters and the value function model network parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of computer software security analysis, and in particular to a fuzzy test mutation scheduling method and system based on an Actor-Critic algorithm. Background Art

[0002] Vulnerability detection is a critical issue in software security. With the continuous advancement of software technology and the increasing complexity of related product functions, the number of vulnerabilities is increasing day by day. Many codes with security risks are inevitably exploited by malicious attackers, causing great trouble for ordinary users and causing huge financial losses and security threats to companies and countries.

[0003] Fuzz testing is an effective method for software testing and discovering unknown vulnerabilities. The concept of fuzz testing was first proposed by Barton Miller et al. in 1988. It is an automated testing technique whose core principle is to input large amounts of randomly generated, unexpected data into the target program while simultaneously collecting and monitoring abnormal information during the execution of test cases. This method aims to detect illegal input that may cause errors in the target program and identify its vulnerabilities.

[0004] However, due to the complexity of software systems, mutation-based fuzz testing techniques are limited in their ability to discover vulnerabilities. Mutation-based fuzz testing techniques randomly select bytes based on an initial seed and perform additions, deletions, and bit flips on them. However, this method is inefficient, lacks quality assurance, and is difficult to trigger deep vulnerabilities.

[0005] With the development of reinforcement learning technology, fuzz testing methods based on reinforcement learning have become widely used. Deep reinforcement learning methods use neural networks to approximate the value function or policy function in reinforcement learning, thereby making decisions in complex environments. This can be applied to optimal decision-making in fuzz testing mutation scheduling. Summary of the Invention

[0006] To achieve the purpose of the present invention, the present application provides a fuzzy testing mutation scheduling method based on the Actor-Critic algorithm, comprising:

[0007] Step S1: Based on the program to be tested and the test sample, perform a taint inference test to obtain the corresponding relationship between the byte changes of the input sample and the program execution path;

[0008] Step S2: Using the corresponding relationship as the state of the reinforcement learning environment, the Actor-Critic algorithm model is used to improve the mutation scheduling strategy, generate a mutation action space, set the reward function, and define the policy model network parameters and the value function model network parameters;

[0009] Step S3: Perform seed fuzz testing using the mutation scheduling strategy, and update the strategy model network parameters and the value function model network parameters based on the program execution path and coverage as reward evaluation indicators.

[0010] In some specific embodiments, step S1 includes:

[0011] Step S11: instrument the program to be tested to monitor changes in program path execution information and related variable values;

[0012] Step S12: taking the initial seed as input and executing the program to be tested, obtaining a value set of variables related to the program path constraints when the initial sample is running through taint analysis;

[0013] Step S13: mutate each byte of the initial seed in sequence to generate a new test sample;

[0014] Step S14: Input the new test sample into the program, obtain the value set of variables related to the program path constraints when the new test sample is running, and if it is the same as the value set of variables related to the program path constraints when the initial sample is running, record the position of the mutated byte and the relationship between the program variables and the path.

[0015] In some specific embodiments, step S2 includes:

[0016] Step S21: Use the Actor-Critic algorithm to build a policy model and a value function model;

[0017] Step S22: using the strategy model, outputting an action probability distribution according to the state of the current input sample, and determining the action strategy to be selected;

[0018] Step S23: generating a variant action space of the selected action strategy through a reinforcement learning algorithm;

[0019] Step S24: Setting a reward function so that the agent selects a mutation action according to the current environment state and the mutation action space, and mutates in the direction of maximizing the final expected reward value.

[0020] In some specific embodiments, step S3 includes:

[0021] Step S31: performing deterministic mutation and taint tracking on the initial input sample to obtain an initial state byte-path mapping relationship graph as the initial state;

[0022] Step S32: Initialize the policy model network parameters and the value function model network parameters, and define the policy model network learning rate and the value function model network learning rate;

[0023] Step S33: mutating the initial input sample and performing fuzz testing;

[0024] Step S34: determining the action probability under the environmental state conditions according to the policy model, and then determining the reward value of the network according to the value function model;

[0025] Step S35: Update the policy model network parameters and the value function model network parameters until the fuzzy test non-deterministic mutation phase ends.

[0026] In some specific embodiments, the reward function is determined according to a seed vulnerability trigger reward function, a seed vulnerability trigger reward function, and a path feature reward function, wherein the seed path coverage reward function is determined according to the following formula:

[0027] ;

[0028] Where trace_bits represents the path coverage bitmap executed by the seed fuzz test, and virgin_bits represents the path coverage bitmap covered by all seeds in the fuzz test. represents the summation function;

[0029] The reward function for triggering a seed vulnerability is determined by the following formula:

[0030] ;

[0031] Where, Indicates the number of times the vulnerability is triggered in the global fuzz test;

[0032] The path feature reward function is determined according to the following formula:

[0033] ;

[0034] Where, represents the edges covered by the seed execution process, Indicates the depth of the program path where the edge is located, Indicates the number of times the edge is covered in the global fuzz test.

[0035] In some specific embodiments, the weight of the reward parameter is changed by adjusting the weight of the reward function to improve the fuzz testing mutation preference.

[0036] To achieve the same invention purpose, the present application also provides a fuzzy testing mutation scheduling system based on the Actor-Critic algorithm, including:

[0037] Taint analysis module: used to perform taint inference testing based on the program to be tested and test samples, and obtain the correspondence between the byte changes of the input sample and the program execution path;

[0038] Mutation space generation module: used to use the corresponding relationship as the state of the reinforcement learning environment, use the Actor-Critic algorithm model to improve the mutation scheduling strategy, generate the mutation action space, set the reward function, and define the policy model network parameters and the value function model network parameters;

[0039] Parameter update module: used to perform seed fuzz testing using the mutation scheduling strategy, and update the strategy model network parameters and value function model network parameters based on the program execution path and coverage as reward evaluation indicators.

[0040] In some specific embodiments, the stain analysis module is configured to perform the following steps:

[0041] Step S11: instrument the program to be tested to monitor changes in program path execution information and related variable values;

[0042] Step S12: taking the initial seed as input and executing the program to be tested, obtaining a value set of variables related to the program path constraints when the initial sample is running through taint analysis;

[0043] Step S13: mutate each byte of the initial seed in sequence to generate a new test sample;

[0044] Step S14: Input the new test sample into the program, obtain the value set of variables related to the program path constraints when the new test sample is running, and if it is the same as the value set of variables related to the program path constraints when the initial sample is running, record the position of the mutated byte and the relationship between the program variables and the path.

[0045] In some specific embodiments, the parameter updating module is configured to perform the following steps:

[0046] Step S21: Use the Actor-Critic algorithm to build a policy model and a value function model;

[0047] Step S22: using the strategy model, outputting an action probability distribution according to the state of the current input sample, and determining the action strategy to be selected;

[0048] Step S23: generating a variant action space of the selected action strategy through a reinforcement learning algorithm;

[0049] Step S24: Setting a reward function so that the agent selects a mutation action according to the current environment state and the mutation action space, and mutates in the direction of maximizing the final expected reward value.

[0050] In some specific embodiments, the parameter updating module is configured to perform the following steps:

[0051] Step S31: performing deterministic mutation and taint tracking on the initial input sample to obtain an initial state byte-path mapping relationship graph as the initial state;

[0052] Step S32: Initialize the policy model network parameters and the value function model network parameters, and define the policy model network learning rate and the value function model network learning rate;

[0053] Step S33: mutating the initial input sample and performing fuzz testing;

[0054] Step S34: determining the action probability under the environmental state conditions according to the policy model, and then determining the reward value of the network according to the value function model;

[0055] Step S35: Update the policy model network parameters and the value function model network parameters until the fuzzy test non-deterministic mutation phase ends.

[0056] Beneficial effects of the above technical solution:

[0057] This application provides a fuzz testing mutation scheduling method based on the Actor-Critic algorithm. This method utilizes a reinforcement learning algorithm to guide the mutation of test samples, significantly improving the low quality of test samples in traditional fuzz testing due to the randomness of mutation strategies, and increasing code coverage. Furthermore, this method uses a taint analysis method to obtain seeds and samples in the program execution state, and uses reinforcement learning to generate mutation strategies, addressing the blindness of mutations and improving fuzz testing efficiency. This method addresses the problem of blind mutations in non-deterministic mutation strategies in fuzz testing, resulting in low test efficiency and difficulty in improving vulnerability coverage and discovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A flowchart of a fuzzy test mutation scheduling method based on the Actor-Critic algorithm provided by one embodiment of the present invention;

[0060] Figure 2 A schematic diagram of the structure of a fuzzy test mutation scheduling system based on the Actor-Critic algorithm provided by one embodiment of the present invention;

[0061] Figure 3An example diagram of a mapping diagram of input sample bytes and paths for a fuzzy test mutation scheduling method based on an Actor-Critic algorithm provided in one embodiment of the present invention;

[0062] Figure 4 A schematic diagram of an Actor strategy network function model for a fuzzy test mutation scheduling method based on an Actor-Critic algorithm provided by one embodiment of the present invention;

[0063] Figure 5 A network diagram of a critic value function model of a fuzzy testing mutation scheduling method based on an Actor-Critic algorithm is provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0065] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0066] Example 1

[0067] An embodiment of the present invention provides a fuzzy test mutation scheduling method based on the Actor-Critic algorithm, referring to Figure 1 Shown, including:

[0068] Step S1: Based on the program to be tested and the test sample, perform a taint inference test to obtain the corresponding relationship between the byte changes of the input sample and the program execution path;

[0069] Step S2: Using the corresponding relationship as the state of the reinforcement learning environment, the Actor-Critic algorithm model is used to improve the mutation scheduling strategy, generate a mutation action space, set the reward function, and define the policy model network parameters and the value function model network parameters;

[0070] Step S3: Perform seed fuzz testing using the mutation scheduling strategy, and update the strategy model network parameters and the value function model network parameters based on the program execution path and coverage as reward evaluation indicators.

[0071] In a specific embodiment of the present invention, step S1 includes:

[0072] Step S11: the program to be tested is instrumented to monitor program path execution information and changes in related variable values;

[0073] Step S12: an initial seed is input and the program to be tested is executed, and an initial sample runtime value set of variables related to program path constraints is obtained through a taint analysis;

[0074] Step S13: each byte of the initial seed is mutated in turn to generate a new test sample;

[0075] Step S14: the new test sample is input to the program, a new test sample runtime value set of variables related to program path constraints is obtained, and if the new test sample runtime value set of variables related to program path constraints is the same as the initial sample runtime value set of variables related to program path constraints, the position of the mutated byte and the relationship with the program variables and the path are recorded.

[0076] Referring to Figure 3 , path information is recorded by AFL using a bitmap, and the position of the mutated byte and the relationship with the program variables are recorded by the method, and are mapped with the program path recorded by the AFL bitmap to form a mapping relationship diagram.

[0077] In one specific embodiment of the present application, step S2 comprises:

[0078] Step S21: an Actor-Critic algorithm is used to construct a policy model and a value function model;

[0079] Specifically, referring to Figure 4 , the policy model network is composed of a deep learning network, including an input layer, a fully connected layer and a Softmax activation function. The input layer has a size of the state space, and the fully connected layer is composed of neural networks with 512 and 64 nodes. The output layer uses a Softmax activation function. The network input is the current state , the first fully connected layer changes the input into a feature vector feature, and the second fully connected layer is used to map the feature vector, and the probability of each action is obtained after normalization processing. The policy model network is symbolically represented as , wherein are network parameters.

[0080] Referring to Figure 5 , the value function network is a deep learning network. First, the state space is input into a fully connected layer with 512 nodes to become a state feature vector, and a fully connected layer with 512 nodes is used to map the action The two feature vectors are then concatenated and passed through a network with 1024 nodes and 128 nodes in a hidden layer. Finally, the expected value of the action strategy is calculated using the sigmoid activation function. The network symbol for the value function model is: in is the network parameter.

[0081] Step S22: using the strategy model, outputting an action probability distribution according to the state of the current input sample, and determining the action strategy to be selected;

[0082] Step S23: generating a variant action space of the selected action strategy through a reinforcement learning algorithm;

[0083] Step S24: Setting a reward function so that the agent selects a mutation action according to the current environment state and the mutation action space, and mutates in the direction of maximizing the final expected reward value.

[0084] In a specific embodiment of the present invention, the reward function is determined based on a seed vulnerability trigger reward function, a seed vulnerability trigger reward function, and a path feature reward function.

[0085] In a specific embodiment of the present invention, the weight of the reward parameter is changed by adjusting the weight of the reward function to improve the fuzz testing mutation preference.

[0086] Specifically, during the fuzz testing process, due to the differences in seed size and program paths, there is insufficient information in defining the reinforcement learning state space based on the seed size and the number of paths, and the state space is difficult to reflect the program characteristics. The method of the present invention uses the byte-path mapping relationship diagram solved by the taint inference analysis module as the state space. Since the program path bitmap space defined by AFL is large, the reinforcement learning state space is too large, which increases the complexity of the solution. The method of the present invention represents the state space with a byte array, the index represents the byte position of the input seed, and the value represents the index of the deepest edge of the path mapped by the byte in the AFL bitmap.

[0087] In fuzz testing, the mutation operator, mutation position, and mutation length all affect the efficiency of fuzz testing in exploring unknown paths. The reinforcement learning algorithm of this invention expands the fuzz testing mutation action space, and the mutation action is defined as a triple ,in It is the mutation operation type, recorded in coding mode. Table 1 describes the mutation operation types. Indicates the mutation position, that is, the starting byte of the mutation. The length of the bytes changed for this mutation. According to the reward function, the reinforcement learning agent intelligently selects the mutation action from the above mutation action space according to the current environment state, so that it mutates in the direction of the maximum expected reward value, that is, wherein is the current action policy.

[0088] Table 1 Mutation operation number Description

[0089]

[0090] The reward function is calculated based on three aspects of seed-generated example path coverage, vulnerability triggering, and execution path features.

[0091] The path coverage of the seed directly affects the effect of fuzz testing. Generally speaking, a higher path coverage seed can achieve higher rewards. The path coverage calculation can be obtained by the execution bitmap of the seed and the global execution bitmap during the entire fuzz testing process. Define the seed path coverage reward function:

[0092] ;

[0093] In the formula, trace_bits represents the path coverage bitmap of the seed fuzz testing execution, virgin_bits represents the path coverage bitmap covered by all seeds of the fuzz testing, indicates the summation function. Since 0 and 1 in the bitmap represent whether the path is covered through each bit, the summation function can be used to calculate the path coverage number, and measures the path coverage reward of the seed in the entire fuzz testing process.

[0094] The fundamental purpose of fuzz testing is to find vulnerabilities, so the seed leakage is an important factor in reward value calculation. However, if the vulnerability discovery reward is too high, it may cause the fuzz testing to be more biased towards triggering the vulnerability, affecting the efficiency of other vulnerability discovery. Therefore, the vulnerability triggering reward is related to the number of times the vulnerability is triggered, and after the vulnerability is triggered multiple times, the vulnerability reward will decrease, and the seed vulnerability triggering reward function is defined as:

[0095] ;

[0096] wherein indicates the number of times the vulnerability is triggered in the global fuzz testing.

[0097] The execution path features of the seed consider the path depth and path execution frequency of the execution path of the seed. The fuzz testing execution process is more biased towards deeper paths and paths with lower execution frequency, so the path feature reward function is defined as:

[0098] ;

[0099] wherein represents the edges covered by the seed execution process, Indicates the depth of the program path where the edge is located, Indicates the number of times the edge is covered in the global fuzz test.

[0100] Based on the above formula, the reward function of the seed during execution can be defined as:

[0101] ;

[0102] in, is the weight of each reward parameter, R i Represents the reward function, where i ranges from 1 to 3, with R1 being the path coverage reward, R2 being the seed vulnerability trigger reward, and R3 being the path feature reward. By adjusting the weights of the reward parameters, the fuzz test mutation preference can be improved, thereby enhancing the pertinence and efficiency of the test.

[0103] In a specific embodiment of the present invention, step S3 includes:

[0104] Step S31: performing deterministic mutation and taint tracking on the initial input sample to obtain an initial state byte-path mapping relationship graph as the initial state;

[0105] Step S32: Initialize the policy model network parameters and the value function model network parameters, and define the policy model network learning rate and the value function model network learning rate;

[0106] Step S33: mutating the initial input sample and performing fuzz testing;

[0107] Step S34: determining the action probability under the environmental state conditions according to the policy model, and then determining the reward value of the network according to the value function model;

[0108] Step S35: Update the policy model network parameters and the value function model network parameters until the fuzzy test non-deterministic mutation phase ends.

[0109] Specifically, the detailed process of the mutation strategy is as follows:

[0110] Step 1: According to Momentary status , the policy network calculates the probability of each action in the action set , select the maximum probability mutation action ;

[0111] Step 2: Use mutation actions Mutate the initial seed and perform fuzz testing. Collect the path coverage of the mutation sample fuzz test execution process and calculate Moment Rewards Update the environment status at the same time ;

[0112] Step 3: Use the policy network to calculate the environment state Conditional action probability ,estimate The optimal action in the state , used for the value function model network to estimate rewards;

[0113] Step 4: and ;

[0114] Step 5: Calculate TD error;

[0115] ;

[0116] in is a discount factor that controls the importance of future rewards;

[0117] Step 6: Derivative the value network and update the value function network parameters ;

[0118] ;

[0119] ;

[0120] Step 7: Update the policy model network parameters using gradient ascent ;

[0121] ;

[0122] ;

[0123] Step 8: Continue executing Step 1 - Step 7 until the non-deterministic mutation phase of fuzz testing ends.

[0124] To verify the effectiveness of the reinforcement learning mutation scheduling algorithm of the present invention, this application selected the classic fuzz testing tool AFL, which uses a randomized strategy, and the fuzz testing tool NEUZZ as comparison objects. Under the same experimental environment, the three fuzz testing tools were used to perform 24-hour fuzz testing on the LAVA-M dataset and a real program dataset. The experiment was repeated five times, and the average of the results was taken as the final experimental result. The experimental results for the LAVA-M test set are shown in Table 2. According to the results in Table 2, the method of the present invention outperforms other testing tools in code coverage and number of crashes detected for the test programs in the LAVA-M dataset. Compared with AFL and NEUZZ, the total path coverage increased by 44.5% and 33.4%, and the number of crashes increased by 173% and 92%, respectively, demonstrating the superiority of the method of the present invention.

[0125] Experimental results show that using reinforcement learning to guide the mutation of test samples can significantly optimize the low quality of test samples caused by the randomness of the mutation strategy in traditional fuzz testing and improve code coverage.

[0126] Table 2 Comparison of the results of the proposed method and the existing fuzzy testing method on the LAVA-M dataset

[0127]

[0128] Example 2

[0129] An embodiment of the present invention provides a fuzzy test mutation scheduling system based on the Actor-Critic algorithm, referring to Figure 2 Shown, including:

[0130] Taint analysis module 10: used to perform taint inference testing based on the program to be tested and the test sample, and obtain the corresponding relationship between the byte changes of the input sample and the program execution path;

[0131] Mutation space generation module 20: used to use the corresponding relationship as the state of the reinforcement learning environment, improve the mutation scheduling strategy using the Actor-Critic algorithm model, generate the mutation action space, set the reward function, and define the policy model network parameters and the value function model network parameters;

[0132] Parameter updating module 30: used to perform seed fuzz testing using the mutation scheduling strategy, and update the strategy model network parameters and value function model network parameters based on the program execution path and coverage as reward evaluation indicators.

[0133] In a specific embodiment of the present invention, the stain analysis module 10 is configured to perform the following steps:

[0134] Step S11: instrument the program to be tested to monitor changes in program path execution information and related variable values;

[0135] Step S12: taking the initial seed as input and executing the program to be tested, obtaining a value set of variables related to the program path constraints when the initial sample is running through taint analysis;

[0136] Step S13: mutate each byte of the initial seed in sequence to generate a new test sample;

[0137] Step S14: Input the new test sample into the program, obtain the value set of variables related to the program path constraints when the new test sample is running, and if it is the same as the value set of variables related to the program path constraints when the initial sample is running, record the position of the mutated byte and the relationship between the program variables and the path.

[0138] In a specific embodiment of the present invention, the parameter updating module 20 is configured to perform the following steps:

[0139] Step S21: Use the Actor-Critic algorithm to build a policy model and a value function model;

[0140] Step S22: using the strategy model, outputting an action probability distribution according to the state of the current input sample, and determining the action strategy to be selected;

[0141] Step S23: generating a variant action space of the selected action strategy through a reinforcement learning algorithm;

[0142] Step S24: Setting a reward function so that the agent selects a mutation action according to the current environment state and the mutation action space, and mutates in the direction of maximizing the final expected reward value.

[0143] In a specific embodiment of the present invention, the parameter updating module 30 is configured to perform the following steps:

[0144] Step S31: performing deterministic mutation and taint tracking on the initial input sample to obtain an initial state byte-path mapping relationship graph as the initial state;

[0145] Step S32: Initialize the policy model network parameters and the value function model network parameters, and define the policy model network learning rate and the value function model network learning rate;

[0146] Step S33: mutating the initial input sample and performing fuzz testing;

[0147] Step S34: determining the action probability under the environmental state conditions according to the policy model, and then determining the reward value of the network according to the value function model;

[0148] Step S35: Update the policy model network parameters and the value function model network parameters until the fuzzy test non-deterministic mutation phase ends.

[0149] In a specific embodiment of the present invention, the reward function is determined according to the seed vulnerability trigger reward function, the seed vulnerability trigger reward function and the path feature reward function, wherein the seed path coverage reward function is determined according to the following formula:

[0150] ;

[0151] Where trace_bits represents the path coverage bitmap executed by the seed fuzz test, and virgin_bits represents the path coverage bitmap covered by all seeds in the fuzz test. represents the summation function;

[0152] The reward function for triggering a seed vulnerability is determined by the following formula:

[0153] ;

[0154] Where, Indicates the number of times the vulnerability is triggered in the global fuzz test;

[0155] The path feature reward function is determined according to the following formula:

[0156] ;

[0157] Where, represents the edges covered by the seed execution process, Indicates the depth of the program path where the edge is located, Indicates the number of times the edge is covered in the global fuzz test.

[0158] Based on the above formula, the reward function of the seed during execution can be defined as:

[0159] ;

[0160] in, is the weight of each reward parameter, R i Represents the reward function, where i ranges from 1 to 3, with R1 being the path coverage reward, R2 being the seed vulnerability trigger reward, and R3 being the path feature reward. By adjusting the weights of the reward parameters, the fuzz test mutation preference can be improved, thereby enhancing the pertinence and efficiency of the test.

[0161] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0162] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between each embodiment can be referred to each other. The embodiments of the present application are described with reference to flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the functions specified in the flowcharts and / or blocks of the flowcharts and / or blocks. Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks The computer program instructions can also be stored in the computer readable storage medium that can guide the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer readable storage medium produce the manufacture including the instruction device, which implements the functions specified in the flowcharts and / or blocks of the flowcharts and / or blocks. Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks The computer program instructions can also be loaded into the computer or other programmable data processing terminal device, so that a series of operation steps are performed on the computer or other programmable terminal device to generate the computer implemented processing, so that the instructions executed on the computer or other programmable terminal device provide the means for implementing the functions specified in the flowcharts and / or blocks of the flowcharts and / or blocks. Figure 1 one or more flows and / or blocks Figure 1The steps of the functions specified in one or more blocks. Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention. Finally, it should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or terminal device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0163] The method and apparatus provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0164] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "a specific embodiment," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fuzzy testing mutation scheduling method based on the Actor-Critic algorithm, characterized by: include: Step S1: Based on the program to be tested and the test sample, perform a taint inference test to obtain the corresponding relationship between the byte changes of the input sample and the program execution path; Step S2: Using the corresponding relationship as the state of the reinforcement learning environment, the Actor-Critic algorithm model is used to improve the mutation scheduling strategy, generate a mutation action space, set the reward function, and define the policy model network parameters and the value function model network parameters; Step S3: performing seed fuzz testing using the mutation scheduling strategy, and updating the strategy model network parameters and the value function model network parameters based on the program execution path and coverage as reward evaluation indicators; Step S2 includes: Step S21: Use the Actor-Critic algorithm to build a policy model and a value function model; Step S22: using the strategy model, outputting an action probability distribution according to the state of the current input sample, and determining the action strategy to be selected; Step S23: generating a variant action space of the selected action strategy through a reinforcement learning algorithm; Step S24: setting a reward function so that the agent selects a mutation action according to the current environment state and the mutation action space, and mutates in the direction of maximizing the final expected reward value; The reward function is determined according to the seed vulnerability trigger reward function, the seed vulnerability trigger reward function and the path feature reward function, Among them, the seed path coverage reward function is determined according to the following formula: Where trace_bits represents the path coverage bitmap executed by the seed fuzz test, virgin_bits represents the path coverage bitmap covered by all seeds in the fuzz test, and sum() represents the summation function; The reward function for triggering a seed vulnerability is determined by the following formula: Where Crash_sum represents the number of times the vulnerability is triggered in the global fuzz test; The path feature reward function is determined according to the following formula: Where, e i Indicates the edge covered by the seed execution process, depth(e i ) represents the depth of the program path where the edge is located, hits(e i ) represents the number of times the edge is covered in the global fuzz test.

2. The fuzzy testing mutation scheduling method based on the Actor-Critic algorithm according to claim 1 is characterized in that: Step S1 includes: Step S11: instrument the program to be tested to monitor changes in program path execution information and related variable values; Step S12: taking the initial seed as input and executing the program to be tested, obtaining a value set of variables related to the program path constraints when the initial sample is running through taint analysis; Step S13: mutate each byte of the initial seed in sequence to generate a new test sample; Step S14: Input the new test sample into the program, obtain the value set of variables related to the program path constraints when the new test sample is running, and if it is the same as the value set of variables related to the program path constraints when the initial sample is running, record the correspondence between the position of the mutated byte and the program variables and the program path.

3. The fuzzy testing mutation scheduling method based on the Actor-Critic algorithm according to claim 1 is characterized in that: Step S3 includes: Step S31: performing deterministic mutation and taint tracking on the initial input sample to obtain an initial state byte-path mapping relationship graph as the initial state; Step S32: Initialize the policy model network parameters and the value function model network parameters, and define the policy model network learning rate and the value function model network learning rate; Step S33: mutating the initial input sample and performing fuzz testing; Step S34: determining the action probability under the environmental state conditions according to the policy model, and then determining the reward value of the network according to the value function model; Step S35: Update the policy model network parameters and the value function model network parameters until the fuzzy test non-deterministic mutation phase ends.

4. The fuzzy testing mutation scheduling method based on the Actor-Critic algorithm according to claim 1 is characterized in that: The weight of the reward parameter is changed by adjusting the weight of the reward function to improve the fuzz testing mutation preference.

5. A fuzzy testing mutation scheduling system based on the Actor-Critic algorithm, characterized by: include: Taint analysis module: used to perform taint inference testing based on the program to be tested and test samples, and obtain the correspondence between the byte changes of the input sample and the program execution path; Mutation space generation module: used to use the corresponding relationship as the state of the reinforcement learning environment, use the Actor-Critic algorithm model to improve the mutation scheduling strategy, generate the mutation action space, set the reward function, and define the policy model network parameters and the value function model network parameters; Parameter update module: used to perform seed fuzz testing using the mutation scheduling strategy, and update the strategy model network parameters and value function model network parameters based on the program execution path and coverage as reward evaluation indicators; The parameter updating module is used to perform the following steps: Step S21: Use the Actor-Critic algorithm to build a policy model and a value function model; Step S22: using the strategy model, outputting an action probability distribution according to the state of the current input sample, and determining the action strategy to be selected; Step S23: generating a variant action space of the selected action strategy through a reinforcement learning algorithm; Step S24: setting a reward function so that the agent selects a mutation action according to the current environment state and the mutation action space, and mutates in the direction of maximizing the final expected reward value; The reward function is determined according to the seed vulnerability trigger reward function, the seed vulnerability trigger reward function and the path feature reward function, wherein the seed path coverage reward function is determined according to the following formula: Where trace_bits represents the path coverage bitmap executed by the seed fuzz test, virgin_bits represents the path coverage bitmap covered by all seeds in the fuzz test, and sum() represents the summation function; The reward function for triggering a seed vulnerability is determined by the following formula: Where Crash_sum represents the number of times the vulnerability is triggered in the global fuzz test; The path feature reward function is determined according to the following formula: Where, e i Indicates the edge covered by the seed execution process, depth(e i ) represents the depth of the program path where the edge is located, hits(e i ) indicates the number of times the edge is covered in the global fuzz test.

6. The fuzzy testing mutation scheduling system based on the Actor-Critic algorithm according to claim 5 is characterized in that: The stain analysis module is used to perform the following steps: Step S11: instrument the program to be tested to monitor changes in program path execution information and related variable values; Step S12: taking the initial seed as input and executing the program to be tested, obtaining a value set of variables related to the program path constraints when the initial sample is running through taint analysis; Step S13: mutate each byte of the initial seed in sequence to generate a new test sample; Step S14: Input the new test sample into the program, obtain the value set of variables related to the program path constraints when the new test sample is running, and if it is the same as the value set of variables related to the program path constraints when the initial sample is running, record the position of the mutated byte and the relationship between the program variables and the path.

7. The fuzzy testing mutation scheduling system based on the Actor-Critic algorithm according to claim 5 is characterized in that: The parameter updating module is used to perform the following steps: Step S31: performing deterministic mutation and taint tracking on the initial input sample to obtain an initial state byte-path mapping relationship graph as the initial state; Step S32: Initialize the policy model network parameters and the value function model network parameters, and define the policy model network learning rate and the value function model network learning rate; Step S33: mutating the initial input sample and performing fuzz testing; Step S34: determining the action probability under the environmental state conditions according to the policy model, and then determining the reward value of the network according to the value function model; Step S35: Update the policy model network parameters and the value function model network parameters until the fuzzy test non-deterministic mutation phase ends.

Images