A Software Intelligent Testing Method Based on DQN Neural Network

Abstract

This invention discloses a software intelligent testing method based on a DQN neural network, primarily addressing the difficulty of detecting control strategy vulnerabilities in large-scale system software using existing methods. The solution includes: 1) extracting a parameter list and the control strategies the software must follow from the software development manual; 2) converting the control strategies into violation examples, while extracting parameters as supervisory parameters; 3) constructing and improving a deep Q-learning neural network (DQN) model; 4) interfaceing the improved model with the software under test; 5) setting the maximum number of fuzzing rounds N and the judgment conditions for the software under test; and 6) optimizing the fuzzing input to enhance fuzzing efficiency. This invention effectively improves the detection effect of software control strategy vulnerabilities and can be used for targeted testing of control strategy vulnerabilities in large-scale system software.

Description

Technical Field

[0001] This invention belongs to the field of network and information security technology, and further relates to software testing methods, specifically a software intelligent testing method based on deep Q-learning neural network (DQN), which can be used to perform targeted testing on control policy vulnerabilities in software. Background Technology

[0002] With the rapid development of Internet technology, people's demand for application software is increasing year by year, and they have higher requirements for the purposefulness and security of software. The importance of software security testing is self-evident.

[0003] Fuzzing is the process of discovering security vulnerabilities in input processing code by repeatedly testing it with modified or fuzzed input. Fuzzing is an effective method for discovering software security vulnerabilities and is becoming a standard practice in software development. Existing fuzzing tools differ in how they handle program input, but in practice, no single tool can exhaustively explore the entire input space of a real program. Therefore, fuzzing heuristics are typically used to determine which parts of the input to fuzz next. These heuristics may be completely random, or they may attempt to be optimized for specific goals, such as maximizing code coverage.

[0004] Traditional fuzzing methods fail to consider the entire input space of the software, thus hindering the effective detection of specific software vulnerabilities, such as violations of control policies in control software of large industrial or commercial systems. From a static analysis perspective, personnel facing massive amounts of software code must rely on experience to identify control policy vulnerabilities. However, this approach demands extensive experience and familiarity with various attack methods. When detecting control policy vulnerabilities in large industrial or commercial system control software, this is not only time-consuming and inefficient but also prone to oversights due to insufficient experience or negligence on the part of the personnel. Therefore, there is an urgent need to develop an efficient method for detecting software control policy vulnerabilities. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the present invention aims to solve the problem of difficulty in detecting control strategy vulnerabilities in large-scale system software, and provides a software intelligent testing method based on DQN neural network. By obtaining an input selection model that continuously learns fuzzy testing experience, the efficiency of software control strategy vulnerability detection is effectively improved.

[0006] The specific steps of this invention to achieve the above objectives are as follows:

[0007] (1) Extract the list of input parameters in the fuzzy testing work from the software development manual of the software under test, and extract the control strategy that the software needs to follow. The control strategy is divided into two types: first control strategy and second control strategy. The first control strategy consists of the operation to be performed and the preconditions to be met for the operation. The second control strategy consists of the current state of the software and the parameter operation to be performed.

[0008] (2) The two types of control strategies are converted into violation examples respectively, and the parameters in the control strategies are extracted as supervision parameters; the violation examples are the target value combinations of the supervision parameters.

[0009] (3) Using the current state of the software as the input of the DQN model input layer and the list of input parameters in the fuzzing test as the output of the DQN model output layer, a deep Q-learning neural network DQN model is constructed. The number of neurons in the model output layer is set to the number of input parameters in the list of input parameters. The instant reward R of the DQN model is set to the normalized approximation of the value of the supervision parameter to the value of its corresponding target under the current round of fuzzing test.

[0010] (4) Add a softmax function module after the output layer of the DQN model constructed in step (3) to obtain the improved DQN model;

[0011] (5) Connect the improved DQN model with the software under test. The software under test transmits the current state of the software to the input layer of the model through the software interface, that is, the combination of values ​​of the supervision parameters in the current round.

[0012] (6) The model inputs the weights Q of the parameters in the output parameter list of the output layer according to the current state of the received software, and calculates the probability of each neuron in the output layer of the model being selected through the softmax function module based on the weights Q of the input parameters.

[0013] (7) Set the maximum number of rounds N for fuzz testing and the judgment conditions for the software under test. The judgment conditions are as follows: the operation to be performed in the first control strategy can be performed or the parameter operation required in the second control strategy is not performed; the maximum number of rounds N is greater than or equal to 50.

[0014] (8) Randomly select the input parameters of the fuzzy test from the probabilities calculated in step (6), and assign values ​​to the input parameters of the fuzzy test using the principle of bisection.

[0015] (9) Determine whether the values ​​of each monitoring parameter in the current fuzzy test round meet the target values ​​of the monitoring parameters specified by the violation sample. If they meet, further determine whether the conditions set in step (7) are met. If they are met, report that the control strategy programming corresponding to the current violation sample has a vulnerability. Otherwise, report that the current control strategy programming is correct and execute step (10). If not, determine whether the maximum round N has been reached. If yes, execute step (10). Otherwise, return to step (8).

[0016] (10) The DQN model uses the optimal input parameters, supervision parameters, current values ​​of supervision parameters, and optimal immediate reward R in the current fuzzy test round as test experience. It records one experience every 5 rounds to achieve experience accumulation. And every 10 accumulated experiences are used to update the DQN model in reverse.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] First, it can automatically learn from fuzz test input experience, resulting in high input selection efficiency: Because this invention uses the softmax function to improve the DQN model, it improves the input selection method of fuzz testing, avoiding the inefficiency caused by blindly and randomly selecting inputs in traditional automated fuzz testing methods. It can continuously learn from past selection experience during the automated testing process, and continuously improve the input selection efficiency of the model.

[0019] Secondly, it is sensitive to the detection of software control strategy vulnerabilities: Since the present invention transforms the two types of control strategies into violation examples as the software target state for fuzzing, and extracts the parameters in the control strategies as supervision parameters to determine whether the fuzzing has reached the target state, the fuzzing method of the present invention has a heuristic to guide the testing direction and a result judgment standard, which can sensitively identify the control vulnerabilities in the software. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;

[0021] Figure 2 This is a flowchart illustrating the workflow of the DQN input selection model in the method of this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the present invention will be further described clearly and completely below in conjunction with specific embodiments.

[0023] Example 1: Refer to Figure 1 The present invention proposes a software intelligent testing method based on DQN neural network, which specifically includes the following steps:

[0024] Step 1: Extract the list of input parameters for fuzzy testing from the software development manual. At the same time, extract the control strategies that the software needs to follow and divide the control strategies into two types: first control strategy and second control strategy. The first control strategy consists of the operation to be performed and the preconditions that need to be met to perform the operation. The second control strategy consists of the current state of the software and the parameter operations that need to be performed.

[0025] The list of input parameters includes the value range of each input parameter as determined by the development manual;

[0026] The control strategy is in natural language form.

[0027] Step 2: Convert the two types of control strategies into violation examples respectively, and extract the parameters in the control strategies as monitoring parameters; the violation example is the target value combination of the monitoring parameters; the violation example is the software prerequisite state that triggers the programming vulnerability of the control strategy, and is used to specify the target software state of the fuzzing work.

[0028] The method of transforming violation examples is used to convert the two types of control policies into violation examples, as detailed below:

[0029] For the first control strategy: the security preconditions that must be met in the strategy are used as indicators, and any operation that is not met and is attempted is considered a violation example;

[0030] For the second control strategy: the required parameter operations in the strategy are used as indicators, and the failure to perform the required parameter operations when the software is in a state with security risks is considered a violation example.

[0031] Step 3: Using the current state of the software as the input of the DQN model input layer and the list of input parameters in the fuzzing test as the output of the DQN model output layer, construct a deep Q-learning neural network (DQN) model. Set the number of neurons in the model output layer to the number of input parameters in the list of input parameters, and set the instantaneous reward R of the DQN model to the normalized approximation of the value of the supervision parameter to its corresponding target value under the current round of fuzzing test.

[0032] The Deep Q-Learning Neural Network (DQN) model is described, where the neural network is divided into three layers according to their positions: an input layer, a hidden layer, and an output layer. The first layer is the input layer, the last layer is the output layer, and the intermediate neural network layers are all hidden layers. The model contains two networks with identical structures: a behavior network and a target network. During initialization, the parameter values ​​of the two networks are the same. During training, the behavior network is trained to update its parameters, and the weights of the target network are updated by direct assignment. The model's input is the current state of the software; in fuzzing tests, the list of input parameters represents the actions. The neurons in the output layer output the Q-values ​​corresponding to each input parameter based on the current state. The Softmax function calculates the probability of each input parameter being selected based on the given Q-values, and the DQN model randomly selects an input parameter according to the probability.

[0033] The instantaneous reward R of the DQN model is set as the normalized approximation of the value of the supervised parameter to its corresponding target value under the selected input parameters, as follows:

[0034] Let R j This represents the normalized approximation of the j-th supervisory parameter value in the current round to its corresponding target value. The immediate reward R is then determined by R1, R2, ..., R... j The value of ... determines R. j The values ​​can be either continuous or discrete, and the specific rules for classification are as follows:

[0035] If the j-th supervisory parameter is a continuous variable:

[0036] R j =(y t,j -y t-1,j ) / y t,j ,

[0037] If the j-th supervision parameter is a discrete variable:

[0038] R j =Bool(Y) t,j =y j，target )-Bool(Y t-1,j =y j，target ),

[0039] Where t represents the current round of the fuzzy test, t-1 represents the previous round, and y j，target Y represents the target value of the j-th supervision parameter in the strategy; t,j Indicates the supervisory parameter; y t,j Corresponding to Y t,j The value to be taken in the current round; Bool() represents a Boolean function;

[0040] Set the instant reward R as follows:

[0041] R = max(|R1|,...,|R) i |,...,|R j |,...),

[0042] Where || represents taking the absolute value, R1, R2, ..., R j... This is the normalized approximation of the values ​​of each monitoring parameter to their corresponding target values.

[0043] Step 4: Add a softmax function module after the output layer of the DQN model constructed in step 3 to obtain the improved DQN model;

[0044] Step 5: Connect the improved DQN model to the software under test. The software under test transmits the current state of the software to the input layer of the model through the software interface, that is, the combination of values ​​of the supervision parameters in the current round.

[0045] Step 6: The model inputs the weights Q of the parameters in the output parameter list of the output layer according to the received current state of the software, and calculates the probability of each neuron in the output layer of the model being selected through the softmax function module based on the weights Q of the input parameters.

[0046] Step 7: Set the maximum number of rounds N for fuzz testing and the judgment conditions for the software under test. The judgment conditions are as follows: the operation to be performed in the first control strategy can be executed or the parameter operation required in the second control strategy is not executed; the maximum number of rounds N is greater than or equal to 50.

[0047] Step 8: Randomly select the input parameters for fuzzy testing from the probabilities calculated in Step 6, and assign values ​​to the input parameters for fuzzy testing using the bisection principle. Specifically, take the median value of the range of the input parameters selected in the current round of fuzzy testing and assign a value to the input parameter. If the assignment result helps the supervisory parameter move closer to the target value specified by the violation example, then take that value. Otherwise, divide the median value of the range of the input parameter into two intervals and continue to take the median value from the new intervals until the value assigned to the input parameter makes the supervisory parameter move closer to the target value specified by the violation example.

[0048] Step 9: Determine whether the values ​​of each monitoring parameter in the current fuzzy test round meet the target values ​​of the monitoring parameters specified by the violation sample. If they meet, further determine whether the conditions set in Step 7 are met. If they are met, report that the control strategy programming corresponding to the current violation sample has a vulnerability; otherwise, report that the current control strategy programming is correct and execute Step 10. If not, determine whether the maximum round N has been reached. If yes, execute Step 10; otherwise, return to Step 8.

[0049] Step 10: The DQN model uses the optimal input parameters, supervision parameters, current values ​​of the supervision parameters, and optimal immediate reward R from the current fuzzy test round as test experience. One experience point is recorded every 5 rounds to accumulate experience; and every 10 accumulated experiences are used to update the DQN model in reverse. Figure 2 As shown.

[0050] The reverse update is used to continuously increase the probability of the DQN model selecting input parameters related to the supervision parameters, so that the software state tends to the target state specified by the violation sample, thereby realizing the detection of programming vulnerabilities in the control strategy.

[0051] The parts of this invention not described in detail are common knowledge to those skilled in the art.

[0052] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A software intelligent testing method based on DQN neural network, characterized in that, The method comprises the following steps: (1) extracting the input parameter list in the fuzzy test from the software development manual to be tested, and extracting the control strategy required by the software, and dividing the control strategy into two types of first control strategy and second control strategy, wherein the first control strategy is composed of an operation to be performed and a prerequisite condition required to perform the operation, and the second control strategy is composed of a current state of the software and a parameter operation required to be performed; (2) converting the two types of control strategies into violation examples respectively, and extracting the parameters in the control strategies as supervision parameters; the violation example is a target value combination of the supervision parameters; (3) constructing a deep Q-learning neural network DQN model by using the current state of the software as the input of the input layer of the DQN model and the input parameter list in the fuzzy test as the output of the output layer of the DQN model, setting the number of neurons in the output layer of the model as the number of input parameters in the input parameter list, and setting the immediate reward R of the DQN model as the normalized proximity of the value of the supervision parameter to the corresponding target value in the current round of fuzzy test; (4) adding a softmax function module to the output layer of the DQN model constructed in step (3) to obtain an improved DQN model; (5) connecting the improved DQN model with the software to be tested, and transmitting the current state of the software to the input layer of the model through the software interface, i.e. the value combination of the supervision parameters in the current round; (6) outputting the weight Q of the input parameters in the parameter list in the output layer of the model according to the received current state of the software, and calculating the probability of each neuron in the output layer of the model being selected by the softmax function module according to the weight Q of the input parameters; (7) setting the maximum number of rounds N of the fuzzy test and the judgment condition of the software to be tested, wherein the judgment condition is that the operation to be performed in the first control strategy can be performed or the parameter operation required to be performed in the second control strategy is not performed, and the maximum number of rounds N is greater than or equal to 50; (8) randomly selecting the input parameters of the fuzzy test from the probabilities calculated in step (6), and assigning values to the input parameters of the fuzzy test by using the bisection method; (9) judging whether the values of the monitoring parameters in the current round of fuzzy test meet the target values of the monitoring parameters specified in the violation example, if yes, further judging whether the condition set in step (7) is established, if yes, reporting that the control strategy corresponding to the current violation example has a programming bug, otherwise, reporting that the current control strategy is programmed correctly, and executing step (10); if not, judging whether the maximum number of rounds N is reached, if yes, executing step (10), otherwise, returning to step (8); (10) accumulating the experience by taking the optimal input parameters, supervision parameters, current values of the supervision parameters and optimal immediate rewards R in the current round of fuzzy test as test experience, recording one experience every 5 rounds, and updating the DQN model once every 10 experiences.

2. The method of claim 1, wherein: The input parameter list in step (1) comprises a value range of each input parameter determined according to a development manual; and the control strategy is in a natural language form.

3. The method of claim 1, wherein: In step (2), the two types of control strategies are respectively converted into violation examples by using a violation example conversion method, and the conversion method is as follows: For the first control strategy, a safety precondition required to be met in the strategy is taken as an index, and a violation example is taken as an operation that is not performed due to not meeting any index; For the second control strategy, a parameter operation required to be performed in the strategy is taken as an index, and a violation example is taken as a state of the software in which a security risk exists and the required parameter operation is not performed.

4. The method of claim 3, wherein: The violation example in step (2) is a software precondition state triggering a control strategy programming vulnerability, and is used to specify a target software state of the fuzzy testing work.

5. The method of claim 1, wherein: In step (3), the deep Q learning neural network DQN model comprises an input layer, a hidden layer and an output layer according to different positions; the first layer is the input layer, the last layer is the output layer, and the neural network layers in the middle are the hidden layers.

6. The method of claim 5, wherein: The deep Q learning neural network DQN model comprises two networks with the same structure: a behavior network and a target network; during initialization, the parameter values of the two networks are the same, and during the training process, the behavior network is trained to update the parameters, and the target network weight is updated by direct assignment.

7. The method of claim 6, wherein: The deep Q learning neural network DQN model takes the current state of the software as the input, the input parameter list in the fuzzy testing work as the action, and the neurons in the output layer output the Q values corresponding to each input parameter based on the current state; The Softmax function calculates the probability of each input parameter being selected based on the given Q values, and the DQN model randomly selects an input parameter according to the probability.

8. The method of claim 1, wherein: In step (3), the immediate reward R of the DQN model is set as the normalized approximation of the value of the supervised parameter to the corresponding target value under the selected input parameter in this round, and the setting is as follows: Let R j denote the normalized closeness of the value of the jth supervision parameter in the current round to its corresponding target value, i.e. the immediate reward R is jointly determined by the values of R1, R2,..., R j ..., and R j has both continuous and discrete values, and the specific rules for classification are as follows: If the jth supervised parameter is a continuous variable: R j = (y t,j - y t-1,j ) / y t,j , If the jth supervised parameter is a discrete variable: R j = Bool(Y t,j = y j，target ) - Bool(Y t-1,j = y j，target ), where t represents the current round of fuzz testing, t-1 represents the previous round, y j，target represents the target value of the jth supervision parameter in the strategy; Y t,j represents the supervision parameter; y t,j corresponds to Y t,j the value in the current round; Bool() represents a Boolean function; The immediate reward R is set as follows: R = max(|R1|,...,|R i |,...,|R j |,...), wherein || denotes taking the absolute value, R1, R2,... R j ... is the normalized proximity of the value of each monitoring parameter to its corresponding target value.

9. The method of claim 1, wherein: In step (8), the input parameter is valued by using the bisection method, and the valuation is as follows: the middle value of the value range of the input parameter selected in the current round of fuzzy testing is taken as the value of the input parameter; if the valuation result is conducive to the supervised parameter approaching the target value specified by the violation example, the value is taken; otherwise, the middle value of the input parameter value range is divided into two intervals, and the middle value in the new interval is continuously taken until the value of the input parameter makes the supervised parameter approach the target value specified by the violation example.

10. The method of claim 1, wherein: In step (10), the reverse update is used to make the probability of the DQN model selecting the input parameter related to the supervised parameter constantly increase, so that the software state tends to the target state specified by the violation example, and the control strategy programming vulnerability is detected.

Images