An automated penetration testing method, system, device and medium based on deep reinforcement learning

Abstract

The application discloses an automatic penetration testing method and system based on deep reinforcement learning, and a device and a medium. The method comprises the following steps: performing vulnerability scanning and baseline checking on a device by using a vulnerability scanning and baseline checking tool to obtain vulnerability information of a system under test; generating an attack graph according to the obtained vulnerability information, determining a vulnerability exploitation probability according to the vulnerability information, and assigning values to edges of the attack graph to obtain a Bayesian attack graph; designing a reward value of a target node of an attacker on the generated Bayesian attack graph, simulating possible attack behaviors of the attacker, and calculating an attack strategy of the attacker by using a deep reinforcement learning method; simulating potential attack behaviors of the attacker in the attack graph according to the attack strategy to generate a penetration attack path most likely to be adopted by the attacker; and performing automatic penetration testing to generate a penetration testing report. The application can intelligently generate a penetration testing scheme according to the vulnerability of a system, and greatly improves the precision and efficiency of network security evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and particularly relates to an automatic penetration testing method, system, device and medium based on deep reinforcement learning. BACKGROUND

[0002] With the continuous and rapid development of computer technology and Internet technology, people are provided with many conveniences, but also many network security problems. On the one hand, network applications and operating systems are becoming more and more complex, and vulnerabilities are emerging in the design or configuration. On the other hand, the improvement of interconnection degree and transmission speed provides more attack entry and utilization conditions for network attacks, resulting in a substantial increase in the number of network attacks. And network attack behavior is no longer an isolated event, but shows a multi-stage characteristic. Attackers often lurk into the target network through hidden means, use the causal relationship between business processes and vulnerabilities in the network, and use different devices as a stepping stone to ultimately achieve information theft and device destruction and other goals. In the face of such complex and hidden, more professional and organized network attacks, as well as continuously updated advanced threats, security personnel often use penetration testing methods to improve network security capabilities. Penetration testing is a proactive security assessment method that simulates the attack behavior of malicious hackers to assess the security of computer network systems. Penetration testing analyzes the weaknesses, technical defects or vulnerabilities of the system, with the purpose of discovering and exploiting these vulnerabilities from the perspective of a potential attacker, to verify the security of the system, and handle and avoid potential security vulnerabilities and risks.

[0003] Penetration testing requires a lot of professional knowledge and is difficult to formalize. Therefore, the existing penetration testing is usually manually executed by professional testers according to certain penetration testing strategies. The specific testing steps include: 1. Determine the target and scope of penetration testing: determine the IP address range, domain name, application program, system and service of the target system with the test organizer; 2. Information collection and threat modeling: use public information sources and scanning tools to collect target information, analyze the collected information, determine the attacker's interest point, create an attacker model, and find an attack path; 3. Vulnerability analysis: use testing tools to identify vulnerabilities in systems, networks and applications, verify vulnerabilities and evaluate vulnerability availability; 4. Penetration attack: find or develop related exploitation tools according to the results of vulnerability analysis, exploit vulnerabilities to obtain system access (such as obtaining shell, privilege escalation, etc.), break through the system boundary and move horizontally within the system, find more vulnerabilities and sensitive data, and try to obtain sensitive data, and leave a backdoor in the system; 5. Write a test report: record all findings in the test in detail, provide repair suggestions and improvement measures according to the test results, and write a formal test report.

[0004] The above process constitutes a complete penetration test. The penetration test is a complex task that is laborious and time-consuming, highly dependent on the skills of the operator, and the accuracy and reliability of the test cannot be guaranteed. All strategies during the operation process, including configuration, execution, monitoring and maintenance, need to be manually performed, and the efficiency of the penetration test is low, which cannot meet the requirements of attack effectiveness. In addition, the size of the network is growing now, and the network topology is becoming more and more complex. In this case, enumerating and testing all penetration paths requires a large amount of resources and time, and it is difficult to complete through manpower or some simple automated test scripts.

[0005] Based on the above-mentioned defects, there is an urgent need in the industry for a method capable of intelligently generating a penetration test strategy to quickly and accurately perform automated penetration testing on information systems. The present application proposes an automated penetration testing method and system based on deep reinforcement learning. SUMMARY

[0006] The purpose of the present application is to provide an automated penetration testing method, system, device and medium based on deep reinforcement learning, which can intelligently generate a penetration test scheme according to the vulnerability of the system, realize automated penetration testing, and greatly improve the accuracy and efficiency of network security evaluation.

[0007] To achieve the above-mentioned purpose, the technical solution provided by the present application is:

[0008] The first aspect of the present application provides an automated penetration testing method based on deep reinforcement learning, comprising the following steps:

[0009] Determine the network, operating system and application information range of the penetration test, use vulnerability scanning and baseline checking tools to perform vulnerability scanning and baseline checking on all devices within the determined range, and obtain the vulnerability information of the system under test;

[0010] Generate an attack graph according to the obtained vulnerability information, determine the vulnerability exploitation probability according to the vulnerability information, and assign values to the edges of the attack graph to obtain a Bayesian attack graph;

[0011] Design the reward value of the attacker target node on the generated Bayesian attack graph, simulate the possible attack behavior of the attacker, and calculate the attack strategy of the attacker using a deep reinforcement learning method;

[0012] Simulate the potential attack behavior of the attacker in the attack graph according to the attack strategy, and generate the penetration attack path most likely to be used by the attacker;

[0013] In the penetration test test environment, perform automated penetration testing on the test object according to the penetration attack path most likely to be used by the attacker, and generate a penetration test report.

[0014] To optimize the above technical solutions, the specific measures taken also include:

[0015] An attack network model containing formalized network security elements is established, the network security elements including host information, network topology and network vulnerability information; the host information is expressed by a triple <Host, Services, Vuls>, wherein Host is the unique identification of the host; Services is the software and service information running on the host; and Vul refers to the vulnerability information of the host; the network topology is expressed by an undirected graph G<N, E>, wherein N represents the set of all physical hosts in the network; and E represents the physical connections between the hosts, each edge being composed of a quadruple <Start, Destination, Protocol, Port>, wherein Start, Destination, Protocol and Port represent the source host, the destination host, the protocol and the port number of the connection, respectively; the network vulnerability information is expressed by a quintuple <ID, IP, Probability, Complex, Impact>, wherein ID is the unique corresponding number of the vulnerability in the vulnerability library; IP is the host address where the vulnerability exists; Probability is the probability of the vulnerability being exploited by an attacker, and the value range is (0, 1); Complex is the complexity of the vulnerability, and the value range is (0, 1); and Impact is the severity of the harm caused after the vulnerability is exploited, and the vulnerability score calculation formula provided by CVSS is used for calculation, and the score range is (0, 10), the higher the score, the greater the vulnerability. a > represents, wherein S is a state node; A is a set of attack actions available for the attacker; E is a directed edge; and p a is a non-zero probability value, representing the probability of success of the attack action a.

[0016] The obtained Bayesian attack graph is processed as follows: the vulnerability information in the attack graph is deleted, and only the state nodes and the transition directions are retained; the secure states are reduced according to the monotonicity principle, so that the attacker will not acquire a secure state again after having acquired the secure state, so as to avoid repeated attack paths; the attack is formalized through a directed "hyperedge", and the secure states linked by the attack actions are focused on.

[0017] The penetration testing process is converted into a Markov decision process, the attacker is corresponding to an agent, and the training of the attack strategy is abstracted as a problem of learning and improving the strategy of the agent in the interaction with the environment, and the attack strategy is corresponding to the probability distribution of the next action selected by the agent in different states.

[0018] The Markov decision MDP is composed of a four-tuple <S, A, P, R>, wherein S is a state space composed of attack graphs at different times in the Bayesian attack graph; A is an action space representing that a penetration tester activates a target node using a vulnerability in the BAG; P is a transition probability representing a probability of using different edges in the Bayesian attack graph; and R is a reward, which is obtained after the agent activates a target node, and in order to prevent the agent from repeating the same action indefinitely, the agent will obtain a value as a negative reward for taking an action as a cost each time the action is taken.

[0019] Further, the attack strategy of the attacker is trained using a deep reinforcement learning method, an optimal action value function of the attacker is defined according to a time difference function, and the optimal action value function is used as an optimization target.

[0020]

[0021] wherein Q ＊ (s t , a t ) represents the optimal action value function of the attacker; represents an expected value of a reward obtained in a next state after an attack action is taken; r t represents a reward obtained by the attacker after an action is taken at time t; and γ represents a discount rate of learning, and the discount rate is greater for a more distant future. represents a maximum Q value in a next round; s t represents a state of an attack graph at time t; and a t represents an action taken by the attacker at time t.

[0022] The Q-learning method is used to cause the attacker to continuously update the optimal action value function value in interaction with the environment, so as to approximate a theoretical optimal value.

[0023] Further, the attack strategy training adopts a deep neural network Q(s t , a t ; θ), wherein θ is a parameter of the deep neural network; and the deep neural network Q(s t , a t ; θ) is composed of an input layer, a hidden layer, a softmax layer, and an output layer, the input layer inputs all states observed by the agent at present, including all node states and successful attack actions, the hidden layer is composed of two fully connected layers, and a ReLU activation function is used; the softmax layer is before the output layer, and is responsible for converting a result into a set of probability distributions; and the output layer includes all actions and a null node, the null node represents that the attacker does not take any action in the round, and the purpose is to prevent the attacker from randomly performing an action when there is no suitable attack target.

[0024] Further, the use of deep reinforcement learning method to calculate the attack strategy of the attacker, the process is as follows:

[0025] The deep neural network parameters of the attacker are initialized as random values, the initial attack graph state s0 is randomly selected from the training data set, and the algorithm is run from the initial attack graph state s0;

[0026] The attacker selects actions according to the current attack strategy, and adds exploration elements in the selection process, so that the attacker as an agent can sample new states with a certain probability, and the state transition of the attack graph is promoted by continuously selecting actions;

[0027] After each state transition occurs, the reward value obtained by the agent is calculated in real time, and the reward value includes the reward obtained after activating the target node and the cost paid by the agent each time the action is taken;

[0028] According to the reward value after state transition, the gradient of the loss function of the deep neural network is calculated, the parameters of the deep neural network are updated using the gradient descent method, and the attack strategy of the attacker is updated based on this.

[0029] The second aspect of the application provides an automatic penetration testing system based on deep reinforcement learning, comprising:

[0030] The scanning module is used to determine the network, operating system and application information range of the penetration test, and uses vulnerability scanning and baseline checking tools to perform vulnerability scanning and baseline checking on all devices in the determined range, and obtains the vulnerability information of the tested system;

[0031] The attack graph generation module is used to generate an attack graph according to the obtained vulnerability information, determine the vulnerability exploitation probability according to the vulnerability information, and assign values to the edges of the attack graph to obtain a Bayesian attack graph;

[0032] The strategy training module is used to design the reward value of the attacker target node on the generated Bayesian attack graph, simulate the possible attack behavior of the attacker, and calculate the attack strategy of the attacker using a deep reinforcement learning method;

[0033] The simulated attack module is used to simulate the potential attack behavior of the attacker in the attack graph according to the attack strategy, and generate the penetration attack path most likely to be adopted by the attacker;

[0034] The penetration testing module is used to perform automatic penetration testing on the test object according to the penetration attack path most likely to be adopted by the attacker in the penetration testing test environment, and generate a penetration testing report.

[0035] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the deep reinforcement learning based automated penetration testing method according to the first aspect of the present application when executing the computer program.

[0036] The fourth aspect of the present application provides a computer readable storage medium storing a computer program, wherein the computer program causes a computer to execute the deep reinforcement learning based automated penetration testing method according to the first aspect of the present application.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] In the scenario of penetration testing on a target object, the present application firstly performs vulnerability scanning and baseline checking on the target object to obtain network asset information and vulnerability information of the target object. The success probability of exploiting the vulnerability is queried, and the asset information and the vulnerability information are combined and uniformly input into an attack graph generation tool to generate a Bayesian attack graph of the target object. A deep reinforcement learning algorithm is run on the Bayesian attack graph to train an optimal attack strategy of an attacker, generate an attacker strategy neural network, and output attack actions of the attacker in different situations. In the initial state, each attack action of the attacker is simulated according to the trained attacker strategy network to generate an optimal attack path most likely to be adopted by the attacker. The real network environment can be automatically penetrated by using the attack path.

[0039] Compared with the prior art of penetration testing by artificial means, the above-mentioned method of the present application generates a Bayesian attack graph in advance according to the vulnerability of the target object to be tested, uses a deep reinforcement learning model to train the penetration method, obtains an automatic attack strategy network, and can automatically adjust the attack behavior according to the environment. After obtaining the real network environment, the real network information can be input into the automatic penetration testing model of the present application to obtain an attack graph and attack actions, generate an attack path, and directly use the real attack actions and attack programs in the attack weapon library to penetrate the real network, thereby quickly and accurately performing automatic penetration testing on the network. The present application improves the limitations of the conventional scheme that the penetration testing is highly dependent on the professional level of the tester and consumes too much time, improves the accuracy and efficiency of the penetration testing, and improves the testing effect.

[0040] The deep reinforcement learning algorithm used in the present application successfully solves the problem of large state space and difficulty in searching for an optimal strategy, can provide support for penetration testing of large networks, and the dynamic model used is closer to the actual situation. The present application optimizes the traditional attack graph, the attack graph generation speed is faster, can process larger network structures, and the visualization effect and expandability of the attack graph are better. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 Flowchart of the deep reinforcement learning-based automatic penetration testing method of the present application.

[0042] Figure 2 Network topology diagram in the embodiment of the present application.

[0043] Figure 3 Bayesian attack graph in the embodiment of the present application.

[0044] Figure 4 State transition diagram of the Bayesian attack graph in the embodiment of the present application.

[0045] Figure 5 Flowchart of the attack strategy training of the present application.

[0046] Figure 6 Deep neural network diagram of the attack strategy training of the present application.

[0047] Figure 7 Penetration testing result in the embodiment of the present application.

[0048] Figure 8 Structural schematic diagram of the deep reinforcement learning-based automatic penetration testing system of the present application. DETAILED DESCRIPTION

[0049] The above content of the present application is further described in detail in the form of examples below, but this should not be understood as the scope of the above subject matter of the present application being limited to the following examples only, and any technology realized based on the above content of the present application falls within the scope of the present application.

[0050] Some nouns in the present application are explained as follows:

[0051] AG (Attack Graph, attack graph): a directed graph showing the attack sequence and attack effect that an attacker may launch, composed of vertices and directed edges. The attack graph can build a complete network security model, reflecting the vulnerability of each node in the network and depicting all ways for an attacker to break into important nodes.

[0052] BAG (Bayesian Attack Graph, Bayesian attack graph): a network security analysis tool combining Bayesian network and attack graph, which is a directed acyclic graph and can represent the probability of attacker behavior and attack progress in the network.

[0053] MDP (Markov decision process): A mathematical framework for modeling the process of decision-making by an agent in an uncertain environment, the agent perceives the current state, affects the environment through actions and gets rewards.

[0054] DRL (Deep Reinforcement Learning): A machine learning method that combines deep learning (DL) and reinforcement learning (RL). It uses deep neural networks to process high-dimensional sensory inputs and learn how to make decisions in complex environments.

[0055] The present application proposes an automatic penetration testing method based on deep reinforcement learning, as shown in Figure 1 The method comprises the following steps:

[0056] Determine the network, operating system and application information range of the penetration test, use vulnerability scanning and baseline checking tools to perform vulnerability scanning and baseline checking on all devices within the determined range, and obtain the vulnerability information of the tested system;

[0057] Generate an attack graph according to the obtained vulnerability information, determine the vulnerability exploitation probability according to the vulnerability information, and assign values to the edges of the attack graph to obtain a Bayesian attack graph;

[0058] Design the reward value of the attacker target node on the generated Bayesian attack graph, simulate the possible attack behavior of the attacker, and use deep reinforcement learning method to calculate the attack strategy of the attacker;

[0059] Simulate the potential attack behavior of the attacker in the attack graph according to the attack strategy, and generate the penetration attack path most likely to be adopted by the attacker;

[0060] In the penetration testing test environment, according to the penetration attack path most likely to be adopted by the attacker, the test object is automatically penetrated and tested, and a penetration testing report is generated.

[0061] The following specific embodiments further explain the present application from the following five parts: network security element modeling, constructing Bayesian attack graph, using deep reinforcement learning method to train attack strategy, generating attack path and implementing penetration test:

[0062] (1) Network security element modeling

[0063] Large-scale network systems have dynamic and changeable connection structures, containing a large number of hosts, servers, routers, firewalls and other network devices. Different devices run complex and diverse operating systems and application software. From the perspective of penetration testers, the first step to establish a model of attacking network is to formalize these network elements. Generally speaking, the network model needs to show the network elements including host information, network topology, network vulnerability information. The following will define these elements one by one:

[0064] Host information: Host information includes the address of the host, the services and open ports of the host, and the vulnerability information of the host, reflecting the attributes of different types of hosts in the network system. It can be represented by a triple <Host, Services, Vuls>, where Host is the unique identifier of the host, which can be the host name, IP address or Mac address; Services is the software and service information running on the host, including operating system, application software, etc.; Vul refers to the vulnerability information of the host; For example, the information triple of the host <127.0.0.1, {Windows7, SQLServer}, CVE-2012-1123> has an IP address of 127.0.0.1, uses Windows7 operating system and runs SQL Server database service, and has a vulnerability with CVE number CVE-2012-1123.

[0065] Vulnerability information: All information of a vulnerability v can be represented by a five-tuple <ID, IP, Probability, Complex, Impact>, where ID is the unique corresponding number of the vulnerability in the CVE vulnerability database; IP is the address of the host that exists the vulnerability;

[0066] Probability is the probability of the vulnerability being exploited by the attacker, with a value range of (0, 1); Complex is the complexity of the vulnerability, with a value range of (0, 1); Impact is the severity of the damage caused by the vulnerability after being exploited, which can be calculated using the vulnerability score calculation formula provided by CVSS, with a score range of (0, 10), the higher the score, the greater the vulnerability; For example, the system has a "EternalBlue" vulnerability v, whose six-tuple is <CVE-2017-0144, 127.0.0.1, 0.6, 0.5, 9.3>, indicating that the "EternalBlue" vulnerability exists in the host with IP address 127.0.0.1, the probability of the vulnerability being exploited is 0.6, the complexity is 0.5, and the severity score after being exploited is 9.3.

[0067] Topology information: refers to the connection structure between all hosts in the network. Physically, all devices form an undirected graph G < N, E >, where N represents the set of all physical hosts in the network; E represents the physical connection between hosts. Each edge consists of a four-tuple < Start, Destination, Protocol, Port >, representing the source host, destination host, protocol, and port number of the connection, respectively.

[0068] The penetration tester can use vulnerability scanning devices and baseline checking devices to obtain the above information system data. In the present embodiment, the network topology used is as shown in Figure 2 Fig. 1, which has two subnets, a DMZ area, and the like, four areas, 13 hardware devices such as web servers and database servers, and the vulnerability information scanned by the vulnerability scanning device in the embodiment is as shown in the following table:

[0069] Table 1

[0070]

[0071]

[0072] (2) Constructing a Bayesian attack graph

[0073] According to the formalized network security elements, a corresponding Bayesian attack graph (BAG) can be generated. First, the BAG is defined.

[0074] BAG: a directed acyclic Bayesian attack graph consists of a four-tuple: BAG (N) = < S, A, E, p a > S is a set of state nodes, including system vulnerabilities, insecure system and network attributes, etc.; A is a set of attack actions available to the attacker; E is a set of directed edges; p a is a non-zero probability value representing the probability of success of attack action a.

[0075] S = {s1, s2, …, s N} is the N security states that the BAG has, each state s i comes from a vulnerability of the system, contains a state value, true (s i = 1) or false (s i = 0), indicating whether the security state s i has been (not been) exploited by the attacker; in addition to vulnerabilities, s i may also be insecure system and network attributes, such as using weak passwords for login passwords.

[0076] A = {a1, a2, …, a KA set of atomic attacks, representing the behavior of an attacker exploiting vulnerabilities in a network; an atomic attack is a causal relationship between security states, connecting two different security states; each attack node a has a set of pre-node pre(a) and post-node post(a), an attack action a can only be initiated if the state of the pre-node is true (pre(a) = 1); if a is executed successfully, the state of post(a) will be changed from false to true, and the attack graph moves to the next state.

[0077] E is a set of directed edges; for any node e = (u, v) in E, if u is a security state node, then v must be an attack action node, and vice versa. No loop is allowed in BAG, and if it occurs, it needs to be manually eliminated.

[0078] p a is the probability of the success of atomic attack a; the probability of the success of an attack action of a penetration attacker depends on the complexity of the vulnerability and other vulnerability basic properties, and on the configuration of the defense measures in the network; the success probability of using the same vulnerability in different network environments may be different.

[0079] Take the network topology of Figure 2 for example, the following problems will be encountered when generating a standard Bayesian attack graph: 1, generating a Bayesian attack graph requires enumerating all possible system security states, which will reach dozens in this embodiment, and the cost will increase exponentially with the growth of the system size, and the scalability is poor; 2, the Bayesian attack graph contains a lot of information that penetration testers do not need to know, and most penetration testers may not need to know the details of the vulnerability, but only need to know the consequences of the exploitation of the vulnerability and the direction of state transition; 3, the visualization effect will be greatly reduced.

[0080] Therefore, the standard Bayesian attack graph is simplified and modified as follows: on the one hand, unnecessary system states in the attack graph are reduced, and on the other hand, the information required to describe the attack is reduced, and the visualization effect is enhanced:

[0081] All vulnerability information is deleted, and only state nodes and transition directions are retained in the attack graph;

[0082] According to the monotonicity principle, the security states are reduced, that is, the attacker will not acquire a certain security state again to avoid repeated attack paths;

[0083] The concept of attack is formalized using directed "hyper-edges", and the means of attack are no longer analyzed, but only the security states linked by the attack action are focused on.

[0084] "Hyper-edge" refers to an edge abstracted to a higher level, containing more information and being able to link multiple nodes.

[0085] The BAG modified by the above steps is more concise and intuitive. In this embodiment, the BAG is generated according to the network topology of Figure 2 and the vulnerability information in Table 1, and the generation result is as shown in Figure 3 The Bayesian attack Figure 1 has 14 state nodes and 20 attack action nodes, and the state nodes include 3 initial nodes of the attacker entering the system and 3 final target nodes. The edges in the figure connect the action nodes and the state nodes, and each edge has a corresponding attack success probability.

[0086] (3) Training the attack strategy using a deep reinforcement learning method

[0087] First, the penetration testing process needs to be converted into a Markov Decision Process (MDP), and the training of the attack strategy is abstracted as a problem of an agent learning and improving its own strategy in interaction with the environment. Here, the strategy is the probability distribution of the agent selecting the next action in different states, and the agent's selection of action in a state depends entirely on its strategy. The MDP is composed of a four-tuple <S, A, P, R>, where S is the state space, which is composed of the attack graphs at different times in the BAG; A is the action space, which represents the penetration tester continuously exploiting the vulnerabilities in the BAG to activate new state nodes; P is the transition probability, which uses the probabilities of different edges in the BAG; R is the reward, which is defined by the tester, and different reward values may train different attack strategies; in addition, to prevent the agent from repeating the same action indefinitely, the agent will obtain a negative reward value as the cost of taking action each time. Figure 4 The process of state transition of the BAG when the attacker takes action is described.

[0088] The action strategy of the attacker is trained using a deep reinforcement learning method. First, the optimal action value function Q of the attacker is defined according to the time difference function, which is used as the optimization goal:

[0089]

[0090] Where represents the expected value of the reward obtained by the next state after taking the attack action; r t represents the reward obtained by the attacker after taking action at time t; γ represents the discount rate of learning, and the farther the future, the greater the discount; represents the maximum Q value of the next round; s t represents the state of the attack graph at time t; a trepresents the action taken by the attacker at time t; the attacker agent uses the Q-learning method to constantly update the Q value in the interaction with the environment to approach the theoretical optimal value, and the strategy of the attacker is to select the action with the highest Q value. The deep Q learning method is used in the application, and a deep neural network Q(s t , a t ; theta) is used to approximate the theoretical Q value, and theta is the parameter of the deep neural network. The TD target y t and the TD error delta t are defined based on the time difference function:

[0091]

[0092] delta t = Q(s t , a t ; theta) - y t

[0093] The TD target y t is the estimation of the best Q value at time t by the Q function at time t+1, which is based on the actual observation of the next round, and is more reliable than the Q function, so the difference between the TD error and the Q function can be used as the loss function L:

[0094]

[0095] The gradient descent method can be used to take the partial derivative of L to calculate the gradient of the loss function L with respect to theta:

[0096]

[0097] Further gradient descent can make the value of Q(s t , a t ; theta) closer to the TD target y t , where alpha is the learning rate, which is a manually adjusted hyperparameter:

[0098]

[0099] A complete algorithm training process is shown in Figure 5 , which can be roughly divided into four steps:

[0100] (1) Environment initialization: the neural network parameters of the attacker agent are initialized as random values, the initial attack graph state s0 is randomly selected from the training data set, and the algorithm is run from s0;

[0101] (2) Action selection: the attacker selects the action according to the current strategy, and a certain exploration element is added in the selection process to allow the intelligent agent to sample new states with a certain probability. The state transition of the attack graph is promoted by continuously selecting actions;

[0102] (3) Calculate rewards: The reward value obtained by the agent must be calculated in real time after each state transition. The reward value includes the reward obtained after activating the target node and the cost incurred by the agent for each action.

[0103] (4) Parameter update: Calculate the gradient of the loss function based on the reward value after the state transition, update the parameters of the neural network using the gradient descent method, and update the attacker's strategy based on this.

[0104] The attacker agent's trained neural network Q(s) t a t ;θ) as Figure 6 As shown, this neural network consists of an input layer, hidden layers, a softmax layer, and an output layer. The input layer takes into account all the states currently observed by the agent, including all node states and successful attack actions. The hidden layer consists of two fully connected layers (FC), each containing 64 neurons with ReLU activation. The softmax layer, preceding the output layer, transforms the results into a set of probability distributions. The output layer includes all actions and a null node. The null node indicates that the attacker will not take any action in this round, preventing the attacker from arbitrarily executing actions when there is no suitable target.

[0105] This invention includes an experiment to verify the method, with the experimental environment referring to... Figure 2 In the experiment, the attacker's reward for activating the target's secure state node was set to +100, and the cost of the attacker performing the attack action was uniformly set to -5. To prevent unexpected situations in the initial training phase, the total number of rounds was set to 30, after which the game would be forcibly terminated and the current reward calculated. Algorithms compared to this invention are all widely used in the field of automated penetration testing, including the genetic algorithm NSGA-II, the game theory algorithm PF, the Nash Q algorithm combining game theory and artificial intelligence, and randomized policy methods. The evaluation criterion for the experiment was the attacker's total reward value, and the number of iterations was set to 2000. The experimental results are as follows: Figure 7 As shown, DDQN is the deep reinforcement learning algorithm used in this experiment; the random strategy algorithm has consistently performed poorly; as the number of iterations increases, the performance of DDQN, Nash-Q, and NSGA-II gradually improves; after 2000 rounds, DDQN has the best performance.

[0106] (4) Generate attack paths

[0107] After the attack strategy model is trained, a model that can give the probability distribution of the next attack action in any state is obtained. The attack graph in the initial state is input into the model, and the state is transferred step by step until the termination (the target node is activated or the state transition times reach the threshold); this step is run 100 times, and the attack paths of each time are recorded, and finally the three attack paths with the highest frequency of strategy model use are selected as the candidate paths for penetration testing.

[0108] (5) Implementing penetration testing

[0109] The system is subjected to penetration testing using the three attack paths selected in the previous step. Since there is a certain success probability for exploiting system vulnerabilities, if one attack path encounters an obstacle, it can be switched to the other two attack paths in time. The vulnerabilities, attack methods, impact range, etc. found during penetration testing are recorded in detail, and repair suggestions are proposed to form a formal penetration testing report. Based on the results of penetration testing, targeted security reinforcement suggestions are provided to help users strengthen defense measures.

[0110] The application also provides an automatic penetration testing system based on deep reinforcement learning, comprising:

[0111] A scanning module is configured to determine the network, operating system, and application information range of penetration testing, use vulnerability scanning and baseline checking tools to perform vulnerability scanning and baseline checking on all devices within the determined range, and obtain vulnerability information of the system under test.

[0112] An attack graph generation module is configured to generate an attack graph based on the obtained vulnerability information, determine the vulnerability exploitation probability based on the vulnerability information, and assign values to the edges of the attack graph to obtain a Bayesian attack graph.

[0113] A strategy training module is configured to design the reward value of the target node of the attacker on the generated Bayesian attack graph, simulate the possible attack behavior of the attacker, and calculate the attack strategy of the attacker using a deep reinforcement learning method.

[0114] A simulated attack module is configured to simulate the potential attack behavior of the attacker in the attack graph based on the attack strategy, and generate the penetration attack path most likely to be adopted by the attacker.

[0115] A penetration testing module is configured to perform automatic penetration testing on the test object according to the penetration attack path most likely to be adopted by the attacker in a penetration testing test environment, and generate a penetration testing report.

[0116] The application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the automatic penetration testing method based on deep reinforcement learning is implemented.

[0117] The application also provides a computer readable storage medium storing a computer program, the computer program enabling a computer to execute the above deep reinforcement learning based automated penetration testing method.

[0118] In the embodiments disclosed in the application, the computer storage medium can be a tangible medium which can contain or store programs for use by or in connection with an instruction execution system, apparatus or device. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, apparatus or devices, or any suitable combination of the above. More specific examples of the computer storage medium can include one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0119] In summary, the method models the penetration testing attack process as a Markov decision process based on the Bayesian attack graph, defines the reward and state transition function of the attacker behavior and environment according to the four elements of MDP, trains the optimal attack strategy from the perspective of dynamic attack, and compared with the existing mainstream technology, the deep reinforcement learning algorithm used in the application successfully solves the problem of large state space and difficult optimal strategy search, can provide support for penetration testing of large networks, and the dynamic model used is more close to the actual situation.

[0120] The application modifies the traditional attack graph, adds a probability element, and deletes some unnecessary information and possible loops. Now the mainstream method mainly uses the traditional attack graph. Compared with the traditional method, the attack graph generation speed of the application is faster, the larger scale network structure can be processed, and the visualization effect and expansibility of the attack graph are better.

[0121] The application can intelligently generate a penetration testing scheme according to the vulnerability of the system, realize automated penetration testing, and greatly improve the precision and efficiency of network security evaluation.

[0122] The above is only a preferred embodiment of the application, and does not limit the application in any form. Any skilled person in the art can make any simple modification, equivalent replacement and improvement to the above embodiment according to the technical essence of the application without departing from the scope of the technical solution of the application.

Claims

1. A method for automated penetration testing based on deep reinforcement learning, characterized in that, The method comprises the following steps: determining the network, operating system, and application information range of the penetration test, using a vulnerability scanning and baseline checking tool to perform vulnerability scanning and baseline checking on all devices within the determined range, and obtaining vulnerability information of the tested system; generating an attack graph according to the obtained vulnerability information, determining a vulnerability exploitation probability according to the vulnerability information, and assigning values to edges of the attack graph to obtain a Bayesian attack graph; designing a reward value of a target node of an attacker on the generated Bayesian attack graph, simulating possible attack behaviors of the attacker, and using a deep reinforcement learning method to calculate an attack strategy of the attacker; simulating potential attack behaviors of the attacker in the attack graph according to the attack strategy to generate a penetration attack path most likely to be adopted by the attacker; performing automated penetration testing on the test object according to the penetration attack path most likely to be adopted by the attacker in the penetration test test environment to generate a penetration test report; An attack network model containing formalized network security elements is established, the network security elements include host information, network topology and network vulnerability information; the host information is represented by a triple <Host, Services, Vuls>, wherein Host is the unique identifier of the host; Services is the software and service information running on the host; and Vul refers to the vulnerability information of the host; the network topology is represented by an undirected graph G<N, E>, wherein N represents the set of all physical hosts in the network; E represents the physical connection between hosts, and each edge is composed of a quadruple <Start, Destination, Protocol, Port>; Start, Destination, Protocol and Port represent the source host, destination host, protocol and port number of the connection, respectively; the network vulnerability information is represented by a quintuple <ID, IP, Probability, Complex, Impact>, wherein ID is the unique corresponding number of the vulnerability in the vulnerability library; IP is the host address where the vulnerability exists; Probability is the probability of the vulnerability being exploited by an attacker, and the value range is (0, 1); Complex is the complexity of the vulnerability, and the value range is (0, 1); The obtained Bayesian attack graph is processed as follows: the vulnerability information in the attack graph is deleted, and only the state nodes and transition directions are retained; the security states are reduced according to the monotonicity principle, so that the attacker will not acquire a security state again after having acquired it, to avoid repeated attack paths; the attack is formalized through directed hyperedges, and the security states linked by the attack actions are focused on. Impact is the severity of the damage caused after the vulnerability is exploited, and the formula provided by CVSS is used to calculate the score, the score range is(0, 10), the higher the score, the greater the vulnerability damage; According to the formal network security elements, the corresponding Bayesian attack graph can be generated, which is represented by the four-tuple BAG(N)=<S, A, E, p a >, where S is the state node; A is the set of attack actions available to the attacker; E is the directed edge; p a is a non-zero probability value, indicating the probability of success of the attack action a; The penetration test process is converted into a Markov decision process, the attacker corresponds to an agent, the training of the attack strategy is abstracted as a problem of learning and improving the strategy of the agent in the interaction with the environment, and the attack strategy corresponds to the probability distribution of the next action selected by the agent in different states.

2. The deep-reinforcement learning based automated penetration testing method of claim 1, wherein: ​ The Markov decision MDP is composed of a four-tuple <S, A, P, R>, wherein S is a state space composed of attack graphs at different times in the Bayesian attack graph; A is an action space representing that a penetration tester activates a target node using a vulnerability in the BAG; P is a transition probability representing a probability of using different edges in the Bayesian attack graph; and R is a reward, which is obtained after the agent activates the target node, and in order to prevent the agent from repeating the same action infinitely, the agent will obtain a value as a negative reward as a cost of taking an action each time.

3. The deep-reinforcement learning based automated penetration testing method of claim 2, wherein: The attack strategy of the attacker is trained using a deep reinforcement learning method, an optimal action value function of the attacker is defined according to a time difference function, and the optimal action value function is used as an optimization target. wherein, represents the optimal action value function of the attacker; represents the expected value of the reward obtained in the next state after taking an attack action; represents t represents the reward obtained by the attacker at time t after taking an action; represents the discount rate of learning, the more distant the future, the greater the discount; represents the maximum Q value of the next round; represents t represents the state of the attack graph at time t; represents t represents the action taken by the attacker at time t; The Q-learning method is used to enable the attacker to continuously update the optimal action value function value in interaction with the environment, so as to approach a theoretical optimal value.

4. The deep-reinforcement learning based automated penetration testing method of claim 3, wherein: The attack strategy training employs a deep neural network Q s t , a t ; The deep neural network parameters of the attacker are initialized as random values, an initial attack graph state s0 is randomly selected from a training data set, and the algorithm is run from the initial attack graph state s0. ), wherein, The attacker selects an action according to the current attack strategy, and an exploration element is added in the selection process, so that the attacker as an agent can have a probability of sampling a new state, and the state transition of the attack graph is promoted by continuously selecting actions. are parameters of a deep neural network; the deep neural network Q( s t , a t ; After each state transition, the reward value obtained by the agent is calculated in real time, and the reward value includes a reward obtained after activating a target node and a cost paid by the agent each time an action is taken. ) is composed of an input layer, a hidden layer, a softmax layer and an output layer, the input layer inputs all the states observed by the agent at present, including all the node states and the successful attack actions, the hidden layer is composed of two fully connected layers, and the activation function is ReLU; the softamax layer is before the output layer, responsible for converting the result into a set of probability distribution; the output layer includes all the actions and a null node, the null node represents that the attacker does not take any action in this round, the purpose is to prevent the attacker from randomly performing actions when there is no suitable attack target.

5. The deep-reinforcement learning based automated penetration testing method of claim 4, wherein: The gradient of the loss function of the deep neural network is calculated according to the reward value after the state transition, the parameters of the deep neural network are updated using the gradient descent method, and the attack strategy of the attacker is updated based on this. The scanning module is configured to determine a network, an operating system, and application information range of the penetration test, perform vulnerability scanning and baseline checking on all devices in the determined range using a vulnerability scanning and baseline checking tool, and obtain vulnerability information of the system under test. The attack graph generation module is configured to generate an attack graph according to the obtained vulnerability information, determine a vulnerability exploitation probability according to the vulnerability information, and assign values to edges of the attack graph to obtain a Bayesian attack graph. The strategy training module is configured to design a target node reward value of the attacker on the generated Bayesian attack graph, simulate possible attack behaviors of the attacker, and calculate an attack strategy of the attacker using a deep reinforcement learning method. The simulated attack module is configured to simulate potential attack behaviors of the attacker in the attack graph according to the attack strategy, and generate a penetration attack path most likely to be used by the attacker.

6. An automated penetration testing system based on deep reinforcement learning for the method of claim 1, characterized in that, The penetration test module is configured to perform automated penetration testing on a test object according to the penetration attack path most likely to be used by the attacker in a penetration test test environment, and generate a penetration test report. The memory, the processor, and the computer program stored in the memory and executable on the processor are included, and the processor implements the deep reinforcement learning-based automated penetration test method according to any one of claims 1-5 when executing the computer program. ​ ​ ​ ​ 7. An electronic device, comprising: ​ ​ 8. A computer readable storage medium storing a computer program, the computer program causing a computer to perform the deep reinforcement learning based automated penetration testing method of any one of claims 1-5.

Images