Automatic penetration testing method, system and equipment based on deep reinforcement learning and medium

Abstract

The invention discloses an automatic penetration test method, system and device based on deep reinforcement learning, and a medium, and the method comprises the steps: carrying out the vulnerability scanning and baseline verification of a device through employing a vulnerability scanning and baseline verification tool, and obtaining the vulnerability information of a tested system; generating an attack graph according to the obtained vulnerability information, determining a vulnerability utilization probability according to the vulnerability information, and assigning values to edges of the attack graph to obtain a Bayesian attack graph; designing an attacker target node reward value on the generated Bayesian attack graph, simulating a possible attack behavior of an attacker, and calculating an attack strategy of the attacker by using a deep reinforcement learning method; simulating potential attack behaviors of an attacker in the attack graph according to an attack strategy, and generating a penetration attack path which is most likely to be adopted by the attacker; and performing an automatic penetration test to generate a penetration test report. According to the invention, the penetration test scheme can be intelligently generated according to the vulnerability of the system, and the precision and efficiency of network security evaluation are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an automated penetration testing method, system, equipment and medium based on deep reinforcement learning. Background Art

[0002] The continuous and rapid development of computer technology and Internet technology has provided people with many conveniences, but also caused many network security problems. On the one hand, network applications and operating systems are becoming more and more complex, and vulnerabilities generated during design or configuration are endless. On the other hand, the increase in the degree of interconnection and transmission speed provides network attacks with more abundant attack entrances and exploitation conditions, resulting in a significant increase in the number of network attacks. In addition, network attack behaviors are no longer isolated events, but present multi-stage characteristics. Attackers often lurk into the target network through covert means, use the causal relationship between business processes and vulnerabilities in the network, use different devices as springboards, and ultimately achieve goals such as information theft and device destruction. Faced with such complex, covert, more professional and organized network attacks, as well as continuously updated advanced threats, security personnel often use penetration testing methods to improve network security capabilities. Penetration testing is an active security assessment method that simulates the attack behavior of malicious hackers to evaluate the security of computer network systems. Penetration testing conducts in-depth analysis of system weaknesses, technical defects or vulnerabilities. The purpose is to discover and exploit these vulnerabilities from the perspective of a potential attacker to verify the security of the system and deal with and avoid potential security vulnerabilities and risks.

[0003] Penetration testing requires a lot of expertise and is difficult to formalize. Therefore, existing penetration testing is usually performed manually by professional testers according to certain penetration testing strategies. Specific test steps include: 1. Determine the target and scope of the penetration test: determine the IP address range, domain name, application, system and service of the target system with the test organizer; 2. Information collection and threat modeling: use public information sources and scanning tools to collect target information, analyze the collected information, determine the attacker's interest points, create an attacker model, and find attack paths; 3. Vulnerability analysis: use test tools to identify vulnerabilities in systems, networks and applications, verify vulnerabilities and evaluate the exploitability of vulnerabilities; 4. Penetration attack: find or develop relevant exploit tools based on the results of vulnerability analysis, use vulnerabilities to obtain system access rights (such as obtaining shells, elevating permissions, etc.), break through the system boundaries and move laterally within the system, find more vulnerabilities and sensitive data, and try to obtain sensitive data, leaving backdoors in the system; 5. Write a test report: record all findings in the test in detail, provide repair suggestions and improvement measures based on the test results, and write a formal test report.

[0004] The above process constitutes a complete penetration test. Penetration testing is a laborious and time-consuming task that is highly dependent on the skills of the operator. The accuracy and reliability of the test cannot be guaranteed. During the operation, all policy configuration, execution, monitoring and maintenance need to be done manually. The efficiency of penetration testing is low and cannot meet the requirements of attack effectiveness. In addition, the scale of networks is growing and the network topology is becoming more and more complex. In this case, enumerating all penetration paths and testing them one by one requires a lot of resources and time, which is difficult to complete only by manpower or some simple automated test scripts.

[0005] Based on the above defects, the industry is currently in urgent need of a method that can intelligently generate penetration testing strategies to quickly and accurately perform automated penetration testing on information systems. The present invention proposes an automated penetration testing method and system based on deep reinforcement learning. Summary of the invention

[0006] The purpose of the present invention is to provide an automated penetration testing method, system, equipment and medium based on deep reinforcement learning, which can intelligently generate penetration testing plans according to the vulnerability of the system, realize automated penetration testing, and greatly improve the accuracy and efficiency of network security assessment.

[0007] To achieve the above purpose, the technical solution provided by the present invention is:

[0008] The first aspect of the present application provides an automated penetration testing method based on deep reinforcement learning, comprising the following steps:

[0009] Determine the scope of the network, operating system, and application information for the penetration test, and use vulnerability scanning and baseline verification tools to perform vulnerability scanning and baseline verification on all devices within the determined scope to obtain vulnerability information of the system under test;

[0010] Generate an attack graph based on the acquired vulnerability information, determine the probability of vulnerability exploitation based on the vulnerability information, assign values ​​to the edges of the attack graph, and obtain a Bayesian attack graph;

[0011] Design the attacker's target node reward value on the generated Bayesian attack graph, simulate the attacker's possible attack behavior, and use deep reinforcement learning methods to calculate the attacker's attack strategy;

[0012] Simulate the attacker's potential attack behavior in the attack graph according to the attack strategy, and generate the penetration attack path that the attacker is most likely to adopt;

[0013] In the penetration test environment, the test object is automatically penetrated according to the penetration attack path that the attacker is most likely to use, and a penetration test report is generated.

[0014] To optimize the above technical solutions, the specific measures taken also include:

[0015] Establish an attack network model containing formalized network security elements, wherein the network security elements include host information, network topology and network vulnerability information; the host information is represented by a triple<Host,Services,Vuls> Indicates, where Host is the unique identifier of the host; Services is the software and service information running on the host; Vul refers to the vulnerability information of the host; the network topology is represented by an undirected graph G<N,E> Indicates, where N represents the set of all physical hosts in the network; E represents the physical connection between hosts, and each edge consists of a four-tuple<Start,Destination,Protocol,Port> Start, Destination, Protocol, and Port represent the source host, destination host, protocol, and port number of the connection respectively; the network vulnerability information is represented by a five-tuple<ID,IP,Probability,Complex,Impact> It is represented by, where ID is the unique number of the vulnerability in the vulnerability library; IP is the host address where the vulnerability exists; Probability is the probability of the vulnerability being exploited by an attacker, and its value range is (0, 1); Complex is the complexity of the vulnerability, and its value range is (0, 1); Impact is the severity of the harm caused by the vulnerability after being exploited, and is calculated using the vulnerability score calculation formula provided by CVSS, with a score range of (0, 10). The higher the score, the greater the harm of the vulnerability. According to the formalized network security elements, the corresponding Bayesian attack graph can be generated, using the four-tuple BAG(N)= <S,A,E,p a > indicates that S is a state node; A is a set of attack actions available to the attacker; E is a directed edge; p a is a non-zero probability value, indicating the probability of success of attack action a.

[0016] The obtained Bayesian attack graph is processed as follows: the vulnerability information in the attack graph is deleted, and only the state nodes and transfer directions are retained; the security state is deleted according to the monotonicity principle, so that the attacker will not obtain the security state again after obtaining the security state, so as to avoid repeated attack paths; the attack is formalized through directed "hyperedges", focusing on the security state linked to the attack action.

[0017] The penetration test process is transformed into a Markov decision process, the attacker corresponds to the intelligent agent, and the training of the attack strategy is abstracted as the problem of the intelligent agent learning to improve its own strategy in the interaction with the environment. The attack strategy corresponds to the probability distribution of the intelligent agent's choice of the next action in different states.

[0018] The Markov decision MDP consists of four tuples<S,A,P,R> In this paper, S is the state space, which is composed of the attack graphs at different times in the Bayesian attack graph; A is the action space, which represents the penetration tester using the vulnerability in the BAG to activate the target node; P is the transition probability, which represents the probability of using different edges in the Bayesian attack graph; R is the reward, which is obtained after the agent activates the target node, and in order to prevent the agent from repeating the same action infinitely, each time the agent acts, it will receive a negative reward as the cost of taking the action.

[0019] Furthermore, the attacker's attack strategy is trained using a deep reinforcement learning method, and the attacker's optimal action value function is defined according to the temporal difference function as the optimization goal:

[0020]

[0021] Among them, Q ＊ (s t , a t ) represents the optimal action value function of the attacker; Represents the expected value of the reward in the next state after taking the attack action; r t represents the reward obtained by the attacker after taking action at time t; γ represents the discount rate of learning, and the discount is larger for the farther future; Indicates the maximum Q value of the next round; s t represents the state of the attack graph at time t; a t represents the action taken by the attacker at time t;

[0022] The Q-learning method is used to allow the attacker to continuously update the optimal action-value function value in the interaction with the environment to approach the theoretical optimal value.

[0023] Furthermore, the attack strategy training uses a deep neural network Q(s t , a t ; θ), where θ is a parameter of the deep neural network; the deep neural network Q(s t , a t ; θ) consists of an input layer, a hidden layer, a softmax layer, and an output layer. The input layer inputs all the states currently observed by the agent, including all node states and successful attack actions. The hidden layer consists of two fully connected layers with ReLU as the activation function. The softmax layer is before the output layer and is responsible for converting the results into a set of probability distributions. The output layer includes all actions and a null node. The null node indicates that the attacker does not take any action in this round. The purpose is to prevent the attacker from performing actions at will when there is no suitable attack target.

[0024] Furthermore, the process of using the deep reinforcement learning method to calculate the attacker's attack strategy is as follows:

[0025] Initialize the attacker’s deep neural network parameters to random values ​​and randomly select the initial attack graph state s from the training dataset 0 , from the initial attack graph state s 0 Start running the algorithm;

[0026] The attacker selects actions based on the current attack strategy and adds exploration elements to the selection process, so that the attacker as an intelligent agent can sample new states with probability and promote the state transition of the attack graph by continuously selecting actions;

[0027] After each state transition, the agent's reward value is calculated in real time. The reward value includes the reward obtained after activating the target node and the cost paid by the agent for each action.

[0028] The gradient of the loss function of the deep neural network is calculated according to the reward value after the state transfer, and the parameters of the deep neural network are updated using the gradient descent method, and the attacker's attack strategy is updated based on this.

[0029] The second aspect of the present application provides an automated penetration testing system based on deep reinforcement learning, comprising:

[0030] Scanning module, used to determine the scope of network, operating system, and application information for penetration testing, and use vulnerability scanning and baseline verification tools to perform vulnerability scanning and baseline verification on all devices within the determined scope to obtain vulnerability information of the system under test;

[0031] An attack graph generation module is used to generate an attack graph based on the acquired vulnerability information, determine the probability of vulnerability exploitation based on the vulnerability information, assign values ​​to the edges of the attack graph, and obtain a Bayesian attack graph;

[0032] The strategy training module is used to design the attacker's target node reward value on the generated Bayesian attack graph, simulate the attacker's possible attack behavior, and calculate the attacker's attack strategy using deep reinforcement learning methods;

[0033] The attack simulation module is used to simulate the attacker's potential attack behavior in the attack graph according to the attack strategy and generate the penetration attack path that the attacker is most likely to adopt;

[0034] The penetration test module is used to perform automated penetration testing on the test object in the penetration test environment according to the penetration attack path most likely to be adopted by the attacker, and generate a penetration test report.

[0035] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the automated penetration testing method based on deep reinforcement learning as described in the first aspect of the present application.

[0036] The fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the automated penetration testing method based on deep reinforcement learning as described in the first aspect of the present application.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] In the scenario of penetration testing of a target object, the present invention first performs vulnerability scanning and baseline verification on the target object to obtain network asset information and vulnerability information of the target object. The probability of successful exploitation of the vulnerability is queried, combined with the asset information and vulnerability information, and uniformly input into the attack graph generation tool to generate a Bayesian attack graph of the target object; a deep reinforcement learning algorithm is run on the Bayesian attack graph to train the attacker's optimal attack strategy, generate an attacker strategy neural network, and output the attacker's attack actions under different situations; in the initial state, each attack action of the attacker is simulated according to the trained attacker strategy network to generate an optimal attack path that the attacker is most likely to adopt; this attack path can be used to perform automated penetration testing on the real network environment.

[0039] Compared with the existing scheme of manual penetration testing, the above method of the present invention generates a Bayesian attack graph in advance according to the vulnerability of the target object to be tested, uses a deep reinforcement learning model to train the penetration method, and obtains an automated attack strategy network that can automatically adjust the attack behavior according to different environments. After obtaining the real network environment, the real network information can be input into the automated penetration testing model of the present invention to obtain the attack graph and attack action, generate the attack path, and directly use the real attack action and the attack program in the attack arsenal to penetrate the real network, and quickly and accurately perform automated penetration testing on the network. The present invention improves the limitations of the traditional scheme penetration test that is highly dependent on the professional level of the tester and the test consumes too much time, improves the accuracy and efficiency of the penetration test, and improves the test effect.

[0040] The deep reinforcement learning algorithm used in the present invention successfully solves the problem of large state space and difficult optimal strategy search, can provide support for penetration testing of larger networks, and the dynamic model used is closer to the actual situation. The present invention optimizes the traditional attack graph, the attack graph generation speed is faster, it can handle larger-scale network structures, and the visualization effect and scalability of the attack graph are better. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 :Schematic diagram of the process of the automated penetration testing method based on deep reinforcement learning of the present invention.

[0042] Figure 2 : Network topology diagram in an embodiment of the present invention.

[0043] Figure 3 : Bayesian attack graph in an embodiment of the present invention.

[0044] Figure 4 : State transition diagram of the Bayesian attack graph in an embodiment of the present invention.

[0045] Figure 5 : Flowchart of attack strategy training of the present invention.

[0046] Figure 6 : The attack strategy of the present invention trains a deep neural network graph.

[0047] Figure 7 : Penetration test results in an embodiment of the present invention.

[0048] Figure 8 : Schematic diagram of the structure of the automated penetration testing system based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0049] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0050] Some of the terms in the present invention are explained as follows:

[0051] AG (Attack Graph): A directed graph that shows the attack sequence and attack effects that an attacker may launch. It consists of vertices and directed edges. The attack graph can build a complete network security model, reflect the vulnerability of each node in the network, and depict all the ways for attackers to capture important nodes.

[0052] BAG (Bayesian Attack Graph): A network security analysis tool that combines Bayesian networks and attack graphs. It is a directed acyclic graph that can represent the probability of attacker behavior and attack progress in the network.

[0053] MDP (Markov decision process): A mathematical framework used to model the process of decision makers making decisions in an uncertain environment. Decision makers perceive the current state, influence the environment through actions and obtain rewards.

[0054] DRL (Deep Reinforcement Learning): is a machine learning method that combines deep learning (DL) and reinforcement learning (RL). It uses deep neural networks to process high-dimensional perceptual inputs and learn how to make decisions in complex environments.

[0055] The present invention proposes an automated penetration testing method based on deep reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0056] Determine the scope of the network, operating system, and application information for the penetration test, and use vulnerability scanning and baseline verification tools to perform vulnerability scanning and baseline verification on all devices within the determined scope to obtain vulnerability information of the system under test;

[0057] Generate an attack graph based on the acquired vulnerability information, determine the probability of vulnerability exploitation based on the vulnerability information, assign values ​​to the edges of the attack graph, and obtain a Bayesian attack graph;

[0058] Design the attacker's target node reward value on the generated Bayesian attack graph, simulate the attacker's possible attack behavior, and use deep reinforcement learning methods to calculate the attacker's attack strategy;

[0059] Simulate the attacker's potential attack behavior in the attack graph according to the attack strategy, and generate the penetration attack path that the attacker is most likely to adopt;

[0060] In the penetration test environment, the test object is automatically penetrated according to the penetration attack path that the attacker is most likely to use, and a penetration test report is generated.

[0061] The following specific embodiments further explain the present invention in detail from five aspects: network security element modeling, construction of Bayesian attack graph, training attack strategy using deep reinforcement learning method, generation of attack path and implementation of penetration test:

[0062] (1) Modeling of network security elements

[0063] Large-scale network systems have a dynamic and changeable interconnection structure, including a large number of hosts, servers, routers, firewalls and other network devices. Different devices run complex and diverse operating systems and application software. From the perspective of penetration test attackers, to establish a model of the attack network, these network elements must first be formalized. Generally speaking, the network elements that the network model needs to display include host information, network topology, and network vulnerability information. These elements are defined one by one below:

[0064] Host information: Host information includes the host address, the services and ports opened by the host, and the vulnerability information of the host. It reflects the properties of different types of hosts in the network system and can be expressed as a triplet:<Host,Services,Vuls> to represent the information of a host, where Host is the unique identifier of the host, which can be a host name, IP address or Mac address; Services is the software and service information running on the host, including the operating system, application software, etc.; Vul refers to the vulnerability information of the host; for example, the host with the information triplet of <127.0.0.1, {Windows7, SQLServer}, CVE-2012-1123> has an IP address of 127.0.0.1, uses the Windows7 operating system and runs the SQL Server database service, and has a vulnerability with the CVE number CVE-2012-1123.

[0065] Vulnerability information: All information about a vulnerability v can be expressed as a five-tuple<ID,IP,Probability,Complex,Impact> To represent, ID is the unique number of the vulnerability in the CVE vulnerability database; IP is the host address where the vulnerability exists;

[0066] Probability is the probability of a vulnerability being exploited by an attacker, and its value range is (0, 1); Complexity is the complexity of the vulnerability, and its value range is (0, 1); Impact is the severity of the damage caused by the vulnerability after it is exploited, which can be calculated using the vulnerability score calculation formula provided by CVSS, and the score range is (0, 10). The higher the score, the greater the vulnerability damage. For example, if there is an "Eternal Blue" vulnerability v in the system, its sextuple is<CVE-2017-0144,127.0.0.1,0.6,0.5,9.3> , indicating that the "EternalBlue" vulnerability exists on the host with IP address 127.0.0.1. The probability of the vulnerability being exploited is 0.6, the complexity is 0.5, and the severity score after being exploited is 9.3.

[0067] Topology information: refers to the connection structure between all hosts in the network. Physically, all devices form an undirected graph G.<N,E> , where N represents the set of all physical hosts in the network; E represents the physical connection between hosts. Each edge consists of a four-tuple<Start,Destination,Protocol,Port> It is composed of the source host, destination host, protocol and port number of the connection.

[0068] Penetration testers can use vulnerability scanning equipment and baseline verification equipment to obtain the above information system data. In this embodiment, the network topology used is as follows: Figure 2 As shown, there are four areas including two subnets and a DMZ area, and 13 hardware devices including a Web server and a database server. The vulnerability information scanned by the missed scanning devices in the embodiment is shown in the following table:

[0069] Table 1

[0070]

[0071]

[0072] (2) Constructing a Bayesian attack graph

[0073] According to the formalized network security elements, the corresponding Bayesian Attack Graph (BAG) can be generated. First, BAG is defined.

[0074] BAG: A directed acyclic Bayesian attack graph consists of a four-tuple: BAG(N) = <S,A,E,p a >, where S is a set of state nodes, including system vulnerabilities, unsafe systems and network properties, etc.; A is a set of attack actions that can be used by attackers; E is a set of directed edges; p a is a non-zero probability value indicating the probability of success of attack action a.

[0075] S={s 1 ,s 2 ,…,s N} is the N security states owned by BAG, each state s i All come from a vulnerability of the system, including a state value, true (s i =1) or false(s i =0), respectively representing the safe state s i has been (not) exploited by attackers; in addition to the vulnerability, s i It can also be insecure system and network properties, such as using weak passwords for login passwords.

[0076] A={a 1 , a 2 , …, aK} is a set of atomic attacks, which represent the attacker's behavior of exploiting vulnerabilities in the network; atomic attacks are causal relationships between security states, linking two different security states; each attack node a has a predecessor node set pre(a) and a post-node set post(a), and attack action a can only be initiated when the state of the predecessor node is true (pre(a) = 1); if a is executed successfully, the state of post(a) will change from false to true, and the attack graph will move to the next state.

[0077] E is a set of directed edges; for any node e = (u, v) ∈ E in E, if u is a safe state node, then v must be an attack action node, and vice versa. Cycles are not allowed in BAG and need to be manually eliminated if they occur.

[0078] p a is the probability of success of atomic attack a; the probability of success of the penetration attacker's attack depends on the basic attributes of the vulnerability such as the complexity of the vulnerability on the one hand, and on the configuration of defense measures in the network on the other hand; the probability of success of exploiting the same vulnerability in different network environments may also be different.

[0079] by Figure 2 Taking the network topology as an example, the following problems will be encountered when generating a standard Bayesian attack graph: 1. Generating a Bayesian attack graph requires enumerating all possible system security states. In this embodiment, the number of states will reach dozens, and the cost will increase exponentially with the growth of the system scale, and the scalability is poor; 2. The Bayesian attack graph contains a lot of information that penetration testers do not need. Most penetration testers may not need to know the details of the vulnerability, but only need to know the consequences of the vulnerability being exploited and the direction of state transfer; 3. The visualization effect will be greatly reduced.

[0080] Therefore, the present invention makes the following simplified modifications to the standard Bayesian attack graph, which can reduce unnecessary system states in the attack graph on the one hand, and reduce the information required to describe the attack on the other hand, thereby enhancing the visualization effect:

[0081] All vulnerability information is deleted, and only state nodes and transfer directions are retained in the attack graph;

[0082] The security state is pruned according to the monotonicity principle, that is, once an attacker obtains a certain security state, he will not obtain it again to avoid repeated attack paths;

[0083] We use directed “hyperedges” to formalize the concept of attack, no longer analyzing the means of attack, but only focusing on the security status linked to the attack actions.

[0084] "Hyperedge" refers to an edge that is abstracted to a higher level, contains more information and can link multiple nodes.

[0085] The BAG modified by the above steps is more concise and intuitive. Figure 2 The network topology and vulnerability information in Table 1 are generated, and the generated results are as follows Figure 3 Bayesian attack Figure 1 There are 14 state nodes and 20 attack action nodes. The state nodes include 3 initial nodes where the attacker enters the system and 3 final target nodes. The edges in the graph connect the action nodes and the state nodes, and each edge has a corresponding attack success probability.

[0086] (3) Using deep reinforcement learning methods to train attack strategies

[0087] First, we need to transform the penetration test process into a Markov decision process (MDP), and abstract the training of attack strategies into a problem in which an agent learns to improve its own strategy in the interaction with the environment. The strategy here is the probability distribution of the agent's choice of the next action in different states. The choice of action by the agent in a certain state depends entirely on its strategy. MDP consists of a four-tuple<S,A,P,R> In this paper, S is the state space, which is composed of the attack graphs at different times in the BAG; A is the action space, which represents the penetration tester's continuous use of the vulnerabilities in the BAG to activate new state nodes; P is the transition probability, which uses the probability of different edges in the BAG; R is the reward, which is formulated by the tester. Different reward values ​​may train different attack strategies; in addition, in order to prevent the agent from repeating the same action infinitely, each time the agent acts, it will receive a negative reward as the cost of taking the action. Figure 4 The process of BAG state transition when the attacker takes action is described.

[0088] The attacker’s action strategy is trained using deep reinforcement learning. First, the attacker’s optimal action value function Q is defined based on the temporal difference function, which is used as the optimization goal:

[0089]

[0090] in Represents the expected value of the reward in the next state after taking the attack action; r t represents the reward obtained by the attacker after taking action at time t; γ represents the discount rate of learning, and the discount is larger for the farther future; Indicates the maximum Q value of the next round; s t represents the state of the attack graph at time t; a trepresents the action taken by the attacker at time t; the Q-learning method is used to allow the attacker Agent to continuously update the Q value in the interaction with the environment to approach the theoretical optimal value. The attacker's strategy is to choose the action with the highest Q value. t , a t ; θ) to approximate the theoretical Q value, θ is the parameter of the deep neural network. Define the TD target y based on the time difference function t and TD error δ t :

[0091]

[0092] δ t =Q(s t , a t ;θ)-y t

[0093] Where TD target y t It is the Q function's estimate of the best Q value at time t at time t+1. It is based on the actual observation results of the next round and is more reliable than the Q function. Therefore, the difference between the TD error and the Q function can be used as the loss function L:

[0094]

[0095] The gradient descent method can be used to find the partial derivative of L and calculate the gradient of the loss function L with respect to θ:

[0096]

[0097] Further gradient descent can make Q(s t , a t ; The value of θ) is closer to the TD target y t , where α is the learning rate, which is a manually adjusted hyperparameter:

[0098]

[0099] A complete round of algorithm training process is as follows Figure 5 As shown, it can be roughly divided into four steps:

[0100] (1) Environment initialization: Initialize the neural network parameters of the attacker Agent to random values ​​and randomly select the initial attack graph state s from the training dataset. 0 , from s 0 Start running the algorithm;

[0101] (2) Action selection: The attacker selects actions based on the current strategy. A certain exploration element is added to the selection process, allowing the agent to sample new states with probability. The state transition of the attack graph is promoted by continuously selecting actions;

[0102] (3) Calculate rewards: After each state transition, the agent’s reward value must be calculated in real time. The reward value includes the reward obtained after activating the target node and the cost paid by the agent for each action;

[0103] (4) Parameter update: Calculate the gradient of the loss function based on the reward value after state transfer, use the gradient descent method to update the parameters of the neural network, and use this as a basis to update the attacker's strategy.

[0104] The attacker's trained neural network Q(s t , a t ; θ) as Figure 6 As shown in the figure. The neural network consists of an input layer, a hidden layer, a softmax layer, and an output layer. The input layer inputs all the states currently observed by the Agent, including all node states and successful attack actions; the hidden layer consists of two fully connected layers (FC), each of which contains 64 neurons and the activation function is ReLU; the softmax layer is before the output layer and is responsible for converting the results into a set of probability distributions; the output layer includes all actions and a null node; the null node indicates that the attacker does not take any action in this round, in order to prevent the attacker from performing actions at will when there is no suitable attack target.

[0105] The present invention designs an experiment to verify the method, and the experimental environment refers to Figure 2 . In the experiment, the payoff after the attacker activates the target security state node is set to +100, and the cost for the attacker to perform the attack action is uniformly set to -5. In order to prevent unexpected situations in the initial stage of training, the total number of rounds of the game is set to 30. After 30 rounds, the game will be forced to end and the current reward will be calculated. The algorithms compared with the algorithm of the present invention are all algorithms widely used in the field of automated penetration testing, including genetic algorithm NSGA-II, game theory algorithm PF, algorithm Nash Q that combines game theory with artificial intelligence, and random strategy method. The evaluation criterion of the experiment is the total reward value of the attacker, and the number of iterations is set to 2000. The experimental results are as follows Figure 7 As shown in the figure, DDQN is the deep reinforcement learning algorithm used in this experiment; the algorithm with random strategy has always performed poorly; with the increase in the number of iterations, the performance of DDQN, Nash-Q and NSGA-II gradually improves; after 2000 rounds, DDQN has the best performance.

[0106] (4) Generate attack path

[0107] After the attack strategy model is trained, a model is obtained that can give the probability distribution of the next attack action in any state. The attack graph in the initial state is input into the model, and the state is transferred step by step until the end (the target node is activated or the number of state transfers reaches the threshold); this step is run 100 times, and each attack path is recorded. Finally, the three attack paths with the highest frequency of use in the strategy model are selected as the alternative paths for penetration testing.

[0108] (5) Conduct penetration testing

[0109] Use the three attack paths selected in the previous step to conduct a penetration test on the system. Since there is a certain probability of success in exploiting system vulnerabilities, if one attack path encounters an obstacle, you can switch to the other two attack paths in a timely manner. Record the vulnerabilities, attack methods, and impact ranges found during the penetration test in detail, and make repair suggestions to form a formal penetration test report. Based on the results of the penetration test, provide targeted security reinforcement suggestions to help users strengthen defense measures.

[0110] This application also provides an automated penetration testing system based on deep reinforcement learning, including:

[0111] Scanning module, used to determine the scope of network, operating system, and application information for penetration testing, and use vulnerability scanning and baseline verification tools to perform vulnerability scanning and baseline verification on all devices within the determined scope to obtain vulnerability information of the system under test;

[0112] An attack graph generation module is used to generate an attack graph based on the acquired vulnerability information, determine the probability of vulnerability exploitation based on the vulnerability information, assign values ​​to the edges of the attack graph, and obtain a Bayesian attack graph;

[0113] The strategy training module is used to design the attacker's target node reward value on the generated Bayesian attack graph, simulate the attacker's possible attack behavior, and calculate the attacker's attack strategy using deep reinforcement learning methods;

[0114] The attack simulation module is used to simulate the attacker's potential attack behavior in the attack graph according to the attack strategy and generate the penetration attack path that the attacker is most likely to adopt;

[0115] The penetration test module is used to perform automated penetration testing on the test object in the penetration test environment according to the penetration attack path most likely to be adopted by the attacker, and generate a penetration test report.

[0116] The present application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned automated penetration testing method based on deep reinforcement learning is implemented.

[0117] The present application also provides a computer-readable storage medium storing a computer program, which enables a computer to execute the above-mentioned automated penetration testing method based on deep reinforcement learning.

[0118] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0119] In summary, based on the Bayesian attack graph, this method models the penetration test attack process as a Markov decision process, defines the attacker's behavior, the environment's reward and the state transfer function according to the four elements of the MDP, and trains the optimal attack strategy from the perspective of dynamic attack. Compared with the existing mainstream technology, the deep reinforcement learning algorithm used in the present invention successfully solves the problem of large state space and difficult optimal strategy search, and can provide support for penetration testing of larger networks. The dynamic model used is also closer to the actual situation.

[0120] The present invention modifies the traditional attack graph, adds probability elements, and deletes some unnecessary information and possible loops. The current mainstream method mainly uses the traditional attack graph. Compared with the traditional method, the present invention generates the attack graph faster, can handle larger-scale network structures, and has better visualization and scalability of the attack graph.

[0121] The present invention can intelligently generate a penetration test plan according to the vulnerability of the system, realize automated penetration testing, and greatly improve the accuracy and efficiency of network security assessment.

[0122] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. An automated penetration testing method based on deep reinforcement learning, characterized in that: The following steps are involved: Determine the scope of the network, operating system, and application information for the penetration test, and use vulnerability scanning and baseline verification tools to perform vulnerability scanning and baseline verification on all devices within the determined scope to obtain vulnerability information of the system under test; Generate an attack graph based on the acquired vulnerability information, determine the probability of vulnerability exploitation based on the vulnerability information, assign values ​​to the edges of the attack graph, and obtain a Bayesian attack graph; Design the attacker's target node reward value on the generated Bayesian attack graph, simulate the attacker's possible attack behavior, and use deep reinforcement learning methods to calculate the attacker's attack strategy; Simulate the attacker's potential attack behavior in the attack graph according to the attack strategy, and generate the penetration attack path that the attacker is most likely to adopt; In the penetration test environment, the test object is automatically penetrated according to the penetration attack path that the attacker is most likely to use, and a penetration test report is generated.

2. The automated penetration testing method based on deep reinforcement learning according to claim 1, characterized in that: Establish an attack network model containing formalized network security elements, wherein the network security elements include host information, network topology and network vulnerability information; the host information is represented by a triple<Host,Services,Vuls> Indicates, where Host is the unique identifier of the host; Services is the software and service information running on the host; Vul refers to the vulnerability information of the host; the network topology is represented by an undirected graph G<N,E> Indicates, where N represents the set of all physical hosts in the network; E represents the physical connection between hosts, and each edge consists of a four-tuple<Start,Destination,Protocol,Port> Start, Destination, Protocol, and Port represent the source host, destination host, protocol, and port number of the connection respectively; the network vulnerability information is represented by a five-tuple<ID,IP,Probability,Complex,Impact> It is represented by, where ID is the unique number of the vulnerability in the vulnerability database; IP is the host address where the vulnerability exists; Probability is the probability of the vulnerability being exploited by an attacker, and its value range is (0, 1); Complex is the complexity of the vulnerability, ranging from (0, 1); Impact is the severity of the damage caused by the vulnerability after it is exploited. It is calculated using the vulnerability score calculation formula provided by CVSS. The score range is (0, 10). The higher the score, the greater the harm of the vulnerability. According to the formalized network security elements, the corresponding Bayesian attack graph can be generated, using the four-tuple BAG(N) = <S,A,E,p a > indicates that S is a state node; A is a set of attack actions available to the attacker; E is a directed edge; p a is a non-zero probability value, indicating the probability of success of attack action a.

3. The automated penetration testing method based on deep reinforcement learning according to claim 1, characterized in that: The obtained Bayesian attack graph is processed as follows: the vulnerability information in the attack graph is deleted, and only the state nodes and transfer directions are retained; the security state is deleted according to the monotonicity principle, so that the attacker will not obtain the security state again after obtaining the security state, so as to avoid repeated attack paths; the attack is formalized through directed "hyperedges", focusing on the security state linked to the attack action.

4. The automated penetration testing method based on deep reinforcement learning according to claim 1, characterized in that: The penetration test process is transformed into a Markov decision process, the attacker corresponds to the intelligent agent, and the training of the attack strategy is abstracted as the problem of the intelligent agent learning to improve its own strategy in the interaction with the environment. The attack strategy corresponds to the probability distribution of the intelligent agent's choice of the next action in different states. The Markov decision MDP consists of four tuples<S,A,P,R> In this paper, S is the state space, which is composed of the attack graphs at different times in the Bayesian attack graph; A is the action space, which represents the penetration tester using the vulnerability in the BAG to activate the target node; P is the transition probability, which represents the probability of using different edges in the Bayesian attack graph; R is the reward, which is obtained after the agent activates the target node, and in order to prevent the agent from repeating the same action infinitely, each time the agent acts, it will receive a negative reward as the cost of taking the action.

5. The automated penetration testing method based on deep reinforcement learning according to claim 4 is characterized in that: The attacker's attack strategy is trained using deep reinforcement learning methods, and the attacker's optimal action value function is defined according to the temporal difference function as the optimization goal: Among them, Q ＊ (s t , a t ) represents the optimal action value function of the attacker; Represents the expected value of the reward in the next state after taking the attack action; r t represents the reward obtained by the attacker after taking action at time t; γ represents the discount rate of learning, and the discount is larger for the farther future; Indicates the maximum Q value of the next round; s t represents the state of the attack graph at time t; a t represents the action taken by the attacker at time t; The Q-learning method is used to allow the attacker to continuously update the optimal action-value function value in the interaction with the environment to approach the theoretical optimal value.

6. The automated penetration testing method based on deep reinforcement learning according to claim 5, characterized in that: Attack strategy training uses a deep neural network Q(s t , a t ; θ), where θ is a parameter of the deep neural network; the deep neural network Q(s t , a t ; θ) consists of an input layer, a hidden layer, a softmax layer, and an output layer. The input layer inputs all the states currently observed by the agent, including all node states and successful attack actions. The hidden layer consists of two fully connected layers with ReLU as the activation function. The softmax layer is before the output layer and is responsible for converting the results into a set of probability distributions. The output layer includes all actions and a null node. The null node indicates that the attacker does not take any action in this round. The purpose is to prevent the attacker from performing actions at will when there is no suitable attack target.

7. The automated penetration testing method based on deep reinforcement learning according to claim 6, characterized in that: The process of using the deep reinforcement learning method to calculate the attacker's attack strategy is as follows: Initialize the attacker's deep neural network parameters to random values, randomly select the initial attack graph state s0 from the training dataset, and run the algorithm from the initial attack graph state s0; The attacker selects actions based on the current attack strategy and adds exploration elements to the selection process, so that the attacker as an intelligent agent can sample new states with probability and promote the state transition of the attack graph by continuously selecting actions; After each state transition, the agent's reward value is calculated in real time. The reward value includes the reward obtained after activating the target node and the cost paid by the agent for each action. The gradient of the loss function of the deep neural network is calculated according to the reward value after the state transfer, and the parameters of the deep neural network are updated using the gradient descent method, and the attacker's attack strategy is updated based on this.

8. An automated penetration testing system based on deep reinforcement learning, characterized in that: include: Scanning module, used to determine the scope of network, operating system, and application information for penetration testing, and use vulnerability scanning and baseline verification tools to perform vulnerability scanning and baseline verification on all devices within the determined scope to obtain vulnerability information of the system under test; An attack graph generation module is used to generate an attack graph based on the acquired vulnerability information, determine the probability of vulnerability exploitation based on the vulnerability information, assign values ​​to the edges of the attack graph, and obtain a Bayesian attack graph; The strategy training module is used to design the attacker's target node reward value on the generated Bayesian attack graph, simulate the attacker's possible attack behavior, and calculate the attacker's attack strategy using deep reinforcement learning methods; The attack simulation module is used to simulate the attacker's potential attack behavior in the attack graph according to the attack strategy and generate the penetration attack path that the attacker is most likely to adopt; The penetration test module is used to perform automated penetration testing on the test object in the penetration test environment according to the penetration attack path most likely to be adopted by the attacker, and generate a penetration test report.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the automated penetration testing method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the automated penetration testing method based on deep reinforcement learning as described in any one of claims 1 to 7.

Images