A method and device for attack tree honeypot deployment defense based on deep reinforcement learning

Abstract

The present invention discloses a deep reinforcement learning-based attack tree honeypot deployment defense method and device. The method comprises the following steps: obtaining network topology information and sequentially constructing an attack tree; combining a convolutional neural network with a Q-Learning algorithm to create a DQN model; training an intelligent agent based on the DQN model, wherein the intelligent agent serves as a penetration attack party and the training goal is to generate a current optimal penetration attack path; according to the current optimal penetration attack path, all nodes traversed by the current optimal attack path are regarded as nodes that may be attacked, and vulnerable nodes are determined by monitoring the traffic changes of each potentially attacked node; calculating the difficulty of the attack vulnerability corresponding to each vulnerable node; obtaining a node with the highest attack probability based on the traffic changes of the vulnerable node and the difficulty of the attack vulnerability, and deploying a honeypot at the node; and continuously updating the deployment of the honeypot to lure attackers into the deployed honeypot, thereby completing the defense.

Description

Technical Field

[0001] The present invention belongs to the field of cyberspace security and deep reinforcement learning defense, and specifically relates to an attack tree honeypot deployment defense method and device based on deep reinforcement learning. Background Art

[0002] Automated penetration testing refers to transforming the manual penetration testing process into an automated process without human involvement, which reduces the manpower and material costs of manual penetration to a certain extent. Early automated penetration testing methods mainly took the form of attack graphs, which model the systems and how they are affected by specific vulnerabilities. In the attack graph, the nodes are usually the state of the system, where the state is determined by the current system configuration, i.e., the operating system, permissions, network connections, etc., and the edges connecting the nodes are known vulnerabilities. Figure 1 Once constructed, the attacker's action sequence (e.g., exploiting a vulnerability) can be searched. The nodes the attacker passes through from the starting point to the end point and the actions taken at each node are called the attack path. Once the attack path is obtained, the attacker will use it to approach the target system. Furthermore, finding the attack path can be combined with AI-based path planning techniques. However, the main drawback of this approach is that it requires prior knowledge of the complete network topology and the configuration information of each machine, making it unrealistic from the attacker's perspective. It also requires manual configuration of the diagram for each new system being evaluated.

[0003] Attack trees, derived from attack graphs, are a promising approach for automated penetration testing. These methods ensure that the actions taken during automated penetration testing closely mirror those of security experts, playing a significant role in the field of automated penetration testing. This method, first proposed by Schneier in 1999, models potential security threats to a given system and represents attacks against a target in the form of a tree structure. Analyzing attack trees can provide a better understanding of the relationships between attack methods. Similarly, applying reinforcement learning (RL) to attack tree analysis, such as using Q-learning to find attack paths, is also a viable approach. However, this approach still suffers from the limitations of a small action space and sample size. Compared to RL, deep reinforcement learning (DRL) is a more suitable approach for analyzing attack trees because it combines deep learning and reinforcement learning and employs a trial-and-error approach to find the optimal solution. This allows penetration testing of larger networks using attack trees.

[0004] Honeypot technology is a trap technology used in network defense. It attracts and deceives attackers and records their attack behavior, thereby studying and learning about their attack objectives and methods, and protecting real service resources. Honeynet technology, derived from honeypot technology, is a highly effective active defense mechanism. Honeynets, composed of multiple honeypots centrally deployed with decoy service modules, are a highly interactive research-oriented honeypot technology. Honeynet technology uses multiple pre-designed honeypot hosts to lure attackers into attacking, misleading them into thinking the target is a real machine. This confuses the attacker and simultaneously captures information about their attack behavior and situation, which can then be analyzed and evaluated. This mechanism is a highly effective active defense mechanism.

[0005] However, traditional honeypot technology suffers from shortcomings such as static configuration and fixed deployment, making it easily identified and bypassed by attackers, rendering it useless as a decoy. Therefore, improving the dynamics and deceptiveness of honeypots has become a key issue in the honeypot field. Similarly, traditional honeynets require physical machine deployment, which makes deployment complex, costly, and difficult to control traffic. One approach to addressing this issue is to apply deep reinforcement learning (DRL) technology to the construction of attack trees. Based on the vulnerability and traffic information of each node in the attack tree, the DRL algorithm is again used to dynamically deploy honeypots, thus intelligently implementing the honeypot deployment process.

[0006] Since its introduction, deep reinforcement learning has been a highly sought-after area of artificial intelligence. With its rapid development and application, reinforcement learning has been widely used in fields such as robotic control, game playing, computer vision, and autonomous driving. Reinforcement learning (RL) is an AI optimization technique whose key advantage lies in not requiring an environmental model to generate attack strategies, but rather learning optimal strategies through interaction with the environment. Deep reinforcement learning, on the other hand, leverages neural networks as parameter structures, combining the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning to optimize deep reinforcement learning strategies. Using deep reinforcement learning to deploy honeypots can dynamically update attack trees while optimizing honeypot placement, attracting attackers into the honeypots and thus preventing penetration attacks. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the present invention proposes an attack tree honeypot deployment defense method and device based on deep reinforcement learning.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows: A first aspect of an embodiment of the present invention provides an attack tree honeypot deployment defense method based on deep reinforcement learning, which specifically includes the following steps:

[0009] S1, obtains network topology information and builds attack trees in sequence;

[0010] S2, combining a convolutional neural network with a Q-Learning algorithm to create a DQN model, and training an agent based on the DQN model, wherein the agent acts as an infiltration attacker, and the training goal is to generate the current optimal infiltration attack path;

[0011] S3, based on the current optimal penetration attack path obtained in step S2, all nodes passed by the current optimal attack path are regarded as nodes that may be attacked, and the vulnerable nodes are determined by monitoring the traffic changes of each node that may be attacked; the difficulty of the attack vulnerability corresponding to each vulnerable node is calculated based on the access matrix, access complexity, and authenticity index; the node with the highest attack probability is obtained based on the traffic changes of the vulnerable node and the difficulty of the attack vulnerability, and a honeypot is deployed at the node; and the deployment of the honeypot is continuously updated to lure attackers into the deployed honeypot, thereby completing the defense.

[0012] Furthermore, obtaining the network topology information in step S1 includes: using the vulnerability scanning tool Shodan to perform a port scan on the target network system, thereby obtaining the IP address of each device, the operating system type of each device, the ports opened by each IP address, the list of services running on each port, the communication protocol used by each port, and the communication relationship between different subnets, including key information for constructing a real network topology.

[0013] Furthermore, the process of constructing the attack tree in step S1 is specifically as follows: based on the obtained network topology information, an attack tree is generated through MulVAL; wherein, the attack tree contains all information of the network topology nodes; all information of the network topology nodes includes the vulnerabilities of the nodes and the operations that can be executed according to the vulnerability exploitation relationship; the executable operations include penetration attacks, lateral movement, obtaining user permissions and further upgrading to root permissions.

[0014] Furthermore, the process of training the agent based on the DQN model in step S2 is specifically as follows:

[0015] S201, the parameters θ of the current network in the DQN model are updated in real time. After every N rounds of iteration, the parameters of the current network are copied to the target network, and then the network parameters are updated by minimizing the mean square error between the current Q value and the target Q value. The mean square error function is:

[0016] L(θ i )=E s,a,r,s' [(Y i -Q(s,a|θ i )) 2 ]

[0017] in, r represents the immediate reward of the DQN model;

[0018] Calculate the network gradient according to the following formula and update the current network parameter θ:

[0019]

[0020] In step S202, the vulnerability information, including the vulnerable service, vulnerability attributes, and vulnerability type, corresponding to each network topology node in the attack tree, is converted into a simplified matrix as the state input of the DQN model trained in step S2. According to the Bellman optimality equation theory, as long as the target Q value is infinitely close to the current Q value through continuous iteration and updating of step S201, the training is finally completed, and the training target is obtained, which is the current optimal penetration attack path:

[0021]

[0022] Furthermore, the calculation process of the instant reward r of the DQN model is as follows: combining the basic score of the vulnerability corresponding to each network topology node in the attack tree and the penetration feasibility score of the general vulnerability scoring system to reward different vulnerabilities that successfully penetrate different nodes. The instant reward r of the DQN model is the reward score Score vul Set to:

[0023]

[0024] Among them, baseScore represents the basic score, and exploitabilityScore represents the penetration feasibility score.

[0025] Furthermore, the step S3 specifically includes the following steps:

[0026] S301, based on the current optimal attack path obtained through training in step S2, all nodes traversed by the current optimal attack path are nodes that may be attacked, and the nodes that may be attacked are screened to obtain vulnerable nodes;

[0027] S302: Score the vulnerability of each node in the attack tree based on the vulnerability information corresponding to the node and the three indicators of the CVSS vulnerability scoring system, namely, access matrix AV, access complexity AC, and authenticity AU. The range of the three indicators is (0, 1), and the three indicators are recorded as V, C, and U respectively.

[0028] S303: Calculate the difficulty D of attacking the vulnerability based on the V, C, and U indicators obtained in step S302. The calculation formula is as follows:

[0029]

[0030] S304: Combine the traffic changes corresponding to the vulnerable node obtained in step S301 with the vulnerability analysis to obtain the node with the highest attack probability, and deploy a honeypot at the node;

[0031] S305, during the defense process of intelligently deploying honeypots, if the agent falls into the honeypot deployed in step S304, a positive reward r is given; if it does not fall into the honeypot, a negative reward -r is given;

[0032] S306, storing the state transition process in the experience replay experience buffer pool as a training data set for the DQN model; sampling N training data sets from the experience buffer pool, and updating and training the current Q network and target Q network in the DQN model;

[0033] S307, repeating the training process of the DQN model in steps S301 to S306, continuously updating the deployment of honeypots, and luring penetration attackers into the deployed honeypots, thereby achieving the purpose of defending against penetration attacks.

[0034] Furthermore, step S301 includes: a process of screening nodes that may be attacked: combining the generated attack tree with the software-defined network, monitoring the traffic changes of each node that may be attacked, calculating the traffic change matrix corresponding to the current moment and the next moment for each node that may be attacked, and performing dimensional comparison on the traffic change matrix corresponding to the current moment and the next moment. If the changes in all dimensional data exceed the custom threshold, the node that may be attacked can be considered a vulnerable node.

[0035] Furthermore, in step S305, if the agent falls into the honeypot pre-deployed in step S304, a positive reward r is given, and if it does not fall into the honeypot, a negative reward -r is given, including:

[0036] If the agent falls into a pre-set honeypot, the agent will be given the 2D reward set in step S303. If it does not fall into the honeypot but bypasses the honeypot to perform a penetration attack on the hosts around the honeypot, the agent will be given a negative reward of -2D.

[0037] A second aspect of an embodiment of the present invention provides an attack tree honeypot deployment defense device based on deep reinforcement learning, comprising one or more processors for the above-mentioned attack tree honeypot deployment defense method based on deep reinforcement learning.

[0038] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it is used for the above-mentioned attack tree honeypot deployment defense method based on deep reinforcement learning.

[0039] The beneficial effects of the present invention are as follows: the present invention combines attack tree generation technology and honeypot deployment technology, and while dynamically constructing the attack tree, deploys virtual honeypot hosts in a dynamic and static manner according to the vulnerability information and traffic information of the tree nodes, and uses deep reinforcement learning to optimize the selection of penetration paths and the deployment of honeypots. By introducing penetration attackers into honeypots deployed according to the vulnerability information of the attack tree nodes and the traffic changes of the nodes, the target system is prevented from being attacked, thereby improving the security of the network system, thereby achieving the purpose of defending against penetration attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the method of the present invention;

[0041] Figure 2 A schematic diagram of the DQN algorithm structure used in the method of the present invention;

[0042] Figure 3 Schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0044] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0046] The overall idea of the present invention is as follows: 1) constructing a real network topology based on the vulnerability information obtained by Shodon scanning, and then constructing an attack tree through MulVAL to optimize the penetration path; 2) constructing the penetration path optimization process as an MDP process and using the DQN algorithm for training. The state input is the vulnerability information of each node, the reward is set according to the basic score and expected score of different vulnerabilities in CVSS, and the action output is the attack path selected by the DQN model; 3) combining the SDN traffic controller with the attack tree, after the optimal attack path is generated, by detecting the changes in the traffic matrix on each node on the optimal attack path, inferring the vulnerable nodes, and then combining it with the vulnerability analysis of the corresponding nodes to determine the most vulnerable nodes and deploying honeypots at these nodes; 4) modeling the honeypot deployment process as an MDP process, the state input is the traffic load of each node, and the reward is set according to whether the node is trapped in the honeypot. If the node is trapped in the honeypot, a double CVSS score reward is given, otherwise a negative reward is given. The purpose is to make the penetration attacker fall into the deployed honeypot, thereby achieving the purpose of defending against penetration attacks. 5) For the first time, the attack tree and SDN traffic control are combined. The vulnerable nodes are inferred based on the penetration attack path obtained by attack tree training, and then the vulnerable nodes are inferred based on the real-time traffic matrix changes on each node. In this way, honeypots can be deployed on the attack tree in a dynamic and static manner.

[0047] The following is a detailed description of the attack tree honeypot deployment defense method and device based on deep reinforcement learning of the present invention in conjunction with the accompanying drawings. The features of the following embodiments and implementations can be combined with each other unless they conflict.

[0048] like Figures 1 and 2 As shown, the embodiment of the present invention proposes an attack tree honeypot deployment defense method based on deep reinforcement learning, which specifically includes the following steps:

[0049] 1) Obtain network topology information to build an attack tree:

[0050] 1.1) Obtaining network topology information: Using the vulnerability scanning tool Shodan, we perform port scanning on the target network system to obtain key information such as the IP address of each device, the operating system type of each device, the ports open by each IP address, the list of services running on each port, the communication protocol used by each port, and the communication relationships between different subnets, which are used to construct a realistic network topology.

[0051] 1.2) Constructing an attack tree: The acquired network topology information is used to generate an actual attack tree corresponding to the given network topology through the MulVAL tool. This tree is used to find all possible penetration paths for the given input network topology. The attack tree contains all information about the network topology nodes, such as the vulnerabilities existing in the node and the actions that can be performed based on the vulnerability exploitation relationship, such as penetration attacks, lateral movement, obtaining user privileges and further escalating to root privileges, and other related operations.

[0052] 2) Combining a convolutional neural network with a Q-Learning algorithm to create a DQN model, and training an intelligent agent based on the DQN model, where the intelligent agent acts as a penetration attacker, and the training goal is to generate the current optimal penetration attack path.

[0053] In an embodiment of the present invention, the DQN model specifically combines a convolutional neural network with a Q-Learning algorithm to create a DQN model. The DQN network is a representative value-based approach. The DQN model's input is the current state, which undergoes nonlinear transformations through three convolutional layers and two fully connected layers, ultimately generating a Q value for each action at the output layer. The DQN model utilizes a target network mechanism. Specifically, based on the current value network structure, a target value network with an identical structure is constructed to form the overall DQN model framework. During training, the predicted Q value output by the current value network is used to select action a, while another target value network is used to calculate the target Q value.

[0054] Specifically, compared to the Q-Learning algorithm, the DQN model alleviates the problem of unstable representation functions in nonlinear networks. For example, the DQN model uses an experience replay pool to store transfer samples. At each time step t, transfer samples obtained by the agent's interaction with the environment are stored in the experience replay buffer. During training, a small batch of transfer samples is randomly selected and the network parameters θ are updated using the stochastic gradient descent (SGD) algorithm.

[0055] Specifically, the DQN model also modifies the calculation method of Q value compared with the Q-Learning algorithm. Therefore, in the DQN model, Q(s,a|θ i ) represents the output of the current value network, which is used to evaluate the value function of the current state action. i - ) represents the output of the target value network, generally used as the target Q value.

[0056] Among them, s represents the current state, θ i represents the parameters of the current network in the DQN model, s' represents the next state after taking action a, θ i -represents the parameters of the target network in the DQN model, a' represents the possible actions in state s', r represents the immediate reward of the DQN model, and γ is the discount factor. A larger discount factor indicates a greater emphasis on long-term rewards.

[0057] The specific process of training an agent based on the DQN model is as follows:

[0058] 2.1) In the DQN model, the parameters θ of the current network are updated in real time. After every N iterations, the parameters of the current network are copied to the target network. The network parameters are then updated by minimizing the mean squared error between the current Q value and the target Q value. The mean squared error function is:

[0059] L(θ i )=E s,a,r,s' [(Y i -Q(s,a|θ i )) 2 ]

[0060] in, r represents the immediate reward of the DQN model.

[0061] The calculation process of the instant reward r of the DQN model is as follows: combining the basic score of the vulnerability corresponding to each network topology node in the attack tree and the penetration feasibility score of the CVSS (Common Vulnerability Scoring System) to reward different vulnerabilities that successfully penetrate different nodes. The instant reward r of the DQN model is the reward score Score vul Set to:

[0062]

[0063] Among them, baseScore represents the basic score, and exploitabilityScore represents the penetration feasibility score.

[0064] In this way, the impact of each Q-value change on the network parameters is reduced, that is, the correlation between the target Q-value and the current Q-value is reduced, and the stability of the policy training is increased. The network gradient is calculated as follows to update the current network parameter θ:

[0065]

[0066] 2.2) Convert the vulnerability information such as the vulnerable service, vulnerability attribute, and vulnerability type corresponding to each network topology node in the attack tree into a simplified matrix as the state input of the DQN model trained in step (2). According to the Bellman optimal equation theory, as long as the target Q value is infinitely close to the current Q value through continuous iteration and updating of step 2.1), the training is finally completed, and the training target is obtained, which is the current optimal penetration attack path:

[0067]

[0068] 3) According to the current optimal penetration attack path obtained in step S2, all nodes traversed by the current optimal attack path are regarded as nodes that may be attacked. By monitoring the traffic changes of each node that may be attacked, the vulnerable nodes are determined; based on the access matrix, access complexity, and authenticity index, the difficulty of the attack vulnerability corresponding to each vulnerable node is calculated; based on the traffic changes of the vulnerable node and the difficulty of the attack vulnerability, the node with the highest attack probability is obtained, and a honeypot is deployed at the node; and the deployment of the honeypot is continuously updated to lure the attacker into the deployed honeypot, thereby completing the defense.

[0069] 3.1) According to the current optimal attack path obtained by training in step 2), all nodes traversed by the current optimal attack path are nodes that may be attacked. The nodes that may be attacked are screened to obtain vulnerable nodes.

[0070] The process of screening nodes that may be attacked includes: combining the generated attack tree with the software defined network (SDN), obtaining the load flow information of each node that may be attacked, monitoring the flow change of each node that may be attacked, calculating the flow change matrix corresponding to the current moment and the next moment for each node that may be attacked, and performing dimensional comparison on the flow change matrix corresponding to the current moment and the next moment. If the changes in all dimensional data exceed a custom threshold (the custom threshold is selected as 50% in the embodiment of the present invention), the node that may be attacked can be considered a vulnerable node.

[0071] 3.2) Based on the vulnerability information corresponding to each node in the attack tree generated by MulVAL, the vulnerability is scored using the three indicators of the CVSS vulnerability scoring system: access vector (AV), access complexity (AC), and authentication (AU). The range of these three indicators is (0, 1). These three indicators are denoted as V, C, and U respectively.

[0072] 3.3) Calculate the difficulty D of attacking the vulnerability based on the V, C, and U indicators obtained in step 3.2). A larger D value indicates a greater attack difficulty. The calculation formula is as follows:

[0073]

[0074] 3.4) Combine the traffic changes corresponding to the vulnerable nodes obtained in step (3.1) with the vulnerability analysis to obtain the node with the highest attack probability and deploy a honeypot at the node.

[0075] 3.5) During the intelligent honeypot deployment process, the initial state s at the current moment is the traffic load matrix for each node at the current moment. Action a is the selection of an infiltration attack path. If the agent falls into the honeypot pre-deployed in step 3.4), it is given a positive reward r; if it does not fall into the honeypot, it is given a negative reward -r. The next state s' is the traffic load matrix at the next moment.

[0076] The specific setting of giving positive reward r is that if the agent falls into a pre-set honeypot, the agent is given the 2D reward set in step 3.3). If the agent does not fall into the honeypot but bypasses the honeypot to perform a penetration attack on the host around the honeypot, the agent is given a negative reward of -2D.

[0077] 3.6) Store the state transition process (state s, action a, reward r, next state s') in the experience replay experience buffer pool as the training dataset for the DQN model; sample N training datasets from the experience buffer pool and update the current Q network and target Q network in the DQN model;

[0078] 3.7) Repeat steps 3.1) to 3.6) to train the DQN model, continuously update the deployment of honeypots, and lure the attackers into the deployed honeypots, thereby achieving the purpose of defending against penetration attacks.

[0079] Corresponding to the aforementioned embodiment of the attack tree honeypot deployment defense method based on deep reinforcement learning, the present invention also provides an embodiment of the attack tree honeypot deployment defense device based on deep reinforcement learning.

[0080] See also Figure 3 An embodiment of the present invention provides an attack tree honeypot deployment defense device based on deep reinforcement learning, including one or more processors for implementing the attack tree honeypot deployment defense method based on deep reinforcement learning in the above embodiment.

[0081] The embodiment of the attack tree honeypot deployment defense device based on deep reinforcement learning of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown, this is a hardware structure diagram of any device with data processing capabilities where the attack tree honeypot defense device based on deep reinforcement learning is deployed. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0082] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0083] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0084] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the attack tree honeypot deployment defense method based on deep reinforcement learning in the above embodiment is implemented.

[0085] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0086] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A defense method for attack tree honeypot deployment based on deep reinforcement learning, characterized in that: The specific steps include: S1, obtain network topology information and construct attack trees in sequence; the attack trees are used to optimize the penetration path; S2, combining a convolutional neural network with a Q-Learning algorithm to create a DQN model, and training an agent based on the DQN model, wherein the agent acts as an infiltration attacker, and the training goal is to generate the current optimal infiltration attack path; The penetration path optimization process is constructed as an MDP process and trained using the DQN algorithm. The state input is the vulnerability information of each node. The reward is set according to the basic score and expected score of different vulnerabilities in CVSS. The action output is the attack path selected by the DQN model. S3 combines the SDN traffic controller with the attack tree. Based on the current optimal penetration attack path obtained in step S2, all nodes traversed by the current optimal attack path are regarded as nodes that may be attacked. By monitoring the traffic changes of each node that may be attacked, the vulnerable nodes are determined; the difficulty of the attack vulnerability corresponding to each vulnerable node is calculated based on the access matrix, access complexity, and authenticity index; based on the traffic changes of the vulnerable node and the difficulty of the attack vulnerability, the node with the highest attack probability is obtained, and a honeypot is deployed at the node; and the deployment of the honeypot is continuously updated to lure attackers into the deployed honeypot, thereby completing the defense.

2. The attack tree honeypot deployment defense method based on deep reinforcement learning according to claim 1 is characterized in that: Acquiring network topology information in step S1 includes: using the vulnerability scanning tool Shodan to perform port scanning on the target network system, thereby obtaining the IP address of each device, the operating system type of each device, the ports opened by each IP address, the list of services running on each port, the communication protocol used by each port, and the communication relationship between different subnets, which are key information for building a real network topology.

3. The attack tree honeypot deployment defense method based on deep reinforcement learning according to claim 1 is characterized in that: The process of constructing the attack tree in step S1 is specifically as follows: based on the obtained network topology information, an attack tree is generated through MulVAL; wherein, the attack tree contains all information of the network topology nodes; all information of the network topology nodes includes the vulnerabilities of the nodes and the operations that can be performed based on the vulnerability exploitation relationship; the executable operations include penetration attacks, lateral movement, obtaining user permissions and further upgrading to root permissions.

4. The attack tree honeypot deployment defense method based on deep reinforcement learning according to claim 1 is characterized in that: The process of training the agent based on the DQN model in step S2 is specifically as follows: S201, parameters of the current network in the DQN model It is updated in real time. After every N rounds of iteration, the parameters of the current network are copied to the target network, and then the current Values and Goals The mean square error between the values is used to update the network parameters. The mean square error function is: ； in, , r represents the immediate reward of the DQN model; Calculate the network gradient according to the following formula and update the parameters of the current network : ； S202, convert the vulnerability service, vulnerability attribute and vulnerability type corresponding to each network topology node of the attack tree into a simplified matrix as the state input of the DQN model trained in step S2; according to the Bellman optimal equation theory, as long as the target is updated through continuous iterations of step S201, Value and current The value is infinitely close, thus completing the training and obtaining the training target, that is, generating the current optimal penetration attack path: 。 5. The attack tree honeypot deployment defense method based on deep reinforcement learning according to claim 4 is characterized in that: The calculation process of the instant reward r of the DQN model is as follows: combining the basic score of the vulnerability corresponding to each network topology node in the attack tree and the penetration feasibility score of the general vulnerability scoring system to reward different vulnerabilities that successfully penetrate different nodes. The instant reward r of the DQN model is the reward score Set to: ； in, Indicates the basic score, Indicates the penetration feasibility score.

6. The attack tree honeypot deployment defense method based on deep reinforcement learning according to claim 1 is characterized in that: The step S3 specifically includes the following steps: S301, based on the current optimal attack path obtained through training in step S2, all nodes traversed by the current optimal attack path are nodes that may be attacked, and the nodes that may be attacked are screened to obtain vulnerable nodes; S302, based on the vulnerability information corresponding to each node in the attack tree and combined with the three indicators of the CVSS vulnerability scoring system, the vulnerability is scored, namely, access matrix AV, access complexity AC and authenticity AU. The range of the three indicators is And the above three indicators are recorded as V, C, and U respectively; S303, calculate the difficulty of attacking the vulnerability based on the V, C, and U indicators obtained in step S302 , the calculation formula is as follows: ； S304, combining the traffic changes corresponding to the vulnerable nodes obtained in step S301 with the vulnerability analysis to obtain the node with the highest attack probability, and deploying a honeypot at the node; S305, during the defense process of intelligent deployment of honeypots, if the agent falls into the honeypot deployed in step S304, a positive reward will be given. , if not trapped in the honeypot, a negative reward is given ; S306, store the state transition process in the experience replay experience buffer pool as the training data set of the DQN model; sample from the experience buffer pool training datasets, for the current DQN model Network and Target The network is updated and trained; S307, repeat the training process of the DQN model from steps S301 to S306, continuously update the deployment of honeypots, and lure penetration attackers into the deployed honeypots, thereby achieving the purpose of defending against penetration attacks.

7. The attack tree honeypot deployment defense method based on deep reinforcement learning according to claim 6 is characterized in that: The step S301 includes: a process of screening nodes that may be attacked: combining the generated attack tree with the software-defined network, monitoring the traffic changes of each node that may be attacked, calculating the traffic change matrix corresponding to the current moment and the next moment for each node that may be attacked, and performing dimensional comparison on the traffic change matrix corresponding to the current moment and the next moment. If the changes in all dimensional data exceed a custom threshold, the node that may be attacked can be considered a vulnerable node.

8. The attack tree honeypot deployment defense method based on deep reinforcement learning according to claim 6 is characterized in that: In the step S305, if the agent falls into the honeypot deployed in advance in step S304, a positive reward is given. , if not trapped in the honeypot, a negative reward is given include: If the agent falls into the pre-set honeypot, the agent is given the Reward: If the agent does not fall into the honeypot but bypasses the honeypot to conduct penetration attacks on the host around the honeypot, it will be given Negative rewards.

9. An attack tree honeypot deployment defense device based on deep reinforcement learning, characterized in that: The method comprises one or more processors for implementing the attack tree honeypot deployment defense method based on deep reinforcement learning according to any one of claims 1 to 8.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the attack tree honeypot deployment defense method based on deep reinforcement learning according to any one of claims 1 to 8.

Images