Strategic honeypot deployment defense method based on intelligent penetration testing

Abstract

A reward defense method based on intelligent penetration testing comprises the following steps: step 1, gradually penetrating each subnet of a target network to discover the location of its sensitive host and attack it; using a deep neural network as a function approximator to generate state-action values; performing penetration testing on the target sensitive host by performing vulnerability exploitation, scanning, permission extraction and other operations; step 2, deploying honeypots on the target network to protect the sensitive host, using a reinforcement learning QL algorithm to obtain the optimal number of honeypots while considering the benefit-cost of defenders and attackers, and verifying the defense effect based on the above method; step 3, setting firewalls with different reward values ​​in the target network to which the honeypot host is added, setting a firewall with a high reward value in the honeypot host subnet to lure the attacker into the honeypot host, and setting a firewall with a low reward value in the subnet containing the sensitive host to protect the sensitive host from being attacked, thereby achieving a better defense effect.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent agent reinforcement learning penetration testing and honeypot deployment defense, and specifically relates to a strategic honeypot deployment defense method based on intelligent penetration testing. Background Art

[0002] In recent years, with the development and application of internet technology, online attacks have posed numerous security risks. Penetration testing (PT or PenTesting) is a proactive, authorized simulated network attack designed to assess network security and uncover hidden vulnerabilities. Currently, penetration testing plays a vital role in strengthening computer systems' defenses against cyberattacks, as digital assets are more exposed than ever to persistent, diverse, and complex threats from hackers. Penetration testing can simulate attackers invading sensitive hosts, identifying system vulnerabilities and hidden risks, and conducting security assessments. However, the illegal use of penetration testing by some individuals poses a threat to network security, necessitating appropriate defensive measures against penetration attacks.

[0003] Network deception defense involves disrupting and misleading the attacker's cognitive decisions, forcing them to take actions that are detrimental to successful penetration. This helps the defender detect, delay, or interrupt the attack, thereby enhancing the target network's security. The defender's focus is on providing specific protection against the attacker's actions.

[0004] Deploying honeypots on a network plays a significant role in maintaining network security. Honeypots act as decoys, misleading attackers while protecting real assets and thus defending against penetration attacks. Setting reward values ​​is a key step in training reinforcement learning algorithms. Rewards provide a direct source of experience for agents to continuously improve themselves and achieve their goals autonomously. Agents use environmental rewards to judge the quality of their own behavior, thereby increasing the likelihood of choosing high-reward behaviors and aligning themselves with their target state. Changing reward values ​​can influence the penetration strategies learned by reinforcement learning agents, thereby defending against penetration attacks.

[0005] In summary, in order to prevent the malicious use of intelligent penetration testing based on reinforcement learning, exploring a strategic honeypot deployment method for intelligent penetration testing can reduce the success rate of intelligent penetration testing. By adopting firewalls with different reward values, it is of great significance to induce attackers to fall into the set honeypot more easily, thereby improving the defense effect. Summary of the Invention

[0006] The present invention aims to overcome the above-mentioned shortcomings of the prior art and, based on the principle of intelligent penetration testing of reinforcement learning, provides a strategic honeypot deployment defense method for intelligent penetration testing to achieve the purpose of maintaining network security.

[0007] Unlike other defense methods, this invention uses reinforcement learning to deploy honeypots to determine the optimal number and location of honeypots required for a target network to defend against penetration attacks. By setting up different types of firewalls and modifying the reward value to defend against penetration tests, luring attackers into the honeypots, this method achieves even better defenses.

[0008] The present invention provides a reward defense method based on intelligent penetration testing, comprising the following steps:

[0009] Step 1: Penetration attack. Penetration testing involves progressively penetrating each subnet of the target network to locate and attack sensitive hosts. Finally, the results are plotted as a report and provided to the tester, who can use it to strengthen system defenses and improve network security. Based on the QL algorithm, a deep neural network is used as a function approximator to generate state-action values. Penetration testing is performed by exploiting vulnerabilities, scanning, and extracting permissions on sensitive target hosts.

[0010] Step 2: Honeypot Deployment. Honeypots are deployed on the target network to protect sensitive hosts. A reinforcement learning (QL) algorithm is used to strategically deploy honeypots to achieve the optimal number of honeypots, taking into account the benefits and costs of both defenders and attackers. The effectiveness of the defense is then verified based on this method.

[0011] Step 3: Firewall defense. Firewalls with different reward values ​​are set up in the target network where the honeypot host is added. High-reward firewalls in the honeypot subnet lure attackers into the honeypot host, while low-reward firewalls in the subnet containing sensitive hosts protect sensitive hosts from attack, achieving a more effective defense. This step still utilizes the QL algorithm framework, but by modifying the firewalls to utilize different reward values, the success rate of penetration by attackers is ultimately reduced.

[0012] The above steps are iterated until the algorithm model fully converges, completing the training of the QL algorithm model. After the training is completed, a better reward defense method based on intelligent penetration testing is obtained.

[0013] The technical concept of the present invention is: 1. Network penetration testing is an effective method for evaluating the security of network systems. Relevant penetration testers can simulate hacker attack behaviors to mine vulnerabilities in the network and its hosts and conduct security assessments. 2. Compared with manual penetration testing and automated penetration attacks, penetration attacks based on reinforcement learning have stronger attack performance, so it is important to defend against such penetration attacks. 3. In order to prevent the illegal use of penetration testing based on reinforcement learning, it is called an attack method for network security. Based on the QL algorithm to penetrate the target network and strategically deploy honeypots to defend against attackers, the penetration process and the honeypot deployment process can be established as a Markov decision process to describe: MDP can be expressed as a four-tuple<S,A,R,P> , where S represents the state space set, A represents the action space set, and P represents the state transfer matrix, that is, from the current state s at time t t Take action a t The probability of going to the next state is P(s t+1 |s t ,a t ), R represents the reward function.

[0014]

[0015] Where γ∈(0,1) represents a discount factor, which measures the importance of current rewards relative to future rewards. The main idea of ​​the QL algorithm is to construct a Q-table that stores Q(s,a) values ​​based on the agent's state and actions at different times. The action with the highest benefit is then selected based on the Q values ​​in the Q-table. The update mechanism is as follows:

[0016]

[0017] Where s represents the attacker’s current state, a represents the attacker’s current action, s, and a, represent the next state and possible actions after the attacker takes action a. r represents the immediate reward, and α and γ represent the learning rate and discount factor, respectively.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: the reward defense method for intelligent penetration testing provided by the present invention uses the QL algorithm as the network foundation. Intelligently adding an optimal number of honeypots to the target network for defense based on reinforcement learning can reduce the cost of defense against penetration attacks and improve defense effectiveness. Firewalls are viewed as obstacles in the target network's terrain. By adding firewalls with different reward values ​​to the subnets of the honeypot network, attackers can be trapped in the honeypot hosts they designed, hindering the attackers from infiltrating sensitive hosts in the target network, thereby improving defense efficiency and success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a diagram of the reward defense model of the present invention.

[0021] Figure 2 It is the overall flow chart of the present invention.

[0022] Figure 3 This is a diagram showing the actual application of this solution in the constructed network. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0024] The present invention provides a reward defense method based on intelligent penetration testing, which uses the reward data of the intelligent agent in reinforcement learning as the entry point to perform reward defense for intelligent penetration testing when the attacker launches a penetration attack. Figure 3 As shown, there are 4 subnets and 8 hosts. The goal is to penetrate the sensitive host (4, 0) in subnet 4. Figure 2 The method comprises the following steps:

[0025] S1. Conduct penetration attacks on the target network.

[0026] S11. In the process of realizing intelligent penetration attack based on Q-learning algorithm, the attacker will continuously optimize the automatically generated attack path and finally obtain the best attack path. The intelligent agent is regarded as the penetration attacker, S={s1,s2,...,s i} represents the attacker's state set, where s i It is the information about the host that the attacker obtains from the external network using scanning tools at a specific moment.

[0027] S12. Based on the acquired state information, the attacker will infiltrate the target host. The attacker's action set can be expressed as A = {a1, a2, ..., a i}, where a i is the state s obtained by the attacker based on his interaction with the environment i The actions taken during the penetration attack are shown in the following table.

[0028] name type Cost consumption Probability Access Rights SSH penetration 3 0.9 User FTP penetration 1 0.6 User HTTP penetration 2 0.9 User Tomcat Privilege Escalation 1 1 Root Daclsvc Privilege Escalation 1 1 Root Service Scan scanning 1 1 / Operating System Description scanning 1 1 / Subnet Scan scanning 1 1 / Process Scanning scanning 1 1 /

[0029] S13. Take action a according to the attacker i Whether the target host is successfully penetrated, the attacker will be rewarded with feedback. i ,R={r1,r2,...,r n} represents the reward set obtained by the attacker. The reward value is calculated as follows:

[0030]

[0031] S14. During the penetration process, the attacker's goal is to maximize the accumulated reward value, that is, to use as few operations as possible to penetrate the most valuable sensitive host, as shown in the following formula:

[0032]

[0033] S2. After the target network is penetrated and attacked according to the method of step S1, a honeypot host is set up on the target network for defense.

[0034] S21. Consider the honeypot deployment problem as a security confrontation between two adversarial players. N is the total number of connected stations that can be used as honeypots or access points. The ratio of honeypot utilization to N is denoted by θ. D,i ∈{-1,1} represents the defender’s strategy: when the network attack targets a real access point or a honeypot, s D,i = -1 and 1 respectively, S A,i Denotes the attacker's strategy, and the defender's utility function U in time interval t D [t] can be expressed as:

[0035]

[0036] Where δ1 represents the profit of the defender for each attack against the honeypot, δ2 represents the profit of the defender for each attack detected without using the honeypot, and δ3 is the cost of each attack that the defender fails to detect in time. r,i =1, when not attacked I r,i =0, when S D,i = -1 means that the attacking device is a honeypot, and C is related to the cost caused by using the honeypot.

[0037] S22. In order to make honeypots have a better defense effect, the best strategy for the defender is to randomly distribute honeypots so that attackers cannot recognize their existence. D Denotes the utility function of the defender in the time interval t. Since the defender cannot know the number of attacks a priori, the goal is to optimize U D Expected value This is achieved by the probability φ of each connected device being attacked being related to the probability θ of the asset corresponding to the honeypot:

[0038]

[0039] P r is the probability of detecting an attack in the absence of a honeypot. P r = 1, the honeypot will not provide any benefits to the defender. In order to achieve the best defense effect, the appropriate θ value will be selected to maximize the utility function U D [t].

[0040] S23. The number of honeypots that can be deployed in subsequent work represents the possible action a, and the number of honeypots represents the current state s. Based on the QL algorithm update mechanism, the most appropriate number of honeypots for the target network can be obtained.

[0041] S24. Based on the method in step S1, a penetration attack is carried out on the sensitive host of the target network. If the attacker is trapped in the honeypot within the specified number of rounds and cannot successfully penetrate the sensitive host, it proves that the present invention has a good defense effect.

[0042] S3. By setting firewalls with different reward values, it is easier for attackers to fall into the honeypot designed by themselves and better protect sensitive hosts.

[0043] S31. We design rewards to incentivize realistic attack activities by using the term k, so that the reward of state s after taking action a becomes:

[0044] R(s,a)=R(s,a)+k(s) (7)

[0045] S32. The term k is used to reduce the reward to encourage fire prevention. Its value is as follows:

[0046]

[0047] Where w≤0 is a parameter that adjusts the intensity of the incentive. The reward can be modified by the security of the communication protocol. The k multiplier for FTP is 0.8, while for SSH it is 0.2. Therefore, penetrating a host with an FTP-based firewall may be more difficult than with SSH.

[0048] S33. Based on the above principle, attackers are more likely to attack and penetrate subnets containing SSH. By utilizing the honeypot network in step S2, an SSH firewall can be set up in the subnet containing honeypot hosts, while an FTP firewall can be set up in the subnet containing sensitive hosts. This makes it easier for attackers to fall into the honeypot hosts and more difficult to penetrate sensitive hosts.

[0049] S34. Perform penetration attacks on sensitive hosts in the target network based on the QL algorithm. If the number of rounds required to penetrate the sensitive hosts is greater than that in step S2, it can be proved that the model has a better defense effect.

[0050] This example provides a reward-based defense method based on intelligent penetration testing for defending against reinforcement learning-based network penetration attacks. The attacked network model is defined as hosts, connections, and configurations on the network, as represented by the tuple {subnet, topology, host, service, firewall}. To interact with the network model, a reinforcement learning Markov decision process based on the QL algorithm models the interaction between the attacker agent and the environment, thereby enabling interaction between the network and the penetration environment. In application, by modifying the target network model within the penetration environment to generate different penetration environments, the practicality and defensive effectiveness of this method can be verified using agent-based reinforcement learning.

[0051] Specific actual defense models such as Figure 1 As shown, the defender captures and modifies the target network reward data from the attacker's agent training. First, all firewalls in the network are set to SSH-type firewalls. Based on step S2, honeypot hosts are embedded in the subnet for penetration defense. The defense's effectiveness is verified by conducting penetration attacks on sensitive hosts. If the attacker fails to gain access to the sensitive host after the training round, the defense is considered successful. The attacker is then set to FTP-type firewalls for the sensitive host and the other firewalls remain SSH-type, making it easier for the attacker to access the sensitive host. The attacker then conducts penetration attacks on the sensitive host using the same attack method, verifying that this model has better penetration defense effectiveness.

[0052] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A reward-based defense method based on intelligent penetration testing, comprising the following steps: Step 1: Penetration attack. Penetration testing involves gradually penetrating each subnet of the target network to discover the location of sensitive hosts and attack them. Finally, the results are plotted as a report and provided to the tester, who can use the report to strengthen the system's defenses and improve network security. Based on the QL algorithm, a deep neural network is used as a function approximator to generate state-action values. Penetration testing is performed by exploiting vulnerabilities, scanning, and extracting permissions on the target sensitive hosts. Step 2: Honeypot deployment. Honeypots are deployed on the target network to protect sensitive hosts. The reinforcement learning QL algorithm is used to strategically deploy honeypots to obtain the optimal number of honeypots while considering the benefits and costs of defenders and attackers. The defense effect is verified based on the above methods, including: S21. Consider the honeypot deployment problem as a security confrontation between two hostile players. is the total number of connected stations that can be used as honeypots or access points; the utilization of honeypots The ratio of express; Represents the defender's strategy: When a network attack targets a real access point or a honeypot, are equal to -1 and 1 respectively, Denotes the attacker's strategy, the defender's utility function in time interval t It can be expressed as: (5) in represents the defender’s profit from each attack on the honeypot, represents the defender’s revenue per attack detected without using a honeypot, is the cost of each attack that the defender fails to detect in time; when the attack is detected, , when not attacked ,when This means the attacking device is a honeypot. Related to the costs incurred by using honeypots; S22. In order to make honeypots more effective, the best strategy for the defender is to randomly assign honeypots so that attackers cannot identify their existence; represents the utility function of the defender in the time interval t. Since the defender cannot know the number of attacks a priori, the goal is to optimize Expected value , and the probability of being attacked by each connected device The probability of being associated with the asset portion corresponding to the honeypot To achieve: (6) is the probability of detecting an attack in the absence of a honeypot; In this case, the honeypot will not provide any benefits to the defender; in order to achieve the best defense effect, the appropriate value to maximize the utility function ; S23. The number of honeypots that can be deployed in subsequent work indicates possible actions , the number of honeypots indicates the current state ; Based on the QL algorithm model update mechanism, the most suitable number of honeypots for the target network can be obtained; S24, based on the method in step S1, conduct penetration attacks on sensitive hosts of the target network. If the attacker is trapped in the honeypot within the specified number of rounds and cannot successfully penetrate the sensitive hosts, it proves that the defense effect is good; Step 3, firewall defense; by setting up firewalls with different reward values ​​in the target network where the honeypot host is added; by setting up a firewall with a high reward value in the honeypot host subnet to lure attackers into the honeypot host, and setting up a firewall with a low reward value in the subnet containing sensitive hosts to protect sensitive hosts from being attacked, thereby achieving better defense effects; in this step, the QL algorithm framework is still adopted, and defense is performed by modifying and utilizing firewalls with different reward values, ultimately reducing the penetration success rate of infiltration attackers.

2. The reward defense method based on intelligent penetration testing according to claim 1, characterized in that: Step 1 specifically includes: S11. In the process of implementing intelligent penetration attacks based on the Q-learning algorithm, the attacker will continuously optimize the automatically generated attack path and eventually obtain the optimal attack path; the intelligent agent is regarded as a penetration attacker. represents the attacker's state set, where The attacker at a specific moment Information about the host obtained from the external network using scanning tools; S12. Based on the acquired status information, the attacker will infiltrate the target host. The attacker's action set can be expressed as ,in is the state obtained by the attacker based on his interaction with the environment The actions taken during the penetration attack are shown in the following table; S13. Take action based on the attacker Whether the target host is successfully penetrated, the attacker will be rewarded , Represents the set of rewards obtained by the attacker; the reward value is calculated as follows: (3) S14. During the penetration process, the attacker's goal is to maximize the accumulated reward value, that is, to use as few operations as possible to penetrate the most valuable sensitive host, as shown in the following formula: (4)。 3. The reward defense method based on intelligent penetration testing according to claim 1, characterized in that: Step 3 includes: S31. Through the use of terminology Design rewards to incentivize realistic attack activities so that when actions are taken Post-state The reward becomes: （7） S32. Terminology Used to reduce rewards to incentivize firewall avoidance; its values ​​are as follows: （8） in It is a parameter to adjust the intensity of incentives; the change of rewards can be changed through the security of the communication protocol; the FTP The multiplier value is 0.8, while SSH The multiplier value is 0.2, so it is more difficult for an attacker to penetrate a host containing an FTP-based firewall than an SSH-based one. S33. Based on the principle of intelligent penetration testing based on reinforcement learning, attackers are more likely to attack and penetrate subnets containing SSH. By utilizing the honeypot network in step S2, an SSH firewall can be set up in the subnet with honeypot hosts, while an FTP firewall can be set up in the subnet with sensitive hosts. This makes it easier for attackers to fall into the honeypot hosts and more difficult to penetrate sensitive hosts. S34. Perform a penetration attack on sensitive hosts in the target network based on the QL algorithm. If the number of rounds required to penetrate the sensitive hosts is greater than that in step S2, it can be proved that the QL algorithm model has a better defense effect.

Images