A security exploration reinforcement learning method based on deployment of a honeypot and a pseudo honeypot

Abstract

The present application relates to a kind of security exploration reinforcement learning method based on honeypot and pseudo honeypot deployment.It relates to network attack and defense security field, in the security attack and defense scene of software defined network architecture, network defense system has only incomplete observation information under the premise of unknown external attacker, without predicting the attack model of attacker, using security reinforcement learning algorithm can dynamically change defense deployment strategy, to cope with the change of attack strategy, improve attack detection, capture and defense performance.The method can be through the deployment cost of observation current defense strategy, detection cost and last network defense benefit etc. Feedback information, adopt security reinforcement learning algorithm to select honeypot and pseudo honeypot etc. Defense deployment strategy, can reduce the resource cost of network defense system, improve security defense efficiency, to guarantee the security of equipment in software defined network.

Description

Technical Field

[0001] This invention relates to the field of network attack and defense security, specifically to a security exploration reinforcement learning method based on honeypot and pseudo-honeypot deployment. Background Technology

[0002] In recent years, cyberattacks have become increasingly frequent. These attacks exploit vulnerabilities and security flaws in network information systems to carry out various types of attacks, causing serious security threats and enormous economic losses worldwide. Honeypot-based network defense technology has demonstrated significant potential in mitigating cyberattacks. Honeypot technology is a deception technique that uses bait to lure attackers into attacks and gather intelligence on them. Conversely, anti-honeypot technology involves disguising legitimate hosts as honeypots to mislead attackers into detecting them as honeypots and bypassing the attack.

[0003] Honeypot technology is widely used in modern cybersecurity. For example, Chinese patent publication number CN116599733A proposes a honeypot-based network attack protection method and system. This method analyzes low, medium, and high-risk network attacks, the honeypot's trapping rate, and defense success rate to determine the number of honeypots to deploy and ensure communication security. Chinese patent publication number CN116346430A proposes a network threat management system based on highly interactive honeypots, using these honeypots as information gathering nodes to lure potential attackers. Chinese patent publication number CN116545675A proposes a method for capturing and countering phishing emails based on computer deception and trapping technology. It establishes an open proxy server honeypot to attract attackers to send phishing emails, then identifies the attacker's IP address for capture and countermeasures.

[0004] W. Tian et al. [W. Tian, ​​M. Du, X. Ji, G. Liu, Y. Dai and Z. Han, "Honeypot Detection Strategy Against Advanced Persistent Threats in Industrial Internet of Things: A Prospect Theoretic Game," in IEEE Internet of Things Journal, vol. 8, no. 24, pp. 17372-17381, 15 Dec. 15, 2021, doi: 10.1109 / JIOT.2021.3080527.] proposed a dynamic bounded rational honeypot-APT game model based on SDN, in which the defense system maximizes utility by choosing a periodic strategy for collecting and analyzing data in the honeypot, while the attacker maximizes utility by choosing a periodic strategy for its delay and attack. B.Li et al. [B.Li, Y.Xiao, Y.Shi, Q.Kong, Y.Wu and H.Bao, "Anti-Honeypot Enabled Optimal Attack Strategy for Industrial Cyber-Physical Systems," in IEEE OpenJournal of the Computer Society, vol.1, pp.250-261, 2020, doi:10.1109 / OJCS.2020.3030825.] proposed an anti-honeypot optimization strategy. This method introduces hybrid signaling and repeated game theory, derives Nash equilibrium based on various possible defenses and the attack behaviors of different attackers, and finds the optimal attack strategy to achieve the goal of attacking honeypot-protected targets. Summary of the Invention

[0005] The purpose of this invention is to provide a security exploration reinforcement learning method based on honeypot and pseudo-honeypot deployment. Based on honeypot and pseudo-honeypot deployment technologies, this method utilizes security reinforcement learning to maximize defense effectiveness while meeting given security constraints, avoiding the defense system's exploration and selection of high-risk deployment strategies. This helps resist unknown external network attacks and improves the overall security of software-defined networks. This method does not require predicting the attacker's attack model. Against attackers using ordinary attacks or anti-honeypot attack strategies, the defense system can explore the optimal defense deployment strategy through security reinforcement learning during frequent network attack and defense confrontations.

[0006] To achieve the above objectives, the technical solution of the present invention is: a security exploration reinforcement learning method based on honeypot and pseudo-honeypot deployment, comprising the following steps:

[0007] Step 1: Network Initialization and Parameter Definition

[0008] Suppose there are N servers and one network defense system in a software-defined network; in the k-th time slot, a visitor accesses the system, and the probability of this visitor being present is P. (k) It is an external attacker, with a probability of 1-P. (k) The user is legitimate; external attackers can use strategies including normal attacks and anti-honeypot attacks to attack the software-defined network; the network defense system can defend by deploying H honeypots and P fake honeypots, and adopting strategies that provide normal services, honeypot services, and fake honeypot services; where Ω = [Ω1, Ω2] represents whether the network defense system provides services, i.e., Ω1 provides services and Ω2 does not provide services; suppose the i-th server in the software-defined network suffers an unknown network attack in the k-th time slot, and the network defense system adopts defense deployment strategy a. i (k) Deploying this strategy will generate C i (k) Defense costs consist of resource costs and time costs; in the initial stage of defense system deployment, attackers, network defense systems, and security reinforcement learning networks are initialized.

[0009] Step 2: In the k-th time slot, an external attacker launches an attack on the software-defined network; to counter this unknown network attack, the network defense system adopts defense deployment strategy a for each of the N servers. i (k) The defense deployment strategy space set is represented as A. (k) =[a i (k) ] 0≤i≤N ;

[0010] Step 3: In the k-th time slot, the network defense system observes the network system state of the previous time slot k-1, including the probability P that the visitor is an external attacker. (k-1) Successfully captured the benefits R gained by external attackers (k-1) The loss L caused by a successful attack by an external attacker (k-1) Security assessment level of network defense system Safe (k-1) Risk level of network defense system adopting three strategies: normal service, honeypot service, and fake honeypot service. (Risk1) (k-1) Risk2 (k-1) Risk3 (k-1) And the cost C of obtaining the defense deployment strategy adopted in the current k-slot. d (k)Therefore, the current observation state of the network defense system in time slot k is:

[0011] Step 4: Use the supervisor F(x) in the security reinforcement learning algorithm to determine the defense deployment strategy space set A for the current k time slots. (k) The resulting safety assessment level Safe (k) Is it within the safety constraint region D? s {Safe (k) Within ≤threshold}; where, Safe (k) The safety assessment level is indicated by the following formula: Where P_A i (k) P_E represents the probability that an external attacker will use attack strategy i to launch an attack. i (k) C represents the probability that a network defense system successfully defends against attack strategy i by an external attacker. i (k) This represents the defense cost used by the network defense system to defend against attack strategy i from external attackers; if the security constraint region D is satisfied... s Then the network defense system calculates the immediate benefit u. N (k) Its formula is β represents the probability that the network defense system successfully detects an external attacker; if it is not within the security constraint zone D s If the network defense system is breached by external attackers, the network defense system will need to update and upgrade its defense deployment strategies to cope with the next round of attacks.

[0012] Step 5: Observation state function s based on the k-th time-slot network defense system (k) and safety assessment level Safe (k) Update the defense deployment strategy set A of the next time-slot network defense system. (k+1) ;

[0013] Step 6: Repeat steps 2 to 5. When the iteration ends at k = K, terminate the iteration.

[0014] Compared to existing technologies, this invention offers the following advantages: This invention provides a security exploration reinforcement learning method based on honeypot and pseudo-honeypot deployment. Leveraging honeypot and pseudo-honeypot deployment technologies, it utilizes security reinforcement learning to maximize defense effectiveness while meeting given security constraints. This avoids the defense system exploring and selecting high-risk deployment strategies, thereby resisting unknown external network attacks and improving the overall security of software-defined networks. This method does not require predicting the attacker's attack model. Against attackers using ordinary attacks or anti-honeypot attack strategies, the defense system can explore the optimal defense deployment strategy through security reinforcement learning during frequent network attack and defense confrontations. Detailed Implementation

[0015] The technical solution of the present invention will now be described in detail.

[0016] This invention provides a security exploration reinforcement learning method based on honeypot and pseudo-honeypot deployment, and the specific implementation steps are as follows:

[0017] Step 1: Network Initialization and Parameter Definition: Assume there are N = 8 servers and one network defense system in the software-defined network. In the k-th time slot, a visitor accesses the system; the probability of this visitor is P. (k) It is an external attacker, with a probability of 1-P. (k) The user is legitimate. In this software-defined network (SDN), external attackers can use strategies such as ordinary attacks or anti-honeypot attacks to attack the SDN system. The defense system can deploy H = 2 honeypots and P = 6 fake honeypots, and use strategies such as providing normal service, honeypot service, and fake honeypot service for defense; where Ω = [Ω1, Ω2] represents whether the defense system provides normal user service, that is, Ω1 = 1 means providing service, and Ω2 = 0 means not providing service. Suppose that the i-th server in the SDN system suffers an unknown network attack in the k-th time slot, and the defense system adopts defense deployment strategy a. i (k) =[a 1,i (k) ,a 2,i (k) ,a 3,i (k) Deploying this strategy will generate C i (k) Defense costs consist of resource costs and time costs. In the initial phase of deploying a defense system, attackers, the defense system, and the security reinforcement learning network are initialized.

[0018] Step 2: In the k-th time slot, an external attacker launches an attack on the system within the software-defined network. To counter this unknown network attack, the defense system employs the following defense deployment strategy a for each of the N servers. i(k) The defense deployment strategy space set is A (k) =[a i (k) ] 0≤i≤8 .

[0019] Step 3: In the k-th time slot, the defense system observes the network system state of the previous time slot k-1, including the probability P that the visitor is an external attacker. (k-1) Successfully captured the benefits R gained by external attackers (k-1) The loss L caused by a successful attack by an external attacker (k-1) Safe assessment level of the defense system i (k-1) Risk level 1 for the defense system employing three strategies: normal service, honeypot service, and pseudo-honeypot service. (k-1) Risk2 (k-1) Risk3 (k-1) And the cost C of obtaining the defense deployment strategy adopted in the current k-slot. d (k) Therefore, the current observation state of the defense system in time slot k is:

[0020] Step 4: Use the supervisor F(x) in the security reinforcement learning algorithm to determine the defense deployment strategy space set A for the current k time slots. (k) The resulting safety assessment level Safe (k) Is it within the safety constraint region D? s {Safe (k) Within ≤100}. Where, Safe (k) The safety assessment level is indicated by the following formula: Where P_A i (k) P_E represents the probability that an external attacker will use attack strategy i to launch an attack. i (k) C represents the probability that the defense system successfully defends against attack strategy i by an external attacker. i (k) This represents the defense cost used by the defense system to defend against attack strategy i from external attackers. If the security constraint region D is satisfied... s The defense system then calculates the immediate benefit u. N (k) Its formula is Specifically, β represents the probability that the defense system successfully detects an external attacker, and 0 ≤ β ≤ 1. If the attacker is not within the security constraint region D... s If the defense system is breached by external attackers, the defense system will need to update and upgrade its defense deployment strategies to cope with the next round of attacks.

[0021] Step 5: Based on the observation state function s of the current k-th time slot defense system (k) and safety assessment level Safe (k) Update the defense deployment strategy set A for the next time slot defense system. (k+1) .

[0022] Step 6: Repeat steps 2 to 5. When the iteration ends at k = 1000, terminate the iteration.

[0023] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A security exploration reinforcement learning method based on honeypot and pseudo-honeypot deployment, characterized in that, Includes the following steps: Step 1: Network Initialization and Parameter Definition Suppose there are N servers and one network defense system in a software-defined network; in the k-th time slot, a visitor accesses the system, and the probability of this visitor being present is P. (k) It is an external attacker, with a probability of 1-P. (k) The user is legitimate; external attackers can use strategies including normal attacks and anti-honeypot attacks to attack the software-defined network; the network defense system can defend by deploying H honeypots and P fake honeypots, and adopting strategies that provide normal services, honeypot services, and fake honeypot services; where Ω = [Ω1, Ω2] represents whether the network defense system provides services, i.e., Ω1 provides services and Ω2 does not provide services; suppose the i-th server in the software-defined network suffers an unknown network attack in the k-th time slot, and the network defense system adopts defense deployment strategy a. i (k) Deploying this strategy will generate C i (k) Defense costs consist of resource costs and time costs; in the initial stage of defense system deployment, attackers, network defense systems, and security reinforcement learning networks are initialized. Step 2: In the k-th time slot, an external attacker launches an attack on the software-defined network; to counter this unknown network attack, the network defense system adopts defense deployment strategy a for each of the N servers. i (k) The defense deployment strategy space set is represented as A. (k) =[a i (k) ] 0≤i≤N ; Step 3: In the k-th time slot, the network defense system observes the network system state of the previous time slot k-1, including the probability P that the visitor is an external attacker. (k-1) Successfully captured the benefits R gained by external attackers (k-1) The loss L caused by a successful attack by an external attacker (k-1) Security assessment level of network defense system Safe (k-1) Risk level of network defense system adopting three strategies: normal service, honeypot service, and fake honeypot service. (Risk1) (k-1) Risk2 (k-1) Risk3 (k-1) And the cost C of obtaining the defense deployment strategy adopted in the current k-slot. d (k) Therefore, the current observation state of the network defense system in time slot k is: Step 4: Use the supervisor F(x) in the security reinforcement learning algorithm to determine the defense deployment strategy space set A for the current k time slots. (k) The resulting safety assessment level Safe (k) Is it within the safety constraint region D? s {Safe (k) Within ≤threshold}; where, Safe (k) The safety assessment level is expressed by the following formula: Where P_A i (k) P_E represents the probability that an external attacker will use attack strategy i to launch an attack. i (k) C represents the probability that a network defense system successfully defends against attack strategy i by an external attacker. i (k) This represents the defense cost used by the network defense system to defend against attack strategy i from external attackers; if the security constraint region D is satisfied... s Then the network defense system calculates the immediate benefit u. N (k) Its formula is β represents the probability that the network defense system successfully detects an external attacker; if it is not within the security constraint zone D s If the network defense system is breached by external attackers, the network defense system will need to update and upgrade its defense deployment strategies to cope with the next round of attacks. Step 5: Observation state function s based on the k-th time-slot network defense system (k) and safety assessment level Safe (k) Update the defense deployment strategy set A of the next time-slot network defense system. (k+1) ; Step 6: Repeat steps 2 to 5. When the iteration ends at k = K, terminate the iteration.

Images