APT network killing chain dynamic defense method and system based on double deep reinforcement learning

Abstract

The invention discloses an APT network killing chain dynamic defense method and system based on double deep reinforcement learning, and the method comprises the steps: configuring a network system in advance, and enabling the network system to comprise a host node, a security defender, a dangerous attacker and a security defense device; a dangerous attacker executes an attack behavior on a host node according to a network killing chain, a security defender monitors the attack behavior and obtains a network state, and the security defense device dynamically generates an optimal defense strategy based on a dual-depth reinforcement learning algorithm in combination with the network state and feeds back the optimal defense strategy to the security defender. And the security defender dynamically responds to the attack behavior by using the optimal defense strategy so as to realize protection of the network killing chain. According to the method, efficient recognition and accurate response of the APT attack are realized, the security and resource utilization efficiency of the system are effectively improved, an innovative solution is provided for dynamic defense in a complex network environment, and the resisting effect on the APT attack is remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the field of network attack and defense security, and in particular to an APT network kill chain dynamic defense method and system based on dual deep reinforcement learning. Background Art

[0002] In recent years, with the increasing complexity and concealment of network attacks, especially the frequent occurrence of advanced persistent threat attacks, how to effectively detect and defend against such attacks has become a key problem in the field of network security. Advanced persistent threat attacks gradually achieve their attack targets through a multi-stage network kill chain. The attack chain stages include reconnaissance, weaponization, delivery, utilization, installation, command and control, and achieving attack targets, which makes the defender face huge challenges in identifying and intervening in advanced persistent threat attack behaviors. The Chinese patent with publication number CN117395072A discloses a method for generating a network kill chain, which generates multi-source logs and security alarm logs by analyzing network traffic and terminal behavior, and aggregates the alarm logs, and then associates the multi-source logs to generate user behavior sequences. The method identifies the attack traffic in the behavior sequence through a pre-trained detection model. If the attack traffic is found, the corresponding network kill chain is generated using the generation model. This method significantly improves the efficiency and accuracy of attack traffic identification, and accurately restores the network kill chain of attack traffic, helping network security personnel to comprehensively evaluate and respond to potential threats in a timely manner.

[0003] Although existing technologies can improve the accuracy and comprehensiveness of generating kill chains, how to reasonably and dynamically allocate limited defense resources in the multi-stage chain of advanced persistent threat attacks to ensure the best defense effect at each stage is still a key problem. Network deception technology has been introduced into the APT defense system as an innovative defense method. The Chinese patent with publication number CN116614296A discloses a honeypot deception defense method, which helps security personnel efficiently locate attack methods and optimize defense responses by identifying attacker behavior and displaying it based on the ATT&CK matrix view. AHAnwar et al. [AHAnwar, CA Kamhoua, NO Leslie and C. Kiekintveld,"Honeypot Allocation for Cyber ​​Deception Under Uncertainty,"in IEEE Transactions on Network and Service Management,vol.19,no.3,pp.3438-3452,Sept.2022,doi:10.1109 / TNSM.2022.3179965.] proposed a deception method that combines game theory and reinforcement learning models. They modeled the reactive deception problem as a partially observable Markov decision process based on a game-theoretic dynamic model to cope with the incomplete monitoring of the attacker's behavior.

[0004] However, the implementation of network deception defense technology requires a lot of computing resources and network bandwidth. Improper resource allocation may lead to high costs and even affect normal system performance. Therefore, the reasonable allocation of network deception defense resources and the optimization of defense strategies have become important issues in the defense against advanced persistent threat attacks. Zhang, L et al. [L. Zhang, T. Zhu, F. K. Hussain, D. Ye and W. Zhou,"A Game-Theoretic Method for Defending Against Advanced Persistent Threats in Cyber ​​Systems," in IEEE Transactions on Information Forensics and Security, vol. 18, pp. 1349-1364, 2023, doi: 10.1109 / TIFS.2022.3229595.] help defenders achieve the best defense effect when responding to APT attacks by optimizing the timing and resource allocation of defense strategies, thereby maximizing defense efficiency and reducing resource waste. B.Peng et al. [B.Peng, J.Liu and J.Zeng,"Dynamic Analysis of Multiplex Networks With HybridMaintenance Strategies,"in IEEE Transactions on Information Forensics and Security,vol.19,pp.555-570,2024,doi:10.1109 / TIFS.2023.3324386.] proposed a propagation suppression strategy based on the RCMO model, combining static and dynamic control methods, effectively suppressing the spread of malware through a hybrid maintenance strategy, optimizing resource allocation to improve network security and reduce security risks.

[0005] As an intelligent optimization method, dual deep reinforcement learning can perform adaptive learning in complex unknown environments and gradually optimize strategies through continuous feedback. It is extremely suitable for complex and dynamic defense tasks such as advanced persistent threat attacks. Y. Yu et al. [Y. Yu, W. Yang, W. Ding and J. Zhou, "Reinforcement Learning Solution for Cyber-Physical Systems Security Against Replay Attacks," in IEEE Transactions on Information Forensics and Security, vol. 18, pp. 2583-2595, 2023, doi: 10.1109 / TIFS.2023.3268532.] proposed an attack detection method based on reinforcement learning. Through a model-free learning framework, it can automatically identify and respond to the attacker's dynamic strategy. At the same time, it designs a defense strategy for optimized learning, which effectively improves the network security protection capability. W. He et al. [W. He, J. Tan, Y. Guo, K. Shang and H. Zhang,"A Deep Reinforcement Learning-Based Deception Asset Selection Algorithmin Differential Games," in IEEE Transactions on Information Forensics and Security, vol. 19, pp. 8353-8368, 2024, doi: 10.1109 / TIFS.2024.3451189.] proposed a differential game deception asset selection algorithm based on multi-agent deep reinforcement learning. By constructing network security state evolution analysis and differential game model, the deployment of deception assets in advanced persistent threat attack defense is optimized, and more efficient attack defense strategy selection is achieved.

[0006] In an APT attack, dangerous APT attackers can only obtain information about the damaged node and its neighboring nodes through tools (such as Burp Suite and OpenVAS), and choose the optimal path based on the trade-off between the minimum attack cost and the maximum attack benefit based on the importance of the node and the characteristics of the vulnerability. Subsequently, the attacker launches a variety of threat behaviors on the target node, including scanning attacks, phishing attacks, botnet attacks, DDoS attacks, zero-day attacks, key damage attacks, fake key attacks, and data leakage attacks. If the attack is successful, the attacker will advance to the next stage of the network kill chain and continue to interact with defenders who adopt a bundling strategy. Summary of the invention

[0007] The main purpose of the present invention is to overcome the network deception defense problem in the environment of the prior art, and propose a dynamic defense method and system for APT network kill chain based on dual deep reinforcement learning, which comprehensively considers network topology changes, node importance and vulnerability information, and aims to dynamically optimize resource allocation and defense strategy selection by obtaining network status and dangerous attacker behavior information in real time.

[0008] The present invention adopts the following technical solution:

[0009] A dynamic defense method for APT network kill chain based on dual deep reinforcement learning, pre-configuring the network system, the network system includes host nodes, security defenders, dangerous attackers and security defense devices;

[0010] The dangerous attacker executes attack behavior on the host node according to the network kill chain, the security defender monitors the attack behavior and obtains the network status, the security defense device is based on a dual deep reinforcement learning algorithm, combines the network status to dynamically generate an optimal defense strategy and feeds back to the security defender, and the security defender uses the optimal defense strategy to dynamically respond to the attack behavior to achieve protection of the network kill chain.

[0011] The dangerous attacker performs attack actions on the host node according to the network kill chain specifically referring to:

[0012] The dangerous attacker gradually utilizes the zombie node set according to each stage of the network kill chain, selects targets for attack according to the characteristics of the host nodes adjacent to the zombie nodes; then allocates dangerous attack resources according to the attack budget and launches a dangerous attack. The set of successfully damaged host nodes will be converted into a new set of zombie nodes.

[0013] The security defender models the dynamic risk level of the host node being attacked by zombie nodes as an exponential function; a response indication vector is introduced to describe the resources allocated by the security defender to the damaged host node using a limited defense budget.

[0014] At time slot t, the security defender obtains the network status of the previous time slot t-1 by observation, and constructs the observation state function S[t] of time slot t, S[t] = {I[t-1], V[t-1], c A [t-1],r A [t-1],d D [t-1],c D [t-1],r D[t-1], TD[t-1], p[t-1]}, I[t-1] is the importance set of the host nodes of the network system at time slot t-1, V[t-1] is the vulnerability set of the host nodes of the network system at time slot t-1, c A [t-1] is the cost set of the dangerous attacker adopting the dangerous attack strategy at time slot t-1, r A [t-1] is the reward set obtained by the dangerous attacker at time slot t-1, d D [t-1] is the set of resources allocated by the security defender to the damaged host node at time slot t-1, c D [t-1] is the cost set of the security defender adopting the security defense strategy at time slot t-1, r D [t-1] is the reward obtained by the security defender at time slot t-1, TD[t-1] is the set of damaged host nodes at time slot t-1, and p[t-1] is the risk level set of the damaged host nodes being subjected to dangerous attacks at time slot t-1.

[0015] The security defense device uses the double deep reinforcement learning algorithm to obtain the network deception defense deployment strategy space set A[t]={d D [t],π[t]},d D [t] represents the resource allocation of the security defender at time slot t, π[t] represents the network deception defense strategy selected by the security defender at time slot t, and the utility of the security bundling defense strategy is calculated as DR i [t] represents the response indicator vector.

[0016] Where TD[t] is the set of host nodes damaged at time slot t, DR i [t] indicates whether the security defender takes action to protect the damaged host node i, p at time slot t i [t] is the risk level of the damaged host node i being attacked by a dangerous attack at time slot t, is the amount of resources allocated by the security defender to the damaged host node i at time slot t, It is the cost of the security defender adopting the security defense strategy for the damaged host node i at time slot t.

[0017] The dual deep reinforcement learning algorithm includes a main network and a target network. The main network selects the optimal execution action according to the current state, and the target network is used to evaluate the value of the action selected by the main network and calculate the target Q value matrix accordingly; the main network is updated and trained at each time step, and the target network synchronously updates its weights at fixed time intervals to track the changes of the main network.

[0018] The Q value matrix is ​​Q a (S[t], A[t]), which represents the Q value when the security defender selects the network deception defense deployment strategy space set A[t] under the observation state function S[t] of the network state. The expression for updating the Q value matrix is ​​as follows:

[0019] Q a (S[t],A[t])=(1-α)Q a (S[t],A[t])+α(R D [t+1]+γ·Q b (S[t+1],max a∈A Q a (S[t+1],A[t])));

[0020] Among them, α is the learning factor, γ is the discount factor, and R D [t+1] and S[t+1] are the benefits and status of the security defender in time slot t+1 respectively, A[t] is the action of the security defender in time slot t, and the max(.) function is the maximum value function.

[0021] The security defense device also receives defense results fed back from the security defender to adjust the optimal defense strategy to form a closed-loop feedback mechanism.

[0022] It also includes judging whether the network system fails according to a failure condition threshold, wherein the network system failure means that the operation logic of the network system is terminated and the attack and defense interaction can no longer be continued; the failure threshold condition is as follows:

[0023]

[0024] Among them, Th1 and Th2 are the information importance leakage threshold and the available node threshold respectively, I i is the importance value of the host node i in the network system, |VN0| is the number of all host nodes in the initial network system when it is not attacked by dangerous attackers, |VN t | is the number of all available host nodes in the network system at time slot t, N i Indicates the safety, damage or expulsion status of the host node, N i =1 means the host node is damaged or expelled, otherwise N i =0.

[0025] If the network system fails or reaches the maximum number of iterations, it ends. Otherwise, it is determined whether the dangerous attacker has successfully attacked. If the attack is unsuccessful, the current stage of the network kill chain continues. If the attack is successful, the next stage of the network kill chain is entered.

[0026] A dynamic defense system for APT network kill chain based on dual deep reinforcement learning, including:

[0027] Host node,

[0028] Dangerous attackers, who perform attack actions on the host node according to the network kill chain;

[0029] A security defender monitors the attack behavior and obtains the network status, and dynamically responds to the attack behavior using the optimal defense strategy to achieve protection of the network kill chain;

[0030] The security defense device, based on a dual deep reinforcement learning algorithm, dynamically generates an optimal defense strategy in combination with the network status and feeds back to the security defender.

[0031] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention analyzes and obtains the network status through the real-time network traffic imported by the security defender, and uses the dual deep reinforcement learning algorithm to iteratively update the bundled defense strategy, so as to select the optimal deception asset deployment strategy in the multi-stage attack scenario, effectively respond to complex attacks, and reduce the potential profitability of dangerous attackers while protecting the system host nodes and improving the system security.

[0033] The present invention systematically discusses how dangerous attackers transfer at different stages of the network kill chain and interact with security defenders who adopt bundling strategies. Based on this design, the security defense device helps defenders accurately predict the attacker's behavior, assess the risk intensity, and select corresponding optimized resource allocation and defense strategies through intelligent analysis modules and algorithm modules, thereby significantly improving the network defense capability and enhancing the resistance to APT attacks, which has significant advantages and application prospects.

[0034] The present invention innovatively discusses the vulnerability of nodes, defining it as a dynamic comprehensive vulnerability related to time and the allocation of attack and defense resources in interaction, and considering that when the network topology changes, dangerous attackers can only know the information of adjacent nodes, and use the importance and vulnerability of nodes to select the optimal path.

[0035] The present invention realizes efficient identification and accurate response to APT attacks, effectively improves the security and resource utilization efficiency of the system, provides an innovative solution for dynamic defense in complex network environments, significantly enhances the resistance to APT attacks, and has important technical innovation and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram of the network system architecture of the present invention.

[0037] Figure 2 This is a module diagram of the security defense device of the present invention.

[0038] Figure 3 The figure is a flow chart of the method of the present invention.

[0039] Figure 4 This is a state transition diagram of the network kill chain of the present invention.

[0040] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0041] The present invention is further described below through specific implementation modes.

[0042] See also Figure 3 , a dynamic defense method for APT network kill chain based on dual deep reinforcement learning, pre-configured network system, network system includes host nodes, security defenders, dangerous attackers and security defense devices. The model of the network system is built on the SDN architecture, including multiple host nodes with different importance and known vulnerabilities, and a core component in which the SDN controller acts as a security defender. The security defender deploys a network intrusion detection system to monitor network traffic in real time, and responds to changes in complex network topology through dynamic security defense devices. In response to the diverse attacks launched by dangerous APT attackers at different stages of the network kill chain, the security defense device can dynamically adjust the output defense strategy. For example, the output honeypot defense strategy allows the security defender to use the honeypot to capture dangerous attackers and analyze attack behaviors, generate decoy data, etc.

[0043] Initialize the network system and related parameters, initialize the Q value matrix, learning factor α and discount factor γ, etc. Configure the number of host nodes in the system (e.g. 100), the number of security defenders (e.g. 1), and the number of dangerous attackers (e.g. 2). Determine the network system failure condition threshold (e.g. set to and ), determine the total amount of resources owned by the security defender (e.g., 50), including time, computing resources, the importance of initializing each host node, and the vulnerability.

[0044] The method of the present invention further comprises the steps of:

[0045] S1 dangerous attackers perform attack behaviors on host nodes according to the network kill chain.

[0046] Specifically, dangerous attackers gradually utilize a set of zombie nodes (i.e., a set of damaged nodes) according to various stages of the network kill chain (e.g., from reconnaissance to action), and select targets for attack based on the characteristics of host nodes adjacent to the zombie nodes; then, they allocate dangerous attack resources according to the attack budget and launch a dangerous attack. The set of successfully damaged host nodes will be transformed into a new set of zombie nodes to carry out the next dangerous attack.

[0047] The neighboring node path selection weights of the zombie node set controlled by the dangerous attacker are modeled as:

[0048]

[0049] Among them, β1, β2, and β3 are used to balance the importance of nodes, node vulnerability, and the impact of attack costs when dangerous attackers select paths. i is the importance value of host node i, V i is the vulnerability value of host node i, is the attack cost required for a dangerous attacker to attack host node i.

[0050] The state transition diagram of the network kill chain is as follows Figure 4 As shown, its transfer matrix is ​​modeled as:

[0051]

[0052] Among them, λ i and μ i is the eigenvalue of the transfer matrix of the i-th layer network kill chain, P i [t], i∈[0,7] is the transition probability of the i-th layer network kill chain at time slot t.

[0053] S2 security defender monitors attack behaviors and obtains network status.

[0054] The security defender uses the network intrusion detection system deployed on it to obtain abnormal dangerous attack information, and uses the dangerous attack information to model the characteristics of the attacked host node, that is, the damaged node i, as a binary θ i (t i ,risk i ), where t i Indicates the time of being attacked, risk i Indicates the initial risk level of being attacked.

[0055] The security defender calculates the dynamic risk level p of the damaged node i being attacked by the zombie node in the current time slot t i [t] is modeled as an exponential function Introducing Response Indicator Vector DR i[t] to describe the use of limited defense budget by security defenders D Assign to damaged node i resources.

[0056] In this step, at time slot t, the security defender obtains the network status of the previous time slot t-1 by observation, and constructs the observation state function S[t]={I[t-1],V[t-1],c A [t-1],r A [t-1],d D [t-1],c D [t-1],r D [t-1], TD[t-1], p[t-1]}, I[t-1] is the importance set of the host nodes of the network system at time slot t-1, V[t-1] is the vulnerability set of the host nodes of the network system at time slot t-1, c A [t-1] is the cost set of the dangerous attacker adopting the dangerous attack strategy at time slot t-1, r A [t-1] is the reward set obtained by the dangerous attacker at time slot t-1, d D [t-1] is the set of resources allocated by the security defender to the damaged host node at time slot t-1, c D [t-1] is the cost set of the security defender adopting the security defense strategy at time slot t-1, r D [t-1] is the reward obtained by the security defender at time slot t-1, TD[t-1] is the set of damaged host nodes at time slot t-1, and p[t-1] is the risk level set of the damaged host nodes being subjected to dangerous attacks at time slot t-1.

[0057] At time slot t, the dynamic change formula of the vulnerability k of host node i is defined as follows:

[0058]

[0059] in, is the initial value of vulnerability k of host node i, δ is the discount factor of the vulnerability time-varying function, The interactive result of dangerous attackers and security defenders allocating resources to host node i. is the amount of resources allocated by the security defender to the damaged host node i at time slot t, is the amount of resources allocated by the dangerous attacker to the host node i at the time slot t. m is used to control the nonlinear effect of the ratio of attack resources to defense resources. When m is large, the ratio of attack to defense in the formula has a more significant impact on the result. On the contrary, when m is small, the impact of the ratio of attack to defense is relatively gentle, and the change has a smaller impact on the result. The constant value It is the maximum vulnerability growth time of the system's host node.

[0060] The S3 security defense device is based on a dual deep reinforcement learning algorithm, which dynamically generates the optimal defense strategy based on the network status and feeds it back to the security defender.

[0061] The security defense device uses the observation state function S[t] of the network state at time slot t and uses the double deep reinforcement learning algorithm to obtain the network deception defense deployment strategy space set A[t]={d D [t],π[t]},d D [t] represents the resource allocation of the security defender at time slot t, and π[t] represents the network deception defense strategy selected by the security defender at time slot t, where π=Pπ i , i∈[0,7]}, where i=0 refers to firewall, i=1 refers to patching, i=2 refers to redefining encryption keys, i=3 refers to expelling damaged nodes, i=4 refers to deploying low-interaction honeypots and high-interaction honeypots, i=5 refers to setting honey information, i=6 refers to setting false keys, and i=7 refers to hiding the edges of the network topology. The utility of the security bundling defense strategy is calculated as DR i [t] represents the response indicator vector.

[0062] Where TD[t] is the set of host nodes damaged at time slot t, DR i [t] indicates whether the security defender takes action to protect the damaged host node i, p at time slot t i [t] is the risk level of the damaged host node i being attacked by a dangerous attack at time slot t, is the amount of resources allocated by the security defender to the damaged host node i at time slot t, It is the cost of the security defender adopting the security defense strategy for the damaged host node i at time slot t.

[0063] The dual-depth reinforcement learning algorithm of the present invention adopts two neural network architectures, including a main network and a target network. The main network selects the optimal execution action according to the current state, and the target network is used to evaluate the value of the action selected by the main network and calculate the target Q value matrix accordingly. The main network is updated and trained at each time step, and the target network synchronously updates its weights at fixed time intervals to track the changes of the main network. By reducing the frequent updates of the target network, dual-depth reinforcement learning effectively reduces the estimation bias, thereby significantly improving the stability and convergence speed of the strategy.

[0064] The Q value matrix is ​​Q a(S[t], A[t]), which represents the Q value when the security defender selects the network deception defense deployment strategy space set A[t] under the network state function S[t]. The specific Q value matrix expression is updated as follows:

[0065] Q a (S[t],A[t])=(1-α)Q a (S[t],A[t])+α(R D [t+1]+γ·Q b (S[t+1],max a∈A Q a (S[t+1],A[t])));

[0066] Among them, α is the learning factor, γ is the discount factor, 0≤α≤1, 0≤γ≤1, R D [t+1] and S[t+1] are the benefits and status of the security defender in time slot t+1 respectively, A[t] is the action of the security defender in time slot t, and the max(.) function is the maximum value function.

[0067] S4 Security Defender uses the best defense strategy to dynamically respond to attack behaviors to protect the network kill chain.

[0068] In this step, the security defender adopts the network deception defense strategy to respond to the surge of dangerous attacks. At time slot t, the network deception defense deployment strategy space set adopted by the security defender is defined as A[t] = {d D [t],dπ[t]},d D [t] represents the resource allocation of the security defender at time slot t, and π[t] represents the network deception defense strategy selected by the security defender at time slot t.

[0069] The security defense device also receives defense results feedback from security defenders to adjust the optimal defense strategy to form a closed-loop feedback mechanism.

[0070] The present invention repeats the above steps until the system fails or the time slot t is equal to the iteration termination condition T (for example, T=1000), and then stops iterating. Among them, whether the network system fails is determined according to the failure condition threshold. The network system failure here means that the network system cannot work normally or provide the expected service due to some reason during operation (for example, the node is attacked, data is leaked, or the number of nodes is insufficient to meet the operation requirements). That is, the system operation logic is terminated, and the attack and defense interaction can no longer continue. It will also cause the network service to be paralyzed or interrupted, affecting the actual business or functional requirements. The failure threshold conditions are as follows:

[0071]

[0072] Among them, Th1 and Th2 are the information importance leakage threshold and the available node threshold respectively, I i is the importance value of the host node i in the network system, |VN0| is the number of all host nodes in the initial network system when it is not attacked by dangerous attackers, |VN t | is the number of all available host nodes in the network system at time slot t, N i Indicates the safety, damage or expulsion status of the host node, N i =1 means the host node is damaged or expelled, otherwise N i =0.

[0073] If the network system fails or reaches the maximum number of iterations, it ends. Otherwise, it determines whether the dangerous attacker has successfully attacked. If the attack is unsuccessful, it continues with the current stage of the network kill chain. If the attack is successful, it enters the next stage of the network kill chain.

[0074] Based on this, see Figure 1 , Figure 2 The present invention also proposes an APT network kill chain dynamic defense system based on dual-depth reinforcement learning, which is built on the SDN architecture and adopts the above-mentioned APT network kill chain dynamic defense method based on dual-depth reinforcement learning, including:

[0075] Host nodes, including multiple host nodes with different importance and known vulnerabilities.

[0076] Dangerous attackers perform attack actions on host nodes according to the network kill chain.

[0077] Security defenders monitor attack behaviors and obtain network status, and use the best defense strategy to dynamically respond to attack behaviors to protect the network kill chain. The SDN controller serves as the core component of the security defender. The defender deploys a network intrusion detection system to monitor network traffic in real time and responds to changes in complex network topology through dynamic security defense devices.

[0078] The security defense device, based on the dual deep reinforcement learning algorithm, dynamically generates the optimal defense strategy based on the network status and feeds back to the security defender for the diverse attacks launched by dangerous APT attackers at different stages of the network kill chain. For example, it outputs the honeypot defense strategy, allowing the security defender to use the honeypot to capture dangerous attackers, analyze attack behaviors, and generate decoy data.

[0079] The security defense device of the present invention can obtain the network status in real time and generate security defense strategies. The specific device schematic diagram is shown in FIG. Figure 2As shown, it mainly includes an analysis module and a defense strategy generation module. The analysis module is responsible for data preprocessing of the real-time network traffic input into the system by the security defender, extracting and generating real-time network status information, including the importance and vulnerability value of each node, etc. The defense strategy generation module is based on a dual-depth reinforcement learning algorithm, combined with the threat level of dangerous attack behaviors and the network status, to dynamically generate the optimal defense strategy, and send it to the security defender for real-time protection. At the same time, the defense strategy generation module receives feedback on the defense results, forming a closed-loop feedback mechanism, and continuously improving the defense performance of the system through continuous learning and optimization. In the dynamic defense device of the present invention, the dual-depth reinforcement learning algorithm is its core component, which can optimize the allocation of computing resources and time resources and select the deployment of network deception defense strategies such as honeypots. The specific deployment scheme is common to the specific steps of the above method.

[0080] Finally, security defenders use the generated optimal defense strategy to dynamically respond to dangerous attacks, thereby achieving precise protection against APT attacks and improving the intelligence and adaptive capabilities of defense. The device integrates real-time traffic detection, automatic generation and distribution of defense strategies, and dynamic optimization of defense effects, effectively improving the intelligence level and response efficiency of network security systems, and significantly enhancing the ability to deal with complex network threats.

[0081] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A dynamic defense method for APT network kill chain based on dual deep reinforcement learning, characterized in that: Pre-configure a network system, the network system includes a host node, a security defender, a dangerous attacker, and a security defense device; The dangerous attacker executes attack behavior on the host node according to the network kill chain, the security defender monitors the attack behavior and obtains the network status, the security defense device is based on a dual deep reinforcement learning algorithm, combines the network status to dynamically generate an optimal defense strategy and feeds back to the security defender, and the security defender uses the optimal defense strategy to dynamically respond to the attack behavior to achieve protection of the network kill chain.

2. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 1, characterized in that: The dangerous attacker performs attack actions on the host node according to the network kill chain specifically referring to: The dangerous attacker gradually utilizes the zombie node set according to each stage of the network kill chain, selects targets for attack according to the characteristics of the host nodes adjacent to the zombie nodes; then allocates dangerous attack resources according to the attack budget and launches a dangerous attack. The set of successfully damaged host nodes will be converted into a new set of zombie nodes.

3. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 1, characterized in that: The security defender models the dynamic risk level of the host node being attacked by zombie nodes as an exponential function; a response indication vector is introduced to describe the resources allocated by the security defender to the damaged host node using a limited defense budget.

4. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 1, characterized in that: At time slot t, the security defender obtains the network status of the previous time slot t-1 by observation, and constructs the observation state function S[t] of time slot t, S[t] = {I[t-1], V[t-1], c A [t-1],r A [t-1],d D [t-1],c D [t-1],r D [t-1], TD[t-1], p[t-1]}, I[t-1] is the importance set of the host nodes of the network system at time slot t-1, V[t-1] is the vulnerability set of the host nodes of the network system at time slot t-1, c A [t-1] is the cost set of the dangerous attacker adopting the dangerous attack strategy at time slot t-1, r A [t-1] is the reward set obtained by the dangerous attacker at time slot t-1, d D [t-1] is the set of resources allocated by the security defender to the damaged host node at time slot t-1, c D [t-1] is the cost set of the security defender adopting the security defense strategy at time slot t-1, r D [t-1] is the reward obtained by the security defender at time slot t-1, TD[t-1] is the set of damaged host nodes at time slot t-1, and p[t-1] is the risk level set of the damaged host nodes being subjected to dangerous attacks at time slot t-1.

5. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 4, characterized in that: The security defense device uses the double deep reinforcement learning algorithm to obtain the network deception defense deployment strategy space set A[t]={d D [t],π[t]},d D [t] represents the resource allocation of the security defender at time slot t, π[t] represents the network deception defense strategy selected by the security defender at time slot t, and the utility of the security bundling defense strategy is calculated as DR i [t] represents the response indicator vector. Where TD[t] is the set of host nodes damaged at time slot t, DR i [t] indicates whether the security defender takes action to protect the damaged host node i, p at time slot t i [t] is the risk level of the damaged host node i being attacked by a dangerous attack at time slot t, is the amount of resources allocated by the security defender to the damaged host node i at time slot t, It is the cost of the security defender adopting the security defense strategy for the damaged host node i at time slot t.

6. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 1, characterized in that: The dual deep reinforcement learning algorithm includes a main network and a target network. The main network selects the optimal execution action according to the current state, and the target network is used to evaluate the value of the action selected by the main network and calculate the target Q value matrix accordingly; the main network is updated and trained at each time step, and the target network synchronously updates its weights at fixed time intervals to track the changes of the main network.

7. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 6, characterized in that: The Q value matrix is ​​Q a (S[t], A[t]), which represents the Q value when the security defender selects the network deception defense deployment strategy space set A[t] under the observation state function S[t] of the network state. The expression for updating the Q value matrix is ​​as follows: Q a (S[t],A[t])=(1-α)Q a (S[t],A[t])+α(R D [t+1]+γ·Q b (S[t+1],max a∈A Q a (S[t+1],A[t]))); Among them, α is the learning factor, γ is the discount factor, and R D [t+1] and S[t+1] are the benefits and status of the security defender in time slot t+1 respectively, A[t] is the action of the security defender in time slot t, and the max(.) function is the maximum value function.

8. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 1, characterized in that: The security defense device also receives defense results fed back from the security defender to adjust the optimal defense strategy to form a closed-loop feedback mechanism.

9. The APT network kill chain dynamic defense method based on dual deep reinforcement learning as claimed in claim 1, characterized in that: It also includes judging whether the network system fails according to a failure condition threshold, wherein the network system failure means that the operation logic of the network system is terminated and the attack and defense interaction can no longer be continued; the failure threshold condition is as follows: Among them, Th1 and Th2 are the information importance leakage threshold and the available node threshold respectively, I i is the importance value of the host node i in the network system, |VN0| is the number of all host nodes in the initial network system when it is not attacked by dangerous attackers, and |VN t | is the number of all available host nodes in the network system at time slot t, N i Indicates the safety, damage or expulsion status of the host node, N i =1 means the host node is damaged or expelled, otherwise N i =0. If the network system fails or reaches the maximum number of iterations, it ends. Otherwise, it is determined whether the dangerous attacker has successfully attacked. If the attack is unsuccessful, the current stage of the network kill chain continues. If the attack is successful, the next stage of the network kill chain is entered.

10. A dynamic defense system for APT network kill chain based on dual deep reinforcement learning, characterized in that: include: Host node, Dangerous attackers, who perform attack actions on the host node according to the network kill chain; A security defender monitors the attack behavior and obtains the network status, and dynamically responds to the attack behavior using the optimal defense strategy to achieve protection of the network kill chain; The security defense device, based on a dual deep reinforcement learning algorithm, dynamically generates an optimal defense strategy in combination with the network status and feeds back to the security defender.

Images