Honey array defense resource allocation optimization method based on self-game reinforcement learning

Abstract

The invention provides a honey array defense resource allocation optimization method based on self-game reinforcement learning, and relates to the technical field of network security defense. The optimization method comprises the following steps: deploying defense resources based on network topology, and establishing a mapping relationship between a host and the defense resources; performing attack and defense confrontation simulation based on the mapping relation to obtain an initial attack strategy and an initial defense strategy; constructing a Markov game model based on the initial attack strategy and the initial defense strategy; performing self-game reinforcement learning on attack and defense training of the Markov game model based on a depth deterministic strategy gradient algorithm; and carrying out value evaluation on the attack and defense training based on an Actor-Critic algorithm to obtain an optimal defense resource allocation strategy. By implementing the optimization method provided by the invention, the defense party and the attack party can perform self-game reinforcement learning, the defense strategy is dynamically adjusted in confrontation, and the defense effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of network security defense technology, and in particular to a honey array defense resource allocation optimization method based on self-game reinforcement learning. Background Art

[0002] Traditional static defense solutions pre-allocate network resources and configure defense mechanisms. Although they have the advantages of simple deployment and easy implementation, they lack dynamism and are easily breached when faced with new attack strategies or attack methods with changing locations.

[0003] The commonly used dynamic network defense resource allocation schemes today can make strategic and dynamic adjustments based on factors such as network conditions, threat intelligence, and defense targets, improving the adaptability of network resources, but they still have certain limitations. These schemes usually rely on pre-established rules and logic, and are pre-designed for specific network scenarios or threat types. When faced with new attack methods, they may be too dependent on past attack data and difficult to adjust in time, resulting in limited defense effectiveness.

[0004] Therefore, it is urgent to provide a solution to improve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a honey array defense resource allocation optimization method based on self-game reinforcement learning, so that the defender can dynamically adjust the defense strategy in the confrontation with the attacker.

[0006] The present invention provides a honey array defense resource allocation optimization method based on self-game reinforcement learning, which adopts the following technical solutions: Deploy defense resources based on network topology and establish a mapping relationship between hosts and defense resources; Perform attack and defense confrontation simulation based on the mapping relationship to obtain an initial attack strategy and an initial defense strategy; Construct a Markov game model based on the initial attack strategy and the initial defense strategy; Based on the deep deterministic policy gradient algorithm, self-game reinforcement learning is performed on the attack and defense training of the Markov game model; The attack and defense training is evaluated based on the Actor-Critic algorithm to obtain the optimal defense resource allocation strategy.

[0007] Optionally, the process of deploying defense resources based on network topology includes: defining the network topology, dividing the hosts into high-value hosts, ordinary hosts and honeypot hosts, setting the location of the honeypot host based on the location of the high-value host, and allocating computing resources, bandwidth resources, and security resources to the high-value hosts, ordinary hosts and honeypot hosts respectively.

[0008] Optionally, initial attack and defense strategies are obtained through offense-defense confrontation simulation based on the mapping relationship, including: the attacker scans network nodes, detects the positions of honeypots, and attacks the high-value hosts based on the mapping relationship to generate an initial attack strategy; the defender allocates computing resources, bandwidth resources, and security resources, and deploys honeypot hosts based on the mapping relationship to generate an initial defense strategy, and obtains the behavior information of the attacker by luring attacks on the honeypot hosts.

[0009] Optionally, a Markov game model is constructed based on the initial attack strategy and the initial defense strategy, including: obtaining the system state set, the action set of the defender, the action set of the attacker, and the probability of system state transition based on the initial attack strategy and the initial defense strategy; defining the cost function and reward function of the defender based on the system state set and the action set of the defender; defining the cost function and reward function of the attacker based on the system state set and the action set of the attacker; constructing a Markov game model based on the system state set, the action set of the defender, the action set of the attacker, the probability of system state transition, and the reward functions of the defender and the attacker.

[0010] Optionally, the reward functions of the defender and the attacker are respectively defined as: ; where is the reward obtained by the defender when performing action in system state , is the revenue value obtained by successfully protecting host , is the probability of the defender's defense failure, are computing resources, bandwidth resources, and security resources respectively, is the revenue of the attacker's behavior information obtained by the defender through the honeypot, is the system loss caused by the defender's failure to successfully defend, are weight coefficients respectively; ; where is the reward obtained by the attacker when performing action in system state , is the revenue value obtained by successfully attacking host , is the probability of successful attack, are computing resources, bandwidth resources, and security resources respectively, is the attack cost of the attacker, is the weight of the attack cost.

[0011] Optionally, self-play reinforcement learning is performed on the offense and defense training of the Markov game model based on the deep deterministic policy gradient algorithm, including: generating a defender policy and an attacker policy respectively based on the deep deterministic policy gradient algorithm, and calculating the immediate reward of the defender based on the cost function and reward function of the defender, and calculating the immediate reward of the attacker based on the cost function and reward function of the attacker.

[0012] Optionally, in the process of generating a defender policy and an attacker policy respectively based on the deep deterministic policy gradient algorithm, it includes: The generation formula of the defender policy is: ; Where is the defense action generated by the defender at time , is the system state at the current moment, is the attack action at the previous moment, is the policy function of the defender, is the parameter of the defender policy network; The generation formula of the attacker policy is: ; Where is the defense action generated by the attacker at time , is the system state at the current moment, is the defense action at the previous moment, is the policy function of the attacker, is the parameter of the attacker policy network.

[0013] Optionally, in the process of obtaining the optimal defense resource allocation strategy by evaluating the value based on the Actor-Critic algorithm for the offense and defense training, it includes: Initializing the parameters of the Actor network and the Critic network. The Actor network generates the defender action policy based on the deep deterministic policy gradient algorithm, and the Critic network estimates the Q value based on the Bellman equation to evaluate the long-term benefit of the action policy generated by the Actor in the current state; Continuously adjusting and optimizing the attacker policy and the defender policy based on the deep deterministic policy gradient algorithm and the long-term benefit. When the defender policy corresponds to the Nash equilibrium, it is the optimal defense resource allocation strategy, and output the optimal defense resource allocation strategy.

[0014] Optionally, when evaluating the long-term benefit of the strategy generated by the Actor by the Critic network, the evaluation formula is as follows: ; Among them, is the long-term benefit of the defender performing an action in the system state under and the long-term benefit, is the immediate reward, is the discount factor, is the expected value, is the Q value at the next moment, and the Q value is the cumulative benefit that can be obtained by continuing to adopt the strategy at a future moment.

[0015] An optimized method for allocating honeypot defense resources based on self-play reinforcement learning proposed by the present invention has the beneficial effects that: 1. By constructing a Markov model to design a reward function and combining the deep deterministic policy gradient algorithm and Actor-Critic, the present invention enables the defender and the attacker to perform self-play reinforcement learning, enabling the defender to make more intelligent and flexible defense decisions in a complex and changeable network environment.

[0016] 2. Through the deep deterministic policy gradient algorithm and the Actor-Critic algorithm, the present invention enables the defender to find the optimal defense resource allocation strategy, reasonably allocate the defense resources, not only improving the efficiency of network defense, but also reducing the operating cost of the defense system.

[0017] 3. By optimizing the position and quantity of honeypots in the defense strategy through self-play reinforcement learning, as well as calculating the reasonable allocation of computing resources, bandwidth resources, and security resources, the present invention enables the honeypot to be more efficient as an active defense means in an attack. Description of the Drawings

[0018] Figure 1 is a flowchart of an optimized method for allocating honeypot defense resources based on self-play reinforcement learning provided by an embodiment of the present invention. Detailed Embodiments

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein shall have the ordinary meaning as understood by those of ordinary skill in the art in the field to which the present invention belongs. The words such as "including" used herein mean that the elements or items appearing before the word cover the elements or items listed after the word and their equivalents, without excluding other elements or items.

[0020] An embodiment of the present invention provides an optimization method for honeypot defense resource allocation based on self-play reinforcement learning. Refer to Figure 1 , including: S1. Deploy defense resources based on the network topology and establish a mapping relationship between the hosts and the defense resources; S2. Conduct offense-defense confrontation simulation based on the mapping relationship to obtain an initial attack strategy and an initial defense strategy; S3. Construct a Markov game model based on the initial attack strategy and the initial defense strategy; S4. Conduct self-play reinforcement learning on the offense-defense training of the Markov game model based on the deep deterministic policy gradient algorithm; S5. Conduct value evaluation on the offense-defense training based on the Actor-Critic algorithm to obtain an optimal defense resource allocation strategy.

[0021] In some embodiments, during the execution of step S1, it includes: S1.1. Deploy defense resources based on the network topology; S1.2. Establish a mapping relationship between the hosts and the defense resources.

[0022] Specifically, when executing step S1.1 and deploying defense resources based on the network topology, it includes: defining the network topology, dividing the hosts into high-value hosts, ordinary hosts, and honeypot hosts, setting the positions of the honeypot hosts based on the positions of the high-value hosts, and respectively allocating computing resources, bandwidth resources, and security resources to the high-value hosts, ordinary hosts, and honeypot hosts.

[0023] Specifically, when executing step S1.2 and establishing a mapping relationship between the hosts and the defense resources, establish a mapping relationship between the ordinary hosts and high-value hosts and the honeypot hosts and the allocated computing resources, bandwidth resources, and security resources.

[0024] In some embodiments, during the execution of step S2, it includes: S2.1. Initialize the attack strategy; S2.2. Initialize the defense strategy.

[0025] Specifically, when executing step S2.1 and initializing the attack strategy, the attacker scans the network nodes based on the mapping relationship, detects the positions of the honeypot hosts, and attacks the high-value hosts to generate an initial attack strategy.

[0026] Specifically, when executing step S2.2 and initializing the defense strategy, the defender allocates computing resources, bandwidth resources, and security resources based on the mapping relationship, arranges the honeypot hosts to generate an initial defense strategy, and obtains the behavior information of the attacker based on the deception of the honeypot hosts.

[0027] In some embodiments, during the execution of step S3, it includes: S3.1. Obtain the parameters of the Markov game model based on the initial attack strategy and the initial defense strategy; S3.2. Define the cost function and the reward function of the defender; S3.3. Define the cost function and the reward function of the attacker; S3.4. Construct the Markov game model.

[0028] Specifically, when executing step S3.1 to obtain the parameters of the Markov game model based on the initial attack strategy and the initial defense strategy, obtain the system state set, the action set of the defender, the action set of the attacker, and the probability of system state transition based on the initial attack strategy and the initial defense strategy.

[0029] Specifically, when executing step S3.2 to define the cost function and the reward function of the defender, it includes: S3.2.1. Define the cost function of the defender; S3.2.2. Define the reward function of the defender.

[0030] Specifically, when executing step S3.2.1 to define the cost function of the defender, the cost function of the defender mainly comes from the allocation and use of resources, the deployment and maintenance of honeypots, and the operation and maintenance of the entire system. Therefore, the cost function of the defender is defined as: ; Where is the cost consumed by the defender when executing action in state , is the initial defense deployment cost, is the number of honeypots, is the maintenance cost required for each honeypot; is the maintenance cost of the defense system running over time , is the consumption of computing resources, bandwidth resources, and security resources.

[0031] Specifically, when executing step S3.2.2 to define the reward function of the defender, the reward function of the defender mainly consists of the corresponding benefits obtained by successfully protecting the host and the benefits obtained by the defender successfully luring the attacker through the honeypot. Therefore, the reward function of the defender is defined as: ; Where is the reward obtained by the defender when executing action in system state , For successfully protecting the host The obtained revenue value The probability of the defender's defense failure Are computing resources, bandwidth resources, and security resources respectively The information gain of the attacker's behavior obtained by the defender through the honeypot The system loss caused by the defender's failure to successfully defend Are weight coefficients respectively

[0032] Specifically, when performing step S3.3 and defining the cost function and reward function of the attacker, it includes: S3.2.1. Define the cost function of the attacker; S3.2.2. Define the reward function of the attacker.

[0033] Specifically, when performing step S3.2.1 and defining the cost function of the attacker, the cost function of the attacker mainly comes from the time, resources, and number of attempts to bypass the defense system, as well as the resources consumed by the attacker during the attack process. Therefore, the cost function of the attacker is defined as: ; Among them, Is the cost consumed by the attacker when performing action In state , Is the cost coefficient, used to control the weights of resources, time, etc. consumed by the attacker to bypass the defense system; And Are the comprehensive time and resource costs paid by the attacker to bypass the defense system, Is the attack attempt cost coefficient, Is the number of attempts made by the attacker to successfully attack the target.

[0034] Specifically, when performing step S3.2.2 and defining the reward function of the attacker, the reward function of the attacker mainly consists of bypassing the defense strategy and successfully attacking a high-value host, obtaining sensitive information or causing system damage from it, and at the same time avoiding being deceived by the honeypot. Therefore, the reward function of the attacker is defined as: ; Among them, Is the reward obtained by the attacker when performing action In the system state , Is the obtained revenue value for successfully attacking the host , Represents the probability of successful attack, Are computing resources, bandwidth resources, and security resources respectively represents the attack cost of the attacker, represents the weight of the attack cost.

[0035] Specifically, when performing step S3.4 to construct the Markov game model, the Markov game model is constructed based on the system state set, the defender's action set, the attacker's action set, the probability of system state transition, and the reward functions of the defender and the attacker.

[0036] In some embodiments, during the execution of step S4, it includes: S4.1. Generate the defender's strategy based on the deep deterministic policy gradient algorithm; S4.2. Generate the attacker's strategy based on the deep deterministic policy gradient algorithm; S4.3. Calculate the immediate reward of the defender based on the defender's cost function and reward function; S4.4. Calculate the immediate reward of the attacker based on the attacker's cost function and reward function.

[0037] Specifically, when performing step S4.1 to generate the defender's strategy based on the deep deterministic policy gradient algorithm, the generation formula of the defender's strategy is: ; where is the defense action generated by the defender at time , is the system state at the current moment, is the attack action at the previous moment, is the strategy function of the defender, is the parameter of the defender's policy network.

[0038] Specifically, when performing step S4.2 to generate the attacker's strategy based on the deep deterministic policy gradient algorithm, the generation formula of the attacker's strategy is: ; where is the defense action generated by the attacker at time , is the system state at the current moment, is the defense action at the previous moment, is the strategy function of the attacker, is the parameter of the attacker's policy network.

[0039] Specifically, when performing step S4.3 to calculate the immediate reward of the defender based on the defender's cost function and reward function, the calculation method is: ; where The immediate reward for the defender The reward function for the defender The cost function of the defender

[0040] Specifically, when performing step S4.4 and calculating the immediate reward of the attacker based on the attacker's cost function and reward function, the calculation method is as follows: ; Wherein, Is the immediate reward of the attacker Is the reward function of the attacker Is the cost function of the attacker

[0041] Furthermore, in the deep deterministic policy gradient algorithm, after each round of game between the defender and the attacker, they will evaluate their own strategy performance according to the current immediate reward. When the reward increases, they will tend to retain the current strategy. Otherwise, they will tend to adjust the strategy and reduce unfavorable actions.

[0042] In some embodiments, during the execution of step S5, it includes: S5.1. Perform value evaluation on the strategy based on the Actor-Critic algorithm; S5.2. Iteratively update to obtain the optimal defense resource allocation strategy.

[0043] Specifically, when performing step S5.1 and performing value evaluation on the strategy based on the Actor-Critic algorithm, it includes: initializing the parameters of the Actor network and the Critic network. The Actor network generates the action strategy of the defender based on the deep deterministic policy gradient algorithm, and the Critic network estimates the Q value based on the Bellman equation to evaluate the long-term benefit of the action strategy generated by the Actor in the current state.

[0044] Furthermore, the evaluation formula is as follows: ; Wherein, Is the long-term benefit of the defender when executing the action Under the system state , Is the immediate reward Is the discount factor Is the expected value Is the Q value at the next moment. The Q value is the cumulative benefit that can be obtained by continuing to adopt the strategy In the future.

[0045] Specifically, when performing step S5.2 to iteratively update and obtain the optimal defense resource allocation strategy, the attacker's strategy and the defender's strategy are continuously adjusted and optimized based on the deep deterministic policy gradient algorithm and the long-term reward. When the defender's strategy corresponds to the Nash equilibrium, it is the optimal defense strategy, and the optimal defense strategy is output.

[0046] Further, when adjusting and optimizing the attacker's strategy and the defender's strategy, the following formula is used: ; where is the gradient of the strategy, which is the gradient of the parameters of the strategy with respect to the objective function , is the expected value, is the strategy, is the executed action, is the system state, is the long-term reward when the defender executes the action in the system state .

[0047] In fact, updating the parameters of the strategy along the direction of the gradient can gradually improve the performance of the strategy until the defender obtains the optimal defense resource allocation strategy. Then, based on the optimal defense resource allocation strategy, computing resources, bandwidth resources, and security resources are reasonably allocated, and the deployment and adjustment of the honeypot array are guided, so as to achieve the best defense effect with the least resource consumption.

[0048] Although the embodiments of the present invention have been described in detail above, it is obvious to those skilled in the art that various modifications and changes can be made to these embodiments. However, it should be understood that such modifications and changes are all within the scope and spirit of the present invention described in the claims. Moreover, the present invention described herein can have other embodiments and can be implemented or realized in various ways.

Claims

1. A honey array defense resource allocation optimization method based on self-game reinforcement learning, characterized in that: The following steps are involved: Deploy defense resources based on network topology and establish a mapping relationship between hosts and defense resources; Perform attack and defense confrontation simulation based on the mapping relationship to obtain an initial attack strategy and an initial defense strategy; Construct a Markov game model based on the initial attack strategy and the initial defense strategy; Based on the deep deterministic policy gradient algorithm, self-game reinforcement learning is performed on the attack and defense training of the Markov game model; The attack and defense training is evaluated based on the Actor-Critic algorithm to obtain the optimal defense resource allocation strategy.

2. According to the method for optimizing honey array defense resource allocation based on self-game reinforcement learning in claim 1, it is characterized in that: The process of deploying defense resources based on network topology includes: Define the network topology, divide the hosts into high-value hosts, ordinary hosts and honeypot hosts, set the location of the honeypot host based on the location of the high-value hosts, and allocate computing resources, bandwidth resources and security resources to the high-value hosts, ordinary hosts and honeypot hosts respectively.

3. According to the method for optimizing honey array defense resource allocation based on self-game reinforcement learning in claim 2, it is characterized in that: Based on the mapping relationship, an attack and defense confrontation simulation is performed to obtain an initial attack strategy and an initial defense strategy, including: The attacker scans the network nodes based on the mapping relationship, detects the location of the honeypot, and attacks the high-value host to generate an initial attack strategy; The defender allocates computing resources, bandwidth resources, and security resources based on the mapping relationship, and arranges the honeypot host to generate an initial defense strategy, and obtains the attacker's behavior information based on the honeypot host deception attack.

4. According to the method for optimizing honey array defense resource allocation based on self-game reinforcement learning in claim 1, it is characterized in that: A Markov game model is constructed based on the initial attack strategy and the initial defense strategy, including: Based on the initial attack strategy and the initial defense strategy, a system state set, a defender's action set, an attacker's action set, and the probability of system state transition are obtained; Define the defender's cost function and reward function based on the system state set and the defender's action set; Define the attacker's cost function and reward function based on the system state set and the attacker's action set; A Markov game model is constructed based on the system state set, the defender's action set, the attacker's action set, the probability of system state transition, and the defender's and attacker's reward functions.

5. The method for optimizing honey array defense resource allocation based on self-game reinforcement learning according to claim 4 is characterized in that: The reward functions of the defender and attacker are defined as: ； in, For the defender in the system state Next action Rewards received, To successfully protect the host The value of the income obtained, is the probability of the defender’s defense failure, They are computing resources, bandwidth resources, and security resources. The attacker's behavior information benefit obtained by the defender through the honeypot. System losses caused by the defender's failure to successfully defend. are the weight coefficients respectively; ； in, The attacker is in the system state Next action Rewards received, To successfully attack the host The value of the income obtained, is the probability of successful attack, They are computing resources, bandwidth resources, and security resources. is the attack cost of the attacker, is the weight of the attack cost.

6. The method for optimizing honey array defense resource allocation based on self-game reinforcement learning according to claim 4 is characterized in that: Based on the deep deterministic policy gradient algorithm, the attack and defense training of the Markov game model is self-game reinforced learning, including: Based on the deep deterministic policy gradient algorithm, the defender strategy and the attacker strategy are generated respectively, and the defender's immediate reward is calculated based on the defender's cost function and reward function, and the attacker's immediate reward is calculated based on the attacker's cost function and reward function.

7. The method for optimizing honey array defense resource allocation based on self-game reinforcement learning according to claim 6, characterized in that: The process of generating the defender strategy and attacker strategy based on the deep deterministic policy gradient algorithm includes: The generation formula of the defender strategy is: ； in, For the defender in time Generated defensive actions, is the system status at the current moment, is the attack action of the previous moment, is the defender’s strategy function, are the parameters of the defense strategy network; The attacking party's strategy generation formula is: ； in, For the attacker at time Generated defensive actions, is the system status at the current moment, For the defensive action of the previous moment, is the attacker’s strategy function, are the parameters of the attacker’s strategy network.

8. The method for optimizing honey array defense resource allocation based on self-game reinforcement learning according to claim 6, characterized in that: The process of evaluating the attack and defense training based on the Actor-Critic algorithm to obtain the optimal defense resource allocation strategy includes: Initialize the parameters of the Actor network and the Critic network. The Actor network generates the defender's action strategy based on the deep deterministic policy gradient algorithm, and the Critic network estimates the Q value based on the Bellman equation to evaluate the long-term benefits of the action strategy generated by the Actor in the current state. Based on the deep deterministic policy gradient algorithm and long-term benefits, the attacker's strategy and the defender's strategy are continuously adjusted and optimized. When the defender's strategy corresponds to the Nash equilibrium, it is the optimal defense resource allocation strategy, and the optimal defense resource allocation strategy is output.

9. The method for optimizing honey array defense resource allocation based on self-game reinforcement learning according to claim 8, characterized in that: Based on the Critic network, the long-term benefits of the strategy generated by the Actor in the current state are evaluated. The evaluation formula is as follows: ； in, For the defender in the system state Next action long-term benefits, It’s an instant reward. is the discount factor, is the expected value, is the Q value of the next moment, and the Q value is to continue to adopt the strategy in the future The accumulated income that can be obtained later.

Images