Network security defense method based on hierarchical reinforcement learning

Abstract

The invention discloses a hierarchical reinforcement learning-based network security defense method, which comprises the following steps of: firstly, constructing a target network training sub-agent to obtain a sub-agent library, and on the basis of the sub-agent library, constructing another target network training main agent, and selecting the sub-agent in the sub-agent library for defense according to an environment when the main agent is used for defense. After training of the sub-agents and the main agent is finished, defense can be carried out by combining hierarchical reinforcement learning, experience bars are obtained in the defense process and are put into an experience buffer area, strategy optimization is carried out, an estimated value network of the main agent is updated, a network security defense strategy is further optimized, various challenges in the field of network security can be better coped with, and the network security defense effect is improved. And the overall network security defense capability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and more specifically, relates to a network security defense method based on hierarchical reinforcement learning. Background Art

[0002] Network information systems are now deeply integrated into people's daily lives, government operations, industrial development, and academic research, and the importance of network security has become increasingly prominent. With the rapid development of science and technology, various network security threats are also growing, especially Advanced Persistent Threats (APTs), which have brought severe challenges to various fields. At the same time, network information systems are becoming more and more complex and interconnected, which has exacerbated the severity and destructiveness of network security issues. Therefore, ensuring network security has become a major issue that needs to be solved urgently, and automated network security defense is an important measure to ensure network security.

[0003] Currently, automated network security defense faces many challenges. In order to effectively defend against malicious attacks, continuous monitoring, traffic analysis, and vulnerability patching are required, which is not only costly but also requires a lot of manpower. In addition, there is an obvious asymmetry in the field of network security. Attackers can often launch attacks easily, while defenders face great difficulties, which increases the difficulty of network security defense.

[0004] Traditional network security defense measures, such as firewalls, intrusion detection systems (IDS) and intrusion prevention systems (IPS), have shown certain limitations in the rapid development of the Internet. With the increasing complexity of the network environment and the increasing security risks, the development of efficient and intelligent automated defense methods has become an important research direction in the field of network security. Automated defense mechanisms can effectively monitor network activities, detect and prevent various types of network attacks in a timely manner, and provide strong protection for the safe operation of the network environment. However, how to automatically, accurately and effectively select the best defense strategy in this complex situation has become a major challenge that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a network security defense method based on hierarchical reinforcement learning. When facing attackers with different attack modes, the main intelligent agent identifies the attack mode and selects the most appropriate pre-trained reinforcement learning sub-agent for defense, so as to better cope with various challenges in the field of network security and enhance the overall network security defense capabilities.

[0006] In order to achieve the above-mentioned object of the invention, the network security defense method based on hierarchical reinforcement learning of the present invention is characterized by comprising the following steps:

[0007] (1) Sub-agent training

[0008] For M different attack modes, the corresponding sub-agents are trained to select the optimal defense actions in the current environment state under each attack mode, and then stored in the sub-agent library; the sub-agent is an evaluation network Enet child , whose action value function is Q childEvaluat (s, a; ω), select its action value function Q according to the current environment state s childEvaluati (s, a; ω) is the maximum output of defense action a, where ω is the evaluation network Enet child Network parameters;

[0009] To train the sub-agent, construct the target action value function as Q childTarge (s, a; ω′), with a target network Tnet child Implementation, where ω′ is the target network Tnet child Network parameters;

[0010] The training process of each attack mode sub-agent is as follows:

[0011] 1.1) Set an attack mode and treat it as part of the environment, then initialize the sub-agent’s experience buffer D child The capacity is N child , initialize the valuation network Enet child Initialize the target network Tnet with the network parameters ω child The network parameters ω′;

[0012] The sub-agent performs N rounds of interactions with the environment. Each round of interaction consists of T time steps. At each time step t:

[0013] 1.2) The state of the environment is s, and the sub-agent has a probability of ∈ to randomly select a defense action a in the sub-agent's defense action space, and has a probability of 1-∈ according to the action value function Q childEvaluati (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluat (s, a; ω);

[0014] 1.3) After selecting the defensive action, the sub-agent performs the defensive action a in the environment, and the environment returns a reward r and the next state s′ of the environment to the sub-agent. The experience condition d is composed of the current state s, the defensive action a, the reward r and the next state s′. child =(s, a, r, s′, t), stored in the experience buffer D child middle;

[0015] 1.4) When the experience buffer D child Medium Experience child After reaching the set number, each time step t is stored in the experience bar d child After that, from the experience buffer D child Randomly select a batch of experience conditions child , used to update the valuation network Enet child The network parameter ω is the updated action value function Q childEvaluation (s, a; ω):

[0016] First, calculate each empirical term d child The target value y′ is:

[0017]

[0018] Where a′=argmax a′ Q childEvaluat (s′, a′; ω) is based on the action value function Q childEvaluation (s′, a′; ω) The maximum value selects the defense action a′ in the defense action space of the sub-agent;

[0019] Then calculate the difference L child :

[0020] L child =(y′-Q childEvaluat (s, a; ω) 2

[0021] All experience items in this batch child The difference L child Take the average as the loss value, minimize the loss value, and update the valuation network Enet through gradient descent child The network parameters ω;

[0022] At the same time, every C time steps, the target network Tnet child The network parameter ω′ is set as the estimation network Enet child The network parameter ω is the empirical buffer D child When the amount is full, the number of experience points reaches N child , save the new experience bar childWhen the experience bar first deposited is removed, the experience bar first deposited is then removed;

[0023] After N rounds of interaction, the sub-agent is stored in the sub-agent library;

[0024] (2) Main agent training

[0025] For M different attack modes, a master agent is trained to identify the attack mode in the current environment. The master agent is an estimation network Enet main , whose action value function is Q mainEvaluation (s, A; θ), select its action value function Q according to the current environment state s mainEvaluation (s, A; θ) The largest sub-agent selects action A to output, where θ is the valuation network Enet main Network parameters;

[0026] To train the main agent, construct the target action value function as Q mainTarge (s, A; θ′), with a target network Tnet main Implementation, where θ′ is the target network Tnet main Network parameters;

[0027] The training process of the main agent is:

[0028] 2.1) Initialize the main agent's experience buffer D main The capacity is N main , initialize the valuation network Enet main Initialize the target network Tnet with the network parameters θ main The network parameters θ′;

[0029] The main agent performs N rounds of interaction with the environment. At the beginning of each round of interaction, it selects one of the M attack modes and regards it as part of the environment until the end of this round. Each round of interaction contains T time steps. At each time step t:

[0030] 2.2) The state of the environment is s, and the main agent has a probability of ∈ to randomly select a sub-agent in the sub-agent selection action space of the main agent to select action A, and has a probability of 1-∈ according to the action value function Q mainEvaluation (s, A; θ) The maximum value in the sub-agent selection action space of the main agent selects the sub-agent to select action A, that is, A = argmax A Q mainEvaluation (s, A; θ);

[0031] 2.3) After selecting the agent to select the action, the main agent executes the sub-agent selection action A and selects the corresponding sub-agent. The selected sub-agent is based on the environment state s and the action value function Q childEvaluatio (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluat (s, a; ω) and execute, the environment returns to the main agent the reward r and the next environment state s′ of the environment, and the experience condition d main =(s, A, r, s′, t), stored in the experience buffer D main middle;

[0032] 2.4) When the experience buffer D main Medium Experience main After reaching the set number, each time step t is stored in the experience bar d main After that, from the experience buffer D main Randomly select a batch of experience conditions mian , used to update the valuation network Enet main The network parameters θ are updated action value function Q childEvalua (s, A; θ):

[0033] First, calculate each empirical term d main The target value Y′ is:

[0034]

[0035] Where A′=argmax A′ Q childEvaluati (s′, A′; θ) is based on the action value function Q childEvaluati (s′, A′; θ) The maximum value is in the sub-agent selection action space of the main agent. The sub-agent selects the action A′;

[0036] Then calculate the difference:

[0037] L main =(Y′-Q mainEvaluation (s, A; θ)) 2

[0038] All experience items in this batch main The difference L main Take the average as the loss value, minimize the loss value, and update the valuation network Enet through gradient descent main The network parameters θ;

[0039] At the same time, every C time steps, the target network Tnet main The network parameters are set as θ′, which is the estimated network Enet main The network parameter θ, when the experience buffer D main When the amount is full, the number of experience points reaches N main , save the new experience bar main When the experience bar first deposited is removed, the experience bar first deposited is then removed;

[0040] After N rounds of interaction, the main agent is trained and saved;

[0041] (3) Layered reinforcement learning for defense

[0042] 3.1) Identify the attacker pattern in the current environment and send the best sub-agent to defend

[0043] The recognition method is to input the state s of the current environment into the main agent, and then mainEvaluation (s, A; θ) The maximum value in the sub-agent selection action space of the main agent selects the sub-agent to select action A, that is, A = argmax A Q mainEvaluation (s, A; θ);

[0044] After selecting the agent selection action, the main agent executes the sub-agent selection action A and selects the corresponding sub-agent;

[0045] 3.2) Execution of defensive actions

[0046] The selected sub-agent is based on the environment state s and the action value function Q childEvaluation (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluation (s, a; ω) and execute, the environment returns to the main agent the reward r and the next state s′ of the environment;

[0047] 3.3) Strategy Optimization

[0048] The experience main =(s, A, r, s′, t) is stored in the experience buffer D main , update the valuation network Enet according to the method in step 2.4) main The network parameters θ.

[0049] The object of the invention of the present invention is achieved in this way:

[0050] The network security defense method based on hierarchical reinforcement learning of the present invention first trains sub-agents by constructing a target network to obtain a sub-agent library. On this basis, another target network is constructed to train the main agent. When used for defense, the sub-agent in the sub-agent library is selected for defense according to the environment. After the training of the sub-agent and the main agent is completed, the defense can be combined with hierarchical reinforcement learning. The experience bars obtained during the defense process are put into the experience buffer and the strategy is optimized. The valuation network of the main agent is updated, and the network security defense strategy is further optimized to better cope with various challenges in the field of network security and enhance the overall network security defense capability.

[0051] The present invention also has the following beneficial effects:

[0052] 1. The present invention can dynamically respond to changes in the network environment. Since the network environment is complex and changeable, and there are various attack methods, it is necessary to adopt dynamic learning defense methods to deal with these attacks. The present invention combines reinforcement learning to dynamically learn network features and use flexible defense methods. For example, for M types of attacker patterns, we can use M different defense strategies to deal with them;

[0053] 2. The present invention can carry out targeted network defense. The efficiency of defense depends on the accuracy of attack identification. The main intelligent agent in the present invention can dynamically identify the attack mode and continuously evolve, so as to accurately identify the attack mode and select the corresponding sub-intelligent agent to perform defense actions. Since the identification process of the present invention is accurate and targeted, it can significantly improve the defense performance. ;

[0054] 3. The present invention has good scalability, which is reflected in two aspects: first, the action value function of the main agent can be fitted using different types of neural networks, such as LSTM and Pointer Network. Second, the number of pre-trained sub-agent libraries can be adjusted according to the judgment of the number of attacker patterns in the network, thereby achieving flexible expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a schematic diagram of the simulation environment;

[0056] Figure 2 It is a flowchart of a specific implementation method of the network security defense method based on hierarchical reinforcement learning of the present invention;

[0057] Figure 3 It is a specific schematic diagram of the network security defense architecture;

[0058] Figure 4 It is the training flowchart of the sub-agent;

[0059] Figure 5It is the training flowchart of the main agent. DETAILED DESCRIPTION

[0060] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0061] Cybersecurity has a huge impact on business and industry, such as the ByteDance database leak and the damage to drone systems on the battlefield in Israel. At the same time, cybersecurity issues are becoming increasingly serious, and we need to pay more attention to this area. However, cybersecurity issues are difficult to deal with because the network environment is complex and the attack methods are diverse and difficult to grasp. We urgently need new tools to meet these challenges.

[0062] Reinforcement learning is widely used to solve sequential decision-making problems, which are similar in network security scenarios, especially in terms of dynamic environmental changes. Reinforcement learning has been successful in fields such as games and drone control. Inspired by this, this invention aims to use reinforcement learning to solve dynamic defense problems. In addition, for a variety of network attack methods, we can also use reinforcement learning to identify attack patterns and implement corresponding defense measures, thereby improving the overall network security defense capabilities.

[0063] In this embodiment, if Figure 1 As shown, there are 3 subnets and 13 machines. In subnet 1, there are 5 user hosts; in subnet 2, there are 3 enterprise servers and 1 defense host; in subnet 3, there are 3 operation hosts and 1 operation server.

[0064] Each machine exposes some network services that other machines can connect to, and there may also be vulnerabilities that can be exploited. However, there is a firewall in this network, and machines in subnet 1 cannot directly connect to machines in subnet 3. Only the operation host has the authority to access the operation server, and the security of the operation server is crucial to the entire manufacturing process.

[0065] Figure 2 This is a flowchart of a specific implementation of the network security defense method based on hierarchical reinforcement learning of the present invention. Figure 3 Schematic diagram of the corresponding network security defense architecture. In this embodiment, Figure 2 , 3 As shown, the network security defense method based on hierarchical reinforcement learning of the present invention includes the following steps:

[0066] Step S1: Sub-agent training

[0067] For M different attack modes, the corresponding sub-agents are trained to select the optimal defense actions in the current environment state under each attack mode, and then stored in the sub-agent library; the sub-agent is an evaluation network Enet chil , whose action value function is Q childEvaluati (s, a; ω), select its action value function Q according to the current environment state s childEvaluat (s, a; ω) is the maximum output of defense action a, where ω is the evaluation network Enet child network parameters.

[0068] In this embodiment, all attack modes are abstracted, and corresponding D3QN (Dueling Double Deep Q-Learning Network) sub-agents are trained for each attack mode, and then these sub-agents are stored in a sub-agent library.

[0069] In this embodiment, the attack mode in the network is abstracted into two different types, namely M=2. The first attack mode is to randomly diffuse in the network, randomly attacking one node from another until the operating server in the network is obtained. The second attack mode is based on the diffusion of prior information, which includes the node distribution in the network and the optimal attack path.

[0070] To train the sub-agent, construct the target action value function as Q childTarge (s, a; ω′), with a target network Tnet chi Implementation, where ω′ is the target network Tnet child network parameters.

[0071] In this embodiment, if Figure 4 As shown in Figure 2, the training process of the sub-agent for each attack mode is as follows:

[0072] Step S1.1: Set the attack mode and initialize network parameters

[0073] Set an attack mode and treat it as part of the environment, then initialize the sub-agent's experience buffer D child The capacity is N child , initialize the valuation network Enet child Initialize the target network Tnet with the network parameters ω child The network parameters ω′.

[0074] In this embodiment, the sub-agent experience buffer D is initialized. chil The capacity is 10 5 . Valuation Network Enet childis a neural network, where the input layer has 52 neurons, the output layer has 54 neurons, the number of neurons in the middle hidden layer is 256, 512, 1024, 512 and 256 respectively, and the network parameters ω are randomly initialized. Target network Tnet child With Valuation Network Enet child The structure is the same.

[0075] The sub-agent performs N rounds of interaction with the environment, and each round of interaction includes T time steps. In this embodiment, N=50,000, T=100.

[0076] At each time step t:

[0077] Step S1.2: Select defense action a according to the ∈-greedy strategy

[0078] The state of the environment is s, and the sub-agent has probability ∈ to randomly select a defense action a in the sub-agent's defense action space, and has probability 1-∈ according to the action value function Q childEvaluation (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluatio (s, a; ω), that is, the ∈-greedy strategy is adopted to select the defense action a.

[0079] The ∈-greedy strategy is a commonly used strategy in reinforcement learning to balance exploration and exploitation. In this strategy, ∈ represents the probability of exploration, that is, random actions are selected with a certain probability, and 1-∈ represents the probability of exploitation, that is, the action currently considered optimal is selected with a certain probability.

[0080] In this embodiment, if Figure 3 As shown, the state of the environment is s, which represents the set of descriptions of each machine. There are 13 machines (hosts) in total, and each machine is described by four bits. The first two bits represent the malicious attack on the machine: 00 means no malicious attack, 10 means the machine is being scanned, and 11 means the machine has been maliciously exploited. The last two bits indicate the degree of damage to the machine: 00 means the machine is not damaged, 10 means the machine state is unknown, 01 means the machine has been obtained user privileges, and 11 means the machine has been obtained administrator privileges. Therefore, the entire state is represented by 52 bits.

[0081] In this embodiment, the ∈-greedy strategy is an ∈-greedy strategy with decreasing detection rate. The exploration probability ∈ is initialized to 0.95 and is updated once ∈=∈×0.99 after every 1000 rounds of interaction.

[0082] In this embodiment, the defense action space of the sub-agent is shown in Table 1.

[0083]

[0084]

[0085] Table 1

[0086] Among the defense actions in Table 1, the first two are non-directional, while the last four require a specific machine. Therefore, the sub-agent has a total of 54 independent defense actions, which can be represented by a one-hot bit with a length of 54. The sub-agent has a probability of ∈ to randomly select a defense action a from the defense action space of the sub-agent, i.e., the 54 defense actions. There is a probability of 1-∈ according to the action value function Q childEvaluati The maximum of (s, a; ω) selects the defense action a in the defense action space of the sub-agent.

[0087] In this embodiment, the action value function Q childEvaluat The output of (s, a; ω) is a vector of length 54, which represents the value of each action in the current state.

[0088] Step S1.3: The sub-agent performs defensive action a and obtains experience d child And store it in the experience buffer D child

[0089] After selecting a defensive action, the sub-agent performs a defensive action a in the environment, and the environment returns a reward r and the next state s′ of the environment to the sub-agent. The experience condition d child =(s, a, r, s′, t), stored in the experience buffer D child middle.

[0090] In this embodiment, the reward r is designed according to the following principle: when the attacker controls the host, the sub-agent will be punished, and the closer the controlled machine is to the operation server, the greater the punishment. When the attacker controls the operation server, the sub-agent will face a greater punishment. In addition, when the sub-agent restarts a machine, it will also be punished.

[0091] Step S1.4: Update the valuation network Enet child The network parameters ω

[0092] When the experience buffer D child Medium Experience child After reaching the set number, each time step t is stored in the experience bar d chi After that, from the experience buffer D child Randomly select a batch of experience conditions child , used to update the valuation network Enet childThe network parameter ω is the updated action value function Q childEvaluat (s, a; ω).

[0093] In this embodiment, the number of experience items is 3000. child After reaching 3000, the valuation network Enet is updated at each time step t chil The network parameters ω. The specific update steps are:

[0094] First, calculate each empirical term d child The target value y′ is:

[0095]

[0096] Where a′=argmax a′ Q childEvaluat (s′, a′; ω) is based on the action value function Q childEvaluation The maximum value of (s′, a′; ω) selects the defensive action a′ in the sub-agent defensive action space. That is, the target value y′ is updated in the time step less than T using the formula y′=r+Q childTarge (s′, a′; ω′), when the time step t=T, then y′=r.

[0097] Then calculate the difference L chi :

[0098] L child =(y′-Q childEvaluation (s, a; ω) 2

[0099] All experience items in this batch child The difference L child Take the average as the loss value, minimize the loss value, and update the valuation network Enet through gradient descent child The network parameters ω.

[0100] At the same time, every C time steps, the target network Tnet chil The network parameter ω′ is set as the estimation network Enet child The network parameter ω is the empirical buffer D chil When the amount is full, the number of experience points reaches N child , save the new experience bar child When the experience bar that was first deposited is removed, the next experience bar that was deposited is removed.

[0101] In this embodiment, every C = 5000 time steps, the target network Tnet child The network parameter ω′ is set as the estimation network Enet child The network parameters ω.

[0102] After N rounds of interaction, the sub-agent is stored in the sub-agent library;

[0103] Step S2: Main agent training

[0104] For M different attack modes, a master agent is trained to identify the attack mode in the current environment. The master agent is an estimation network Enet main , whose action value function is Q mainEvaluation (s, A; θ), select its action value function Q according to the current environment state s mainEvaluation (s, A; θ) The largest sub-agent selects action A to output, where θ is the valuation network Enet main network parameters.

[0105] To train the main agent, construct the target action value function as Q mainTarge (s, A; θ′), with a target network Tnet main Implementation, where θ′ is the target network Tnetm ain network parameters.

[0106] In this embodiment, the main agent experience buffer D is initialized. main The capacity is 10 5 . Valuation Network Enet main is a neural network, where the input layer has 52 neurons, the output layer has 2 neurons, the number of neurons in the middle hidden layer is 256, 512, 1024, 512 and 256 respectively, and the network parameters θ are randomly initialized. Target network Tnet main With Valuation Network Enet main The structure is the same.

[0107] The main agent performs N rounds of interaction with the environment, and each round of interaction includes T time steps. In this embodiment, N = 50,000, T = 100.

[0108] In this embodiment, if Figure 5 As shown in Figure 2, the training process of the main agent is:

[0109] Step S2.1: Initialize network parameters and select an attack mode in each round

[0110] Initialize the main agent's experience buffer D main The capacity is N main , initialize the valuation network Enet main Initialize the target network Tnet with the network parameters θ main The network parameters θ′;

[0111] The main agent performs N rounds of interaction with the environment. At the beginning of each round of interaction, it selects one of the M attack modes and regards it as part of the environment until the end of this round. Each round of interaction contains T time steps. In this embodiment, one of the M=2 attack modes is selected.

[0112] At each time step t:

[0113] Step S2.2: Select the agent to take action A according to the ∈-greedy strategy

[0114] The state of the environment is s, and the main agent has a probability of ∈ to randomly select a sub-agent in the sub-agent selection action space of the main agent to select action A, and has a probability of 1-∈ according to the action value function Q mainEvaluation (s, A; θ) The maximum value in the sub-agent selection action space of the main agent selects the sub-agent to select action A, that is, A = argmax A Q mainEvaluation (s, A; θ), that is, the ∈-greedy strategy is adopted to select the agent to take action A.

[0115] In this embodiment, the ∈-greedy strategy is a ∈-greedy strategy with decreasing detection rate, just like the sub-agent training. The exploration probability ∈ is initialized to 0.95 and is updated once ∈=∈×0.99 after every 1000 rounds of interaction. At the same time, there are only two possibilities for the sub-agent to select action A: 00 means selecting the first sub-agent, and 10 means selecting the second sub-agent. Therefore, the sub-agent selection action A of the main agent can be represented by a one-hot vector of length 2.

[0116] Step S2.3: The main agent executes the action A selected by the sub-agent and obtains the experience condition d main And store it in the experience buffer D main

[0117] After the agent is selected, the main agent executes the sub-agent selection action A and selects the corresponding sub-agent. The selected sub-agent selects the action value function Q according to the environment state s. childEvaluat (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluatio (s, a; ω) and execute, the environment returns to the main agent the reward r and the next environment state s′ of the environment, and the experience condition d main =(s, A, r, s′, t), stored in the experience buffer D main middle.

[0118] Step S2.4: Update the valuation network Enetmain The network parameters θ

[0119] When the experience buffer D main Medium Experience main After reaching the set number, each time step t is stored in the experience bar d main After that, from the experience buffer D main Randomly select a batch of experience conditions mian , used to update the valuation network Enet main The network parameters θ are updated action value function Q childEvaluat (s, A; θ).

[0120] In this embodiment, the number of experience items is 3000. main After reaching 3000, the valuation network Enet is updated at each time step t main The network parameters θ. The specific update steps are:

[0121] First, calculate each empirical term d main The target value Y′ is:

[0122]

[0123] Where A′=argmax A′ Q childEvaluation (s′, A′; θ) is based on the action value function Q childEvaluati (s′, A′; θ) The maximum value is in the sub-agent selection action space of the main agent. The sub-agent selects the action A′. That is, the new target value Y′ adopts the formula Y′=r+Q in the time step less than T. mainTarget (s′, A′; θ′), when the time step t=T, then Y′=r.

[0124] Then calculate the difference:

[0125] L main =(Y′-Q mainEvaluation (s, A; θ)) 2

[0126] All experience items in this batch main The difference L main Take the average as the loss value, minimize the loss value, and update the valuation network Enet through gradient descent main The network parameters θ;

[0127] At the same time, every C time steps, the target network Tnet main The network parameters are set as θ′, which is the estimated network Enet main The network parameter θ, when the experience buffer D mainWhen the amount is full, the number of experience points reaches N main , save the new experience bar main When the experience bar that was first deposited is removed, the next experience bar that was deposited is removed.

[0128] In this embodiment, every C = 5000 time steps, the target network Tnet main The network parameters are set as θ′, which is the estimated network Enet main The network parameters θ.

[0129] After N rounds of interaction, the main agent is trained and saved.

[0130] Step S3: Hierarchical reinforcement learning for defense

[0131] Once the training of the sub-agent and the main agent is completed, you can start combining hierarchical reinforcement learning for defense.

[0132] Step S3.1: Identify the attacker pattern in the current environment and send the best sub-agent to defend

[0133] The recognition method is to input the state s of the current environment into the main agent, and then mainEvaluation (s, A; θ) The maximum value in the sub-agent selection action space of the main agent selects the sub-agent to select action A, that is, A = argmax A Q mainEvaluation (s, A; θ).

[0134] After selecting the agent selection action, the main agent executes the sub-agent selection action A to select the corresponding sub-agent.

[0135] In this embodiment, the sub-agent selection action A represents the number of the selected sub-agent. For example, if the attack mode in the current network is identified as random diffusion, then A=0, and the first sub-agent is selected.

[0136] Step S3.2: Defensive action execution

[0137] When the main agent identifies the attacker pattern in the current environment and selects the sub-agent to perform specific defensive actions, the selected sub-agent will perform actions based on the environment state s and the action value function Q childEvaluatio (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluat (s, a; ω) and execute, the environment returns to the main agent the reward r and the next environmental state s′ of the environment.

[0138] Step S3.3: Strategy Optimization

[0139] The process in which the main agent identifies the attack pattern in the current environment, selects the sub-agent based on the identification result, and the sub-agent performs defensive actions in the environment can be regarded as the main process of network defense combined with hierarchical reinforcement learning. This process includes two steps: sub-agent selection and action execution. Both steps need to be repeated and experience bars such as (s, A, r, s′, t) are continuously accumulated to fill the experience buffer D main , used for strategy optimization of the main agent.

[0140] The experience main =(s, A, r, s′, t) is stored in the experience buffer, and the estimation network Enet is updated according to the method of step S2.4: main The network parameters θ.

[0141] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A network security defense method based on hierarchical reinforcement learning, characterized in that: The following steps are involved: (1) Sub-agent training For M different attack modes, the corresponding sub-agents are trained to select the optimal defense actions for the current environment state under each attack mode, and then stored in the sub-agent library; The sub-agent is a valuation network Enet child , whose action value function is Q childEvaluation (s, a; ω), select its action value function Q according to the current environment state s childEvaluation (s, a; ω) is the maximum output of defense action a, where ω is the evaluation network Enet child Network parameters; To train the sub-agent, construct the target action value function as Q childTarget (s, a; ω′), with a target network Tnet child Implementation, where ω′ is the target network Tnet child Network parameters; The training process of each attack mode sub-agent is as follows: 1.1) Set an attack mode and treat it as part of the environment, then initialize the sub-agent’s experience buffer D child The capacity is N child , initialize the valuation network Enetchi l Initialize the target network Tnet with the network parameters ω child The network parameters ω′; The sub-agent performs N rounds of interactions with the environment. Each round of interaction consists of T time steps. At each time step t: 1.2) The state of the environment is s, and the sub-agent has a probability of ∈ to randomly select a defense action a in the defense action space of the sub-agent, and a probability of 1-∈ according to the action value function Q childEvalu (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluati (s, a; ω); 1.3) After selecting the defensive action, the sub-agent performs the defensive action a in the environment, and the environment returns a reward r and the next state s′ of the environment to the sub-agent. The experience condition d is composed of the current state s, the defensive action a, the reward r and the next state s′. child =(s, a, r, s′, t), stored in the experience buffer D child middle; 1.4) When the experience buffer D chil Medium Experience child After reaching the set number, each time step t is stored in the experience bar d child After that, from the experience buffer D child Randomly select a batch of experience conditions child , used to update the valuation network Enet chil The network parameter ω is the updated action value function Q childEvaluat (s, a; ω): First, calculate each empirical term d child The target value y′ is: Where a′=argmax a′ Q childEvaluat (s′, a′; ω) is based on the action value function Q childEvaluation (s′, a′; ω) The maximum value selects the defense action a′ in the defense action space of the sub-agent; Then calculate the difference L chil : L child =(y′-Q childEvaluatio (s,a;ω)) 2 All experience items in this batch child The difference L child Take the average as the loss value, minimize the loss value, and update the valuation network Enet through gradient descent childd The network parameters ω; At the same time, every C time steps, the target network Tnet child The network parameter ω′ is set as the estimation network Enet child The network parameter ω is the empirical buffer D child When the amount is full, the number of experience points reaches N child , save the new experience bar child When the experience bar first deposited is removed, the experience bar first deposited is then removed; After N rounds of interaction, the sub-agent is stored in the sub-agent library; (2) Main agent training For M different attack modes, a master agent is trained to identify the attack mode in the current environment. The master agent is an estimation network Enet main , whose action value function is Q mainEvaluation (s, A; θ), select its action value function Q according to the current environment state s mainEvaluation (s, A; θ) The largest sub-agent selects action A to output, where θ is the valuation network Enet main Network parameters; To train the main agent, construct the target action value function as Q mainTarget (s, A; θ′), with a target network Tnet main Implementation, where θ′ is the target network Tnet main Network parameters; The training process of the main agent is: 2.1) Initialize the main agent's experience buffer D main The capacity is N main , initialize the valuation network Enet main Initialize the target network Tnet with the network parameters θ main The network parameters θ′; The main agent performs N rounds of interaction with the environment. At the beginning of each round of interaction, it selects one of the M attack modes and regards it as part of the environment until the end of this round. Each round of interaction contains T time steps. At each time step t: 2.2) The state of the environment is s, and the main agent has a probability of ∈ to randomly select a sub-agent in the sub-agent selection action space of the main agent to select action A, and has a probability of 1-∈ according to the action value function Q mainEvaluation (s, A; θ) The maximum value in the sub-agent selection action space of the main agent selects the sub-agent to select action A, that is, A = argmax A Q mainEvaluation (s, A; θ); 2.3) After selecting the agent to select the action, the main agent executes the sub-agent selection action A and selects the corresponding sub-agent. The selected sub-agent is based on the environment state s and the action value function Q childEvaluati (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluati (s, a; ω) and execute, the environment returns to the main agent the reward r and the next environment state s′ of the environment, and the experience condition d main =(s, A, r, s′, t), stored in the experience buffer D main middle; 2.4) When the experience buffer D main Medium Experience main After reaching the set number, each time step t is stored in the experience bar d main After that, from the experience buffer D main Randomly select a batch of experience conditions mian , used to update the valuation network Enet main The network parameters θ are the updated action value function Q childEvaluati (s, A; θ): First, calculate each empirical term d main The target value Y′ is: Where A′=argmax A′ Q childEvaluation (s′, A′; θ) is based on the action value function Q childEvaluati (s′, A′; θ) The maximum value is in the sub-agent selection action space of the main agent. The sub-agent selects the action A′; Then calculate the difference: L main =(Y′-Q mainEvaluation (s,A;θ)) 2 All experience items in this batch main The difference L main Take the average as the loss value, minimize the loss value, and update the valuation network Enet through gradient descent main The network parameters θ; At the same time, every C time steps, the target network Tnet main The network parameters are set as θ′, which is the estimated network Enet main The network parameter θ, when the experience buffer D main When the amount is full, the number of experience points reaches N main , save the new experience bar main When the experience bar first deposited is removed, the experience bar first deposited is then removed; After N rounds of interaction, the main agent is trained and saved; (3) Layered reinforcement learning for defense 3.1) Identify the attacker pattern in the current environment and send the best sub-agent to defend The recognition method is to input the state s of the current environment into the main agent, and then mainEvaluation (s, A; θ) The maximum value in the sub-agent selection action space of the main agent selects the sub-agent to select action A, that is, A = argmax A Q mainEvaluation (s, A; θ); After selecting the agent selection action, the main agent executes the sub-agent selection action A and selects the corresponding sub-agent; 3.2) Execution of defensive actions The selected sub-agent is based on the environment state s and the action value function Q childEvaluation (s, a; ω) The maximum value selects the defense action a in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluatio (s, a; ω) and execute, the environment returns to the main agent the reward r and the next state s′ of the environment; 3.3) Strategy Optimization The experience main =(s, A, r, s′, t) is stored in the experience buffer D main , update the valuation network Enet according to the method in step 2.4) main The network parameters θ.

2. The network security defense method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The probability ∈ is initialized to 0.95 and is updated once ∈=∈×0.99 after every 1000 rounds of interaction.

3. The network security defense method based on hierarchical reinforcement learning according to claim 1 is characterized in that: Every C = 5000 time steps, the target network Tnet child The network parameter ω′ is set as the estimation network Enet child The network parameters ω are set every C = 5000 time steps. main The network parameters are set as θ′, which is the estimated network Enet main The network parameters θ.

4. The network security defense method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The defense actions include sleep, monitor, analyze, honeypot, remove, and recover.

Images