A Network Security Defense Method Based on Hierarchical Reinforcement Learning

Abstract

The present invention discloses a network security defense method based on hierarchical reinforcement learning. First, a target network is constructed to train sub-agents to obtain a sub-agent library. On this basis, another target network is constructed to train the main agent. When used for defense, sub-agents in the sub-agent library are selected according to the environment for defense. After the training of the sub-agents and the main agent is completed, defense can begin by combining hierarchical reinforcement learning. During the defense process, experience bars are obtained and put into the experience buffer for policy optimization, updating the value network of the main agent, further optimizing the network security defense strategy, so as to better cope with various challenges in the field of network security and improve the overall network security defense ability.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and more specifically, relates to a network security defense method based on hierarchical reinforcement learning. Background Art

[0002] Network information systems are now deeply integrated into people's daily lives, government operations, industrial development, and academic research, making cybersecurity increasingly important. With the rapid advancement of technology, various cybersecurity threats are also growing. Advanced Persistent Threats (APTs), in particular, pose severe challenges to various sectors. Furthermore, the increasing complexity and interconnectedness of network information systems exacerbate the severity and destructiveness of cybersecurity issues. Therefore, ensuring network security has become a major issue that urgently needs to be addressed, and automated network security defense is a key measure to ensure network security.

[0003] Currently, automated cybersecurity defense faces numerous challenges. Effectively defending against malicious attacks requires continuous monitoring, traffic analysis, and vulnerability patching, which is both costly and labor-intensive. Furthermore, the cybersecurity landscape is characterized by significant asymmetry: attackers often find it easy to launch attacks, while defenders face significant challenges. This complicates cybersecurity defense.

[0004] Traditional network security defense measures, such as firewalls, intrusion detection systems (IDS), and intrusion prevention systems (IPS), have shown their limitations amid the rapid development of the internet. With the increasing complexity of network environments and the increasing security risks, developing efficient and intelligent automated defense methods has become a key research direction in the field of network security. Automated defense mechanisms can effectively monitor network activity and promptly detect and prevent various types of network attacks, providing strong protection for the secure operation of network environments. However, how to automatically, accurately, and effectively select the optimal defense strategy in this complex environment has become a major challenge that needs to be addressed. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a network security defense method based on hierarchical reinforcement learning. When facing attackers with different attack modes, the main intelligent agent identifies the attack mode and selects the most appropriate pre-trained reinforcement learning sub-agent for defense, so as to better cope with various challenges in the field of network security and enhance the overall network security defense capabilities.

[0006] To achieve the above-mentioned invention object, the network security defense method based on hierarchical reinforcement learning of the present invention is characterized by including the following steps:

[0007] (1) Sub-agent training

[0008] For M different attack modes, sub-agents corresponding to each attack mode for selecting the optimal defense action in the current environmental state are trained respectively, and then stored in the sub-agent library; the sub-agent is an evaluation network Enet child , whose action value function is Q childEvaluat (s, a; ω), according to the current environmental state s, selects the defense action a with the maximum value of its action value function Q childEvaluati (s, a; ω) and outputs it, where ω is the network parameter of the evaluation network Enet child ;

[0009] To train the sub-agent, a target action value function Q childTarge (s, a; ω′) is constructed and implemented by a target network Tnet child , where ω′ is the network parameter of the target network Tnet child ;

[0010] The training process of the sub-agent for each attack mode is as follows:

[0011] 1.1) Set an attack mode and regard it as part of the environment. Then, initialize the experience buffer D child of the sub-agent to have a capacity of N child , initialize the network parameter ω of the evaluation network Enet child , and initialize the network parameter ω′ of the target network Tnet child ;

[0012] The sub-agent conducts a total of N rounds of interaction with the environment. Each round of interaction contains T time steps. At each time step t:

[0013] 1.2) When the environmental state is s, the sub-agent has a probability of ∈ to randomly select a defense action a in the defense action space of the sub-agent, and a probability of 1 - ∈ to select the defense action a in the defense action space of the sub-agent according to the maximum value of the action value function Q childEvaluati (s, a; ω), that is, a = argmax a Q childEvaluat (s, a; ω);

[0014] 1.3) After selecting a defensive action, the sub-agent executes the defensive action a in the environment. The environment returns the reward r and the next environmental state s' to the sub-agent. The experience tuple d composed of the current environmental state s, the defensive action a, the reward r, and the next environmental state s' child =(s, a, r, s', t) is stored in the experience buffer D child ;

[0015] 1.4) When the number of experience tuples d child in the experience buffer D child reaches the set quantity, after each time step t deposits the experience tuple d child , a batch of experience tuples d child is randomly sampled from the experience buffer D child for updating the network parameters ω of the evaluation network Enet child , that is, for updating the action-value function Q childEvaluation (s, a; ω):

[0016] First, calculate the target value y' of each experience tuple d child :

[0017]

[0018] where a' = argmax a′ Q childEvaluat (s', a'; ω), that is, select the defensive action a' in the sub-agent's defensive action space according to the maximum value of the action-value function Q childEvaluation (s', a'; ω);

[0019] Then calculate the difference L child :

[0020] L child =(y' - Q childEvaluat (s, a; ω)) 2

[0021] Take the average of the differences L child obtained from all the experience tuples d in this batch as the loss value. With the goal of minimizing the loss value, update the network parameters ω of the evaluation network Enet child through gradient descent; child ;

[0022] Meanwhile, every C time steps, set the network parameter ω' of the target network Tnet child to the network parameter ω of the evaluation network Enet child . When the experience buffer D child is full, that is, when the number of experience tuples reaches N child , deposit a new experience tuple d childWhen it is time, the earliest stored experience bar is removed and then stored again;

[0023] After N rounds of interaction, the sub-agent is stored in the sub-agent library;

[0024] (2), Main agent training

[0025] For M different attack patterns, train a main agent to identify the attack pattern in the current environment. The main agent is a valuation network Enet main , and its action-value function is Q mainEvaluation (s, A; θ), according to the current environmental state s, select the sub-agent with the largest action-value function Q mainEvaluation (s, A; θ) to select the action A and output it. Among them, θ is the network parameter of the valuation network Enet main ;

[0026] To train the main agent, construct the target action-value function as Q mainTarge (s, A; θ'), which is implemented by a target network Tnet main , where θ' is the network parameter of the target network Tnet main ;

[0027] The training process of the main agent is as follows:

[0028] 2.1), Initialize the experience buffer D of the main agent main with a capacity of N main , initialize the network parameter θ of the valuation network Enet main , initialize the network parameter θ' of the target network Tnet main ;

[0029] The main agent conducts N rounds of interaction with the environment in total. At the beginning of each round of interaction, select one from M attack patterns and regard it as part of the environment until the end of this round and then reselect. Each round of interaction contains T time steps. At each time step t:

[0030] 2.2), The state of the environment is s. The main agent has a probability of ∈ to randomly select a sub-agent selection action A in the sub-agent selection action space of the main agent, and a probability of 1 - ∈ to select the sub-agent selection action A according to the maximum value of the action-value function Q mainEvaluation (s, A; θ), that is, A = argmax A Q mainEvaluation (s, A; θ);

[0031] 2.3) After the agent selects an action, the main agent executes the sub-agent selection action A to select the corresponding sub-agent. The selected sub-agent selects a defense action a according to the environmental state s based on the action-value function Q childEvaluatio (s, a; ω) at the maximum value in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluat (s, a; ω) and executes it. The environment returns the reward r to the main agent and the next environmental state s' of the environment. The experience bar d composed of the current environmental state s, the sub-agent selection action A, the reward r, and the next environmental state s' main =(s, A, r, s', t) is stored in the experience buffer D main ;

[0032] 2.4) When the number of experience bars d main in the experience buffer D main reaches the set quantity, after each time step t deposits the experience bar d main into it, a batch of experience bars d main is randomly drawn from the experience buffer D mian for updating the network parameters θ of the evaluation network Enet main , that is, updating the action-value function Q childEvalua (s, A; θ):

[0033] First, calculate the target value Y' of each experience bar d main :

[0034]

[0035] Among them, A' = argmax A′ Q childEvaluati (s', A'; θ), that is, based on the action-value function Q childEvaluati (s', A'; θ) at the maximum value, select the sub-agent selection action A' in the sub-agent selection action space of the main agent;

[0036] Then calculate the difference:

[0037] L main =(Y' - Q mainEvaluation (s, A; θ)) 2

[0038] Take the average of the differences L main obtained from all the experience bars d main in this batch as the loss value, and update the network parameters θ of the evaluation network Enet main with the goal of minimizing the loss value;

[0039] Meanwhile, every C time steps, set the network parameters θ′ of the target network Tnet main to the network parameters θ of the evaluation network Enet main When the experience buffer D main is full, that is, when the number of experience entries reaches N main , when a new experience entry d main is stored, remove the earliest stored experience entry and then store it;

[0040] After N rounds of interaction, the main agent is trained and the main agent is saved;

[0041] (3) Hierarchical reinforcement learning for defense

[0042] 3.1) Identify the attacker mode in the current environment and dispatch the optimal sub-agent for defense

[0043] The identification method is to input the state s of the current environment into the main agent, and select the sub-agent action A in the action space of the sub-agents of the main agent according to the maximum value of the action value function Q mainEvaluation (s, A; θ), that is, A = argmax A Q mainEvaluation (s, A; θ);

[0044] After the selected agent selects an action, the main agent executes the sub-agent selected action A and selects the corresponding sub-agent;

[0045] 3.2) Execution of defense actions

[0046] The selected sub-agent selects the defense action a according to the environment state s, according to the maximum value of the action value function Q childEvaluation (s, a; ω), that is, a = argmax a Q childEvaluation (s, a; ω) and executes it, and the environment returns the reward r and the next environment state s′ of the environment to the main agent;

[0047] 3.3) Policy optimization

[0048] Store the experience entry d main =(s, A, r, s′, t) into the experience buffer D main , and update the network parameters θ of the evaluation network Enet main according to the method in step 2.4).

[0049] The invention object of the present invention is realized as follows:

[0050] The network security defense method based on hierarchical reinforcement learning of the present invention first trains sub-agents by constructing a target network to obtain a sub-agent library. On this basis, another target network is constructed to train the main agent. When used for defense, the sub-agent in the sub-agent library is selected according to the environment for defense. After the training of the sub-agent and the main agent is completed, the defense can be started by combining hierarchical reinforcement learning. During the defense process, the experience bar is obtained and put into the experience buffer for policy optimization, and the valuation network of the main agent is updated to further optimize the network security defense strategy to better cope with various challenges in the field of network security and improve the overall network security defense ability.

[0051] The present invention also has the following beneficial effects:

[0052] 1. The present invention can dynamically respond to changes in the network environment. Since the network environment is complex and changeable and there are various attack methods, it is necessary to adopt dynamic learning defense means to cope with these attacks. The present invention combines reinforcement learning, can dynamically learn network features, and uses flexible defense methods. For example, for M attacker modes, we can adopt M different defense strategies to cope with them;

[0053] 2. The present invention can perform network defense in a targeted manner. The efficiency of defense depends on the accuracy of attack recognition. The main agent in the invention can dynamically identify attack patterns and continuously evolve, so as to accurately identify attack patterns and select the corresponding sub-agent to execute defense actions. Since the recognition process of the present invention is accurate and targeted, the defense performance can be significantly improved.

[0054] 3. The present invention has good scalability, which is reflected in two aspects: First, the action value function of the main agent can be fitted using different types of neural networks, such as LSTM and Pointer Network. Second, the number of pre-trained sub-agent libraries can be adjusted according to the judgment of the number of attacker modes in the network, so as to achieve flexible expansion. Description of the Drawings

[0055] Figure 1 is a schematic diagram of the simulation environment;

[0056] Figure 2 is a flowchart of a specific implementation manner of the network security defense method based on hierarchical reinforcement learning of the present invention;

[0057] Figure 3 is a schematic diagram of a specific example of the network security defense architecture;

[0058] Figure 4 is a flowchart of the training of the sub-agent;

[0059] Figure 5It is the training flowchart of the main agent. DETAILED DESCRIPTION

[0060] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0061] Cybersecurity has a significant impact on business and industry, as exemplified by incidents like the ByteDance database leak and the damage to drone systems on Israeli battlefields. Furthermore, cybersecurity issues are becoming increasingly serious, necessitating greater attention to this area. However, cybersecurity challenges are challenging to address due to the complexity of the network environment and the diverse and complex nature of attack methods. New tools are urgently needed to address these challenges.

[0062] Reinforcement learning is widely used to solve sequential decision-making problems, which bear similarities in cybersecurity scenarios, particularly in the context of dynamic environmental changes. Reinforcement learning has already achieved success in fields such as gaming and drone control. This paper, inspired by this, aims to leverage reinforcement learning to address dynamic defense issues. Furthermore, reinforcement learning can be used to identify attack patterns and implement corresponding defensive measures against diverse cyberattacks, thereby improving overall cybersecurity defense capabilities.

[0063] In this embodiment, if Figure 1 As shown in the figure, there are 3 subnets and 13 machines. In subnet 1, there are 5 user hosts; in subnet 2, there are 3 enterprise servers and 1 defense host; in subnet 3, there are 3 operation hosts and 1 operation server.

[0064] Each machine exposes network services that other machines can connect to, potentially exposing exploitable vulnerabilities. However, this network is firewalled, preventing machines on subnet 1 from directly connecting to machines on subnet 3. Only the operations host has access to the operations server, making the security of the operations server crucial to the entire manufacturing process.

[0065] Figure 2 This is a flowchart of a specific implementation of the network security defense method based on layered reinforcement learning of the present invention. Figure 3 Schematic diagram of the corresponding network security defense architecture. In this embodiment, Figure 2 、 3 As shown, the network security defense method based on hierarchical reinforcement learning of the present invention includes the following steps:

[0066] Step S1: Sub-agent training

[0067] For M different attack patterns, corresponding sub-agents for selecting the optimal defense actions for the current environmental state under their respective attack patterns are trained respectively, and then stored in the sub-agent library; the sub-agent is a valuation network Enet chil , and its action value function is Q childEvaluati (s, a; ω). According to the current environmental state s, the defense action a with the maximum action value function Q childEvaluat (s, a; ω) is selected and output, where ω is the network parameter of the valuation network Enet child .

[0068] In this embodiment, all attack patterns are abstracted, and corresponding D3QN (Dueling Double Deep Q-Learning Network) sub-agents are trained for each attack pattern, and then these sub-agents are stored in the sub-agent library.

[0069] In this embodiment, the attack patterns in the network are abstracted into two different types, i.e., M = 2. The first attack pattern is random diffusion in the network, randomly attacking one node from another node until the operation server in the network is obtained. The second attack pattern is diffusion based on prior information, and these prior information include the node distribution and the optimal attack path in the network.

[0070] To train the sub-agent, a target action value function Q childTarge (s, a; ω′) is constructed and implemented by a target network Tnet chi , where ω′ is the network parameter of the target network Tnet child .

[0071] In this embodiment, as Figure 4 shown, the training process of the sub-agent for each attack pattern is as follows:

[0072] Step S1.1: Set the attack pattern and initialize the network parameters

[0073] Set an attack pattern and regard it as part of the environment. Then, initialize the experience buffer D child of the sub-agent with a capacity of N child , initialize the network parameter ω of the valuation network Enet child , and initialize the network parameter ω′ of the target network Tnet child .

[0074] In this embodiment, the capacity of the initialized sub-agent experience buffer D chil is 10 5 . The valuation network Enet childIt is a neural network, where the input layer has 52 neurons, the output layer has 54 neurons, and the number of neurons in the intermediate hidden layers are 256, 512, 1024, 512, and 256 in sequence. The network parameters ω are randomly initialized. The target network Tnet child and the evaluation network Enet child have the same structure.

[0075] The sub-agent interacts with the environment for a total of N rounds, and each round of interaction contains T time steps. In this embodiment, N = 50,000 and T = 100.

[0076] At each time step t:

[0077] Step S1.2: Select the defense action a according to the ε-greedy strategy

[0078] The state of the environment is s. The sub-agent has a probability of ε to randomly select a defense action a in the defense action space of the sub-agent, and a probability of 1 - ε to select the defense action a according to the maximum value of the action value function Q childEvaluation (s, a; ω) in the defense action space of the sub-agent, that is, a = argmax a Q childEvaluatio (s, a; ω), that is, the defense action a is selected using the ε-greedy strategy.

[0079] The ε-greedy strategy is a commonly used strategy in reinforcement learning for balancing exploration and exploitation. In this strategy, ε represents the probability of exploration, that is, randomly selecting an action with a certain probability, and 1 - ε represents the probability of exploitation, that is, selecting the currently considered optimal action with a certain probability.

[0080] In this embodiment, as Figure 3 shown, the state of the environment s represents a set of descriptions of each machine. There are a total of 13 machines (hosts), and each machine is described by four bits. The first two bits represent the malicious attack situation on the machine: 00 means no malicious attack, 10 means the machine is being scanned, and 11 means the machine has been maliciously exploited. The last two bits represent the degree of harm to the machine: 00 means the machine is not damaged, 10 means the machine state is unknown, 01 means the user permissions on the machine have been obtained, and 11 means the administrator permissions on the machine have been obtained. Therefore, the entire state is represented by 52 bits.

[0081] In this embodiment, the ε-greedy strategy is an ε-greedy strategy with a decreasing detection rate. The exploration probability ε is initially set to 0.95 and updated once after every 1000 rounds of interaction, ε = ε × 0.99.

[0082] In this embodiment, the defense action space of the sub-agent is shown in Table 1.

[0083]

[0084]

[0085] Table 1

[0086] Among the defensive actions in Table 1, the first two are non-directional, while the last four require specifying a machine. Therefore, the sub-agent has a total of 54 independent defensive actions, which can be represented by a one-hot bit vector of length 54. The sub-agent has a probability of ∈ of randomly selecting a defensive action a from the defensive action space of the sub-agent, i.e., 54 defensive actions. There is a probability of 1 - ∈ of selecting the defensive action a according to the maximum value of the action-value function Q childEvaluati (s, a; ω) in the defensive action space of the sub-agent.

[0087] In this embodiment, the output of the action-value function Q childEvaluat (s, a; ω) is a vector of length 54, and these values represent the value of each action in the current state.

[0088] Step S1.3: The sub-agent executes the defensive action a and obtains the experience bar d child and stores it in the experience buffer D child

[0089] After selecting the defensive action, the sub-agent executes the defensive action a in the environment, and the environment returns the reward r to the sub-agent and the next environmental state s' of the environment. The experience bar d composed of the current environmental state s, the defensive action a, the reward r, and the next environmental state s' child =(s, a, r, s', t) is stored in the experience buffer D child .

[0090] In this embodiment, the design principle of the reward r is as follows: When the attacker controls the host, the sub-agent will be punished, and the closer the controlled machine is to the operation server, the greater the punishment. When the attacker controls the operation server, the sub-agent will face a greater punishment. In addition, when the sub-agent restarts a certain machine, it will also suffer a punishment.

[0091] Step S1.4: Update the network parameters ω of the evaluation network Enet child of

[0092] When the number of experience bars d child in the experience buffer D child reaches the set quantity, after each time step t deposits the experience bar d chi , a batch of experience bars d child is randomly sampled from the experience buffer D child for updating the evaluation network Enet childThe network parameter ω of the network updates the action value function Q childEvaluat (s, a; ω).

[0093] In this embodiment, the set number is 3,000, which is the experience bar d child After reaching 3,000, the network parameter ω of the evaluation network Enet is updated at each time step t chil The specific update steps are as follows:

[0094] First, calculate the target value y′ of each experience bar d child :

[0095]

[0096] Among them, a′ = argmax a′ Q childEvaluat (s′, a′; ω) is to select the defense action a′ in the sub-agent defense action space according to the maximum value of the action value function Q childEvaluation (s′, a′; ω). That is: when updating the target value y′, the formula y′ = r + Q childTarge (s′, a′; ω′) is adopted at time steps less than T. When the time step t = T, then y′ = r

[0097] Then calculate the difference L chi :

[0098] L child =(y′ - Q childEvaluation (s, a; ω)) 2

[0099] Take the average of the differences L obtained for all experience bars d in this batch child as the loss value, and update the network parameter ω of the evaluation network Enet with the goal of minimizing the loss value child through gradient descent child of the network parameter ω

[0100] At the same time, every C time steps, set the network parameter ω′ of the target network Tnet chil to the network parameter ω of the evaluation network Enet child When the experience buffer D chil is full, that is, when the number of experience bars reaches N child , when a new experience bar d child is deposited, the earliest deposited experience bar is removed and then deposited

[0101] In this implementation example, every C = 5,000 time steps, set the network parameter ω′ of the target network Tnet child to the network parameter ω of the evaluation network Enet child of the network parameter ω

[0102] After the N - round interaction, the sub - agent is stored in the sub - agent library;

[0103] Step S2: Training of the main agent

[0104] For M different attack modes, train a main agent to identify the attack mode in the current environment. The main agent is a valuation network Enet main , and its action - value function is Q mainEvaluation (s, A; θ). According to the current environmental state s, select the sub - agent with the maximum action - value function Q mainEvaluation (s, A; θ) to select the action A for output. Here, θ is the network parameter of the valuation network Enet main .

[0105] To train the main agent, construct a target action - value function Q mainTarge (s, A; θ′), which is implemented by a target network Tnet main . Here, θ′ is the network parameter of the target network Tnetm ain .

[0106] In this embodiment, initialize the experience buffer D of the main agent main with a capacity of 10 5 . The valuation network Enet main is a neural network. Among them, the input layer has 52 neurons, the output layer has 2 neurons, and the number of neurons in the intermediate hidden layers is 256, 512, 1024, 512, and 256 in sequence. Randomly initialize the network parameter θ. The target network Tnet main has the same structure as the valuation network Enet main .

[0107] The main agent conducts a total of N rounds of interaction with the environment, and each round of interaction contains T time steps. In this embodiment, N = 50,000 and T = 100.

[0108] In this embodiment, as Figure 5 shown, the training process of the main agent is as follows:

[0109] Step S2.1: Initialize network parameters and select an attack mode in each round

[0110] Initialize the experience buffer D of the main agent main with a capacity of N main , initialize the network parameter θ of the valuation network Enet main , and initialize the network parameter θ′ of the target network Tnet main ;

[0111] The main agent interacts with the environment for a total of N rounds. At the beginning of each round of interaction, one of the M attack modes is selected and regarded as part of the environment until the end of this round and then reselected. Each round of interaction contains T time steps. In this embodiment, one is selected from M = 2 attack modes.

[0112] At each time step t:

[0113] Step S2.2: Select the agent to select action A according to the ε-greedy strategy

[0114] The state of the environment is s. The main agent has a probability of ε to randomly select a sub-agent selection action A in the sub-agent selection action space of the main agent, and a probability of 1 - ε to select the sub-agent selection action A according to the action value function Q mainEvaluation (s, A; θ) maximum value in the sub-agent selection action space of the main agent to select the sub-agent selection action A, that is, A = argmax A Q mainEvaluation (s, A; θ), that is, use the ε-greedy strategy to select the agent to select action A.

[0115] In this embodiment, similar to the sub-agent training, the ε-greedy strategy is the ε-greedy strategy with a decreasing exploration rate. The exploration probability ε is initialized to 0.95 and updated once after every 1000 rounds of interaction, ε = ε × 0.99. At the same time, there are only two possibilities for the sub-agent selection action A: 00 means selecting the first sub-agent, and 10 means selecting the second sub-agent. Therefore, the sub-agent selection action A of the main agent can be represented by a one-hot vector of length 2.

[0116] Step S2.3: The main agent executes the sub-agent selection action A and obtains the experience bar d main And store it in the experience buffer D main

[0117] After the agent selects the action, the main agent executes the sub-agent selection action A, selects the corresponding sub-agent, and the selected sub-agent selects the defense action a according to the environment state s, according to the action value function Q childEvaluat (s, a; ω) maximum value in the defense action space of the sub-agent to select the defense action a, that is, a = argmax a Q childEvaluatio (s, a; ω) and execute it. The environment returns the reward r to the main agent and the next environment state s' of the environment. The experience bar d composed of the current environment state s, the sub-agent selection action A, the reward r, and the next environment state s' main =(s, A, r, s', t), stored in the experience buffer D main in.

[0118] Step S2.4: Update the evaluation network Enetmain The network parameters θ

[0119] When the experience buffer D main Medium experience bar main After reaching the set number, each time step t is stored in the experience bar d main After that, from the experience buffer D main Randomly select a batch of experience items mian , used to update the valuation network Enet main The network parameters θ are the updated action value function Q childEvaluat (s, A; θ).

[0120] In this embodiment, the number of set items is 3000, i.e., the number of experience items d main After reaching 3000, the valuation network Enet is updated at each time step t main The network parameters θ. The specific update steps are:

[0121] First, calculate each empirical main The target value Y′:

[0122]

[0123] Where A′=argmax A′ Q childEvaluation (s′, A′; θ) is based on the action value function Q childEvaluati The maximum value of (s′, A′; θ) is in the action space of the sub-agent of the main agent. The sub-agent selects the action A′. That is, the new target value Y′ is less than T time step using the formula Y′=r+Q mainTarget (s′, A′; θ′), when the time step t=T, then Y′=r.

[0124] Then calculate the difference:

[0125] L main =(Y′-Q mainEvaluation (s, A; θ)) 2

[0126] All experience items in this batch main The difference L main Take the average as the loss value, minimize the loss value, and update the valuation network Enet through gradient descent main The network parameters θ;

[0127] At the same time, every C time steps, the target network Tnet main The network parameters are set as θ′, which is the estimated network Enet main The network parameters θ, when the experience buffer D mainAfter being full, that is, when the number of experience entries reaches N main , a new experience bar d is stored main . After removing the earliest stored experience bar, it is stored

[0128] In this embodiment, every C = 5000 time steps, the network parameter setting θ′ of the target network Tnet main is set to the network parameter θ of the evaluation network Enet main .

[0129] After N rounds of interaction end, the main intelligent agent finishes training and saves the main intelligent agent

[0130] Step S3: Hierarchical reinforcement learning for defense

[0131] When the training of the sub-intelligent agent and the main intelligent agent ends, defense can start by combining hierarchical reinforcement learning

[0132] Step S3.1: Identify the attacker mode in the current environment and dispatch the optimal sub-intelligent agent for defense

[0133] The identification method is to input the state s of the current environment into the main intelligent agent. According to the maximum value of the action value function Q mainEvaluation (s, A; θ), select the action A of the sub-intelligent agent in the action space of the sub-intelligent agent of the main intelligent agent, that is, A = argmax A Q mainEvaluation (s, A; θ).

[0134] After the selected intelligent agent selects an action, the main intelligent agent executes the action A selected by the sub-intelligent agent and selects the corresponding sub-intelligent agent

[0135] In this embodiment, the action A selected by the sub-intelligent agent represents the number of the selected sub-intelligent agent. For example, if it is identified that the attack mode in the current network is random diffusion, then A = 0 and the first sub-intelligent agent is selected

[0136] Step S3.2: Execution of defense actions

[0137] When the main intelligent agent identifies the attacker mode in the current environment and selects the sub-intelligent agent to execute specific defense actions, the selected sub-intelligent agent selects the defense action a according to the environmental state s, according to the maximum value of the action value function Q childEvaluatio (s, a; ω), that is, a = argmax a Q childEvaluat (s, a; ω) and executes it. The environment returns the reward r to the main intelligent agent and the next environmental state s′ of the environment

[0138] Step S3.3: Policy optimization

[0139] The process by which the main agent identifies the attack pattern in the current environment and selects sub-agents and the process of the sub-agents performing defense actions in the environment can be regarded as the main process of network defense combined with hierarchical reinforcement learning. This process includes two steps: the selection of sub-agents and the execution of actions. These two steps need to be repeated continuously, and experience bars of (s, A, r, s′, t) are continuously accumulated to fill the experience buffer D main for the policy optimization of the main agent.

[0140] Deposit the experience bar d main =(s, A, r, s′, t) into the experience buffer, and update the network parameters θ of the evaluation network Enet according to the method in step S2.4: main

[0141] Although the above illustrative specific embodiments of the present invention have been described to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

Claims

1. A network security defense method based on hierarchical reinforcement learning, characterized in that, It includes the following steps: (1) Sub-agent training For M different attack patterns, sub-agents for selecting the optimal defense actions for the current environmental state under their respective attack patterns are trained separately, and then stored in the sub-agent library; The sub-agent is a valuation network Enet child , and its action value function is Q childEvaluation (s, a; ω). According to the current environmental state s, it selects and outputs the defense action a with the maximum action value function Q childEvaluation (s, a; ω), where ω is the network parameter of the valuation network Enet child ; (2) Main-agent training For M different attack patterns, train a main agent to identify the attack pattern in the current environment. The main agent is a valuation network Enet main , and its action-value function is Q mainEvaluation (s, A; θ). According to the current environmental state s, select the sub-agent with the maximum action-value function Q mainEvaluation (s, A; θ) to select and output the action A, where θ is the network parameter of the valuation network Enet main ; (3) Hierarchical reinforcement learning for defense 3.1) Identify the attacker pattern in the current environment and dispatch the optimal sub-agent for defense The recognition method is to input the state s of the current environment into the main agent, and select the action A of the sub-agent in the action space of the sub-agent of the main agent according to the maximum value of the action value function Q mainEvaluation (s, A; θ), that is, A = argmax A Q mainEvaluation (s, A; θ); After the selection agent selects an action, the main agent executes the sub-agent selection action A to select the corresponding sub-agent; 3.2) Execution of defense actions The selected sub-agent selects a defensive action a according to the environmental state s based on the action-value function Q childEvaluation (s, a; ω) and selects the defensive action a in the defensive action space of the sub-agent, that is, a = argmax a Q childEvaluation (s, a; ω) and executes it. The environment returns the reward r to the main agent and the next environmental state s' of the environment; 3.3) Policy optimization Store the experience bar d main =(s,A,r,s',t) in the experience buffer D main , and update the network parameters θ of the evaluation network Enet main . t represents each time step 2. The network security defense method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step (1), to train the sub-agent, a target action-value function $Q$ childTarget $(s, a; \omega')$ is constructed and implemented using a target network $T_{net}$ child , where $\omega'$ are the network parameters of the target network $T_{net}$ child . The training process of the sub-agent for each attack pattern is as follows: 1.1), Set an attack mode and regard it as part of the environment, then initialize the experience buffer D of the sub-agent child with a capacity of N child , initialize the evaluation network Enet child with network parameters ω, and initialize the target network Tnet child with network parameters ω'; The sub-agent conducts a total of N rounds of interaction with the environment. Each round of interaction contains T time steps. At each time step t: 1.2) When the state of the environment is s, the sub-agent randomly selects a defense action a from the defense action space of the sub-agent with a probability of ∈, and with a probability of 1 - ∈, it selects the defense action a in the defense action space of the sub-agent according to the maximum value of the action value function Q childEvaluation (s, a; ω), that is, a = argmax a Q childEvaluation (s, a; ω); 1.3), After selecting a defensive action, the sub-agent executes the defensive action a in the environment, and the environment returns the reward r and the next environmental state s of the environment to the sub-agent ' , The experience bar d composed of the current environmental state s, the defensive action a, the reward r, and the next environmental state s' child =(s, a, r, s', t), is stored in the experience buffer D child ; 1.4) When the experience buffer D child and the experience bar d child in it reaches the set quantity, after each time step t deposits the experience bar d child into the experience buffer D child a batch of experience bars d child is randomly selected from the experience buffer D child to update the network parameters ω of the evaluation network Enet childEvaluation i.e., to update the action value function Q (s,a;ω): First, calculate the target value y' of each experience bar d child : where a' = argmax a' Q childEvaluation (s', a'; ω), that is, select the defense action a' in the sub-agent defense action space according to the maximum value of the action value function Q childEvaluation (s', a'; ω); Then calculate the difference L child : L child = (y' - Q childEvaluation (s, a; ω)) 2 For all the experience bars in this batch d child The obtained difference L child Take the average as the loss value. With the goal of minimizing the loss value, update the network parameters ω of the evaluation network Enet through gradient descent child ; Meanwhile, every C time steps, the network parameters ω' of the target network Tnet child are set to the network parameters ω of the evaluation network Enet child When the experience buffer D child is full, that is, when the number of experience entries reaches N child , when a new experience entry d child is stored, the earliest stored experience entry is removed and then stored; After N rounds of interaction, the sub-agent is stored in the sub-agent library.

3. The network security defense method based on hierarchical reinforcement learning according to claim 2, characterized in that, In step (2), to train the main agent, a target action-value function Q mainTarget (s, A; θ') is constructed and implemented using a target network Tnet main where θ' are the network parameters of the target network Tnet main ; The training process of the main agent is as follows: 2.1) Initialize the experience buffer D of the main agent main with a capacity of N main , initialize the evaluation network Enet main with network parameters θ, and initialize the target network Tnet main with network parameters θ'; The main agent conducts a total of N rounds of interaction with the environment. At the beginning of each round of interaction, one of the M attack patterns is selected and regarded as part of the environment until the end of this round and then reselected. Each round of interaction contains T time steps. At each time step t: 2.2) When the state of the environment is s, the main agent randomly selects a sub-agent selection action A from the sub-agent selection action space of the main agent with a probability of ∈, and with a probability of 1 - ∈, it selects the sub-agent selection action A according to the maximum value of the action value function Q mainEvaluation (s, A; θ) in the sub-agent selection action space of the main agent, that is, A = argmax A Q mainEvaluation (s, A; θ); 2.3) After the agent selects an action, the main agent executes the sub-agent selection action A to select the corresponding sub-agent. The selected sub-agent selects a defensive action a from the defensive action space of the sub-agent according to the environmental state s based on the action-value function Q childEvaluation (s, a; ω) at its maximum value and executes it. The environment returns the reward r and the next environmental state s' to the main agent. The experience bar d a Q childEvaluation constituted by the current environmental state s, the sub-agent selection action A, the reward r, and the next environmental state s' is main d = (s, A, r, s', t), which is stored in the experience buffer D main ; 2.4) When the experience buffer D main the experience bar d main reaches the set quantity, after each time step t deposits the experience bar d main into the experience buffer D main randomly extracts a batch of experience bars d main from it, which is used to update the network parameters θ of the evaluation network Enet main and update the action value function Q mainEvaluation (s, A; θ): First, calculate the target value Y' of each experience bar d main : where, A' = argmax A' Q mainEvaluation (s', A'; θ), that is, according to the action value function Q mainEvaluation (s', A'; θ), the sub-agent of the main agent selects the action A' by choosing from the action space of the sub-agent at the maximum value; Then calculate the difference: L main = (Y' - Q mainEvaluation (s, A; θ)) 2 For all the experience bars in this batch d main The obtained difference L main Take the average as the loss value. With the goal of minimizing the loss value, update the network parameters θ of the valuation network Enet through gradient descent main ; Meanwhile, every C time steps, set the network parameters θ' of the target network Tnet main to the network parameters θ of the evaluation network Enet main When the experience buffer D main is full, i.e., the number of experience entries reaches N main , when a new experience entry d main is stored, remove the earliest stored experience entry and then store it; After N rounds of interaction, the main agent finishes training and saves the main agent.

4. The network security defense method based on hierarchical reinforcement learning according to claim 3, characterized in that, The probability ∈ is initialized to 0.95 and updated once after every 1000 rounds of interaction as ∈ = ∈ × 0.

99.

5. The network security defense method based on hierarchical reinforcement learning according to claim 3, wherein Every C = 5000 time steps, set the network parameters ω' of the target network Tnet child to the network parameters ω of the evaluation network Enet child Every C = 5000 time steps, set the network parameters θ' of the target network Tnet main to the network parameters θ of the evaluation network Enet main 。 6. The network security defense method based on hierarchical reinforcement learning according to claim 1, wherein The defense actions include sleep, monitoring, analysis, honeypot, removal, and recovery.

Images