Game reinforcement learning-based network security defense method

Abstract

The invention discloses a network security defense method based on game reinforcement learning, and the method comprises the steps: firstly determining key assets and importance weights, determining an attacker action space, an action occurrence probability and an influence degree, determining a defender action space, an action occurrence probability and cost, then determining an attack and defense game target, and then setting a game environment. On the basis, two PPO intelligent agents are created, a simulation environment is established through a network, and an attacker strategy and a defender strategy are alternately trained through mutual cooperation of reinforcement learning and the game theory in an attack and defense dynamic change scene, so that an optimal network security defense strategy is obtained, various challenges in the field of network security are better coped, and the network security defense method is suitable for being applied to the field of network security. And the overall network security defense capability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and more specifically, relates to a network security defense method based on game reinforcement learning. Background Art

[0002] Network information systems are now deeply integrated into people's daily lives, government operations, industrial development, and academic research, and the importance of network security has become increasingly prominent. With the rapid development of science and technology, various network security threats are also growing, especially Advanced Persistent Threats (APTs), which have brought severe challenges to various fields. At the same time, network information systems are becoming more and more complex and interconnected, which has exacerbated the severity and destructiveness of network security issues. Therefore, ensuring network security has become a major issue that needs to be solved urgently, and automated network security defense is an important measure to ensure network security.

[0003] Currently, automated network security defense faces many challenges. In order to effectively defend against malicious attacks, continuous monitoring, traffic analysis, and vulnerability patching are required, which is not only costly but also requires a lot of manpower. In addition, there is an obvious asymmetry in the field of network security. Attackers can often launch attacks easily, while defenders face great difficulties, which increases the difficulty of network security defense.

[0004] Traditional network security defense measures, such as firewalls, intrusion detection systems (IDS) and intrusion prevention systems (IPS), have shown certain limitations in the rapid development of the Internet. With the increasing complexity of the network environment and the increasing security risks, the development of efficient and intelligent automated defense methods has become an important research direction in the field of network security. Automated defense mechanisms can effectively monitor network activities, detect and prevent various types of network attacks in a timely manner, and provide strong protection for the safe operation of the network environment. However, how to automatically, accurately and effectively select the best defense strategy in this complex situation has become a major challenge that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a network security defense method based on game reinforcement learning. In the dynamic attack and defense scenario, through the mutual cooperation of reinforcement learning and game theory, the optimal network security defense strategy is sought to better cope with various challenges in the field of network security and enhance the overall network security defense capability.

[0006] In order to achieve the above-mentioned object of the invention, the network security defense method based on game reinforcement learning of the present invention is characterized by comprising the following steps:

[0007] (1) Establishing offensive and defensive game objectives

[0008] Determine key assets and importance weights: S = {s1, s2, …s J}, W={w1,w2,…s J}, where S is the key asset set, s j ,j=1,2,…,J is the jth key asset, W is the importance weight set, w j ,j=1,2,…,J is the weight of the jth key asset, J is the number of key assets;

[0009] Determine the attacker's action space, probability of occurrence, and impact: A attacker =

[0010] {a attacker1 ,a attacker ,…a attackerM}, P = {p1, p2, ... p M}, I={i1,i2,…i M}, where A attacker is the attacker's action space, a attackerm ,m=1,2,…,M is the mth attacker action, P is the occurrence probability set, p m ,m=1,2,…,M is the probability of the mth attacker’s action, I is the set of influence levels, i m ,m=1,2,…,M is the impact degree of the mth attacker action on the key assets, and M is the number of attacker actions;

[0011] Determine the defender's action space, action probability, and cost: A defender =

[0012] {a defender1 ,a defender2 ,…a defenderN}, Q={q1,q2,…q N}, C={c1,c2,…d N}, where A defender is the defender's action space, a defendern ,n=1,2,…,N is the nth defender action, Q is the occurrence probability set, q n ,n=1,2,…,N is the probability of the nth defender’s action, C is the cost set, c n ,n=1,2,…,N is the cost of the nth defender action, and N is the number of defender actions;

[0013] Determine the offensive and defensive game objectives: Among them, F is the defense objective function, T is the time step of a complete round of attack and defense interaction, and w(t) is the weight of the key asset targeted by the attacker's action in the tth time step, that is, the weight of the jth key asset s in the tth time step is j , its corresponding importance weight is w(t), then w(t)=w j , p(t) and i(t) are the corresponding probability of the attacker's action at the t-th time step and the degree of impact on key assets, that is, the t-th time step is the m-th attacker action a attackerm , then p(t)=p m , i(t)=i m , q(t) and c(t) are the corresponding probability and cost of the defender's action at the t-th time step, that is, the t-th time step is the n-th attacker action a defendern , then q(t)=q n , c(t)=c n .

[0014] (2) Setting the gaming environment

[0015] In the framework of game reinforcement learning, the attacker’s reward r attacker (t) is:

[0016] r attacker (t) = RA suc (Attacker Status,Attacker Action)-RA cost (attacker state, attacker action), where RA suc (attacker state, attacker action) = p(t) × i(t), RA cost (Attacker state, attacker action) is to take attacker action a attacker (t), the risk of the attacker being detected by the system, a attacker (t) is the attacker action at the t-th time step, i.e., the t-th time step is the m-th attacker action a attackerm , then a attacker (t) = a attackerm ;

[0017] Defender's Reward defende (t) is:

[0018] r defender (t) = RD suc (Defender Status, Defender Action) - RD cost (Defender state, defender action), where RD suc (Defender state, defender action) is to take defensive action a defender(t), the reward for successful defense, RD cost (Defender state, defender action) is the cost of taking the defender action, a defender (t) is the defender action at the t-th time step, i.e., the t-th time step is the n-th defender action a defendern , then a defender (t) = a defendern , R.D. cost (defender state, defender action) = q(t) × c(t);

[0019] (3) Offense and defense game

[0020] Create two PPO agents, the attacker PPO agent and the defender PPO agent. The attacker PPO agent is used to determine the attacker's environment state s according to the t-th time step. attacker (t), the attacker’s reward r attacker (t), get the attacker action a attacker (t), the network parameters of the attacker’s PPO agent are the attacker’s strategy π attacker , the defender PPO agent is used to determine the defender’s environment state s at the t-th time step defender (t), get the defender action a defender (t), the network parameters of the defender PPO agent are the defender strategy π defender ;

[0021] Build a simulation environment E for the network. The input of the simulation environment E is the attacker action a at the tth time step. attacker (t) and the defender action a defender (t), the output of the simulation environment E is the attacker’s environment state s attacker (t+1) and reward r attacker (t+1) and the defender's environmental state s defender (t+1) and reward r attacker (t+1);

[0022] Alternately train the attacker and defender strategies:

[0023] 3.1) Initialization

[0024] Initialize the attacker strategy pool Pool attacker And the defender strategy pool Pool defender , set the maximum number of iterations to K, and initialize the number of iterations k = 0;

[0025] 3.2) Defender Strategy Optimization

[0026] Clear the experience buffer, initialize the simulation environment E, and initialize the network parameters of the defender PPO agent, i.e., the defender strategy π defender , from the attacker strategy pool Pool attacker Select an attacker strategy π by sampling attAckEr Given an attacker PPO agent, where the sampling probability is 80% from the attacker strategy pool Pool attacker Randomly select an attacker strategy π from the first L attacker , 20% probability from the attacker strategy pool Pool attacker Randomly select an attacker strategy π from the remaining attacker strategies attacker ;

[0027] According to the simulation environment E, the attacker's environment state s at the first time step is obtained attacker (1) and the attacker’s reward r attacker (1) and the defender's environmental state s defender (1) and the defender's reward r defender (1), and the environmental state s attacker (1) Input the attacker PPO agent and obtain the attacker action a attacker (1), the defender's environmental state s defendee (1) Input the defender PPO agent and obtain the defender action a defender (1) The attacker action a attacker (1) Defender action a defende (1) Input into the simulation environment E and obtain the attacker's environment state s at the second time step attacker (2) and the attacker’s reward r attacker (2) and the defender's environmental state s defender (2) and the defender's reward r defender (2);

[0028] From the attacker strategy pool Pool attacker Select an attacker strategy π by sampling attacker Give the attacker PPO agent the environment state s attacker (2) Send the attacker’s PPO agent to obtain the attacker’s action a attacker (2) The defender’s environmental state s defender (2) Send it to the defender PPO agent and get the defender action a defender (2) The attacker action a attacker (2) Defender action a defender (2) Input into the simulation environment E and obtain the attacker's environment state s at the third time step attacker (3) and the attacker’s reward r attacker(3) and the defender's environmental state s defender (3) and the defender's reward r defender (3);

[0029] In this way, adversarial training is performed to obtain a series of adversarial samples:

[0030] (s defender (t),a defender (t),r defender (t),s defender (t+1))

[0031] The adversarial samples are placed in the experience buffer. When it is full, all adversarial samples are taken out of the experience buffer and the PPO algorithm is used to update the network parameters of the defender PPO agent, i.e., the defender strategy π. defender ;

[0032] Determine the updated defender strategy π defebder With the current defender strategy π defender Compare whether it is improved. If so, clear the experience buffer and continue to put adversarial samples in. When it is full, continue to take out and update the defender strategy π defender , continue to judge whether it has improved. If so, continue to clear the experience buffer and continue to put adversarial samples until there is no improvement. defender Put it into the defender strategy pool defender the front of;

[0033] 3.3) Attacker strategy optimization

[0034] Clear the experience buffer, initialize the simulation environment E, and initialize the network parameters of the attacker PPO agent, that is, the attacker strategy π attacker , from the defender strategy pool Pool defender Sampling a defender strategy π attAckEr Given a defender PPO agent, where the sampling probability is 80% from the defender strategy pool Pool defender Randomly select a defender strategy π from the first L defender , 20% probability from the defender strategy pool Pool defender Randomly select a defender strategy π from the remaining defender strategies defender ;

[0035] According to the simulation environment E, the attacker's environment state s at the first time step is obtained attacker (1) and the attacker’s reward r attacker (1) and the defender's environmental state s defender (1) and the defender's reward r defender (1), and the environmental state sattacker (1) Input the attacker PPO agent and obtain the attacker action a attacker (1), the defender's environmental state s defendee (1) Input the defender PPO agent and obtain the defender action a defender (1) The attacker action a attacker (1) Defender action a defender (1) Input into the simulation environment E and obtain the attacker's environment state s at the second time step attacker (2) and the attacker’s reward r attacker (2) and the defender's environmental state s defender (2) and the defender's reward r defender (2);

[0036] From the defender strategy pool Pool defender Sampling a defender strategy π defender Give the defender PPO agent the environment state s attacker (2) Send the attacker’s PPO agent to obtain the attacker’s action a attacker (2) The defender’s environmental state s defender (2) Send it to the defender PPO agent and get the defender action a defender (2) The attacker action a attacker (2) Defender action a defender (2) Input into the simulation environment E and obtain the attacker's environment state s at the third time step attacker (3) and the attacker’s reward r attacker (3) and the defender's environmental state s defender (3) and the defender's reward r defender (3);

[0037] In this way, adversarial training is performed to obtain a series of adversarial samples:

[0038] (s attacker (t),a attacker (t),r attacker (t),s attacker (t+1))

[0039] The adversarial samples are placed in the experience buffer. When it is full, all adversarial samples are taken out of the experience buffer and the PPO algorithm is used to update the network parameters of the attacker’s PPO agent, i.e., the attacker’s strategy π. attacker ;

[0040] Determine the updated attacker strategy π attacker With the current attacker strategy π attackerCompare whether it is improved. If so, clear the experience buffer and continue to put adversarial samples in. When it is full, continue to take out and update the attacker strategy π attacker , continue to judge whether it has improved, if so, continue to clear the experience buffer, continue to put in adversarial samples, and continue to judge until there is no improvement, and change the current attacker strategy π attacker Put it into the attacker strategy pool Pool attacker the front of;

[0041] 3.3), k = k + 1, determine whether the number of iterations k is less than the maximum number of iterations K, if not, the iteration ends, and the defender strategy π is obtained defender This is the optimal network security defense strategy. Otherwise, return to step 3.2).

[0042] The object of the invention of the present invention is achieved in this way:

[0043] The network security defense method based on game reinforcement learning of the present invention first determines the key assets and importance weights, determines the attacker's action space, the probability of action and the degree of influence, determines the defender's action space, the probability of action and the cost, and then establishes the attack and defense game goals, and then sets the game environment. On this basis, two PPO agents are created, and the network builds a simulation environment E. In the dynamic attack and defense scenario, through the mutual cooperation of reinforcement learning and game theory, the attacker strategy and the defender strategy are alternately trained to obtain the optimal network security defense strategy, so as to better cope with various challenges in the field of network security and improve the overall network security defense capability.

[0044] The present invention has the following beneficial effects:

[0045] 1. Simulate a real network attack and defense environment. By establishing a flag-grabbing environment and virtual games, the interaction between attackers and defenders can be closer to the interaction in real scenarios. At the same time, the opponents in the game are constantly evolving, which increases the exploratory nature of the strategies of attackers and defenders.

[0046] 2. Improve network security defense capabilities. The trained defender will have the ability to automatically detect and respond to network attacks. At the same time, it continuously learns and optimizes defender strategies, thereby enhancing the overall network protection level;

[0047] 3. Explore new network security strategies. Through the game interaction between attackers and defenders, we can discover the shortcomings of the current defense mechanism and then develop better defender strategies to fight against new attacks;

[0048] 4. Identify network vulnerabilities. By simulating various attack behaviors, attackers can help identify security vulnerabilities in actual systems and provide reference for subsequent patching and defense. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of a specific implementation method of the network security defense method based on game reinforcement learning of the present invention;

[0050] Figure 2 is a schematic diagram of the simulation environment;

[0051] Figure 3 It is a schematic diagram of the relationship between the agent and the environment;

[0052] Figure 4 It is a schematic diagram of attack and defense iterative training;

[0053] Figure 5 It is a schematic diagram of the defender's strategy training;

[0054] Figure 6 This is a diagram of attacker strategy training. DETAILED DESCRIPTION

[0055] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0056] Reinforcement learning (RL) in artificial intelligence technology can help develop autonomous, automatic and accurate network security decision-making algorithms, which is crucial to solving complex attack and defense interaction problems in the field of network security. In addition, game theory, as a mathematical tool, can also be used to select strategies in a highly competitive network security environment, thereby improving the accuracy and efficiency of network security decisions. The present invention focuses on the urgent needs in the field of network security and related automated defense mechanisms, and uses these technologies to better cope with various challenges in the field of network security. Game theory and reinforcement learning are used to find the optimal defender strategy in a dynamically changing network, thereby improving the overall network security defense capability.

[0057] Figure 1 It is a flow chart of a specific implementation method of the network security defense method based on game reinforcement learning of the present invention.

[0058] In this embodiment, if Figure 1 As shown, the network security defense method based on game reinforcement learning of the present invention includes the following steps:

[0059] Step S1: Establish the attack and defense game goal

[0060] Step S1.1: Determine key assets and importance weights: Sort out the organization's key business processes, information systems and data, determine key assets, and assign corresponding importance weights to each key asset. The weights can be determined based on the confidentiality, integrity, availability and other characteristics of the asset. If the key asset involves private data, the corresponding asset has a higher importance weight. Specifically, it can be expressed as:

[0061] S={s1,s2,…s J}, W={w1,w2,…s J}, where S is the key asset set, s j ,j=1,2,…,J is the jth key asset, W is the importance weight set, w j ,j=1,2,…,J is the weight of the jth key asset, and J is the number of key assets.

[0062] In this embodiment, if Figure 2 As shown, there are 3 subnets and 13 machines. In subnet 1, there are 5 user hosts; in subnet 2, there are 3 enterprise servers and 1 defense host; in subnet 3, there are 3 operation hosts and 1 operation server.

[0063] Each machine exposes some network services that other machines can connect to, and there may also be vulnerabilities that can be exploited. However, there is a firewall in this network, and machines in subnet 1 cannot directly connect to machines in subnet 3. Only the operation host has the authority to access the operation server, and the security of the operation server is crucial to the entire manufacturing process.

[0064] Key assets S = {Subnet 1 user machine, Subnet 2 enterprise server, Subnet 3 operation server}, that is, J = 3, s1 = Subnet 1 user machine, s2 = Subnet 2 enterprise server, s3 = Subnet 3 operation server. Importance weight W = {0.05, 0.15, 0.8}, that is, w1 = 0.05, w2 = 0.15, w3 = 0.8. Subnet 1 user machine s1 has a lower weight because it is not directly related to key business; Subnet 2 enterprise server s2 has a higher weight because it stores important data; Subnet 3 operation server s3 has the highest weight because it is critical to the entire manufacturing process.

[0065] Step S1.2: Determine the attacker's action space, probability of occurrence, and impact: Based on historical events, intelligence information, security vulnerability reports, etc., determine the network attack actions that the organization may face, analyze each attack action, determine its probability of occurrence and the impact on key assets. Specifically, it can be expressed as:

[0066] A attacker ={a attacker1 ,a attacker2 ,…aattackerM}, P = {p1, p2, ... p M}, I={i1,i2,…i M}, where A attacker is the attacker's action space, a attackerm ,m=1,2,…,M is the mth attacker action, P is the occurrence probability set, p m ,m=1,2,…,M is the probability of the mth attacker’s action, I is the set of influence levels, i m ,m=1,2,…,M is the impact degree of the mth attacker’s action on the key assets, and M is the number of attacker’s actions.

[0067] In this embodiment, the attacker actions include port scanning, vulnerability exploitation, privilege escalation, and data theft, that is, the attacker action data M=4, a attacker1 =Port scan, a attacker2 =Vulnerability Exploitation,

[0068] a attacker3 =Privilege escalation, a attacker = data theft, the probability of occurrence is 0.5, 0.25, 0.15, 0.1, that is, p1 = 0.5, p2 = 0.25, p3 = 0.15, p4 = 0.1. Among them, port scanning has a higher probability of occurrence because system vulnerabilities need to be discovered first; vulnerability exploitation has the second highest probability, and the system may have unpatched vulnerabilities; privilege escalation has the third highest probability, and key assets can be accessed after privilege escalation; data theft has the lowest probability of occurrence. The impact level is set to i1 = 0.1, i2 = 0.15, i3 = 0.25, i4 = 0.5. Port scanning has the lowest impact level, only exposing system information; vulnerability exploitation has a higher impact level, which may lead to system control; privilege escalation has a higher impact level, and key assets can be accessed; data theft has the highest impact level, which may cause huge losses.

[0069] Step S1.3: Determine the defender's action space, probability of occurrence, and cost: A defender =

[0070] {a defender1 ,a defender2 ,…a defenderN}, Q={q1,q2,…q N}, C={c1,c2,…d N}, where D is the defender action space, a defendern ,n=1,2,…,N is the nth defender action, Q is the occurrence probability set, q n ,n=1,2,…,N is the probability of the nth defender’s action, C is the cost set, c n,n=1,2,…,N is the cost of the nth defender action, and N is the number of defender actions.

[0071] In this embodiment, the defender has four actions, namely N=4, including: a defender =Analyze the machine, collect machine information, identify malicious behavior and permission status, and understand the current system security situation;

[0072] a defendrt2 =Terminate the malicious process, directly preventing the attacker's malicious activities, but will affect system availability in the short term; a defender3 =Recovery system; a defender4 =Deploy honeypot service. Honeypot is a defensive measure with a trapping nature that can attract and observe attackers, but it must be deployed with caution to avoid affecting normal business.

[0073] The costs of the defender's actions are c1 = 0, c2 = 0.1, c3 = 0.7, c4 = 0.2. The probabilities of the defender's actions are q1 = 0.4, q2 = 0.2, q3 = 0.3, q4 = 0.1.

[0074] Step S1.4: Determine the attack and defense game goal: Determine the organization's attack and defense game goal based on key assets and attacker actions. The attack and defense game goal can be defined as: minimizing the potential loss and recovery cost of key assets, which can be expressed by the following formula:

[0075] Among them, F is the defense objective function, T is the time step of a complete round of attack and defense interaction, and w(t) is the weight of the key asset targeted by the attacker's action in the tth time step, that is, the weight of the jth key asset s in the tth time step is j , then w(t)=s j , p(t) and i(t) are the corresponding probability of the attacker's action at the t-th time step and the degree of impact on key assets, that is, the t-th time step is the m-th attacker action a attackerm , then p(t)=p m , i(t)=i m , q(t) and c(t) are the corresponding probability and cost of the defender's action at the t-th time step, that is, the t-th time step is the n-th attacker action a defendern , then q(t)=q n , c(t)=c n .

[0076] Step S2: Setting up the gaming environment

[0077] In the framework of game reinforcement learning, the attacker’s reward r attacker (t) is:

[0078] rattacker (t) = RA suc (Attacker Status,Attacker Action)-RA cost (attacker state, attacker action), where RA suc (attacker state, attacker action) = p(t) × i(t), RA cost (Attacker state, attacker action) is to take attacker action a attacker (t), the risk of the attacker being detected by the system, a attacker (t) is the attacker action at the t-th time step, i.e., the t-th time step is the m-th attacker action a attackerm , then a attacker (t) = a attackerm .

[0079] Defender's Reward defender (t) is:

[0080] r defender (t) = RD suc (Defender Status, Defender Action) - RD cost (Defender state, defender action), where RD suc (Defender state, defender action) is to take defensive action a defender (t), the reward for successful defense can be defined as a large constant in the specific real-time process, indicating the reward for successful defense. cost (Defender state, defender action) represents the cost of taking the defender action, a defender (t) is the defender action at the t-th time step, i.e., the t-th time step is the n-th defender action a defendern , then a defender (t) = a defendern , R.D. cost (Defender state, defender action) = q(t) × c(t).

[0081] In this embodiment, the state space and action space of the attacker and defender in the network security attack and defense game can be described as follows:

[0082] The attacker's state space is divided into four parts. Network topology information, including the number of machines and connections controlled by the attacker; system vulnerability information, including the type of vulnerability and its severity level on each machine detected by the attacker; system asset information, including the important assets on each machine confirmed by the attacker; system access permission information, including the access permission of each user obtained by the attacker. In this embodiment, the attacker's state space is a 52-bit vector. It includes the number of machines N controlled by the attacker. attacker-host =13, number of connections Nattacker-connections =122, etc., which can be represented by the vector [13,122]. The remaining 50 bits of the vector represent the basic status information of the machine that can be obtained, including whether there is abnormal activity on the machine and the degree of damage to the machine.

[0083] The attacker's action space: port scanning, vulnerability exploitation, privilege escalation, data theft.

[0084] The defender's state space is divided into five parts. Network topology information, including the number of machines and connections controlled by the defender; system vulnerability information, including the type of vulnerability and its severity level on each machine discovered by the defender; system security configuration information, including the security protection measures deployed by the defender on each machine; asset information, including the important assets on each machine confirmed by the defender; system access permission information, including the access permission of each user controlled by the defender. In this embodiment, the defender's state space is a 40-bit vector. Network topology information, including the number of machines N controlled by the defender defender-host =13, number of connections N defender-connections = 122. The remaining 38 bits of the vector indicate whether the last attack was successful, which machines were attacked, and the level of authority on the machine.

[0085] Action space for defenders: Analyze machines, terminate malicious processes, terminate malicious processes, and deploy honeypot services.

[0086] Step S3: Attack and Defense Game

[0087] The optimal defender, optimal attacker and game equilibrium in the cybersecurity attack and defense game: We use π attacker Denote the attacker's strategy, using π defender represents the defender's strategy, where strategy refers to the rules or methods for selecting actions in a given state. It can be a deterministic strategy, that is, always choosing the same action in a certain state; or a random strategy, that is, choosing different actions with a certain probability in a certain state. attAcKEr It represents the attacker's benefit under a specific attacker strategy and defender strategy, that is, RA (attacker state, attacker action) accumulated over time. deFender It represents the defender's benefit under specific attacker and defender strategies, that is, RD (defender state, defender action) accumulated over time.

[0088] Optimal Defender. The ultimate goal of the attack and defense game is to find the optimal strategy for the defender. The defender's payoff is J defender reaches the maximum value,

[0089] in is the attacker's optimal strategy. The optimal defender will choose the optimal defender strategy based on the defender state space and the defender action space. To maximize your own benefits defender .

[0090] Optimal Attacker. When we find the optimal strategy for the defender, we will also find the optimal strategy for the attacker. This is a method to help us find the optimal strategy for the defender, so that the attacker's profit J attac reaches the maximum value,

[0091] in is the optimal strategy of the defender. The optimal attacker will choose the optimal attacker strategy based on the attacker state space and attacker action space. To maximize your own benefits attacker .

[0092] When the game is in equilibrium, we find the optimal attacker strategy and defender strategies The following conditions are met.

[0093]

[0094] In the state of attack-defense game equilibrium, neither the attacker nor the defender can increase their own benefits by unilaterally changing their strategies. Specifically:

[0095] Create two PPO agents, the attacker PPO agent and the defender PPO agent. The attacker PPO agent is used to determine the attacker's environment state s according to the t-th time step. attacker (t) Get the attacker's action a attacker (t), the network parameters of the attacker’s PPO agent are the attacker’s strategy π attacker , the defender PPO agent is used to determine the defender’s environment state s at the t-th time step defende (t), the defender's reward r defender (t), get the defender action a defender (t), the network parameters of the defender PPO agent are the defender strategy π defender .

[0096] Build a simulation environment E for the network. The input of the simulation environment E is the attacker action a at the tth time step. attacker (t) and the defender action a defender(t), the output of the simulation environment E is the attacker’s environment state s attacker (t+1) and reward r attacker (t+1) and is the defender's environmental state s defender (t+1) and reward r attacker (t+1).

[0097] In this embodiment, if Figure 3 As shown in Figure 1, it shows the relationship between the two agents created using the PPO algorithm and the environment. The two agents need to learn appropriate strategies through reinforcement learning to cope with the challenges of attack and defense. These agents can continuously optimize their strategies to improve the performance of attack and defense tasks and promote competition and learning between agents: alternately train the attacker strategy and the defender strategy, such as Figure 4 Shown, including

[0098] Step S3.1: Initialization

[0099] Initialize the attacker strategy pool Pool attacker And the defender strategy pool Pool defender , set the maximum number of iterations to K, and initialize the number of iterations k = 0.

[0100] Step S3.2: Defender strategy optimization

[0101] In this embodiment, if Figure 5 As shown, the defender strategy optimization includes the following steps:

[0102] Step S3.2.1: Initialization

[0103] Clear the experience buffer, initialize the simulation environment E, and initialize the network parameters of the defender PPO agent, i.e., the defender strategy π defender ;

[0104] Step S3.2.2: Adversarial Training

[0105] From the attacker strategy pool Pool attacker Select an attacker strategy π by sampling attacker Given an attacker PPO agent, where the sampling probability is 80% from the attacker strategy pool Pool attacker Randomly select an attacker strategy π from the first L attacker , 20% probability from the attacker strategy pool Pool attacker Randomly select an attacker strategy π from the remaining attacker strategies attacker ;

[0106] According to the simulation environment E, the attacker's environment state s at the first time step is obtained attacker(1) and the attacker’s reward r attacker (1) and the defender's environmental state s defender (1) and the defender's reward r defender (1), and the environmental state s attacker (1) Input the attacker PPO agent and obtain the attacker action a attacker (1), the defender's environmental state s defender (1) Input the defender PPO agent and obtain the defender action a defender (1) The attacker action a attacker (1) Defender action a defender (1) Input into the simulation environment E and obtain the attacker's environment state s at the second time step attacker (2) and the attacker’s reward r attacker (2) and the defender's environmental state s defender (2) and the defender's reward r defender (2);

[0107] From the attacker strategy pool Pool attacker Select an attacker strategy π by sampling attacker Give the attacker PPO agent the environment state s attacker (2) Send the attacker’s PPO agent to obtain the attacker’s action a attacker (2) The defender’s environmental state s defender (2) Send it to the defender PPO agent and get the defender action a defender (2) The attacker action a attacker (2) Defender action a defender (2) Input into the simulation environment E and obtain the attacker's environment state s at the third time step attacker (3) and the attacker’s reward r attacker (3) and the defender's environmental state s defender (3) and the defender's reward r defender (3);

[0108] In this way, adversarial training is performed to obtain a series of adversarial samples:

[0109] (s defender (t),a defender (t),r defender (t),s defender (t+1))

[0110] Step S3.2.3: Update the network parameters of the defender PPO agent

[0111] The adversarial samples are placed in the experience buffer. When it is full, all adversarial samples are taken out of the experience buffer and the PPO algorithm is used to update the network parameters of the defender PPO agent, i.e., the defender strategy π. defender ;

[0112] Step S3.2.4: Determine the updated defender strategy π defender With the current defender strategy π defender Compare whether it is improved. If so, clear the experience buffer and continue to put adversarial samples in. When it is full, continue to take out and update the defender strategy π defender , continue to judge whether it has improved. If so, continue to clear the experience buffer and continue to put adversarial samples until there is no improvement. defender Put it into the defender strategy pool defender in front.

[0113] Step S3.3: Attacker strategy optimization

[0114] In this embodiment, if Figure 5 As shown, the attacker strategy optimization includes the following steps:

[0115] Step S3.3.1: Initialization

[0116] Clear the experience buffer, initialize the simulation environment E, and initialize the network parameters of the attacker PPO agent, that is, the attacker strategy π attacker ;

[0117] Step S3.3.2: Adversarial Training

[0118] From the defender strategy pool Pool defender Sampling a defender strategy π defender Given a defender PPO agent, where the sampling probability is 80% from the defender strategy pool Pool defender Randomly select a defender strategy π from the first L ddefender , 20% probability from the defender strategy pool Pool defender Randomly select a defender strategy π from the remaining defender strategies defender ;

[0119] According to the simulation environment E, the attacker's environment state s at the first time step is obtained attacker (1) and the attacker’s reward r attacker (1) and the defender's environmental state s defender (1) and the defender's reward r defender (1), and the environmental state s attacker (1) Input the attacker PPO agent and obtain the attacker action a attacker(1), the defender's environmental state s defender (1) Input the defender PPO agent and obtain the defender action a defender (1) The attacker action a attacker (1) Defender action a defender (1) Input into the simulation environment E and obtain the attacker's environment state s at the second time step attacker (2) and the attacker’s reward r attacker (2) and the defender's environmental state s defender (2) and the defender's reward r defender (2);

[0120] From the defender strategy pool Pool defender Sampling a defender strategy π defender Give the defender PPO agent the environment state s attacker (2) Send the attacker’s PPO agent to obtain the attacker’s action a attacker (2) The defender’s environmental state s defender (2) Send it to the defender PPO agent and get the defender action a defende (2) The attacker action a attacker (2) Defender action a defender (2) Input into the simulation environment E and obtain the attacker's environment state s at the third time step attacker (3) and the attacker’s reward r attacker (3) and the defender's environmental state s defender (3) and the defender's reward r defender (3);

[0121] In this way, adversarial training is performed to obtain a series of adversarial samples:

[0122] (s attecker (t),a attacker (t),r attacker (t),s attacker (t+1))

[0123] Step S3.3.3: Update the network parameters of the attacker PPO agent

[0124] The adversarial samples are placed in the experience buffer. When it is full, all adversarial samples are taken out of the experience buffer and the PPO algorithm is used to update the network parameters of the attacker’s PPO agent, i.e., the attacker’s strategy π. attacker ;

[0125] Step S3.3.4: Determine the updated attacker strategy π attacker With the current attacker strategy π attackerCompare whether it is improved. If so, clear the experience buffer and continue to put adversarial samples in. When it is full, continue to take out and update the attacker strategy π attacker , continue to judge whether it has improved, if so, continue to clear the experience buffer, continue to put in adversarial samples, and continue to judge until there is no improvement, and change the current attacker strategy π attacker Put it into the attacker strategy pool Pool attacker in front.

[0126] Step S3.4: k = k + 1, determine whether the number of iterations k is less than the maximum number of iterations K. If not, the iteration ends and the defender strategy π is obtained. defender This is the optimal network security defense strategy. Otherwise, return to step S3.2.

[0127] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A network security defense method based on game reinforcement learning, characterized in that: The following steps are involved: (1) Establishing offensive and defensive game objectives Determine key assets and importance weights: S = {s1, s2, …s J }, W={w1,w2,…s J }, where S is the key asset set, s j ,j=1,2,…,J is the jth key asset, W is the importance weight set, w j ,j=1,2,…,J is the weight of the jth key asset, J is the number of key assets; Determine the attacker's action space, the probability of action occurrence, and the impact: A attacker = {a attacker1 ,a attacker2 ,…a attackerM }, P = {p1, p2, ... p M }, I={i1,i2,…i M }, where A attacker is the attacker's action space, a attackerm ,m=1,2,…,M is the mth attacker action, P is the occurrence probability set, p m ,m=1,2,…,M is the probability of the mth attacker’s action, I is the set of influence levels, i m ,m=1,2,…,M is the impact degree of the mth attacker action on the key assets, and M is the number of attacker actions; Determine the defender's action space, action probability, and cost: A defender = {a defender1 ,a defender2 ,…a defenderN }, Q={q1,q2,…q N }, C={c1,c2,…d N }, where A defender is the defender's action space, a defendern ,n=1,2,…,N is the nth defender action, Q is the occurrence probability set, q n ,n=1,2,…,N is the probability of the nth defender’s action, C is the cost set, c n ,n=1,2,…,N is the cost of the nth defender action, and N is the number of defender actions; Determine the offensive and defensive game objectives: Among them, F is the defense objective function, T is the time step of a complete round of attack and defense interaction, and w(t) is the weight of the key asset targeted by the attacker's action in the tth time step, that is, the weight of the jth key asset s in the tth time step is j , its corresponding importance weight is w(t), then w(t)=w j , p(t) and i(t) are the corresponding probability of the attacker's action at the t-th time step and the degree of impact on key assets, that is, the t-th time step is the m-th attacker action a attackerm , then p(t)=p m , i(t)=i m , q(t) and c(t) are the corresponding probability and cost of the defender's action at the t-th time step, that is, the t-th time step is the n-th attacker action a defendern , then q(t)=q n , c(t)=c n . (2) Setting the gaming environment In the framework of game reinforcement learning, the attacker’s reward r attacker (t) is: r attacker (t) = RA suc (Attacker Status,Attacker Action)-RA cost (attacker state, attacker action), where RA suc (attacker state, attacker action) = p(t) × i(t), RA cost (Attacker state, attacker action) is to take attacker action a attacker (t), the risk of the attacker being detected by the system, a attacker (t) is the attacker action at the t-th time step, i.e., the t-th time step is the m-th attacker action a attackerm , then a attacker (t) = a attackerm ; Defender's Reward defender (t) is: r defender (t) = RD suc (Defender Status, Defender Action) - RD cost (Defender state, defender action), where RD suc (Defender state, defender action) is to take defensive action a defender (t) after which the reward for successful defense, RD cost (Defender state, defender action) is the cost of taking the defender action, a defender (t) is the defender action at the t-th time step, i.e., the t-th time step is the n-th defender action a defendern , then a defender (t) = a defendern , R.D. cost (defender state, defender action) = q(t) × c(t); (3) Offense and defense game Create two PPO agents, the attacker PPO agent and the defender PPO agent. The attacker PPO agent is used to determine the attacker's environment state s according to the t-th time step. attacker (t), the attacker’s reward r attacker (t), get the attacker action a attacker (t), the network parameters of the attacker’s PPO agent are the attacker’s strategy π attacker , the defender PPO agent is used to determine the defender’s environment state s at the t-th time step defender (t), get the defender action a defender (t), the network parameters of the defender PPO agent are the defender strategy π defender ; Build a simulation environment E for the network. The input of the simulation environment E is the attacker action a at the tth time step. attacker (t) and the defender action a defender (t), the output of the simulation environment E is the attacker’s environment state s attacker (t+1) and reward r attacker (t+1) and the defender's environmental state s defender (t+1) and reward r attacker (t+1); Alternately train the attacker and defender strategies: 3.1) Initialization Initialize the attacker strategy pool Pool attacker And the defender strategy pool Pool defender , set the maximum number of iterations to K, and initialize the number of iterations k = 0; 3.2) Defender Strategy Optimization Clear the experience buffer, initialize the simulation environment E, and initialize the network parameters of the defender PPO agent, i.e., the defender strategy π defebder , from the attacker strategy pool Pool attacker Select an attacker strategy π by sampling attacker Given an attacker PPO agent, where the sampling probability is 80% from the attacker strategy pool Pool attacker Randomly select an attacker strategy π from the first L attacker , 20% probability from the attacker strategy pool Pool attacker Randomly select an attacker strategy π from the remaining attacker strategies attacker ; According to the simulation environment E, the attacker's environment state s at the first time step is obtained attacker (1) and the attacker’s reward r attacker (1) and the defender's environmental state s defende (1) Rewards for defenders r defender (1), and the environmental state s attacker (1) Input the attacker PPO agent and obtain the attacker action a attacker (1), the defender's environmental state s defender (1) Input the defender PPO agent and obtain the defender action a defender (1) The attacker action a attacker (1) Defender action a defender (1) Input into the simulation environment E and obtain the attacker's environment state s at the second time step attacker (2) and the attacker’s reward r attacker (2) and the defender's environmental state s defende (2) Rewards for defenders r defender (2)； From the attacker strategy pool Pool attacker Select an attacker strategy π by sampling attacker Give the attacker PPO agent the environment state s attacker (2) Send the attacker’s PPO agent to obtain the attacker’s action a attacker (2) The defender’s environmental state s defender (2) Send it to the defender PPO agent and get the defender action a defender (2) The attacker action a attacker (2) Defender action a defender (2) Input into the simulation environment E and obtain the attacker's environment state s at the third time step attacker (3) and the attacker’s reward r attacker (3) and the defender's environmental state s defende (3) and the defender's reward r defender (3); In this way, adversarial training is performed to obtain a series of adversarial samples: (s defender (t),a defender (t),r defender (t),s defender (t+1)) The adversarial samples are placed in the experience buffer. When it is full, all adversarial samples are taken out of the experience buffer and the PPO algorithm is used to update the network parameters of the defender PPO agent, i.e., the defender strategy π. defender ; Determine the updated defender strategy π defender With the current defender strategy π defender Compare whether it is improved. If so, clear the experience buffer and continue to put adversarial samples in. When it is full, continue to take out and update the defender strategy π defender , continue to judge whether it has improved. If so, continue to clear the experience buffer and continue to put adversarial samples until there is no improvement. defender Put it into the defender strategy pool defender the front of; 3.3) Attacker strategy optimization Clear the experience buffer, initialize the simulation environment E, and initialize the network parameters of the attacker PPO agent, that is, the attacker strategy π attacker , from the defender strategy pool Pool defender Sampling a defender strategy π attacker Given a defender PPO agent, where the sampling probability is 80% from the defender strategy pool Pool defender Randomly select a defender strategy π from the first L defender , 20% probability from the defender strategy pool Pool defender Randomly select a defender strategy π from the remaining defender strategies defender ; According to the simulation environment E, the attacker's environment state s at the first time step is obtained attacker (1) and the attacker’s reward r attacker (1) and the defender's environmental state s defender (1) and the defender's reward r defender (1), and the environmental state s attacker (1) Input the attacker PPO agent and obtain the attacker action a attacker (1), the defender's environmental state s defender (1) Input the defender PPO agent and obtain the defender action a defender (1) The attacker action a attacker (1) Defender action a defender (1) Input into the simulation environment E and obtain the attacker's environment state s at the second time step attacker (2) and the attacker’s reward r attacker (2) and the defender's environmental state s defender (2) and the defender's reward r defender (2); From the defender strategy pool Pool defender Sampling a defender strategy π defender Give the defender PPO agent the environment state s attacker (2) Send the attacker’s PPO agent to obtain the attacker’s action a attacker (2) The defender’s environmental state s defender (2) Send it to the defender PPO agent and get the defender action a defende (2) The attacker action a attacker (2) Defender action a defender (2) Input into the simulation environment E and obtain the attacker's environment state s at the third time step attacker (3) and the attacker’s reward r attacker (3) and the defender's environmental state s defender (3) and the defender's reward r defender (3); In this way, adversarial training is performed to obtain a series of adversarial samples: (s attacker (t),a attacker (t),r attacker (t),s attacker (t+1)) The adversarial samples are placed in the experience buffer. When it is full, all adversarial samples are taken out of the experience buffer and the PPO algorithm is used to update the network parameters of the attacker’s PPO agent, i.e., the attacker’s strategy π. attacker ; Determine the updated attacker strategy π attacker With the current attacker strategy π attacker Compare whether it is improved. If so, clear the experience buffer and continue to put adversarial samples in. When it is full, continue to take out and update the attacker strategy π attacker , continue to judge whether it has improved, if so, continue to clear the experience buffer, continue to put in adversarial samples, and continue to judge until there is no improvement, and change the current attacker strategy π attacker Put it into the attacker strategy pool Pool attacker the front of; 3.3), k = k + 1, determine whether the number of iterations k is less than the maximum number of iterations K, if not, the iteration ends, and the defender strategy π is obtained defender This is the optimal network security defense strategy. Otherwise, return to step 3.2).

2. The network security defense method of game reinforcement learning according to claim 1 is characterized in that: The attacker action space includes four attacker actions: port scanning, vulnerability exploitation, privilege escalation, and data theft. The attacker state space is divided into four parts: network topology information, system vulnerability information, system asset information, and system access permission information. The defender action space includes four defender actions: analyzing machines, terminating malicious processes, terminating malicious processes, and deploying honeypot services. The defender's state space is divided into five parts: network topology information, system vulnerability information, system security configuration information, asset information, and system access permission information.

Images