SGCS control method based on NPPO reinforcement learning

Abstract

The invention relates to the technical field of intelligent power grids, in particular to an SGCS control method based on NPPO reinforcement learning, and the method comprises the steps: constructing an SGCS network which comprises a switch, a host, a router, a server and a firewall; taking the SGCS network as an environment, and training the PPO model; normalizing advantages, states and rewards in the PPO model to obtain an NPPO model; and constructing a reward of the defender agent by using the NPPO model, taking an action by using the defender agent, updating the state of the SGCS network, and outputting a next reward. The method solves the problem that when an existing PPO algorithm is used for processing complex network attacks, sample efficiency is low, rewards are sparse, robustness is poor, and therefore defense strategies are slow in convergence or lack of effectiveness.

Description

Technical Field

[0001] The present invention relates to the technical field of smart grid, and in particular to a SGCS control method based on NPPO reinforcement learning. Background Art

[0002] Smart grid network security faces increasing risks. Traditional defense strategies such as intrusion detection systems (IDS / IPS), anomaly detection, access control, and malware defense mainly rely on static rules and known attack patterns, which are difficult to deal with unknown or emerging attacks. Due to the complexity of smart grids, centralized defense may cause computing delays and make it difficult to respond to dynamic attacks in real time.

[0003] Sahani, N. et al. pointed out in Machine Learning-Based Intrusion Detection for SmartGrid Computing: A Survey. that the application of intrusion detection system (IDS) in smart grid is very important, especially the IDS driven by machine learning (ML) can provide real-time attack detection and classification functions, and improve the detection ability of zero-day attacks and unknown anomalies. Despite this, traditional IDS still faces many limitations, especially in the complex and dynamic network environment of smart grid, the traditional signature-based detection method and anomaly detection method are insufficient in real-time and adaptability. Therefore, the defense system of smart grid urgently needs more flexible and intelligent solutions to adapt to new attack modes and real-time response requirements. Reinforcement learning (RL) is widely used in network defense, especially the PPO (Proximal Policy Optimization) algorithm. However, PPO has problems such as low sample efficiency, sparse rewards, and poor robustness when dealing with complex network attacks, which can easily lead to slow convergence or lack of effectiveness of defense strategies; existing reward functions are usually simple in design, with problems such as sparse rewards and single targets, making it difficult to cope with complex network security environments. Especially in the face of dynamic attacks, the limitations of reward functions make it difficult for defense strategies to effectively adapt to the strategy changes of attacker agents. In addition, existing studies mostly regard attacker agents as black boxes and lack the identification and classification of attacker agents, which makes it impossible for defender agents to accurately predict and respond to behavioral changes of attacker agents, thereby limiting the flexibility and effectiveness of defense systems. Summary of the invention

[0004] In view of the shortcomings of the existing methods, the present invention solves the problems of low sample efficiency, sparse rewards and poor robustness when the existing PPO algorithm handles complex network attacks, which leads to slow convergence or lack of effectiveness of the defense strategy.

[0005] The technical solution adopted by the present invention is: a SGCS control method based on NPPO reinforcement learning comprises the following steps:

[0006] Step 1: Build an SGCS network including switches, hosts, routers, servers, and firewalls;

[0007] Step 2: Train the PPO model using the SGCS network as the environment; use the NPPO model to construct the reward of the defender agent, use the defender agent to take actions, update the state of the SGCS network, and output the next reward value; and normalize the advantages, states, and rewards in the NPPO model to obtain the NPPO model;

[0008] As a preferred embodiment of the present invention, the construction of the NPPO model includes:

[0009] Initialize the environment, policy network, and evaluation network, set hyperparameters, ReplayBuffer, total time step, maximum episode time step, update period, batch size, and mini-batch size;

[0010] Train for i epochs: For each epoch, execute:

[0011] Reset the environment and get the status of the current environment t ;

[0012] Based on the policy network, using the current state s t Calculate the action probability distribution And according to Sampling gets action a t ;

[0013] Select a reward calculator and perform action a in the environment t , get reward r t and the next state s t+1 ; Then change the status award a t , Next state Store in ReplayBuffer;

[0014] Calculate the advantage value of the current strategy based on TD error

[0015] According to the advantage value Calculate the target value function: And the advantage value Normalization processing to obtain the advantage value

[0016] Sample data from ReplayBuffer in small batch sizes and optimize the policy network π based on the PPO loss function θ , and update the parameters θ by gradient descent;

[0017] Optimize the evaluation network, minimize the value function error, and update the parameter ω;

[0018] After each update is completed, the ReplayBuffer is cleared to prepare data for the next round of training.

[0019] As a preferred embodiment of the present invention, the advantage value The formula is:

[0020]

[0021] in, Indicates the overall average value of the current batch; A std Represents the standard deviation of the current batch; ∈ is a small positive value; is the advantage value.

[0022] As a preferred embodiment of the present invention, the state The formula is:

[0023]

[0024] in, represents the normalized state value at the current time t, represents the normalized reward state value at the current time t, represents the normalized reward state value at time t-1, S t,std Represents the normalized state standard deviation at the current time t, s t Indicates the current state value, S t represents the cumulative variance of the normalized state value at the current time t, S t-1 It represents the cumulative variance of the normalized state value at time t-1, and n represents the number of samples that have been observed.

[0025] As a preferred embodiment of the present invention, reward The formula is:

[0026]

[0027] in, represents the normalized reward value at the current time t, represents the normalized reward average at the current time t, represents the normalized reward average at time t-1, R t,std represents the normalized reward standard deviation at the current time t, rt represents the reward value at the current time t, R t represents the cumulative variance of the normalized reward value at the current time t, R t-1 It represents the cumulative variance of the normalized reward value at time t-1, and n represents the number of samples that have been observed.

[0028] As a preferred embodiment of the present invention, the current reward value r t After improvement, the improved total reward value formula is:

[0029] r' t =r t +Punish+cost total (11)

[0030] Among them, Punish is the reward received by the defender agent, cost total The cost required for the defender agent to take a defensive action.

[0031] As a preferred embodiment of the present invention, the formula for the reward received by the defender agent is:

[0032] punish=(C×w c +V×w v )×Baseline red (12)

[0033] Where C represents the complexity of the attack behavior; w c is the weight assigned to the complexity of the attack; V represents the threat level of the attack behavior; w v is the threat level weight of the attack behavior, Baseline red Baseline score for attacker agents.

[0034] As a preferred embodiment of the present invention, the formula for the cost required for the defender agent to take a defensive action is:

[0035] cost total =(C B ×ω B +C T ×ω T )×Baseline blue (13)

[0036] Among them, C B represents the bandwidth cost, ω B is the weight of bandwidth cost; C T Refers to time cost; B is the weight of time cost; Baseline blue is the baseline score of the defender agent.

[0037] As a preferred implementation manner of the present invention, it also includes: Step 3, constructing a bandit model, and using the bandit model to classify the attacker agent type of the trained NPPO model.

[0038] As a preferred embodiment of the present invention, the SGCS control system based on NPPO reinforcement learning includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the SGCS control method based on NPPO reinforcement learning.

[0039] Beneficial effects of the present invention:

[0040] 1. The present invention designs Normalized PPO to normalize the advantage, state, and reward respectively. Advantage normalization solves the problem that the high variance of the alternative objective function destroys the stability of the strategy network update; state and reward normalization ensures consistency with the standard distribution and provides learning stability;

[0041] 2. Two key innovations are introduced. Two basic elements are added to the reward value: one is that the attacker agent imposes a reward on the defender agent after a successful attack, and the other is to increase the cost of the defense action; it can more accurately simulate the real-world network attack and defense scenarios, focusing on evaluating the specific impact of each attack and defense action and their respective weight ratios to more accurately calculate the final reward; this method enhances the applicability of the model by closely reflecting the complexity of the real world; the effectiveness of the modified reward value, the results show that the modified reward calculator can train strategies that are comparable to or even better than the original algorithm, significantly improving the defense performance. This finding highlights the practical value of theoretical progress and provides a novel perspective for the field of network security;

[0042] 3. Introduce the bandit classification method for effective attacker proxy type identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a control logic block diagram of the SGCS based on NPPO reinforcement learning of the present invention;

[0044] Figure 2 is a schematic diagram of the SGCS system of the present invention;

[0045] Figure 3 It is a comparison chart of average reward learning curves of different models of the present invention in different scenarios. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0047] Reinforcement Learning (RL) is a machine learning method in which an agent interacts with an environment and chooses the best action to maximize the cumulative reward in a specific situation. Reinforcement learning algorithms can be roughly divided into value-based methods and policy gradient-based methods. In the field of reinforcement learning, Markov decision processes (MDPs) are very important, and it is necessary to consider the five-tuple S, A, p, r, γ contained in it. Here, assume that S and A are and A subset of; state transition probability distribution Reward Function and the discount factor γ∈[0,1], are important components in MDPs; let the random strategy be The goal of reinforcement learning is to find the optimal policy π * , so that for all states Value Function Maximize; value function v π (s) represents the expected total discounted reward that can be obtained according to strategy π in state s, and its expression is:

[0048]

[0049] Among them, a t ~π(a t ∣s t ), s t+1 ~p(s t+1 ∣s t ,a t ), and r t = r(s t ,a t ,s t+1 ); In addition, using the state action value function q π The standard definition of (s,a) is:

[0050]

[0051] Value-based reinforcement learning (RL) methods mainly focus on approximating the optimal action value in various scenarios. Notable algorithms in this paradigm include Q-learning, DQN (Deep Q Network) and SARSA.

[0052] Q-learning algorithms strategically identify a sequence of actions that cumulatively optimize reward over a span of time; essentially, as the algorithm iteratively explores various actions, it refines its understanding of the value of the actions based on the outcomes.

[0053] However, Q-learning encounters inherent challenges in multifaceted scenarios characterized by a wide range of potential actions; the DQN algorithm leverages deep learning principles to provide a nuanced ability to interpret and operate in such complex environments.

[0054] Nonetheless, a limitation of DQN is that it occasionally overestimates action values, leading to instability; Dual Deep Q Network (DDQN) alleviates this problem by deploying a dual network: one dedicated to action selection and the other dedicated to evaluating these selected actions; RNN-DDQN advances this concept, seamlessly integrating sequence processing, allowing it to expertly manage dynamic environments with inherent temporal dependencies.

[0055] On the other hand, policy gradient based reinforcement learning methods directly learn the optimal policy by estimating the parameters of the policy; these methods optimize the policy parameters by gradient ascent with respect to the expected reward objective; Actor-Critic (AC) and Proximal Policy Optimization (PPO) are examples of policy gradient based reinforcement learning methods; it is impractical to find the best action for all states in a large state space; instead, policy gradient based methods use a policy π parameterized by a vector θ θ (a|s) and search for the optimal parameter vector θ; the function that determines the state value is also parameterized as V(s,w) using the weight vector w; the goal is to find the optimal θ * To maximize the strategy π θ The average performance is expressed as:

[0056]

[0057] Among them, a t ~π θ (a t ∣s t ),s t+1 ~p(s t+1 ∣s t ,a t ), and r t = r(s t ,a t ,s t+1 ), the basic strategy gradient aims to optimize Equation 3, and its gradient can be expressed as:

[0058]

[0059] in, is πθ The steady-state distribution under is through and Estimated advantage function.

[0060] Based on the policy gradient method, PPO uses an unconstrained surrogate objective function and generalized advantage estimation to achieve faster convergence and superior performance on continuous control tasks; the surrogate objective function in PPO aims to approximate the ratio between the new and old policy distributions; using the unconstrained surrogate objective function, PPO allows for a wider range of updates to policy parameters without a strict clipping threshold; the surrogate objective function can be expressed as the formula:

[0061]

[0062] Among them, the strategy ratio Used to adjust ρ through formula (5) θ′ This constraint prevents the value of ρ θ′ Unstable learning due to large values; advantage function The estimate is expressed as That is, the advantage value, which is formalized as formula (6); where δ l is the TD error, which is r l +γV(s l+1 ,w)-V(s l ,w);r l is the reward at time step l, and V is the value function that estimates the expected cumulative reward for a given state under the current policy.

[0063]

[0064] In the field of network security, the strategies and actions of attacker agents and defender agents are crucial to ensuring information security; various techniques that attacker agents may adopt include system reconnaissance, network service discovery, service vulnerability exploitation, privilege escalation and impact on system services; these actions not only reveal the various ways in which attacker agents can penetrate and control systems, but also reveal the ability of attacker agents to cause damage in cyberspace.

[0065] like Figure 1 As shown, a SGCS control method based on NPPO reinforcement learning includes the following steps:

[0066] Step 1: Build an SGCS network including switches, hosts, routers, servers, and firewalls;

[0067] like Figure 2As shown, the SGCS network is divided into subnet 1, subnet 2 and subnet 3; including: switches, hosts, routers, servers and firewalls and other equipment; servers include power substation center servers and smart grid control center operation servers; hosts include: smart grid user hosts and smart grid control center operation hosts.

[0068] Devices such as switches, hosts, routers, servers and firewalls are considered as the environment in the PPO algorithm.

[0069] The potential use of autonomous defense in enhancing the network security of Smart Grid Control System (SGCS) is studied. The configuration of the SGCS network includes detailed information about the network architecture, computing and operational resources, and the locations of attacker agents and defender agents. In the SGCS network, the goal of the attacker agent is to destroy key assets, such as operational servers, while the defender agent attempts to protect the assets through a series of actions. Throughout the attack and defense process, the defender agent must respond to various attack strategies of the attacker agent, identify potential threats, and implement appropriate defense measures.

[0070] Step 2: Construct the Normalized PPO model (NPPO) and normalize the advantages, states, and rewards in the NPPO model. Use advantage normalization to replace the advantages in the alternative objective function to improve the stability of the policy network update. The normalization of states and rewards is to dynamically adjust the mean and standard deviation of states and rewards to ensure consistency with the standard distribution and improve learning stability. Use the NPPO model to construct the rewards of the defender agent, use the defender agent to take actions, update the state of the SGCS network, and output the next reward value.

[0071] The current state of the SGCS network (i.e., the environment) s t is a vector representing the defender agent’s (i.e., intelligent agent) understanding of the entire network environment; the reward r for the defender agent t is the reward obtained by the attacker agent when it takes over other nodes in the network; the actions of the defender agent include various actions that the defender agent can perform, such as "monitoring", "misleading", and "recovery"; based on the value or policy gradient-based reinforcement learning method, the defender agent takes action a t , resulting in the state of the SGCS network being updated to s t+1 And output the next reward r t+1 , after which the iterative process continues until the current episode terminates; the framework also utilizes a replay buffer to store training data randomly sampled from past training steps and applies periodic parameter updates to the policy network and the critic network as part of the deep reinforcement training process.

[0072] The present invention constructs an autonomous defense NPPO model for the SGCS network. The NPPO model includes advantage normalization, state normalization and reward normalization.

[0073] Advantage Normalization: The alternative objective function in PPO incorporates the advantage, as shown in Equation 5; this can lead to high variance, which can destabilize the policy network update; to address this issue, advantage normalization is used to scale the advantage over the data batch, ensuring stable policy updates by subtracting the mean and dividing by the standard deviation of the advantage across the batch. The advantage value is The normalized formula is:

[0074]

[0075] in, Indicates the overall average value of the current batch; A std Represents the standard deviation of the current batch; ∈ is a small positive value to ensure that division by zero does not occur; is the advantage value.

[0076] In PPO, since the scale and scope may vary, normalization of environment states and rewards is crucial for effective policy learning; this normalization is dynamically adjusted based on the mean and standard deviation of the encountered states and rewards to ensure consistency with the standard distribution and provide learning stability; the state normalization method is shown in formula (8) and the reward normalization is shown in formula (9).

[0077] Given an environment state s t , use the following equation to dynamically adjust it, the state normalization The formula is:

[0078]

[0079] in, represents the normalized reward value at the current time t, represents the normalized reward average at the current time t, represents the normalized reward average at time t-1, S t,std represents the normalized reward standard deviation at the current time t, s t represents the reward value at the current moment, S t represents the cumulative variance of the normalized reward value at the current time t, S t-1 It represents the cumulative variance of the normalized reward value at time t-1, and n represents the number of samples that have been observed.

[0080] Given a reward r t , which is dynamically adjusted using the following equation, reward normalization The formula is:

[0081]

[0082] in, represents the normalized reward value at the current time t, represents the normalized reward average at the current time t, represents the normalized reward average at time t-1, R t,std represents the normalized reward standard deviation at the current time t, r t Represents the reward value at the current moment, R t represents the cumulative variance of the normalized reward value at the current time t, R t-1 It represents the cumulative variance of the normalized reward value at time t-1, and n represents the number of samples that have been observed.

[0083] Normalization can offset the adverse effects of extreme rewards in value function training and enable reinforcement learning algorithms to be flexibly applied in different environments.

[0084] The PPO algorithm includes: setting the environment, policy network, evaluation network and algorithm hyperparameters; during each iteration, observing the current state S t , and use the policy network to determine action a t ; Apply the action to the environment to obtain the subsequent state and its corresponding reward; After normalizing the state and reward, the data is stored in the experience buffer.

[0085] After the network update conditions are met, data is retrieved from the experience buffer, and the temporal difference error, advantage function, target value function, and standardized advantage function are calculated in sequence; in a nested loop, a small batch of data is sampled from the experience buffer, and the policy network and evaluation network are updated. At the end of each iteration, the experience buffer is cleared.

[0086] NPPO algorithm, the specific implementation steps are:

[0087] Parameter initialization: Initialize the environment, policy network π θ And the evaluation network V(s,ω), set the hyperparameters γ,λ,∈, the experience buffer ReplayBuffer, and the total time step T total , maximum episode time step T episode , update cycle T, batch size B, mini-batch size MB; for example: a batch size is 128, and the mini-batch size is 64 or 32, which is 1 / 2 or 1 / 4 of the batch size;

[0088] Train for i epochs: For each epoch, do the following:

[0089] Reset the environment and get the status of the current environment t ;

[0090] Action selection and sampling: Based on the policy network, using the current state s t Calculate the action probability distribution And sample the action a according to the distribution t ;

[0091] Environment interaction and data storage:

[0092] First, select the reward calculator and perform action a in the environment t , get reward r t and the next state s t+1 , apply state normalization and reward normalization to the state and reward respectively, and obtain the state award Next state As shown in formula (8) and (9);

[0093] Will Store in ReplayBuffer (experience buffer);

[0094] Calculate the advantage value of the current strategy based on TD error As shown in formula (6);

[0095] Estimating the target value function: Based on the advantage value Calculate the target value function: And the advantage After normalization, we get As shown in formula (7);

[0096] Update the policy and evaluation network: Sample data from the ReplayBuffer in small batch sizes and optimize the policy network π based on the PPO loss function θ , and update the parameters θ by gradient descent;

[0097] At the same time, optimize the evaluation network, minimize the value function error, and update the parameter ω;

[0098] ReplayBuffer is cleared: After each update is completed, the ReplayBuffer is cleared to prepare data for the next round of training.

[0099] Step 3: Construct a multi-armed bandit model (bandit) to classify the attacker agent types when evaluating NPPO;

[0100] Attacker agent type a includes: Bline attacker agent and Meander attacker agent.

[0101] In the field of network security, it is crucial to accurately identify the type of attacker agent. As network attack methods continue to evolve, traditional security defense strategies often find it difficult to quickly and accurately identify and respond to emerging threats. To meet this challenge, the present invention adopts a multi-armed bandit algorithm, which is a strategy in the field of reinforcement learning that can find the best balance between exploration (trying different strategies) and exploitation (using the best-known strategies). In this study, each type of attacker agent is regarded as an arm of the bandit, and the goal is to train a classifier that can accurately predict the current attacker agent type based on the observed data.

[0102] To achieve this goal, the present invention designs a bandit model based on Q-learning, which dynamically adjusts its strategy to optimize the identification of attacker agent types; in this model, a value function Q is defined to estimate the expected utility of taking a certain action under a specific observation state, and the number of times each action is selected is tracked to adjust and optimize the learning process of the model.

[0103] Specifically, the update of the value function Q follows the following form:

[0104]

[0105] Among them, Q(s t ,a) represents the observed state, that is, the environmental state s t The hashable string obtained by encoding the state of the environment and the value of the value function Q under the selected attacker agent type a, which represents the expected utility of choosing a specific type of attacker agent given the observed state; N[s t ][a] Tracking in s t The number of times a specific attacker agent type is selected in a state, which is used to adjust the learning rate so that the estimate becomes more cautious as a specific attacker agent type is repeatedly selected in a specific state; R bandit The immediate reward value obtained after selecting a specific type of attacker agent, which indicates whether the selection of the bandit model matches the actual attacker agent type; if it matches, then R bandit The reward is positive (+1); if not, then R bandit The reward is a negative number (-1).

[0106] CybORG is a versatile and highly modular tool designed to demonstrate a variety of scenarios with varying degrees of realism. The CybORG simulator provides an interface that allows attacker agents and defender agents to interact with complex network simulation environments. It helps abstract the complex expression of real-world scenarios through a series of wrappers. This approach supports reinforcement learning interfaces while maintaining high realism when simulating real-world networks. The modular interface and customizable configuration make CybORG highly adaptable and able to express a variety of industrial control network architectures.

[0107] The baseline network architecture studied in this paper is as follows Figure 2 As shown in FIG. 1 , the entire network is divided into three subnets; a subnet may consist of a group of nodes of the same type or mixed types (e.g., a smart grid user host, a smart grid control center server, a smart grid control center operation server, and a smart grid control center operation host); an attacker agent and a defender agent may be hidden in these subnets; for example, in Figure 2 The attacker agent can be located in subnet 1, while the defender agent is located in subnet 2; the state space, action space and reward of CybORG are described as follows:

[0108] State space s t : The state space is a 52-bit vector, with 4 bits for each of the 13 hosts representing whether the host is scanned or exploited, and the level of access the attacker agent has to the host: no access, user level, administrator level, or unknown.

[0109] Action a t :The concept of action space is crucial in reinforcement learning, especially when used to simulate the complex interactions between attacker and defender agents in cybersecurity; the defined action space contains a comprehensive set of potential strategies that each actor may deploy; this division goes beyond simply outlining tactical options and provides a policy framework that is based on a broad understanding of cybersecurity threats and the sophisticated algorithms of reinforcement learning; the framework helps facilitate the development of complex policies to strengthen defense mechanisms and guide offensive approaches, adapting to the ever-changing cyber threat landscape.

[0110] The action space of the defender agent includes: Sleep: take no action; Analyze: collect additional information about a specific host to help identify if an attacker agent is present in the system; Monitor: observe network traffic and collect information flagged as malicious activity; Remove: suspend processes identified as malicious by the monitoring system; Restore: restore the host to a known good state; Misinform: enable a decoy service on a specific host;

[0111] The actions of the attacker's agent include: Scan Remote Systems (Discover Remote Systems): Use scanning tools such as ping to locate other hosts or IP addresses in the subnet; Scan Network Services (Discover Network Services): Initiate a connection with the target host to discover the services and ports that the target host responds to; Exploit Network Services vulnerabilities (Exploitation Network Services): Try to exploit discovered ports and services on the remote system; Escalate privileges (Escalate): Increase the privileges of the attacker's agent on the host; Impact (Impact): Reduce network performance to achieve the purpose of the attacker's agent denying access to the target server;

[0112] The current reward value r t , i.e., the baseline reward calculator: the defender agent is rewarded when the attacker agent gains administrator access to the system; the negative reward persists as long as the attacker agent retains administrator access; the reward amount depends on the relative importance of the affected host to the confidentiality or availability of the entire system; in the experimental setting, systems with lower importance (smart grid user hosts, power substation operation hosts) receive a reward of -0.1; conversely, systems with higher importance (smart grid control center servers, power substation operation servers) receive a reward of -1.0; in addition, an attacker agent performing an "influence" action on a running server incurs a reward of -10 or a defender agent performing a "recovery" action on any host incurs a reward of -1; the evaluation criterion is the total cumulative reward received by the defender agent throughout the runtime.

[0113] The reward value r at the current moment t To make improvements, considering the single nature of the current rewards, we add rewards for both attackers and defenders based on the baseline reward calculator; for each successful action of the attacker agent, an additional reward will be introduced to the defender agent; for the defender agent, the cost calculation of the defensive action is added; therefore, the calculation formula for the final total reward should be:

[0114] r' t =r t +Punish+cost total (11)

[0115] Among them, r' t Indicates r t The total reward value after improvement; r tIt consists of two parts, one is the availability reward and the other is the confidentiality reward. The availability reward refers to the reward given based on whether the attacker agent performs the "influence" action on the operation server. The "confidentiality reward" is the reward obtained by the attacker agent for occupying the server and the host (a reward of -1 is obtained for each time step for obtaining the administrator access right of the server, and a reward of -0.1 is obtained for each time step for obtaining the administrator access right of the host). Punish refers to the reward given to the defender agent after the attacker agent's attack action is successful, and Cost refers to the cost required for the defender agent to take a defensive action.

[0116] After a successful attack, the reward received by the defender agent is detailed in Table 1, and the formula is:

[0117] punish=(C×w c +V×w v )×Baseline red (12)

[0118] Where, punish represents the reward received by the defender agent; C represents the complexity of the attack behavior (Attack Complexity); w c is the weight assigned to the attack complexity; V represents the threat level of the attack behavior; w v Baseline is the threat level weight of the attack behavior (vulnerability threat). red It is the benchmark score of the attacker's agent, which is used to calculate the overall reward value; among them, for the privilege escalation action, when targeting users or servers with smart grid control center server information, the attack complexity and asset threat are both high; among them, 'very high' is set to -10, 'high' is set to 0.75, 'medium' is set to 0.5, and 'low' is set to 0.1, and each weight is 50%.

[0119] Table 1 Rewards to defender agents based on the attack complexity and vulnerability threats of different actions after a successful attack by the attacker agent

[0120]

[0121] The cost required for the defender agent to take defensive actions is shown in Table 1, and the calculation formula is:

[0122] cost total =(C B ×ω B +C T ×ω T )×Baseline blue (13)

[0123] Among them, costtotal represents the cost required for the defender agent to take defensive actions, C B represents the bandwidth cost, i.e., the cost of network bandwidth resources consumed by implementing defense actions, ω B is the weight of bandwidth cost, which determines its impact on the total cost; C T refers to the time cost, that is, the cost related to the time required to perform the defensive action; ω B is the weight of time cost, indicating its proportion in the total cost; Baseline blue is the baseline score for the defender agent, which is multiplied by the weighted sum of bandwidth and time costs to calculate the total cost; Baseline is set to 1.0, 'High' is set to 1.0, 'Medium' is set to 0.5, and 'Low' is set to 0.1.

[0124] Table 2 Bandwidth cost, time cost and total cost of different actions taken by the defender agent

[0125]

[0126] Environment Setup

[0127] The baseline scenario is Figure 2 As shown, in the experimental scenarios, the game between the attacker agent and the defender agent occurs in various settings, where the green agents (hosts and servers) simulate user activities and generate false alarms for the defender agent; the attacker agent can scan the subnet, launch lateral movement attacks, escalate privileges, or compromise the infected host in each round; at each step, the defender agent receives a vector of the current network state as an observation value, where a fixed set of bits represents each host and receives a numerical reward; the reward value is determined by the importance of each host visited by the attacker agent, and the defender agent is rewarded when using recovery actions; the reward is always negative, so a game score of 0 indicates perfect defense.

[0128] For this lab scenario, the attacker agent can only use one of three built-in strategies; the first is the Bline attacker agent, which is pre-configured with comprehensive knowledge of the network; the second is the Meander attacker agent, which needs to discover network details during the game and attempts to compromise one subnet before moving to the next; the last is the Sleep attacker agent, which performs no actions at each step.

[0129] The standardized PPO model (Proximal Policy Optimization) is trained in seven scenarios, as shown in Table 3;

[0130] Table 3 Description of smart grid network attack scenarios

[0131]

[0132] accomplish:

[0133] The reinforcement learning algorithm was implemented using Nvidia GeForce RTX 3090 GPU and Pytorch 1.9.0 on the Linux platform; for each scenario in Table 3, the Meander attacker agent was trained for 1500 episodes of 100 steps, for a total of 1.5 million steps; the average reward of these 1500 episodes was calculated; then, such a training process was repeated five times, and the maximum average reward (over the five training processes) was recorded; a comparative evaluation of 100 episodes was also performed, including three different episode lengths (30 steps, 50 steps, and 100 steps) and various attacker agents to test whether the multi-armed bandit controller can accurately judge different attacker agents in actual operation. The training parameters of the reinforcement algorithm are as follows:

[0134] 1. Normalized PPO: learning rate is 0.0003, discount factors γ=0.99, γ=0.99, GAE parameter λ=0.95, PPO truncation parameter ∈=0.2, batch size is 2048, and mini-batch size is 64;

[0135] 2. DQN: learning rate is 0.0001, initial exploration rate ∈ = 1, γ = 0.99, and batch size is 32;

[0136] 3. RNN-DDQN: learning rate is 0.0001, initial exploration probability ∈=1, exploration rate decay rate ∈ dec =0.0000065, minimum exploration rate∈ min =0.05, γ=0.99, and the batch size is 32.

[0137] Results and Discussion:

[0138] Figure 3 The average reward learning curves of five training sessions were compared with a confidence level of 95%; first, the Normalized Proximal Policy Optimization (NPPO) algorithm showed better performance than the Proximal Policy Optimization (PPO) algorithm, and showed higher stability in some tasks where the Proximal Policy Optimization showed high variance; this verifies the effectiveness of the standardization operation in improving the stability of the algorithm; in addition, policy-based algorithms are generally superior to value-based algorithms in most cases, which may be because policy-based algorithms usually improve stability by limiting the amplitude of policy updates, which can improve training stability, especially in environments with complex action spaces or sparse rewards; in contrast, value-based reinforcement learning algorithms may find it difficult to stably learn in such situations.

[0139] Table 4 shows the maximum average results of the defender agent in seven scenarios. Normalized PPO outperforms all other algorithms in all scenarios, with test results of -14.04, 13.48, -11.91, -24.83, -6.61, -30.49, and -24.27, respectively, which are the maximum values ​​in each scenario. Figure 3 It is also obvious in Table 4 that the following observations are made:

[0140] 1. Changing the position of the defender agent results in changes in marginal returns; for example, Normalized PPO changes from -14.04 to -13.48 (B1) and -11.91 (B2), while PPO changes from -23.57 to -29.81 (B1) and -21.39 (B2);

[0141] 2. When the attacker agent is located in subnet 3 (R1), the reward of Normalized PPO is -6.61, which is within -10; this indicates that the damage to the smart grid control center server is small and the defense is effective, because a serious attack will result in a reward of -10;

[0142] 3. Changing the number of attacker agents significantly affects the policy-based algorithms, especially PPO. Introducing additional attacker agents causes the reward of proximal policy optimization to drop from -23.57 to -53.87, indicating an increase in vulnerabilities.

[0143] 4. An increase in the size of the subnetwork generally reduces the reward of the algorithm; for example, Normalized PPO changes from -14.04 to -17.92 and -35.18, while PPO changes more significantly, from -23.57 to -49.47 and -38.86; value-based algorithms such as DDQN change from -49.61 to -86.05 and -101.71, among which RNN-DDQN shows a significant decline and slower convergence speed.

[0144] Table 4 Maximum average results of the trained defender agent against the attacker agent in seven scenarios using different algorithms

[0145]

[0146] Table 5 shows the average rewards for evaluating over 100 episodes of different lengths in several scenarios; the rewards obtained by assigning pre-trained models to different attacker agents using a multi-armed bandit consistently exceed the rewards obtained by randomly assigning pre-trained models in all scenarios; in addition, testing shows that the trained multi-armed bandit model can accurately distinguish between known different attacker agents with 100% accuracy.

[0147]

[0148]

[0149] Table 6 illustrates the changes in the defense strategies adopted by the defender agent against different attacker agents in different experimental scenarios under two different reward calculation methods; this analysis is based on 100 episodes in the test phase, each consisting of 100 time steps; the most notable observation is that under the modified reward calculation method, deception (Misinform) is used more cautiously compared to the baseline reward system; this strategic shift mainly stems from the increase in the cost of deception from 0 to 0.3; for example, in the scenario Baseline, when facing the Bline attacker agent, the frequency of using deception actions decreases from 86.63% to 49.03% under the modified system; similarly, against the Meander attacker agent, the usage rate of deception drops significantly from 68.03% to 14.08%; similar trends are also observed in other experimental settings.

[0150] Table 6 Action distribution of each strategy against different attacker agents under different reward calculation methods; for distinction, (Baseline) is added after the model trained by the baseline reward method.

[0151] Table 6 Action distribution of each strategy against different attacker agents under different reward calculation methods

[0152]

[0153]

[0154] The data in Table 7 show that the defense strategy using the NPPO algorithm has achieved significant success in fighting against attacker agents; this is proven by the fact that all availability rewards are zero, indicating that the operating server has not been attacked by the attacker agent; in addition, it can be observed that the total rewards obtained by the model trained with the modified reward calculator in the test are generally higher than the total rewards obtained by the model trained with the baseline reward calculator; for example, in the Baseline scenario against the Bline attacker agent, the total reward obtained by the model is -165.00, while the total reward of the baseline model is -329.94; similar trends are also observed in other scenarios, indicating that the newly added reward mechanism of the present invention effectively improves the model performance, thereby enhancing the defense effect.

[0155] The total reward calculated using the modified reward calculator is the sum of the availability reward, confidentiality reward, reward, and associated costs; here, given that the baseline reward calculation method does not include the reward for a successful attack by the attacker agent or the cost of certain actions by the defender agent, these values ​​are enclosed in brackets to indicate that they are included in the final reward after additional calculation.

[0156] Table 7 shows the total rewards obtained by the defender agent using the modified reward calculation method and the baseline reward calculation method in different scenarios, in a test of 100 episodes, each containing 100 time steps.

[0157] Table 7 Total reward statistics in different scenarios

[0158]

[0159]

[0160] With the increasing number of cyber attacks on smart grid control systems, fast and effective autonomous defense strategies have become crucial; the method of the present invention is based on reinforcement learning and is enhanced by a bandit module for predicting the type of attacker agent, enabling the defender agent to make wise decisions; a large number of tests have shown that the enhanced Normalized PPO model surpasses the traditional model in terms of robustness and rewards, verifying its effectiveness as an autonomous defense mechanism; in addition, experimental results show that increasing the complexity of the reward can effectively improve the performance of the normalized proximal policy optimization model, surpassing its original iteration and further enhancing the defense effect.

[0161] It was observed in the experiments that while the location of the defender agent has little impact on the performance, the location and number of attacker agents significantly affect the effectiveness of the defense, especially in the case of the proximal policy optimization model algorithm; this highlights that the more attacker agents there are, the greater the risk, and confirms the potential of reinforcement learning algorithms to enhance defense in smart grid control systems, providing valuable insights for practical applications.

[0162] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A SGCS control method based on NPPO reinforcement learning, characterized in that: The following steps are involved: Step 1: Build an SGCS network including switches, hosts, routers, servers, and firewalls; Step 2: Train the PPO model using the SGCS network as the environment; use the NPPO model to construct the reward of the defender agent, use the defender agent to take actions, update the state of the SGCS network, and output the next reward value; and normalize the advantages, states, and rewards in the NPPO model to obtain the NPPO model.

2. The SGCS control method based on NPPO reinforcement learning according to claim 1 is characterized in that: The construction of NPPO model includes: Initialize the environment, policy network, and evaluation network, set hyperparameters, ReplayBuffer, total time step, maximum episode time step, update period, batch size, and mini-batch size; Train for i epochs and for each epoch: Reset the environment and get the status of the current environment t ; Based on the policy network, using the current state s t Calculate the action probability distribution And according to Sampling gets action a t ; Select a reward calculator and perform action a in the environment t , get reward r t and the next state s t+1 ; Then change the status award a t , Store in ReplayBuffer; Calculate the advantage value of the current strategy based on TD error According to the advantage value Calculate the target value function: And the advantage value Normalization processing to obtain the advantage value Sample data from ReplayBuffer according to the mini-batch size and optimize the policy network π based on the PPO loss function θ , and update the parameter θ by gradient descent; Optimize the evaluation network, minimize the value function error, and update the parameter ω; After each update is completed, the ReplayBuffer is cleared to prepare data for the next round of training.

3. The SGCS control method based on NPPO reinforcement learning according to claim 2 is characterized in that: Advantage value The formula is: in, Indicates the overall average value of the current batch; A std Represents the standard deviation of the current batch; ∈ is a small positive value; is the advantage value.

4. The SGCS control method based on NPPO reinforcement learning according to claim 2 is characterized in that: state The formula is: in, represents the normalized state value at the current time t, represents the normalized state average at the current time t, represents the normalized state average at time t-1, S t,std Represents the normalized state standard deviation at the current time t, s t Indicates the current state value, S t represents the cumulative variance of the normalized state value at the current time t, S t-1 It represents the cumulative variance of the normalized state value at time t-1, and n represents the number of samples that have been observed.

5. The SGCS control method based on NPPO reinforcement learning according to claim 2 is characterized in that: award The formula is: in, represents the normalized reward value at the current time t, represents the normalized reward average at the current time t, represents the normalized reward average at time t-1, R t,std represents the normalized reward standard deviation at the current time t, r t represents the reward value at the current time t, R t represents the cumulative variance of the normalized reward value at the current time t, R t-1 It represents the cumulative variance of the normalized reward value at time t-1, and n represents the number of samples that have been observed.

6. The SGCS control method based on NPPO reinforcement learning according to claim 5 is characterized in that: The reward value r at the current moment t After improvement, the improved total reward value formula is: r’ t =r t +Punish+cost total (11) Among them, Punish is the reward received by the defender agent, cost total is the cost required for the defender agent to take defensive actions, r t Represents the reward value at the current time t.

7. The SGCS control method based on NPPO reinforcement learning according to claim 6 is characterized in that: The formula for the reward received by the defender agent is: punish=(C×w c +V×w v )×Baseline red (12) Where C represents the complexity of the attack behavior; w c is the weight assigned to the complexity of the attack; V represents the threat level of the attack behavior; w v is the threat level weight of the attack behavior, Baseline red Baseline score for attacker agents.

8. The SGCS control method based on NPPO reinforcement learning according to claim 6, characterized in that: The formula for the cost required for the defender agent to take a defensive action is: cost total =(C B ×ω B +C T ×ω T )×Baseline blue (13) Among them, C B represents the bandwidth cost, ω B is the weight of bandwidth cost; C T Refers to time cost; B is the weight of time cost; Baseline blue is the baseline score of the defender agent.

9. The SGCS control method based on NPPO reinforcement learning according to claim 1, characterized in that: It also includes: Step 3, building a bandit model, and using the bandit model to classify the attacker agent type of the trained NPPO model.

10. The SGCS control system based on NPPO reinforcement learning is characterized by: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the SGCS control method based on NPPO reinforcement learning as described in any one of claims 1 to 9.

Images