CGCN-PPO-based power grid network security defense method

Abstract

The invention relates to the technical field of network security, in particular to a CGCN-PPO-based power grid network security defense method, which comprises the following steps: constructing a network scene, and carrying out action modeling; modeling the actions of the defender agent into node-level, edge-level and global-level actions; creating a feature matrix and an edge index of an attribute node; modeling an attack and defense process; feature enhancement; carrying out dimension reduction on the feature matrix; optimizing an edge index; inputting the current state of the defender agent into the graph convolutional network to obtain the action of the defender agent; and inputting the action of the defender agent into a graph modeling environment to obtain a state of the defender agent at the next moment and a reward value of the defender agent at the next moment, and performing strategy and evaluation network updating by utilizing a feature matrix of dimension reduction at the next moment and a side index of an attribute node. According to the method, the problems of high feature dimension and high calculation complexity when a traditional RL method is used for processing a highly interconnected and distributed environment are solved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a power grid network security defense method based on CGCN-PPO. Background Art

[0002] As an important infrastructure in the energy field, smart grid is a modern power network that uses communication technology and information processing capabilities to optimize electricity production, transmission and consumption; however, this high degree of interconnectivity also makes smart grids face increasingly severe network security threats; such as APT network attacks, which may cause serious failures in smart grid systems, including power outages, equipment damage and economic losses.

[0003] Through reinforcement learning (RL), smart grid defense systems can achieve adaptability, allowing each defense agent to continuously adjust and optimize defense strategies according to real-time changes in attacker behavior, thereby responding quickly to various complex network threats. Although existing RL-based network defense strategies perform well in some scenarios, they still have shortcomings in accurately modeling complex network structures and capturing dynamic interactions between nodes, especially in smart grid environments where IoT devices are highly interconnected. This limitation greatly limits the efficiency and effectiveness of smart defense strategies.

[0004] In addition, although GNN and RL have made significant progress in improving network security defense capabilities, most studies focus on a single defense agent or static network environment, and lack in-depth discussion on the collaborative work of multiple defense agents in smart grids; in addition, traditional RL methods often face challenges of high feature dimensionality and high computational complexity when dealing with highly interconnected and distributed environments such as smart grids, which limits their application in actual large-scale networks. Summary of the invention

[0005] In view of the shortcomings of the existing methods, the present invention constructs a compact graph convolutional network-proximal policy optimization (Compact GCN-PPO, CGCN-PPO) method; based on multi-defense agents, the number of feature matrices and edge indexes received by reinforcement learning is reduced; based on graph modeling, the complex dependencies and interaction patterns between nodes and edges in smart grids can be effectively captured, and the learning and decision-making capabilities of agents in different network environments are enhanced; different types of attackers are constructed to verify the robustness of CGCN-PPO.

[0006] The technical solution adopted by the present invention is: the power grid network security defense method based on CGCN-PPO includes the following steps:

[0007] Step 1: Construct several attribute node network scenarios and use graph modeling to model the actions of attacker and defender agents in the network scenarios.

[0008] As a preferred implementation of the present invention, the attribute nodes include: port nodes, file nodes, system nodes and gateway nodes.

[0009] As a preferred implementation of the present invention, attacker action modeling includes: discovering remote systems, aggressive service discovery, covert service discovery, exploiting network services, privilege escalation, destruction, service degradation, identifying deception and evacuation.

[0010] As a preferred embodiment of the present invention, defender agent action modeling includes: monitoring, analysis, deployment of bait, removal, recovery, blocking / allowing traffic.

[0011] Step 2: Model the actions of the defender agent into node-level, edge-level, and global-level actions; create a feature matrix of attribute nodes, a matrix of non-attribute features, and an edge index of attribute nodes for the defender agent;

[0012] Step 3: Model the attack and defense process of the attacker and defender agents;

[0013] As a preferred embodiment of the present invention, attackers include: aggressive attackers, who modify the probability of attribute nodes being attacked and the probability of selecting actions when the host status is user authority or user authority and discovered, root authority or root authority and discovered.

[0014] As a preferred embodiment of the present invention, the attacker also includes: an exploratory attacker, which modifies the probability of an attribute node being attacked and the probability of selecting an action when the host status is known or known and discovered, the service is known or the service is known and discovered.

[0015] Step 4: Enhance the features of attribute nodes;

[0016] As a preferred embodiment of the present invention, step 4 specifically includes:

[0017] The network topology is modeled as a graph G = (V, E); where V is the attribute node and E is the connection relationship between the attribute nodes;

[0018] Each attribute node v∈V has an initial feature vector GCN aggregates and transforms the node features of each layer with the features of neighboring nodes through several layers of graph convolution operations, gradually capturing higher-order structural information;

[0019] After L layers of graph convolution operations, the final embedding vector of attribute node V is Embed the generated attribute nodes Perform normalization.

[0020] Step 5: Reduce the dimension of the feature matrix of the attribute node; optimize the edge index of the attribute node;

[0021] As a preferred implementation of the present invention, the dimension reduction of the feature matrix of the attribute node is to calculate the reward ratio of each feature, and screen out the features whose reward ratio to the total reward exceeds the reward threshold.

[0022] As a preferred implementation of the present invention, the optimization of the edge index of the attribute node is to screen the temporary edges and eliminate invalid or redundant connections caused by false positives or noise information.

[0023] Step 6: Input the current state of the defender agent into the graph convolutional network to obtain the action of the defender agent; input the action of the defender agent into the graph modeling to obtain the state of the defender agent at the next moment and the reward value at the next moment, and use the feature matrix of the attribute node of the defender agent at the next moment, the matrix of non-attribute features, and the edge index of the attribute node to update the strategy and evaluation network;

[0024] As a preferred implementation of the present invention, step six specifically includes:

[0025] Initialize the environment and graph convolutional network;

[0026] Set hyperparameters: discount factor γ, PPO clipping range ∈, advantage estimation coefficient λ, batch size B, maximum episode time step T total , the number of defender agents N; the experience buffer ReplayBufferr of the i-th defender agent i ;

[0027] In the rth training round, for each defender agent i, the following operations are performed in sequence:

[0028] Reset the environment and get the state s of the i-th defender agent in the current environment t i ;

[0029] Action selection and sampling: observing the current state of the defender agent Extract feature matrix and edge index Using the feature matrix and edge index Get the probability distribution of each action and select the action with the highest probability in the probability distribution

[0030] Graph-based modeling of environmental interaction and data storage: executing actions Interact with the graphical environment and get rewards and the state at the next moment And extract the feature matrix of the next moment and edge index And using the feature matrix and edge index Build Store into experience buffer;

[0031] Estimating the target value function: Computing discounted returns and advantage functions

[0032] Strategy and Evaluation Network Update;

[0033] ReplayBuffer is cleared.

[0034] Beneficial effects of the present invention:

[0035] 1. Use graph data structures to model environmental observations to ensure the accuracy of environmental representation and the stability of the training process. At the same time, through the multi-defender agent reinforcement learning framework, multiple defender agents can work together to integrate observation data from different subnets through information exchange, thereby improving the coordination and effectiveness of the overall defense strategy.

[0036] 2. Feature matrix and edge index optimization technology effectively reduces the computational complexity of the model by reducing the number of useless and irrelevant features and edges, while maintaining or even improving the defense effect; this method ensures that the model can still run efficiently and provide real-time defense response when processing large-scale complex networks;

[0037] 3. Improve the FSM attacker and construct aggressive attackers and exploratory attackers according to the host permissions. The exploratory attacker is suitable for simulating the attacker's behavior in the initial reconnaissance stage of the network; the aggressive attacker tends to choose destructive actions;

[0038] 4. It not only outperforms the traditional graph convolution enhanced PPO (GCN-PPO), but also surpasses the graph attention network enhanced PPO (GAT-PPO) and the traditional PPO and DQN methods that do not use graph neural networks;

[0039] 5. CGCN-PPO shows excellent adaptability and robustness in dealing with different attackers and random attackers. The introduction of a multi-defense agent architecture further enhances its collaboration and defense capabilities. In this architecture, each defense agent focuses on the subnet area it is responsible for, and can quickly respond to complex cross-subnet attacks through efficient information sharing and real-time collaboration. This design not only improves the defense effect against advanced persistent threats (APT), but also shows the practicality of CGCN-PPO in dynamic and complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the CGCN-PPO-based power grid network security defense logic diagram of the present invention;

[0041] Figure 2 is a smart grid topology diagram of the present invention;

[0042] Figure 3 It is a comparison diagram of training curves under different scenarios between the method of the present invention and the existing method. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0044] The present invention is based on the defense model of common advanced persistent threats (APTs) in smart grid control systems (SGCS), and combines a multi-defense intelligent system to achieve efficient defense; APT attacks use a series of complex and advanced technologies (such as system vulnerability exploits, phishing attacks, and zero-day attacks), with the goal of infiltrating the target network, maintaining a long-term hidden presence, and stealing or destroying critical data; such attacks are characterized by phased execution and high complexity, covering multiple stages such as initial intrusion, privilege escalation, and lateral movement, forming a complete attack chain.

[0045] To address these issues, the defender agent needs to develop an automated and optimal defense strategy based on the real-time changes in the network status, to effectively prevent the attacker's infiltration behavior and protect network resources from damage while ensuring the stable operation of the system; since the attacker's strategy is highly dynamic and uncertain, a single defense subject is difficult to cope with complex threats; therefore, the present invention uses a multi-agent defense system to collaborate and coordinate multiple defense subjects, achieve rapid response and decision optimization, and improve the overall defense effect.

[0046] In the framework of multiple defense agents, each defense agent is responsible for monitoring and defending a specific network area, and autonomously selecting the optimal defense action based on the current status; in addition, through efficient information sharing and coordination mechanisms, each defense agent jointly responds to the dynamic changes of APT attacks to ensure the comprehensiveness and robustness of system defense; this intelligent, multi-agent collaborative defense system can quickly adapt to complex network environments under limited resource constraints, and provide efficient solutions for advanced persistent threats.

[0047] like Figure 1 As shown in the figure, the power grid network security defense method based on CGCN-PPO includes the following steps:

[0048] Step 1: Construct several attribute node network scenarios and use graph modeling to model the actions of attacker and defender agents in the network scenarios.

[0049] Attribute nodes include: port nodes, file nodes, system nodes, and gateway nodes;

[0050] Attribute nodes are nodes abstracted from various elements in the network scenario. Port nodes in attribute nodes represent ports and services opened by the host; file nodes represent file resources associated with the host.

[0051] System nodes include system host nodes, system server nodes, and system router nodes, representing core computing and communication equipment;

[0052] Gateway nodes represent connection points between networks;

[0053] The attribute nodes contain characteristic information such as observation space, node type, subnet, importance, system version, etc. The connection relationship between attribute nodes is modeled through the edges of the graph. In addition, the attacker's intrusion and penetration path is mapped to the edges in the graph, including the escalation of permissions of nodes in the network, scanning subnets, and further penetration of service and port nodes. This graph structure modeling method can accurately reflect the network topology, dependencies between nodes, and attack paths, providing clear network abstraction and support for subsequent defense strategies.

[0054] Attacker action modeling includes: discovering remote systems, aggressive service discovery, covert service discovery, exploiting network services, privilege escalation, sabotage, degrading services, identifying decoys, and evacuation;

[0055] Among them, discovering remote systems is to discover new hosts or IP addresses in the network through active scanning;

[0056] Aggressive service discovery is to establish a connection with the target host and discover the services it responds to;

[0057] Covert service discovery is similar to aggressive service discovery, but is slower and quieter, reducing the chance of being detected;

[0058] Exploit network services is an attempt to exploit a specified service vulnerability on a remote system;

[0059] Privilege escalation is to increase the attacker's privileges on the host and obtain a higher level of control;

[0060] Destruction is the destruction of network performance, completing the attacker's goal of blocking operational services;

[0061] Degrading services means blocking the communication of green users and affecting their normal work;

[0062] Identifying decoys is probing a host to determine if it is running a decoy service;

[0063] Evacuation is the removal of the attacker's presence on the target host.

[0064] Defender agent action modeling includes: monitoring, analysis, deployment of decoys, removal, recovery, blocking / allowing traffic;

[0065] Among them, monitoring is to collect information marked as malicious activities in the network;

[0066] Analysis is the collection of more detailed information about malware on a specific host;

[0067] Deploying a decoy is to set up a decoy service on a specified host;

[0068] Removal attempts to eliminate the attacker's presence on the host by destroying malicious processes, files, and services;

[0069] Recovery is restoring a system to a known good state;

[0070] Block / Allow Traffic: Block or allow network traffic between specified security zones.

[0071] Step 2: Model the actions of the defender agent into node-level, edge-level, and global-level actions; create a feature matrix of attribute nodes, a matrix of non-attribute features, and an edge index of attribute nodes for the defender agent;

[0072] Executing actions in a network scenario may result in new edges or nodes being added to the graph. The defender agent's actions are modeled into three different types of actions that can affect the graph, namely, node-level, edge-level, and global-level actions. The changes that actions cause to the graph may be adding nodes, removing edges, or adjusting the structure of the graph globally. Different actions will affect different types of action levels of the graph, namely, node-level, edge-level, and global-level.

[0073] Three different types of actions are modifications to the graph modeling environment;

[0074] Node-level actions include:

[0075] Restoration removes all edges associated with the file and (non-default) connections and restores to the original nodes;

[0076] Deploying a decoy is to create a new port node (decoy) and add an edge connected to the host where the decoy is deployed;

[0077] Analysis is to add a file node when a host node finds a file and add an edge to the analyzed host.

[0078] Edge-level actions include:

[0079] Blocking / allowing traffic is creating / deleting an edge between two subnets.

[0080] Global-level actions include:

[0081] Monitoring is the process of adding any new port nodes that are observed (including port information) and creating edges between these nodes and the host they are on and the source host.

[0082] In order to accurately capture and represent the status and attributes of various types of nodes in the network, feature matrices are designed for each node type; these feature matrices provide comprehensive network information to the defender agent, enabling it to effectively monitor, analyze and respond to various security threats; through various features, the defender agent can identify key nodes, detect abnormal behaviors and optimize defense strategies;

[0083] The attribute node features include: system node features, port node features, file node features, and gateway node features;

[0084] The characteristics of the system node are the operating system version, distribution, architecture, etc., and the value is a one-hot encoded vector; whether it is a user host, server, or router, and the value is a Boolean value;

[0085] The characteristics of the port node are information such as port number, process name and process type, and the value is a one-hot encoding vector; whether it is a temporary port, default port or decoy port, the value is a Boolean value;

[0086] The characteristics of the file node are the version and type of the file, and the value is a one-hot encoding vector; whether it is scanned by action analysis and whether it is signed, and the value is a Boolean value;

[0087] The gateway node is characterized as a fabric node type and is used to connect subnets with Internet connectivity;

[0088] Non-attribute features include: subnet affiliation features, node type features, observation features, and message features;

[0089] The feature of subnet belonging is the one-hot vector of which subnet the node belongs to, and the value is a one-hot encoded vector;

[0090] The node type feature represents the node type: host, port, file or gateway, and the value is a one-hot encoded vector;

[0091] The observed features are the observation information provided by the wrapper, and the values ​​are one-hot encoded vectors;

[0092] The message features include whether the subnets of other defender agents have been scanned and captured, and the value is a one-hot encoded vector.

[0093] The edge index of the attribute node is used to build the connection relationship between nodes, for example, to build the relationship between the system node and the port node, that is, to create the edge index between the two nodes;

[0094] The edge index construction of the attribute node includes: storing the edge index as a 2×E two-dimensional tensor (or matrix); where E represents the number of all edges in the current environment; each column (i, j) represents a directed edge from node i to node j, which can not only compactly represent the network structure, but also facilitate graph convolution operations in graph convolution networks;

[0095] The edge index of the attribute node includes: permanent edges and temporary edges;

[0096] Permanent edges are connection relationships that exist in the network for a long time and will not change easily with time or operations, such as the basic connection between subnets and routers, fixed backbone links, etc.; these edges constitute the stable skeleton of the network topology and generally remain unchanged throughout the experimental scenario.

[0097] Temporary edges are used to represent those edges that are dynamically generated or deleted as attackers and defenders take action. Attackers may create new edges between hosts and port nodes by means of "scanning", "utilizing services", "lateral movement", etc. Defenders may cut off malicious connections by "blocking traffic" or "removing", or use "deployment bait" to add new edges between host nodes and new port nodes. As these operations are performed, the system will add or delete corresponding entries in the edge index to reflect network changes in real time.

[0098] Step 3: Model the attack and defense process of the attacker and defender agents;

[0099] Figure 1 As shown, the attacker first enters a subnet in the network. Figure 1 In the real environment, there is an attacker on the red host in the left picture. The attacker has obtained certain permissions. The attack process starts from the reconnaissance stage of the network. The attacker identifies the target host and its vulnerabilities through active scanning and service discovery, and collects available information in the network. After identifying the high-value host, the attacker uses known vulnerabilities to infiltrate, and after obtaining user permissions, uses permission escalation actions to elevate its own permissions and gain control over the target node. Among them, the high-value host will change with the importance of the node in different stages of the task and the criticality of the current task. Subsequently, the attacker further expands the scope of influence through lateral movement, gradually infiltrates the entire network, and achieves goals such as data theft, service destruction or system control. At the same time, in response to defense measures, the attacker will also adopt strategies, such as detecting bait services in the network or temporarily evacuating threatened nodes to circumvent defense and maintain concealment.

[0100] During the attack and defense process, the defender agent continuously monitors the network status, identifies potential intrusion activities, and takes response measures based on the real-time network status, including removing malicious files and restoring the system status. In addition, to further interfere with the infiltration path, the defender agent can deploy bait services to guide the attacker to expose his actions and limit the scope of the attack through traffic control.

[0101] In graph modeling, the attacker's penetration path and the defender's intelligent agent's response measures will update the status of nodes and edges in real time, such as the loss, recovery or isolation status of nodes, and the blocking or recovery status of edges, forming a dynamically evolving graph structure.

[0102] The whole process revolves around the dynamic game between attackers and defenders. Attackers constantly adjust their penetration paths and strategies to achieve their goals, while defenders take optimal defensive actions based on real-time changes in attacks to minimize network losses. The graph modeling method not only simulates the complete execution process of the attack chain in a real network environment, but also reveals the evolution and interaction of strategies between attackers and defenders in complex networks, providing a basis for optimizing defense strategies.

[0103] Improve the attackers, including aggressive attackers and exploratory attackers;

[0104] The aggressive attacker modifies the probability of the attribute node being attacked and the probability of selecting an action when the host status is user authority or user authority and discovered, root authority or root authority and discovered.

[0105] The exploratory attacker modifies the probability of the attribute node being attacked and the probability of selecting an action when the host status is known or known and discovered, the service is known, or the service is known and discovered.

[0106] Step 4: Enhance the features of attribute nodes;

[0107] Graph Convolutional Network (GCN) performs convolution operations on the graph structure to fuse the local structure information of the node with the feature information to generate more expressive node embeddings;

[0108] The network topology is modeled as a graph G = (V, E); where V is the attribute node and E is the connection relationship between the attribute nodes;

[0109] Each attribute node v∈V has an initial feature vector Including information such as node type, subnet, and importance; GCN aggregates and transforms the node features of each layer with the features of neighboring nodes through multi-layer graph convolution operations, gradually capturing higher-order structural information;

[0110] After L layers of graph convolution operations, the final embedding vector of attribute node V is Contains information from its L-layer neighbors; in order to ensure that the scale of different node embedding vectors is consistent, improve the stability of training and the performance of the model, the generated node embedding Perform normalization processing; the normalization process can be expressed by the following formula: The specific formula is as follows:

[0111]

[0112] Where μ is the mean of node embeddings; σ is the standard deviation of node embeddings, which is used to measure the dispersion of embedding vectors.

[0113] Step 5: Reduce the dimension of the feature matrix of the attribute node; optimize the edge index of the attribute node;

[0114] Perform dimensionality reduction, i.e., compaction, on the feature matrix of the attribute node; by analyzing the actual impact of each feature on the defense strategy, select the features that contribute most to the defense effect and remove some redundant and irrelevant features; ensure that the model can focus on key features, reduce unnecessary computing burden, and thus improve the training speed and response efficiency of the model;

[0115] The contribution of the defense effect is calculated by calculating the reward ratio of each feature, screening out key features whose reward ratio exceeds the reward threshold, and removing redundant and irrelevant features whose reward ratio is lower than the reward threshold; in this embodiment, the reward threshold = 5%.

[0116] The attribute nodes before dimensionality reduction (such as system nodes, port nodes, and file nodes) contain more and more specific feature information, such as operating system version, port number, process name, file version, etc., and are usually represented in the form of one-hot encoding vectors; while the attribute nodes after dimensionality reduction only retain more direct Boolean features that are more relevant to defense decisions (such as "whether it is a user host, server, or router", "whether it is a temporary port or a decoy port", "whether it is scanned or signed", etc.), thereby streamlining redundant features; in addition, the attribute nodes after dimensionality reduction still retain core features such as subnet affiliation, node type, observation information, and messages to support key defense decisions; the attribute features after dimensionality reduction are more compact and targeted, reducing the computational and storage burden;

[0117] The attribute nodes after dimensionality reduction are as follows:

[0118] The characteristics of the reduced-dimensional system nodes are whether they are user hosts, servers, or routers, and the values ​​are Boolean values;

[0119] The features of the port node after dimensionality reduction are whether it is a temporary port, a default port, or a decoy port, and the value is a Boolean value;

[0120] The features of the reduced-dimensional file nodes are whether they are scanned by action analysis and whether they are signed, and the values ​​are Boolean values;

[0121] The reduced-dimensionality gateway nodes are characterized as structural node types, used to connect subnets with Internet connections;

[0122] Edge index optimization (Compact): When processing large-scale networks, simply recording all possible edges (including temporary edges caused by false alarms or noise) often leads to the expansion of edge indexes, increasing the computing and storage burden. To this end, temporary edges are screened and streamlined to remove invalid or redundant connections caused by false alarms or noise information. For example, when the edge scanned by the monitoring action is actually a temporary edge caused by a false alarm, the system will delete the corresponding temporary edge from the edge index to avoid interfering with the model decision. In this way, the network topology can maintain a more compact structure, greatly reducing the computational complexity of the model in the training and inference stages. In the face of a high-dynamic environment with a large number of temporary edges, it can still maintain efficient operation and achieve real-time defense response.

[0123] Step 6: Input the current state of the defender agent into the graph convolutional network to obtain the action of the defender agent; input the action of the defender agent into the graph modeling to obtain the state of the defender agent at the next moment and the reward value at the next moment. The defender agent obtains the feature matrix of the attribute node with reduced dimension at the next moment, the matrix of non-attribute features, and the edge index of the attribute node, and stores them in the experience buffer, and then estimates the target value function and updates the strategy and evaluation network;

[0124] Specifically include:

[0125] Parameter initialization: Initialize the environment, the Actor network and the Critic network of the graph convolutional network;

[0126] Set hyperparameters: discount factor γ, PPO clipping range ∈, advantage estimation coefficient λ, batch size B, maximum episode time step T total , the number of defender agents N; initialize each experience buffer (ReplayBufferr i );

[0127] In the rth training round, for each defender agent i, the following operations are performed in sequence:

[0128] Reset the environment and get the state s of the i-th defender agent in the current environment t i ;

[0129] Action selection and sampling: observing the current state of the defender agent Extract feature matrix and edge index The optimized feature matrix is ​​obtained by dimensionality reduction and edge index optimization. and edge index Get the probability distribution of each action and select the action with the highest probability in the probability distribution

[0130] Graph-based modeling of environmental interaction and data storage: executing actions Interact with the graphical environment and get rewards and the state at the next moment And extract the feature matrix of the next moment and edge index The optimized feature matrix is ​​obtained through feature dimension reduction and edge index optimization and edge index Will Store into experience buffer;

[0131] Estimating the target value function: Computing discounted returns and advantage functions

[0132] Policy and Criteria Network Update: Extract stored data from the experience buffers of all defender agents; apply clipping operations and combine advantages Get the Actor network loss, back-propagate and update the parameter θ; calculate the Critic network loss and back-propagate to update the parameter ω;

[0133] ReplayBuffer clearing: After each update is completed, clear the ReplayBufferr of each agent i , prepare data for the next round of training.

[0134] Experimental process:

[0135] In order to verify the effectiveness of the CGCN-PPO method in the smart grid control system environment, a dynamic attack and defense experimental scenario based on the smart grid control system is designed in this embodiment.

[0136] The network in the experimental scenario is divided into multiple subnets with specific functions, simulating the actual operating environment of core systems such as power generation, transmission and distribution, enterprise office and external access in the smart grid; each subnet is composed of multiple functional areas, and each area has different security requirements and information interaction modes; Figure 2 Smart grid topology diagram;

[0137] Subnet 1: The power generation control subnet includes the unit monitoring area and the key control area, which are used to monitor the operating status of the generator set and process the core control process on the power generation side. This subnet is the core of the power generation system and stores a large amount of sensitive data. Once attacked, it may cause power production interruption or serious failure.

[0138] Subnet 2: The power transmission and distribution control subnet includes the substation liaison area and the core dispatching area, which are responsible for the dispatching and monitoring tasks of the power transmission and distribution smart network; the substation liaison area mainly handles the monitoring and preliminary operation of basic equipment, and the core dispatching area manages the key tasks of power transmission and distribution. This subnet is the core hub of power transmission. Attackers may interfere with power supply or paralyze key lines through this subnet.

[0139] Subnet 3: The office and public access subnet includes the public access area, the enterprise office area, and the maintenance and diagnosis area, which are mainly used to support the company's daily business, external services, and equipment maintenance. The public access area has a higher risk factor because it faces external interfaces. The enterprise office area is responsible for information management and shares some data with other areas. The maintenance and diagnosis area provides equipment diagnosis and remote patch management, but it may also become a springboard for attackers to penetrate horizontally.

[0140] Subnet 4: The external operator network is used to exchange data with third-party operators or partners. This subnet is open to the public and currently has a host that has been compromised by attackers.

[0141] There will be 1-6 servers and 3-10 user hosts in each area; the network will have 5 defender agents; Subnet 1 and Subnet 2 networks will have two defender agents each, Subnet 3 will have one defender agent that manages all areas, and the contractor network will be undefended; the attacker already exists in Subnet 4 and cannot be removed by the defender agent.

[0142] The tasks in the scenario are linearly advanced in three stages (Stage 1, Stage 2, and Stage 3), simulating the dynamic priority changes and attack and defense processes in the smart grid:

[0143] Phase 1: All tasks running in all areas are given the same low priority. This phase mainly simulates the state of the smart grid under normal operation;

[0144] Phase 2: Tasks in the critical control area of ​​the power generation control subnet are elevated to high priority, external connections are cut off, and only connections with the unit monitoring area are retained; tasks in other subnets maintain low priority; this phase represents centralized protection of critical power generation tasks; if equipment in the critical control area is captured, the defender agent receives a high negative reward.

[0145] Phase 3: The tasks in the core dispatching area of ​​the power transmission and distribution network are promoted to high priority, cutting off external connections and retaining only the connection with the substation liaison area; other tasks remain at low priority. If the equipment in the core dispatching area is captured, the defender agent will receive a high negative reward.

[0146] Defender Agent: In this experimental scenario, a multi-agent architecture is adopted and five defender agents are deployed, which are responsible for monitoring and defending different areas in the network and responding to complex and diverse network attack behaviors through collaboration.

[0147] State: The state in the experiment represents the network information received by the agent, including the feature vectors of the nodes in its responsible area and the network topology information; the node feature vector is constructed based on the description in Table 1, including the core attributes such as the operating system version of the system node, the process type of the port node, and the signature information of the file node, and is represented by one-hot encoding and Boolean values; in addition, the agent can also obtain the node’s subnet affiliation, observation information, and messages shared by other agents to determine whether the subnet has been scanned or captured; these state information provide the agent with a comprehensive view of the real-time network status and support the optimization and execution of defense decisions.

[0148] Actions: The actions in the experiment define the specific operations that the defense agent can perform in network defense, including node-level and edge-level defense behaviors; node-level actions include monitoring node status, analyzing file content, deploying decoy services, removing threats, and restoring node status; edge-level actions involve blocking or restoring communication between two subnets; these actions provide defense agents with defense means for specific areas or connections, enabling them to flexibly respond to diverse attack behaviors in the network.

[0149] Reward: In the experiment, the initial reward of the defender agent is 0 points. When the following situations occur, the defender agent will be punished to different degrees: when the subnet cannot work locally, cannot work remotely, or the attacker chooses to perform an "impact" operation on the target, the defender agent will receive a corresponding reward deduction; the degree of punishment is dynamically adjusted with the changes in the task stage and the operation area to reflect the importance of the nodes in different stages and the criticality of the current task, as shown in Table 1:

[0150] Table 1 Rewards received by the defense agent at each stage at each time step due to the failure of the local host to work properly in different subnets / areas

[0151]

[0152] The core purpose of reward design is to highlight the priority changes of task stages and areas by dynamically adjusting rewards, and to guide defense agents to prioritize resources to key areas, thereby effectively protecting high-priority task nodes and optimizing the overall defense effect.

[0153] Attacker: In the network architecture of the smart grid control system, the attacker uses the finite state machine (FSM) strategy to penetrate and destroy the network, and is called an FSM attacker; this attacker gradually increases the control authority over the host through a series of predefined state transitions and action selections; each host is in different states according to the attacker's control level, including known (K), known and discovered (KD), service known (S), service known and discovered (SD), user authority (U), user authority and discovered (UD), root authority (R), root authority and discovered (RD) and final state (F); the attacker selects corresponding actions based on the current state of the host and the weight table, such as remote system discovery, service discovery, remote service exploitation, authority escalation, discovery deception, service degradation, impact and retreat, etc., to achieve its attack goals.

[0154] Table 2 shows the attack probability distribution of FSM attackers against hosts with different permissions. During the attack, all host states (from known states to root permission states) have an equal probability of being attacked, which is 12.5%, indicating that FSM attackers have no preference for the choice of host states, aiming to ensure the comprehensiveness and coverage of the attack model.

[0155] Table 3 describes the selection probability of various attack actions by the FSM attacker in different states of the host; each host state corresponds to nine possible actions, such as remote system discovery, privilege escalation, etc., where the selection probability of each action reflects the attacker's strategic preference in a specific state; for example, in the K (known) state, the FSM attacker is more inclined to perform remote system discovery (50%) and service discovery (25%); while in the U (user privilege) state, the attacker may try to identify the bait (50%) or continue to discover the remote system (50%); Table 3 provides a clear reference for the behavior pattern of the FSM attacker to gradually penetrate and control the host in the smart grid network.

[0156] Table 2 Attack probability of FSM attackers on hosts with different permissions

[0157]

[0158]

[0159] Table 3 The probability of FSM attacker choosing actions under different host states

[0160]

[0161] Experimental setup: In order to evaluate the performance of the CGCN-PPO method in smart grid network defense, this experiment designed the following experimental environment and configuration. The specific experimental scenario description is shown in Table 4: In the experiment, three new attacker types are designed to simulate different threat scenarios:

[0162] Aggressive Red Agent is a variant of FSM attackers. It prioritizes attacking high-privilege hosts (such as user privileges and administrator privileges) and tends to choose destructive actions (such as privilege escalation and influence) to simulate more aggressive attack behaviors. Compared with FSM attackers, aggressive Red Agents are more aggressive in the selection of attack targets, with priority focused on user privilege (U / UD) and root privilege (R / RD) hosts, while paying less attention to low-privilege hosts (K / KD, S / SD). In addition, their action selection is more destructive. For example, in the user privilege state (U), they are more inclined to perform identification bait (75%) and destructive behaviors, while in the root privilege state (R), they prefer to downgrade services (50%) and influence (25%).

[0163] The exploratory attacker (DiscoveryFSRed) is another variant of the FSM attacker. It gives priority to exploratory actions such as remote system discovery and service discovery, with the goal of quickly expanding the attack surface. Compared with the FSM attacker, DiscoveryFSRed has obvious differences in host state priority. It completely ignores attacks on high-privilege hosts (R / RD) and focuses its priority on low-privilege hosts (K / KD, S / SD) to speed up reconnaissance. At the same time, its action selection is mainly concentrated on remote system discovery (such as 75% in the K state) and service discovery (such as 100% in the KD state), while privilege escalation and destructive actions are almost not performed. This makes DiscoveryFSRed more suitable for simulating the behavior of attackers in the initial reconnaissance stage of the network, while the FSM attacker takes into account both reconnaissance and destructive behaviors, covering the operations of the entire attack chain.

[0164] Table 4 Experimental scenario description

[0165]

[0166] Experimental results: In order to evaluate the performance of the CGCN-PPO algorithm in smart grid defense, the performance of four existing different algorithms: DQN, PPO, GCN-PPO, and GAT-PPO in four experimental scenarios were compared; each algorithm ran 20,000 episodes in each scenario, and recorded the cumulative reward, training convergence speed, and strategy robustness indicators for comparison.

[0167] Table 5 shows the convergence speed of the five algorithms respectively. First, the most obvious gap is that the algorithm based on GNN is better than the algorithm without graph neural network, which shows that graph neural network has great potential in dealing with multi-agent network attack and defense.

[0168] Secondly, in Figure 3 Although GCN-PPO converges slightly faster than CGCN-PPO in a, the final converged reward is not as good as CGCN-PPO, and is worse than GAT-PPO; this proves that after feature dimensionality reduction and edge index simplification, CGCN-PPO can converge rewards better than the original GCN-PPO and GAT-PPO.

[0169] Figure 3 In b, CGCN-PPO still performs better than other algorithms, proving that the algorithm can guarantee better computing efficiency and attack and defense effects after optimization; at the same time, although GAT-PPO also has certain advantages, it is lower than CGCN-PPO in the overall summary.

[0170] Figure 3 In c, CGCN-PPO still performs better than other algorithms, proving that the algorithm can guarantee better computing efficiency and attack and defense effects after optimization; at the same time, although GAT-PPO also has certain advantages, it is lower than CGCN-PPO in the overall summary.

[0171] Finally, in Figure 3 In d, although GAT-PPO converges slightly better than CGCN-PPO and GCN-PPO when processing large-scale scenes, the final reward achieved is lower than that of the other two algorithms, proving that GAT-PPO has limited performance when processing complex scenes.

[0172] Figure 3The maximum values ​​of the training curves of the five algorithms in each scenario are shown. From a horizontal comparison, in all scenarios, the maximum reward of CGCN-PPO is always higher than that of other algorithms, especially in scenario A and scenario C, where the maximum values ​​are -103.5 and -33.6 respectively, which are much better than other algorithms. This shows that CGCN-PPO can maintain excellent defense performance in different scenarios. Although GAT-PPO has a slight advantage in convergence speed in some scenarios (such as scenario D), its maximum reward is lower than that of GCN-PPO. In particular, in scenario B, the maximum value of GCN-PPO is -552.6, while that of GAT-PPO is -1156.8, which shows the stability of GCN-PPO in overall defense capability. Traditional algorithms (PPO and DQN) that do not use GNN perform poorly in all scenarios. For example, in scenario A, the maximum rewards of PPO and DQN are -4520.0 and -10900.0 respectively, which are much lower than those of GNN-based algorithms. This further verifies the significant advantages of GNN in multi-agent network attack and defense.

[0173] From the perspective of scene comparison, in the standardized threat scenario, CGCN-PPO performs best, with a maximum reward of -103.5, while the maximum values ​​of GCN-PPO and GAT-PPO are -159.3 and -298.7 respectively, indicating that CGCN-PPO has a significant advantage in dealing with conventional attack modes; in the aggressive attacker scenario, the reward values ​​of each algorithm have declined, but CGCN-PPO still maintains the lead, with a maximum reward of -428.4, showing its stability and adaptability in high-intensity threats; in the face of random attackers, CGCN- The maximum rewards of PPO and GCN-PPO are almost the same (-33.6 and -33.7), while the reward value of GAT-PPO drops to -260.6, indicating its insufficient robustness; in addition, PPO and DQN perform extremely poorly, at -10680.0 and -1875.0 respectively, showing that they are difficult to adapt to complex and dynamic threat environments; in the expanded large-scale network, the maximum reward of CGCN-PPO is -284.6, which is better than -409.4 of GCN-PPO and -785.3 of GAT-PPO, verifying its good scalability.

[0174] Table 5 Maximum values ​​of training curves in each scenario

[0175] algorithm Scenario Maximum Reward GCN-PPO Scenario A -159.3 GAT-PPO Scenario A -298.7 CGCN-PPO Scenario A -103.5 PPO Scenario A -4520.0 DQN Scenario A -10900.0 GCN-PPO Scenario B -552.6 GAT-PPO Scenario B -1156.8 CGCN-PPO Scenario B -428.4 PPO Scenario B -6375.0 DQN Scenario B -9140.0 GCN-PPO Scenario C -33.7 GAT-PPO Scenario C -260.6 CGCN-PPO Scenario C -33.6 PPO Scenario C -10680.0 DQN Scenario C -1875.0 GCN-PPO Scenario D -409.4 GAT-PPO Scenario D -785.3 CGCN-PPO Scenario D -284.6 PPO Scenario D -9800.0 DQN Scenario D -3750.1

[0176] Table 5 shows the results of testing the trained algorithms in four scenarios. In scenario A, the average reward value of CGCN-PPO in this scenario is -148.6, which is much better than -198.1 of GCN-PPO and -304.7 of GAT-PPO, indicating that it has significant performance advantages in conventional threat environments. In scenario B, the aggressive threat scenario, the aggressive attackers reduce the average reward values ​​of all algorithms, and the reward value of CGCN-PPO is -667.8, which is still better than -780.3 of GCN-PPO and -1290.1 ​​of GAT-PPO, indicating that its It can still maintain good performance under high-intensity threats; in scenario C with random attackers, the average reward value of CGCN-PPO is -115.9, which is significantly better than -869.3 of GCN-PPO and -147.6 of GAT-PPO, indicating that it is more adaptable and robust when dealing with random attackers; in scenario D with extended scenarios, the average reward value of CGCN-PPO is -384.5, which is better than -689.1 of GAT-PPO and -826.4 of GCN-PPO, indicating that CGCN-PPO has better scalability and efficiency when dealing with large-scale networks.

[0177] As shown in Table 6, whether facing standardized threats, radical threats or random attackers, as well as large-scale network scenarios, CGCN-PPO shows excellent defense capabilities and stability; it performs well in random attacker scenarios, but its performance in radical threats and large-scale network scenarios is relatively poor, and its robustness is not as good as CGCN-PPO; although its performance is slightly inferior to CGCN-PPO, its average reward value is always higher than GAT-PPO, showing better stability; feature dimensionality reduction and edge index optimization techniques enable it to achieve better reward values ​​in each scenario with lower computational overhead, which fully verifies the superiority of its algorithm design.

[0178] Table 6 Average reward values ​​for five tests in four scenarios

[0179] algorithm Scenario award GCN-PPO Scenario A -198.1 GAT-PPO Scenario A -304.7 CGCN-PPO Scenario A -148.6 GCN-PPO Scenario B -780.3 GAT-PPO Scenario B -1290.1 CGCN-PPO Scenario B -667.8 GCN-PPO Scenario C -869.3 GAT-PPO Scenario C -147.6 CGCN-PPO Scenario C -115.9 GCN-PPO Scenario D -826.4 GAT-PPO Scenario D -689.1 CGCN-PPO Scenario D -384.5

[0180] The present invention proposes a multi-agent defense that combines CGCN-PPO with GNN and reinforcement learning, which significantly improves the performance and adaptability of smart grid network defense. By optimizing the feature matrix and edge index, the computational complexity is effectively reduced, and it is superior to GCN-PPO and GAT-PPO in convergence speed and reward value. At the same time, the multi-agent collaboration mechanism enhances the system's responsiveness to complex dynamic scenarios. In the experiment, CGCN-PPO exhibits excellent robustness and scalability, especially in random attacks and large-scale network scenarios.

[0181] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A power grid network security defense method based on CGCN-PPO, characterized in that: The following steps are involved: Step 1: Construct several attribute node network scenarios and use graph modeling to model the actions of attacker and defender agents in the network scenarios. Step 2: Model the defender agent’s actions into node-level, edge-level, and global-level actions; Create a feature matrix of attribute nodes, a matrix of non-attribute features, and an edge index of attribute nodes for the defender agent; Step 3: Model the attack and defense process of the attacker and defender agents; Step 4: Enhance the features of attribute nodes; Step 5: Reduce the dimension of the feature matrix of the attribute node; Optimize the edge index of attribute nodes; Step 6. Input the current state of the defender agent into the graph convolutional network to obtain the action of the defender agent. Input the action of the defender agent into the graph modeling to obtain the state of the defender agent at the next moment and the reward value at the next moment. Use the feature matrix of the attribute nodes of the reduced dimension of the defender agent at the next moment, the matrix of non-attribute features, and the edge index of the attribute nodes to update the strategy and evaluation network.

2. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: The attackers include: Aggressive attackers: Aggressive attackers modify the probability of attribute nodes being attacked and the probability of selecting actions when the host status is user authority or user authority and discovered, root authority or root authority and discovered.

3. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: Attackers also include: Exploratory attackers. Exploratory attackers modify the probability of attribute nodes being attacked and the probability of selecting actions when the host status is known or known and discovered, the service is known, or the service is known and discovered.

4. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: Step 4 specifically includes: The network topology is modeled as a graph G = (V, E); where V is the attribute node and E is the connection relationship between the attribute nodes; Each attribute node v∈V has an initial feature vector GCN aggregates and transforms the node features of each layer with the features of neighboring nodes through several layers of graph convolution operations; After L layers of graph convolution operations, the final embedding vector of attribute node V is Embed the generated attribute nodes Perform normalization.

5. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: The dimension reduction of the feature matrix of the attribute node is to calculate the reward ratio of each feature and filter out the features whose reward ratio to the total reward exceeds the reward threshold.

6. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: Optimizing the edge index of attribute nodes is to filter temporary edges and remove invalid or redundant connections caused by false positives or noise information.

7. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: Step 6 specifically includes: Initialize the environment and graph convolutional network; Set hyperparameters: discount factor γ, PPO clipping range ∈, advantage estimation coefficient λ, batch size B, maximum episode time step T total , the number of defender agents N; the experience buffer ReplayBufferr of the i-th defender agent i ; In the rth training round, for each defender agent i, the following operations are performed in sequence: Reset the environment and get the state s of the i-th defender agent in the current environment t i ; Action selection and sampling: observing the current state of the defender agent Extract feature matrix and edge index Using the feature matrix and edge index Get the probability distribution of each action and select the action with the highest probability in the probability distribution Graph-based modeling of environmental interaction and data storage: executing actions Interact with the graphical environment and get rewards and the state at the next moment And extract the feature matrix of the next moment and edge index And using the feature matrix and edge index Build Store into experience buffer; Estimating the target value function: Computing discounted returns and advantage functions Strategy and evaluation network updates; ReplayBuffer is cleared.

8. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: Attribute nodes include: port nodes, file nodes, system nodes, and gateway nodes.

9. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: Attacker action modeling includes: discovering remote systems, aggressive service discovery, covert service discovery, exploiting network services, privilege escalation, sabotage, degrading services, identifying decoys, and evacuation.

10. The power grid network security defense method based on CGCN-PPO according to claim 1 is characterized in that: Defender agent action modeling includes: monitoring, analysis, deployment of decoys, removal, recovery, and blocking / allowing traffic.

Images