Persistent threat mitigation method based on GraphSAGE-PPO

Abstract

The invention relates to the technical field of network security, in particular to a persistent threat mitigation method based on GraphSAGE-PPO, which comprises the following steps: constructing a network node model; carrying out observation information modeling on the network node model by utilizing the graph data structure; the defender agent receives the observation information, transmits the observation information to an Actor network based on GraphSAGE, and decides the action to be taken in the current state; the action is evaluated using a Critic network. According to the method, the problems that iteration is slow when convergence is performed to a reward value and a defense agent cannot comprehensively consider a complex dependency relationship between network hosts in an existing method are solved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method for mitigating persistent threats based on GraphSAGE-PPO. Background Art

[0002] Currently, network attacks have become increasingly covert and complex. Taking Advanced Persistent Threats (APTs) as an example, attackers usually have clear motives and target specific organizations; they maintain a high degree of concealment over a long period of time and use complex techniques to exploit vulnerabilities, thus causing significant damage to network systems.

[0003] Existing methods such as DDQN-DNER, RNN-DDQN, and RNN-PPO. The performance of DDQN-DNER deteriorates significantly when the number of nodes increases, as shown by the reward value in Scenario B. RNN-DDQN is completely unable to provide effective defense when the number of nodes increases, with a very low reward value. RNN-PPO performs poorly in all scenarios and cannot achieve effective defense. All three methods make decisions directly based on the one-dimensional feature information of network node characteristics, and may not effectively consider the interactions between devices. Therefore, the defense performance decreases significantly when the number of devices increases. Summary of the Invention

[0004] Aiming at the deficiencies of existing methods, the present invention solves the problems of slow convergence to the reward value and the inability of the defense agent to comprehensively consider the complex dependencies between network hosts in existing methods.

[0005] The technical solution adopted by the present invention is: a method for mitigating persistent threats based on GraphSAGE-PPO includes the following steps:

[0006] Step 1: Construct a network node model;

[0007] As a preferred embodiment of the present invention, the network node model includes an interconnected network topology constructed by different hosts, switches, firewalls, routers, and servers.

[0008] Step 2: Use a graph data structure to model the observation information of the network node model;

[0009] As a preferred embodiment of the present invention, the graph data structure is G=(V, E), where V represents the set of each node's features and E represents the connections between nodes.

[0010] As a preferred embodiment of the present invention, processing the neighbor nodes in the node includes:

[0011] First, construct the neighbor set of node v S is the sampling size;

[0012] Secondly, aggregate the features of the sampled neighbor nodes through the mean aggregation function is the embedding of neighbor u at the (k-1)-th layer;

[0013] Next, combine the embedding of the current node with the aggregated neighbor embeddings to update the node embedding, σ is the activation function, W k is the weight matrix of the k-th layer, and CONCAT is the concatenation operation;

[0014] Finally, obtain the final embedding of node v after k-layer aggregation.

[0015] Step 3: The defender agent receives the observation information, passes it to the GraphSAGE-based Actor network, and decides the action to take in the current state; uses the Critic network to evaluate this action;

[0016] As a preferred implementation manner of the present invention, Step 3 specifically includes:

[0017] Step 31: Initialize the actor network of GraphSAGE and initialize the critic network;

[0018] Step 32: The defender agent remodels the network observation information according to the current state s t , uses the graph data structure G to obtain v t and e t ; inputs v t and e t into the actor network to obtain the action distribution;

[0019] Step 33: Select arg maxP to execute the optimal defense action a t ; use the critic network to obtain the expected cumulative reward;

[0020] Step 34: Use the advantage function A t to measure the advantage of taking a t under s t relative to the average action under s t ;

[0021] As a preferred implementation manner of the present invention, the cumulative reward is the accumulation of several single-step rewards.

[0022] As a preferred implementation manner of the present invention, the formula for the single-step reward is:

[0023] R = -I·N admin - 10·N impact - C action

[0024] where I represents the importance of the host to the entire network system; N admin represents the number of hosts whose administrator privileges have been obtained by the attacker; N impact represents the number of hosts affected by the "impact" operation; C action reflects the cost of performing defense actions.

[0025] Step 35: Use the objective function L CLIP (θ) to update the policy network and update the value network according to the observed cumulative reward.

[0026] As a preferred embodiment of the present invention, the persistent threat mitigation method system based on GraphSAGE-PPO includes: a memory for storing instructions executable by a processor; a processor for executing the instructions to implement the persistent threat mitigation method based on GraphSAGE-PPO.

[0027] As a preferred embodiment of the present invention, a computer-readable medium storing computer program code, the computer program code implementing the persistent threat mitigation method based on GraphSAGE-PPO when executed by a processor.

[0028] Advantages of the present invention:

[0029] 1. The present invention first uses a graph data structure to model the network node model, effectively represents the states of different nodes using the graph, and clearly describes the connections between nodes; subsequently, integrates the GraphSAGE graph neural network model with the deep reinforcement learning algorithm PPO. GraphSAGE can efficiently analyze and process complex information in graph-structured data through information propagation and feature aggregation mechanisms; at the same time, the reinforcement learning ability of PPO enables the defender agent to learn and optimize the defense strategy, so as to take the most effective defense measures when facing complex network threats;

[0030] 2. Utilize the GraphSAGE model to facilitate the real-time transmission of state information between different hosts in the network, enabling each host to access the state information of the hosts connected to it; this information exchange provides a rich and timely context basis for training the defender agent, enabling it to make more accurate defense decisions in a complex Internet of Things network environment;

[0031] 3. The defender agent uses the PPO algorithm for iterative training, selects the best defense action according to the current state information to deal with various potential network threats;

[0032] 4. Each node feature Include its connection node information, enabling the Actor network to more accurately identify abnormal propagation paths and potential threat nodes in the network during decision-making, thereby achieving rapid detection and precise response to network security incidents; utilize the feature learning mechanism of the GraphSAGE model to significantly enhance the situation awareness ability and decision-making timeliness of the Actor network;

[0033] 5. The Critic network evaluates the value function of the current state by analyzing global observation information; the GraphSAGE model is not used in the Critic network for information transmission and feature aggregation, and the flat node state data is directly used as input. The advantage is that it can avoid the noise and complexity that may be introduced by graph-structured data and reduce the complexity of model training; by calculating the expected cumulative return starting from the current state, the Critic network can generate an efficient and stable value evaluation for the policy generated by the Actor network, thus supporting the rapid optimization and performance improvement of the policy;

[0034] 6. The GraphSAGE-Actor-Critic reinforcement learning network framework makes full use of the advantages of the GraphSAGE graph neural network in capturing topological relationships and feature interactions between nodes, enabling the trained autonomous defender agent to more accurately perceive the network situation, identify potential threats, and make timely and precise defense decisions; combining the information transmission mechanism of GraphSAGE with the Actor-Critic framework can not only effectively learn the local features of nodes but also globally optimize the defense strategy, thus significantly enhancing the adaptive ability and defense effectiveness of the agent in complex network environments;

[0035] 7. The present invention upgrades the defense agent from "equal defense" to "intelligent risk weighing" by introducing I, which essentially mathematizes the "criticality first" principle in network security into a learnable gradient signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is the GraphSAGE-based actor-critic framework diagram of the present invention;

[0037] Figure 2 is the deep reinforcement learning network security framework diagram of the Markov decision process;

[0038] Figure 3 is the process modeling diagram of the advanced persistent threat attack and defense process;

[0039] Figure 4 is the network node model diagram;

[0040] Figure 5 is the confidence interval curve diagram of the average rewards of four algorithms under different scenarios and attack agents. Detailed implementation manners

[0041] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner. Therefore, it only shows the components related to the present invention.

[0042] As Figure 1 shown, a persistent threat mitigation method based on GraphSAGE-PPO includes the following steps:

[0043] Figure 2 Present a network security framework based on the Markov decision process (MDP) and integrated with the deep reinforcement learning (DRL) algorithm logic;

[0044] The Markov decision process (MDP) is defined by the five-tuple <S, A, P, R, γ>. The network defense problem is abstracted as an MDP. Among them, the state information of the nodes of the current network node model is represented as the current state s. For example, the connection situation between nodes (i.e., devices) and the current state s of each node; the defender agent evaluates the current state s according to the policy Π and takes a defensive action a. After executing the action a, the network state changes according to this action, and the change process is described by the transition probability P(s′|s, a); meanwhile, the environment provides a reward r according to the change of the device state before and after the action; if this action effectively prevents further intrusion by the attacker, a positive reward is returned; otherwise, a negative reward is returned; the discount factor γ is used to balance the importance of long-term rewards and immediate rewards; in this way, the network defense process is modeled as an MDP; by continuously interacting with the environment and adjusting the policy according to the obtained rewards, the agent iteratively optimizes its policy and finally converges to a higher cumulative reward. This process optimizes the defense policy, enabling it to proactively respond to and mitigate potential threats and vulnerabilities, thereby enhancing the overall security of the network environment.

[0045] Step 1: Construct a network node model;

[0046] The network node model, that is, the network topology, as Figure 4 shown, includes: network interconnection between switches, firewalls, routers, hosts and servers;

[0047] Step 2: Use the graph data structure to model the observation information of the network node model;

[0048] As Figure 1 shown, a graph is a structured data representation that contains nodes and edges. Among them, nodes represent entities and edges represent the relationships between entities; graphs are widely used in the security field, such as malware detection and malicious website detection; use the graph data structure G=(V, E) to re-model the network observation information; the node set V={v1, v2,..., vn}, which represents the set of features of each node in the network topology, such as the active state and the compromised state; while E = {(u, w)|u ∈ V, w ∈ V} represents the connections between nodes in the network topology;

[0049] By combining these two parts of information, the defender agent can take the best action in the current state; specifically, the node set V provides the defender agent with the state information of each node in the network topology, enabling the defender agent to identify the security state and activities of the nodes; the edge set E provides information about the connections between nodes, enabling the defender agent to understand the relationships between network nodes and potential attack paths; by modeling the state information in this way and using the GraphSAGE graph neural network for processing, compared with the traditional single-dimensional state information method, the accuracy of the defender agent's decision-making can be significantly improved.

[0050] The actor-critic network framework consists of two main parts: the Actor network and the Critic network; as Figure 1 shown, the defender agent receives the observation information and passes it to the Actor network to decide the action to take in the current state; the Critic network then evaluates the action and provides feedback on the quality of the selected action by evaluating the value of the current state;

[0051] For each node v, first sample a fixed-size set of neighbor devices using formula (1). For each node v, reduce the computational complexity by sampling a fixed-size subset of neighbors:

[0052]

[0053] where represents the set of neighbors of node v after sampling, S represents the size of the sampling, and SampleNeighbors refers to all neighbor nodes connected to the current node v.

[0054] As Figure 1 shown, message passing is performed between different neighbor nodes. By only considering a subset of neighbors instead of all neighbors, the computational complexity is reduced and the generalization ability is enhanced;

[0055] Next, use formula (2) to aggregate the features of the sampled neighbor devices through the mean aggregation function. The formula is:

[0056]

[0057] where, is the embedding of neighbor u at the (k - 1)th layer, and AGGREGATE k is the aggregation function used at the kth layer;

[0058] Finally, the embedding of the current node is combined with the aggregated neighbor embeddings through formula (3) to update the node embedding;

[0059]

[0060] where σ is the activation function, such as ReLU; W k is the weight matrix of the k-th layer, and CONCAT represents concatenating the embedding of the node with the aggregated neighbor embeddings; the final embedding aggregated at layer k is the embedding of node v

[0061] The features of each node in the network topology of the present invention contain the information of its connected nodes, enabling the Actor network to more accurately identify the abnormal propagation paths and potential threat nodes in the network during decision-making, thereby achieving rapid detection and precise response to network security events; this feature learning mechanism based on graph neural networks significantly improves the situation awareness ability and decision-making timeliness of the Actor network;

[0062] The Critic network evaluates the value function of the current state by analyzing the global observation information; in this scenario, instead of using the GraphSAGE model for information transmission and feature aggregation, the flat device status data is directly used as the input. The advantage of this method is that it can avoid the noise and complexity that may be introduced by graph-structured data and reduce the complexity of model training; by calculating the expected cumulative return starting from the current state, the Critic network can generate an efficient and stable value evaluation for the policy generated by the Actor network, thereby supporting the rapid optimization and performance improvement of the policy.

[0063] The GraphSAGE Actor-Critic reinforcement learning network framework of the present invention makes full use of the advantages of graph neural networks in capturing the topological relationships and feature interactions between devices, enabling the trained autonomous defense agent to more accurately perceive the network situation, identify potential threats, and make timely and precise defense decisions; by combining the information transmission mechanism of GraphSAGE with the Actor-Critic framework, this model can not only effectively learn the local features of device nodes but also globally optimize the defense strategy, thereby significantly improving the adaptive ability and defense effectiveness of the agent in complex network environments.

[0064] The Actor-Critic framework is applied to the Proximal Policy Optimization (PPO) algorithm; this integration enables the PPO algorithm to take more accurate and effective actions when dealing with complex network observations;

[0065] Step 3: The defender agent receives the observation information, passes it to the Actor network of the graph neural network, and decides on the action to take in the current state; uses the Critic network to evaluate the action;

[0066] The steps of the PPO algorithm based on GraphSAGE are as follows:

[0067] First, initialize the actor (policy network) π θ (a|V, E) based on the graph neural network (GraphSAGE), and initialize the critic (value network) V φ (s); where the parameters are θ and φ respectively;

[0068] The defender agent obtains the current state s t , remodels the network observation information using the graph data structure G to obtain v t and e t ; e t is the network device connection situation in the current state s t ; v t is the node feature in the current state s t ;

[0069] Next, input v t and e t into the policy network π θ to obtain the action distribution:

[0070] π θ (a t |v t , e t ) (4)

[0071] Select arg maxP and execute the optimal defense action a t to prevent the attacker agent from further infiltrating and damaging; The Critic network V φ (s) estimates the value of the given state s; This value represents the expected cumulative reward that the defender agent can obtain after starting from state s and taking actions according to the current policy:

[0072]

[0073] where, R t is the cumulative reward; The advantage function A t measures the quality of taking action a t in state s t relative to the average action in that state;

[0074] The cumulative reward is the accumulation of several single-step rewards;

[0075] The single-step reward formula for the defense agent is:

[0076] R = -I·N admin -10·N impact -C action (6)

[0077] Among them, I represents the importance of the host to the entire network system, which is divided into two categories: low importance (for example, user hosts and operation hosts) will generate a negative reward of -0.1, while high importance (for example, enterprise servers and operation servers) will generate a negative reward of -1.0; N admin represents the number of hosts whose administrator privileges have been obtained by the attacker, and N impact represents the number of hosts affected by the "impact" operation, and C action reflects the cost of performing defense actions, and only the cost of the "recovery" action is 1;

[0078] By introducing I, the present invention upgrades the defense agent from "equal defense" to "intelligent risk weighing", which essentially mathematizes the "criticality priority" principle in network security into a learnable gradient signal.

[0079] The calculation formula of the advantage function is:

[0080] A t = R t - V φ (s t ) (7)

[0081] Among them, R t is the observed cumulative reward, that is, R t is the cumulative value of R; V φ (s t ) is the estimated value of this state.

[0082] Using Critic to estimate the value of the state helps to reduce the variance of the policy gradient estimation, so that the update of the policy network π θ is more stable and reliable.

[0083] Finally, the policy network is updated using the objective function, and at the same time, the value network is updated according to the observed cumulative reward. The objective function formula is:

[0084]

[0085] Among them, is the probability ratio, is the advantage function, and ∈ is a hyperparameter that controls the clipping range; clip restricts the policy update amplitude and constrains the policy ratio r t (θ) within the interval [1 - ∈, 1 + ∈); preventing large and destructive step updates during training, thereby improving the stability of policy optimization.

[0086] Experimental process:

[0087] The present invention constructs a network attack and defense process based on the CybORG environment; CybORG is designed specifically for training and developing security personnel and autonomous agents; it provides a general interface for simulating cloud-based virtual machines and emulated network environments; in this environment, the blue team is the defender agent that is trained by performing defense actions and receiving reward signals from the reward function, enabling it to select the optimal defense action according to the current state.

[0088] The state space, action space, and reward function of this environment are described as follows:

[0089] State: The original state space of the defender agent consists of the state information of all nodes in the network topology. Taking the host as an example for the node, the information of each host includes interfaces, processes, sessions, system information, and user information; during the training process, this host information is further processed, and the processed state space is represented by a vector with a length equal to the number of devices multiplied by 4; these four components respectively represent the state of the host, indicating whether it has been scanned or exploited, and the attacker's access privilege level: no access, user level, administrator level, or unknown; the main focus of the observation information is to check for anomalies in the session data.

[0090] The red team is the attacker agent, and the original observation space of the attacker gradually expands as it continues to penetrate the network topology; initially, only the information of the entry point host is available; however, as the red team performs scans and discovers new hosts, its original observation information continuously increases; during the training process, this observation information is processed according to the same processing steps as the defender agent, and finally, an observation vector consisting of four components is generated for each host.

[0091] Actions: The action spaces of the attacking and defending parties;

[0092] Reward: In this environment, the reward function for each round is calculated based on the actions of the attacker agent and its impact on different important devices in the system; specifically, when the attacking agent obtains administrator privileges, the defending agent calculates the penalty R according to formula (8):

[0093] The effectiveness of the defender agent's strategy is evaluated by the cumulative reward obtained throughout the round. A lower cumulative negative reward indicates that the defense strategy is more timely and effective, and can better protect the network system from being damaged by the attacker; as Figure 3 is the network topology structure diagram;

[0094] Three different scenarios were developed based on the CybORG simulation environment, and three different attack agents were evaluated, as shown in Table 1:

[0095] Table 1 Network Scenario Description

[0096]

[0097] In this environment, the proposed GraphSAGE-PPO algorithm of the present invention is implemented, and its effectiveness is demonstrated through ablation experiments; three attack agents are applied to these three scenarios to evaluate the defense strategy based on GraphSAGE-PPO.

[0098] DDQN-DNER: The DDQN-Dueling Noisy-Experience Replay (DDQN-DNER) algorithm proposed by Zhu et al., which is used to train a defense agent to take appropriate actions to protect the network.

[0099] RNN-DDQN: An algorithm proposed by the CUABv2 team in the TTCP CAGE challenge, which is used to train an agent to take actions to protect the network.

[0100] RNN-PPO: Deep learning methods based on RNN and LSTM mechanisms have been widely applied in the field of network security.

[0101] By comparing the performance of several existing algorithms, it is observed that the GraphSAGE-PPO algorithm can always obtain the lowest negative reward under different attackers and scenarios, thus verifying the effectiveness of its defense strategy.

[0102] Ablation Experiment:

[0103] An ablation experiment is used to evaluate the performance of different modules in the proposed GraphSAGE-PPO algorithm; as shown in Table 2:

[0104] Table 2 Maximum average reward values achieved by different algorithms against various attacker agents in Scenario A

[0105]

[0106] Four sets of experiments were conducted in Scenario A, each set of experiments targeting different types of attacker agents (e.g., Bline, which attacks critical servers through the shortest path in a known network topology; Meander, which discovers and attacks critical services in an unknown network topology by continuous scanning; Sleep, which does not perform any attack actions); the traditional PPO algorithm was used as a benchmark, which showed moderate performance in the four experiments; next, edge index information, representing network topology connections, was introduced as an additional observation variable, and the PPO algorithm was retrained; unexpectedly, the performance of the PPO algorithm was lower than that of the traditional PPO algorithm, indicating that the Actor-Critic network framework of PPO failed to effectively utilize edge index information to consider the connections between hosts, but instead increased the training difficulty of the model due to complex observation information, resulting in performance degradation.

[0107] The network observation was modeled as data containing edge indices and node features, and GraphSAGE was combined with PPO; two variants were considered: applying GraphSAGE to the Actor network (GraphSAGE-PPO), and another applying GraphSAGE to both the Actor and Critic networks simultaneously (GraphSAGE-PPO*); the experimental results showed that GraphSAGE-PPO had the best performance, while the performance of GraphSAGE-PPO* was weaker than that of GraphSAGE-PPO and comparable to that of the traditional PPO algorithm; since the core function of the Critic network is to evaluate state values rather than extract topological features, applying GraphSAGE to the Critic network may lead to overfitting of graph structure information, instead reducing the model's ability to evaluate the global state value; in contrast, applying GraphSAGE only to the Actor network can focus more on policy optimization while maintaining the stability of the Critic network, achieving better overall performance, that is, being able to detect and mitigate potential threats more timely, thereby preventing further network damage and significantly enhancing the capabilities of the autonomous defense agent.

[0108] Figure 5 The confidence intervals of the average rewards obtained by the four algorithms in five training runs (each using a different random seed) under different scenarios and attacker agents were shown; it can be seen from the figure that the GraphSAGE-PPO algorithm was superior to other algorithms in terms of both final performance and robustness;

[0109] The specific situation is as follows:

[0110] In scenario A, all algorithms except RNN-PPO converge rapidly to high reward values during the initial training phase; however, it is worth noting that the reward of the GraphSAGE-PPO algorithm continues to increase as training progresses; indicating that the defense agent based on the GraphSAGE-PPO algorithm can further enhance the network defense ability; specifically, when a host in the network is attacked, the connected hosts can quickly receive relevant information, which helps the defender agent quickly identify and eliminate potential threats, ensuring the security of the network system; this graph-based information dissemination mechanism significantly improves the effectiveness of the defense strategy, making GraphSAGE-PPO perform well in the network defense scenario and having stronger performance and robustness; in contrast, the RNN-PPO algorithm has a slow convergence speed and poor results, indicating that the PPO algorithm based on the RNN and LSTM mechanisms is not suitable for network defense; in addition, when facing the Sleep attacker agent, all algorithms except RNN-DDQN converge rapidly to zero cost at the beginning, while RNN-DDQN still incurs costs by taking unnecessary defense actions in the normal network state.

[0111] In scenario B, when facing the B-line attacker, the performance curves of all algorithms are similar to those in scenario A, and GraphSAGE-PPO still demonstrates superior robustness and high convergence rewards; this is because the B-line attacker already knows the network topology and always attacks the critical servers along the shortest path, so the increased number of hosts does not interfere with its attack strategy; however, when facing the Meander attacker, the stability of the DDQN-DNER algorithm begins to decline, and the final performance of the RNN-DDQN and RNN-PPO algorithms is significantly reduced; in contrast, the GraphSAGE-PPO algorithm still shows the highest performance and stability; this is because the increase in the number of hosts inevitably introduces more exploitable vulnerabilities and provides more potential targets for attackers, making the network defense scenario more complex; the GraphSAGE-PPO algorithm maintains excellent stability and overall performance by fully considering the network topology and connections.

[0112] In scenario C, the overall stability of all algorithms has been improved, and the GraphSAGE-PPO algorithm continues to demonstrate superior performance and robustness; in this scenario, the importance of cross-subnet hosts has been further enhanced, which means that when these critical hosts are attacked, the resulting penalty rewards increase significantly; although this leads to a decrease in the final reward value, the stability of the algorithm has been enhanced; this indicates that the defense agents pay more attention to protecting these critical hosts and can effectively prevent attackers from spreading laterally between subnets; this multi-level host importance enables the agents to have a deeper understanding of the network topology and defense strategies, helping them better identify potential threat patterns during the training process; therefore, the convergence of the defense strategy has been improved, and the training effect has become more effective and efficient.

[0113] Table 3 Comparison of the average convergence rewards of GAT-PPO with Norm-PPO, RNN-PPO, and Sb3-PPO in different scenarios

[0114]

[0115] Table 3 shows the maximum reward values obtained by the defense agents when different algorithms defend against different attackers in various scenarios; the data indicates that the GraphSAGE-PPO algorithm always obtains the highest reward value when defending against Bline and Meander attackers in all scenarios; for the Sleep attacker, all algorithms maintain zero-cost defense from the beginning in each scenario except for RNN-DDQN. The following is a detailed discussion of these results.

[0116] In scenario A, the maximum convergence values of the GraphSAGE-PPO algorithm when defending against Bline attackers are all around -20, indicating the efficiency of its defense strategy; when facing Meander attackers, the maximum performance of all algorithms decreases, but the GraphSAGE-PPO algorithm maintains a maximum reward of -26; this shows that the algorithm effectively considers the interactions between hosts, enabling it to formulate relatively effective defense strategies.

[0117] In scenario B, the increase in the number of hosts leads to a decrease in the performance of all algorithms, especially when facing Meander attackers; this indicates that the number of hosts and the random target selection of attackers are two key factors affecting the training of defense strategies and the corresponding defense costs.

[0118] In scenario C, the importance of cross-subnet hosts is enhanced, which inevitably increases the penalty for successful attacks. However, the performance of the GraphSAGE-PPO algorithm has not changed significantly, indicating that the defense agent trained by this algorithm can identify and prioritize the protection of these critical nodes. In addition, it can detect attack paths through information sharing between hosts, thereby taking preventive defense measures. Therefore, the defense cost has not increased as expected.

[0119] The present invention deeply integrates graph neural networks (GNNs) with deep reinforcement learning (DRL) algorithms, proposes the GraphSAGE-PPO algorithm, and trains defense agents to enhance the EDR system and mitigate APT threats.

[0120] The present invention models the attack and defense process in the real-world Internet of Things network scenario as a Markov decision process (MDP) to explore an efficient and timely automated defense strategy to help cybersecurity personnel protect network systems. First, a graph data structure is used to model the network scenario because a graph can effectively represent the states of different hosts and clearly describe the connections between them. Subsequently, the GraphSAGE model in graph neural networks is integrated with the deep reinforcement learning algorithm PPO. The core advantage of the integration is that GraphSAGE can efficiently analyze and process complex information in graph-structured data through information propagation and feature aggregation mechanisms. At the same time, the reinforcement learning ability of PPO enables the defense agent to learn and optimize the defense strategy, thereby taking the most effective defense measures in the face of complex network threats. Specifically, the GraphSAGE model promotes the real-time transmission of state information between different hosts in the network, enabling each host to access the state information of the hosts connected to it. This information exchange provides a rich and timely context basis for training the defense agent, enabling it to make more accurate defense decisions in the complex Internet of Things network environment. Finally, the defense agent uses the PPO algorithm for iterative training, selects the best defense action based on the current state information to cope with various potential network threats.

[0121] A series of experiments were conducted in the CybORG network security simulation environment, different network scenarios were set, and different types of attackers were used to train the defense agent. Comparative analysis with existing network defense algorithms shows that GraphSAGE-PPO is superior in terms of robustness and overall performance, indicating that the GraphSAGE-PPO algorithm can effectively identify attack paths and take more efficient defense measures in the face of complex network attacks.

[0122] Inspired by the above-described ideal embodiments of the present invention, through the above description, relevant staff can completely make various changes and modifications without departing from the technical idea of this invention. The technical scope of this invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A persistent threat mitigation method based on GraphSAGE-PPO, characterized in that, It includes the following steps: Step 1, construct a network node scenario model; Step 2, use the graph data structure to model the observation information of the network node model; Step 3, the defender agent receives the observation information, transmits it to the Actor network based on GraphSAGE, and decides the actions to be taken in the current state; Use the Critic network to evaluate this action.

2. The persistent threat mitigation method based on GraphSAGE-PPO according to claim 1, characterized in that, The graph data structure is G = (V, E), where V represents the set of each node feature, and E represents the connections between nodes.

3. The method for persistent threat mitigation based on GraphSAGE-PPO according to claim 2, wherein Process the neighbor nodes in the node, including: First, construct the neighbor set of node v S is the sampling size; Secondly, the features of the sampled neighbor nodes are aggregated through the mean aggregation function is the embedding of neighbor u at the (k-1)-th layer; Next, combine the embedding of the current node with the aggregated neighbor embeddings to update the node embedding. σ is the activation function, and W k is the weight matrix of the k-th layer, and CONCAT is the concatenation operation. Finally, obtain the final embedding of node v after k-layer aggregation.

4. The persistent threat mitigation method based on GraphSAGE-PPO according to claim 1, characterized in that Step 3 specifically includes: Step 31, initialize the actor network based on GraphSAGE, and initialize the critic network; Step 32. The defender agent, according to the current state s t , remodels the network observation information using the graph data structure G to obtain v t and e t ; inputs v t and e t into the actor network to obtain the action distribution; Step 33: Select argmaxP to execute the optimal defense action a t ; Obtain the expected cumulative reward using the critic network; Step 34: Use the advantage function A t to measure the advantage of taking action a t relative to the average action under s t compared to the average action under s t in terms of its goodness or badness; Step 35: Update the policy network using the objective function L CLIP (θ), and update the value network according to the observed cumulative reward.

5. The persistent threat mitigation method based on GraphSAGE-PPO according to claim 4, wherein The cumulative reward is the accumulation of several single-step rewards.

6. The persistent threat mitigation method based on GraphSAGE-PPO according to claim 5, characterized in that, The formula for the single-step reward is: R = -I·N admin -10·N impact -C action Among them, I represents the importance of the host to the entire network system; N admin represents the number of hosts whose administrator privileges have been obtained by the attacker; N impact represents the number of hosts affected by the "affected" operation; C action reflects the cost of performing defense actions.

7. The persistent threat mitigation method based on GraphSAGE-PPO according to claim 1, characterized in that The network node model includes: an interconnected network topology constructed by different hosts, switches, firewalls, routers, and servers.

8. A persistent threat mitigation method and system based on GraphSAGE-PPO, characterized in that, It includes: A memory for storing instructions executable by the processor; A processor for executing the instructions to implement the GraphSAGE-PPO-based persistent threat mitigation method according to any one of claims 1-7.

9. A computer-readable medium storing computer program code, characterized in that, The computer program code implements the GraphSAGE-PPO-based persistent threat mitigation method according to any one of claims 1-7 when executed by the processor.

Images