Attack path calculation method and device based on hierarchical reinforcement learning

Abstract

The invention discloses an attack path calculation method and device based on hierarchical reinforcement learning, and the method comprises the steps: carrying out the mixed community division of a network, and obtaining a high-level community and a bottom-level sub-community; constructing a high-level-bottom-level deep reinforcement learning framework, wherein the framework comprises a high-level community reinforcement learning model and a bottom-level community reinforcement learning model; the high-level community reinforcement learning model is used for optimizing a global attack direction, and the bottom-level community reinforcement learning model is used for learning an optimal local vulnerability utilization strategy to balance an attack success rate and defense cost; and fusing a global attack direction output by the high-level community reinforcement learning model and a vulnerability utilization selection result output by the bottom-level community reinforcement learning model, and generating and outputting a complete attack path. The problem of search space explosion can be relieved, and the time complexity of path generation is reduced; in addition, through layered and multi-objective optimization capability, an attack path with higher practicability and concealment is generated.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network security, and particularly relates to an attack path calculation method and device based on hierarchical reinforcement learning. BACKGROUND

[0002] In today's digital era, network security has become a core issue for protecting critical information infrastructure and data assets. With the continuous evolution of complex network threats such as APT attacks (Advanced Persistent Threats) and zero-day vulnerabilities, attackers' means are showing intelligent, concealed and complex characteristics. The traditional network security protection system relying on passive defense has been difficult to cope with the above threats and cannot identify potential risks in advance and verify the effectiveness of protection measures. Under this background, penetration testing, as a key technology for actively assessing network security vulnerabilities, the importance of its core link, attack path planning, is increasingly prominent. Attack path planning can help defenders accurately identify system vulnerabilities and quantify security risks by simulating attacker behavior and automatically generating the optimal attack sequence from the initial node to the target node, providing key support for network security protection strategy formulation.

[0003] Attack graph, as the current mainstream network security analysis tool, is an important technical basis for attack path planning research. This tool visualizes potential attack paths in the network through nodes (representing hosts, vulnerabilities, permissions, etc. network status) and directed edges (representing state transition relationships caused by attack behavior), and realizes quantitative analysis of vulnerability correlation and risk propagation logic. Its core value lies in transforming the scattered vulnerability dependency relationship in a complex network into a calculable attack scenario, helping defenders identify key attack surfaces and defense weak points. Existing attack path planning methods usually focus too much on a single attack effectiveness indicator. Many current methods mainly optimize isolated factors, such as path length (shortest path) or vulnerability criticality (path that exploits the most critical weakness). Although these indicators are important, they fail to provide a comprehensive perspective on attacks. There is currently a lack of comprehensive optimization capabilities for multi-dimensional objectives, including various stages of the attack lifecycle, the stealthiness and evasiveness required to bypass defenses, and interactions with diverse defense mechanisms. This limited scope greatly limits the applicability of these methods in complex real-world scenarios. Ultimately, this deficiency results in reduced practicality and effectiveness of the paths exploited by attackers, as well as reduced overall effectiveness of defense strategies aimed at countering these attacks. SUMMARY

[0004] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide an attack path calculation method and device based on hierarchical reinforcement learning, which automatically and optimizes the attack path planning process in penetration testing by combining network structure analysis, vulnerability assessment and machine learning technology.

[0005] In order to achieve the above object, the application adopts the following technical solutions:

[0006] In a first aspect, the application provides a method for calculating an attack path based on hierarchical reinforcement learning, comprising the following steps:

[0007] Performing mixed community division on the network to obtain high-level communities and bottom-level sub-communities; the mixed community division comprises a high-level community division strategy and a bottom-level community division strategy; the high-level community division strategy uses a meta-path-based community division algorithm to identify high-level communities; and the bottom-level community division strategy uses a label propagation (LPA) algorithm to identify bottom-level communities.

[0008] Constructing a high-level-bottom-level deep reinforcement learning framework, which comprises a high-level community reinforcement learning model and a bottom-level community reinforcement learning model; the high-level community reinforcement learning model is used to optimize a global attack direction; and the bottom-level community reinforcement learning model is used to learn an optimal local exploit strategy to balance an attack success rate and a defense cost.

[0009] Fusing a global attack direction output by the high-level community reinforcement learning model and an exploit selection result output by the bottom-level community reinforcement learning model to generate and output a complete attack path.

[0010] As a preferred technical solution, the meta-path-based community division algorithm performs the following steps:

[0011] Defining at least one meta-path for capturing typical behaviors of attackers according to an application scenario and a network structure of the network; the meta-path is a predefined sequence of relationships connecting two nodes in a heterogeneous network.

[0012] Calculating a path number along each meta-path between any two nodes in the test network, and taking the path number as a quantitative indicator of the similarity between the nodes.

[0013] Constructing a homogeneous graph based on the similarity between the nodes; the nodes in the homogeneous graph are of a uniform type, and the edge weights are the meta-path similarity between the corresponding nodes in the original heterogeneous network.

[0014] Setting the edge weights of the homogeneous graph in combination with a CVSS score, asset importance, and network isolation degree, inputting the homogeneous graph with the set weights into a Louvain algorithm, and identifying high-level communities by iteratively optimizing the network modularity of the Louvain algorithm.

[0015] As a preferred technical solution, the Louvain algorithm performs the following steps:

[0016] Local optimization phase: In the initialization, each node in the network is regarded as an independent community. For each node i in the network, it is moved to the community j where its neighbor nodes are located, and the gain of network modularity is calculated after the movement. If there are multiple neighbor communities that can make the modularity gain positive, node i will be moved to the neighbor community that can bring the maximum positive gain. Repeat the iteration until there is no node in the network that can bring an increase in modularity gain.

[0017] Community aggregation phase: fold each community identified in the local optimization phase into a supernode, update the edge weight between supernodes and the self-loop weight of supernodes, and repeat the local optimization phase for the newly constructed supernode network until a stable high-level community division result is obtained.

[0018] As a preferred technical solution, the label propagation LPA algorithm performs the following steps:

[0019] Assign a unique label to each node in the high-level community subgraph obtained by high-level community division;

[0020] Iteratively update the label of each node in random order, update the node label to the label with the highest frequency among its neighbor nodes, and if there are multiple highest frequency labels, randomly select one, until all node labels are stable or the maximum number of iterations is reached.

[0021] Group nodes with the same label into the same bottom-level subcommunity, and output the bottom-level subcommunity division result within the high-level community.

[0022] As a preferred technical solution, the high-level community reinforcement learning model performs the following steps:

[0023] Construct a first Actor-Critic architecture based on a first Actor network and a first Critic network, the first Actor network inputs the current community state and outputs a deterministic attack action; the first Critic network inputs the current community state and the attack action output by the first Actor network, and outputs the Q value of the state-action pair;

[0024] Introduce a target first Actor network and a target first Critic network to calculate the time difference target value through the target network to stabilize the training process;

[0025] Establish an experience replay buffer to store the (state, action, reward, next state) experience tuples generated by the interaction between the agent and the community environment, and randomly extract experience tuples to update network parameters during training;

[0026] Add Gaussian noise to the action output by the first Actor network to realize the exploration mechanism, and reduce the noise variance as the training process progresses, so that the strategy transitions from exploration to utilization;

[0027] The current community ID, the attacked community set, and the remaining resources are input as state, and a continuous attack strategy at the community level is output through the first Actor network to optimize the global attack direction.

[0028] As a preferred technical solution, the bottom-layer community reinforcement learning model performs the following steps:

[0029] A second Actor-Critic architecture based on a second Actor network and a second Critic network is constructed, the second Actor network inputs the bottom-layer sub-community local state and outputs the action probability distribution, and the second Critic network inputs the bottom-layer sub-community local state and outputs the state value.

[0030] The alternative objective function of PPO is defined as follows:

[0031] ;

[0032] Wherein, is the advantage function, which measures how much better a certain action is than the average level; epsilon is a constant, which is used to define the clipping range; the clip function limits the probability ratio to the range of [1-epsilon, 1+epsilon];

[0033] The current node ID, vulnerability information, node connection attributes, defense mechanism information, and local path history are input as state, and the vulnerability exploitation mode selection, target node selection, and attack operation execution are used as the action space.

[0034] A reward function is designed to give positive rewards for successful vulnerability exploitation, node penetration, and achieving local sub-goals, and negative rewards for being detected by the defense system, attack failure, excessive resource consumption, and repeated attacks.

[0035] The experience data is collected to calculate the advantage function, and the second Actor network and the second Critic network parameters are updated through the gradient ascent method to learn the optimal local vulnerability exploitation strategy.

[0036] As a preferred technical solution, the attack path fusion specifically includes:

[0037] The community attack sequence output by the high-layer community reinforcement learning model and the inter-node attack path output by the bottom-layer community reinforcement learning model are spliced in the attack logic order to form a complete attack path from the starting node to the target node, and the complete attack path includes the global community attack strategy and the local vulnerability exploitation details.

[0038] In a second aspect, the application provides an attack path calculation system based on hierarchical reinforcement learning, applied to the attack path calculation method based on hierarchical reinforcement learning, comprising a network community division module, an attack path planning module and a fusion module.

[0039] The network community division module is configured to perform hybrid community division on the network to obtain high-level communities and bottom-level sub-communities.

[0040] The attack path planning module is configured to construct a high-level-bottom-level deep reinforcement learning framework, which comprises a high-level community reinforcement learning model and a bottom-level community reinforcement learning model.

[0041] The fusion module is configured to fuse the global attack direction output by the high-level community reinforcement learning model and the vulnerability exploitation selection result output by the bottom-level community reinforcement learning model to generate and output a complete attack path.

[0042] In a third aspect, the application provides an electronic device, which comprises:

[0043] at least one processor; and

[0044] a memory connected in communication with the at least one processor; wherein

[0045] the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the attack path calculation method based on hierarchical reinforcement learning.

[0046] In a fourth aspect, the application provides a computer readable storage medium storing a program, and the program is executed by a processor to implement the attack path calculation method based on hierarchical reinforcement learning.

[0047] Compared with the prior art, the application has the following advantages and beneficial effects:

[0048] 1. The present application effectively decomposes complex networks into high-level and low-level communities by introducing a hybrid community partition algorithm. This greatly reduces the search space of attack paths, thus fundamentally solving the problem of low efficiency of traditional methods when facing large-scale networks. Existing technologies often struggle to deal with the combinatorial explosion caused by massive nodes and connections, resulting in excessive algorithm time consumption.

[0049] 2. Existing methods generally only focus on a single attack performance indicator, such as the shortest path or the highest vulnerability hazard, which leads to insufficient practicality and defense effectiveness of the planned attack path. The present application constructs a high-level-low-level deep reinforcement learning framework, which realizes the comprehensive optimization of multi-dimensional goals such as attack phase, concealment, and defense mechanism. This hierarchical and multi-objective optimization capability enables the present application to generate more practical and concealed attack paths.

[0050] 3. The high-level-low-level deep reinforcement learning framework of the present application combines macro-strategic decision-making and micro-detail operation. The high-level learning framework focuses on the overall direction and phased goals of the attack, avoiding getting stuck in local optimization; while the low-level learning framework focuses on specific vulnerability exploitation and defense avoidance, ensuring the success rate and concealment of the attack. This divide-and-conquer and collaborative optimization strategy makes up for the shortcomings of existing technologies that often only emphasize the global or local, and are difficult to balance both. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0052] Figure 1 The flowchart of the attack path calculation method based on hierarchical reinforcement learning of the embodiment of the present application;

[0053] Figure 2 The block diagram of the attack path calculation system based on hierarchical reinforcement learning of the embodiment of the present application.

[0054] Figure 3 The structural diagram of the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0056] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0057] like Figure 1 As shown, this embodiment presents an attack path calculation method based on hierarchical reinforcement learning. To address the problem of search space explosion, it first uses a hybrid community partitioning algorithm to divide the network into high-level and low-level communities, reducing the time complexity of path generation. Then, it constructs a high-level-low-level deep reinforcement learning framework. In this framework, high-level communities optimize the global attack direction using a deep deterministic policy gradient algorithm, while low-level sub-communities select vulnerabilities based on a proximal policy optimization algorithm to balance success rate and defense cost. Finally, it merges the attack paths generated by the high-level and low-level optimizations to output the final planned attack path for the entire network. Specifically, it includes the following:

[0058] S1. Perform hybrid community partitioning on the network to obtain high-level communities and low-level sub-communities; the hybrid community partitioning includes a high-level community partitioning strategy and a low-level community partitioning strategy. The high-level community partitioning strategy uses a meta-path-based community partitioning algorithm to identify high-level communities, and the low-level community partitioning strategy uses the label propagation LPA algorithm to identify low-level communities.

[0059] S11. Community partitioning algorithm based on meta-path, details of which are as follows:

[0060] A metapath is a predefined sequence of relationships connecting two nodes in a heterogeneous network. It does not refer to an actual path in the network, but rather a path pattern. For example, in a heterogeneous network containing a web server (W), a database (D), and a personal host (H), one metapath could be H→W, representing "personal host accesses web server"; another metapath could be W→D, representing "web server accesses database"; and yet another metapath could be H→W→D, representing "personal host accesses database via web server".

[0061] Further, the community division algorithm of the meta-path performs the following steps:

[0062] S111, a series of meaningful meta-paths are manually defined according to application scenarios and network structures. For example, in attack path analysis, the meta-path of personal host→web server→database is very meaningful, which can capture the typical behavior of attackers from personal host to web server and finally to database.

[0063] S112, the number of paths between two nodes along the specified meta-path is calculated. The more the number of paths, the more similar the nodes are. If two “personal host” nodes are connected to the same web server through the meta-path H→W, they are considered to have a high similarity.

[0064] S113, a homogenous graph is constructed using the similarity calculated in the previous step. In the homogenous graph, the node types are the same, but the edges have a weight, which represents the meta-path similarity between the nodes in the original heterogeneous network. The higher the similarity, the greater the edge weight.

[0065] S114, the constructed homogenous graph is inputted, and the Louvain algorithm is applied to identify the community. Since the edge weight of the homogenous graph encodes the semantic information of the meta-path, the finally divided community has more specific meaning.

[0066] Further, the Louvain algorithm is a hierarchical clustering algorithm based on modularity optimization, which can quickly discover community structure in large-scale networks. The core idea is to iteratively optimize the modularity of the network. Modularity is an index to measure the strength of network community structure. The higher the value, the more connected the network is, and the sparser the connection between communities, and the better the community division effect. The Louvain algorithm gradually aggregates nodes in the network into communities through a greedy optimization process, and presents the community structure of the network at multiple levels.

[0067] The calculation formula of modularity in the Louvain algorithm is as follows:

[0068]

[0069] wherein, represents the weight inside the community c, represents the weight of the edge connected to the point inside the community c, including the edge inside the community and the edge outside the community.

[0070] The calculation formula of modularity gain is as follows:

[0071]

[0072] where, is the sum of weights of links within the community C (equal to the degree when the weight is 1), if it is the initial case, i.e. a node as a community, it is for the node itself to its own connection, at this time still need to start point plus weight, the end point plus weight (even if the starting point and the end point for the same node, for acyclic graph weight is 0).

[0073] is the sum of weights of links associated to the nodes in C;

[0074] kiis the sum of weights of links associated to node i;

[0075] ki, inis the sum of weights of links from node i to the nodes in C;

[0076] m is the sum of weights of all links in the network.

[0077] Further, the Louvain algorithm includes a local optimization phase (Greedy Optimization) and a community aggregation phase (Community Aggregation),

[0078] S1141, the local optimization phase includes the following specific content:

[0079] Initialization: when the algorithm starts, each node in the network is regarded as an independent community;

[0080] Iterative merging: for each node i in the network, the algorithm attempts to move it to the community j in which its neighbor nodes are located, after moving, the algorithm calculates the gain AQ of the network modularity;

[0081] Maximum gain: if there are multiple neighbor communities that can make the modularity gain positive, node i will be moved to the neighbor community that can bring the maximum positive gain;

[0082] Repeat: this process is repeated until no node in the network moves to bring further gain in modularity, i.e. a local modularity maximization state is reached.

[0083] S1142, the community aggregation phase includes the following specific content:

[0084] Building a new network: after the local optimization phase ends, each detected community will be "folded" into a single "super node";

[0085] Edge weight update: the edges within the original community form self-loops of the new supernode, whose weight is the sum of all original edges' weights; the edges between original communities become edges between new supernodes, whose weight is the sum of all original edges' weights between the corresponding communities;

[0086] Iteration: the new network composed of supernodes performs the local optimization phase again, repeating the above process.

[0087] In a specific application, weights are set in combination with CVSS scores (vulnerability exploit difficulty), asset importance, network isolation degree, etc. If a connection between two nodes (e.g., a communication path from A to B) involves a high-risk vulnerability or is a critical stepping stone for the next attack (such as privilege escalation or lateral movement), a higher weight can be assigned.

[0088] S12, Label Propagation Algorithm (LPA), the specific content is as follows:

[0089] Label Propagation Algorithm (LPA) is an unsupervised community discovery algorithm based on graph topology. Its core idea is that each node in the network propagates its "label" (representing its community ID) to its neighbor nodes and receives labels from its neighbors. The node will adopt the most frequently appearing label among its neighbors, gradually forming consistent communities. The advantage of LPA is its simplicity, efficiency, and the absence of parameters such as the number of pre-set communities. It identifies community structure by simulating the propagation of information in the network.

[0090] Further, the Label Propagation Algorithm (LPA) performs the following steps:

[0091] S121, initialization: each node i in the network is assigned a unique label Li, at the beginning of the algorithm, each node belongs to an independent community;

[0092] S122, label propagation and update: the algorithm enters the iteration process, in each iteration, for each node i in the network (usually in random order or pre-set order), it checks all its neighbor nodes' current labels, and updates its own label to the most frequently appearing label among its neighbor nodes; if there are multiple labels with the highest frequency (i.e., ties), one of them is usually randomly selected, or according to a pre-set rule (such as selecting the smallest / largest label ID); this process continues until all nodes in the network no longer change their labels in one iteration. This means that each node has adopted the most mainstream community affiliation among its neighbors, forming stable communities; or a maximum number of iterations is reached - to prevent certain special cases from not converging or converging too slowly, a maximum number of iterations can be set;

[0093] S123, community formation: when the algorithm converges or reaches the maximum number of iterations, all nodes with the same label are classified into the same community.

[0094] In one specific application, LPA is used to perform more fine-grained local community discovery or validate existing sub-structures within a high-level community, thus assisting the decision-making of the underlying deep reinforcement learning. The Louvain algorithm divides the entire network into several high-level communities. However, within a high-level community, the number of nodes can still be considerable, and it can contain smaller, highly functionally related "micro-communities" or "attack groups" (e.g., web servers, database servers, and cache servers within a certain web application cluster). In order to enable the underlying reinforcement learning to focus and optimize local exploits more effectively, it is beneficial to identify these smaller groups.

[0095] Input: each high-level community divided by the Louvain algorithm is input to LPA as an independent subgraph.

[0096] Execution: LPA runs within the subgraph and propagates labels to the nodes therein.

[0097] Output: LPA identifies more fine-grained "low-level sub-communities" within the high-level community. For example, within a high-level community containing multiple service modules, LPA can identify a group of servers dedicated to authentication, a group of data storage servers, etc.

[0098] S2, construct a high-level-underlying deep reinforcement learning framework, which includes a high-level community reinforcement learning model and an underlying community reinforcement learning model; the high-level community reinforcement learning model is used to optimize the global attack direction, and the underlying community reinforcement learning model is used to learn the optimal local exploit strategy to balance the attack success rate and defense cost;

[0099] S21, the high-level community reinforcement learning model adopts Deep Deterministic Policy Gradient (DDPG), which is a model-free reinforcement learning algorithm based on Actor-Critic architecture, designed for learning optimal policies in continuous action space. It combines the advantages of Deep Q Network (DQN) (using neural networks to handle high-dimensional state space) and policy gradient method (directly learning policy). DDPG generates deterministic actions through the training of an "actor" network and evaluates the value of these actions through a "critic" network, thus enabling effective learning in continuous control tasks. It also introduces a target network and experience replay mechanism to improve the stability and efficiency of training.

[0100] Furthermore, the high-level community reinforcement learning model performs the following steps:

[0101] S211, Actor-Critic architecture:

[0102] The first Actor network: This is a neural network that takes the current state of the environment as input and outputs a deterministic action. Its goal is to learn a policy that allows the network to choose the optimal action given a particular state.

[0103] First Critic Network: This is another neural network. The input is the current state of the environment and the action output by the first Actor network. The output is the Q-value (or value) of that state-action pair, which is the expected long-term cumulative reward after performing that action. Its goal is to accurately evaluate the quality of the actions performed by the first Actor network.

[0104] S212, Deterministic Policy Gradient: Unlike traditional policy gradient algorithms (such as REINFORCE), DDPG's first Actor network outputs a deterministic action, rather than a probability distribution of actions. This means that for the same input state, the first Actor network will always produce the same action. The first Critic network provides a gradient to the first Actor network by evaluating the Q-value of the actions produced by the first Actor network, guiding the first Actor network to adjust its policy so that it can choose actions that can obtain higher Q-values. The update of the first Actor network is based on the gradient of the first Actor network's output action with respect to the Q-value, which makes the learning process more stable.

[0105] S213. Target Networks: To improve training stability, DDPG introduces two "target networks": the target actor network and the target critic network. The target network is used to calculate the TD (temporal difference) target value, which provides a relatively stable Q-value evaluation benchmark, avoiding oscillations and divergences during training, similar to the target network in DQN.

[0106] S214. Experience Replay: DDPG uses an experience replay buffer to store "experience tuples" (s, a, r, s′) (state, action, reward, next state) of the agent's interactions with the environment. During training, a batch of experiences is randomly selected from the experience replay buffer for training. This breaks down data correlations and avoids excessively high correlations in continuously sampled data, which can lead to unstable neural network training. Furthermore, the same batch of experience data can be used for training multiple times.

[0107] S215. Exploration Mechanism: Since the Actor outputs deterministic actions, to ensure the agent can explore the environment, DDPG typically adds Gaussian noise or other stochastic processes (such as the Ornstein-Uhlenbeck process) to the Actor's output actions. As training progresses, the variance of the noise gradually decreases, thus shifting the policy from exploration to exploitation.

[0108] In a specific application, at the high-level community level of this patent, the "attack direction" can be abstracted as a continuous choice. For example, there may be multiple connecting edges from the current community A to the target community B. Choosing which edge to use, or how to allocate the "attack weights" among multiple possible intermediate communities, can be considered as a continuous decision. DDPG can directly output these continuous strategies, rather than discrete choices.

[0109] The goal of high-level DDPG is to optimize a global strategy, such as "which community should be attacked next at the community level." Its state can include the current community ID, the set of attacked communities, remaining time, or resources. DDPG can learn a stable and efficient strategy to guide attackers in lateral movement within high-level networks.

[0110] Although the Louvain algorithm simplifies the search space, the connections between high-level communities and the potential attack value remain complex. DDPG's deep neural network can handle this high-dimensional and complex state information and guides the first actor network to learn the optimal global attack strategy through evaluation by the first Critic network.

[0111] The presence of experience replay and the target network makes DDPG more stable during training, enabling it to learn more effectively from interactions with the environment, ultimately resulting in a high-performance global attack path decision model.

[0112] S22. Low-level community reinforcement learning model: Proximal Policy Optimization (PPO) is a model-free reinforcement learning algorithm based on the Actor-Critic architecture. It aims to combine the advantages of policy gradient methods (such as REINFORCE and A2C) with the stability of Trust Domain Policy Optimization (TRPO), while avoiding the complexity of TRPO. PPO significantly improves training stability and convergence speed by limiting the magnitude of each policy update, ensuring that the new policy does not deviate too far from the old policy. This makes it a widely used and high-performance algorithm in practice. It can handle both discrete and continuous action spaces.

[0113] Furthermore, the low-level community reinforcement learning model performs the following steps:

[0114] S221, Actor-Critic Architecture: Similar to DDPG, PPO also adopts an Actor-Critic architecture. The second Actor (Policy Network): The input is the current state St, and the output is the probability distribution At (for discrete actions) or the mean and variance (for continuous actions). It is responsible for deciding what action to take in a given state. The second Critic (Value Network): The input is the current state St, and the output is the value V(St) of that state, i.e., the cumulative reward expected to be obtained starting from that state. It is responsible for evaluating the merits of the current state.

[0115] S222, Clipped Surrogate Objective: The core innovation of PPO lies in its clipped surrogate objective function. PPO defines a probability ratio, representing the ratio of the probability of the new policy versus the old policy on a specific action, as follows:

[0116]

[0117] The alternative objective function for PPO is:

[0118]

[0119] in, The advantage function measures how much better a particular action is than the average. It is typically estimated using the TD error.

[0120] ϵ is a small constant (e.g., 0.1 or 0.2) used to define the clipping range;

[0121] The clip function calculates probability ratios. It is limited to the range [1-ϵ, 1+ϵ].

[0122] This pruning mechanism ensures that the policy doesn't deviate too much with each update, thus guaranteeing training stability and preventing performance degradation or even crashes caused by excessively large step sizes. At the same time, it closely approximates the original policy gradient objective, effectively guiding the policy towards its optimal direction.

[0123] S223. The agent interacts with the environment to collect a certain amount of empirical data (usually a single trajectory or a batch of trajectories). The collected data is used to calculate the advantage function. Using the aforementioned alternative objective function, the parameters of the second Actor network and the second Critic network are updated via gradient ascent.

[0124] In a specific application, once High-Level Reinforcement Learning (DDPG) determines the "macro" attack direction from the current high-level community A to the next target high-level community B, the specific "micro" attack steps, namely how to select a node within high-level community A as a springboard, which vulnerability to exploit, and how to reach the entry node of high-level community B, are all completed by PPO.

[0125] i) State Representation: The state space of PPO will focus on local information within the current lower-level sub-communities. This includes:

[0126] The current agent (attacker)'s node ID, information on vulnerabilities discovered on the current node (type, CVSS score, exploit difficulty), connection attributes between the current node and neighboring nodes (such as open ports, service types), defense mechanism information of neighboring nodes (such as firewall rules, IDS / IPS deployment), history of attacked local paths (to avoid loops or repeated attacks), and may include more granular "micro-community" information obtained from LPA to help PPO understand the local structure.

[0127] ii) Action Space: The action space of a PPO will be discrete or mixed, including: selecting an available exploit (e.g., a specific exploit for a service vulnerability), selecting a neighboring node as the next attack target, and performing a specific attack operation (e.g., scanning, authentication bypass, privilege escalation).

[0128] Strategy Learning: PPO learns an optimal local strategy through interaction with the lower-level community environment: given the current node state and exploitable vulnerabilities, it selects which vulnerability exploitation method and springboard node can achieve local goals with the highest efficiency and lowest risk (e.g., successfully penetrating the next critical node or reaching the boundary of the higher-level community B).

[0129] iii) Design of the reward function:

[0130] A: Positive Reward (Success Rate):

[0131] Successfully exploited a vulnerability and gained access.

[0132] Successfully penetrated from one node to another.

[0133] To achieve local sub-goals (e.g., to control a critical service).

[0134] Finally, it successfully connected to the entry node of the target high-rise community.

[0135] B: Negative Rewards / Penalties (Defense Costs / Stealth):

[0136] It is detected by the defense system (e.g., triggering an IDS / IPS alert, or being captured by a honeypot).

[0137] The attack failed or took too long.

[0138] Excessive resource consumption (such as bandwidth and CPU).

[0139] Perform high-noise or high-signature attack operations (easily detected).

[0140] Repeated attacks on nodes that have been controlled or paths that have been probed.

[0141] S3. High-level-low-level attack path fusion: This involves fusing the global attack direction output by the high-level community reinforcement learning model with the vulnerability exploitation selection result output by the low-level community reinforcement learning model to generate and output a complete attack path.

[0142] This invention utilizes a high-level-low-level deep reinforcement learning framework to achieve a core intelligent decision-making mechanism for multi-dimensional target comprehensive optimization. It no longer simply seeks the shortest path but simulates a more complex decision-making process closer to that of a real attacker. The high-level community is responsible for macro-level attack direction and strategy selection, ensuring the strategic nature and overall efficiency of the attack path. The low-level sub-communities are responsible for micro-level vulnerability exploitation selection, achieving a balance between success rate and defense cost. This hierarchical design allows the algorithm to simultaneously consider global optimization and local details, enabling it to plan effective overall attack strategies while ensuring efficient and covert vulnerability exploitation.

[0143] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0144] Based on the same idea as the attack path calculation method based on hierarchical reinforcement learning in the above embodiments, the present invention also provides an attack path calculation system based on hierarchical reinforcement learning, which can be used to execute the above-described attack path calculation method based on hierarchical reinforcement learning. For ease of explanation, the structural diagram of the embodiment of the attack path calculation system based on hierarchical reinforcement learning only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0145] Please see Figure 2In another embodiment of this application, an attack path calculation system 100 based on hierarchical reinforcement learning is provided. The system includes a network community division module 101, an attack path planning module 102, and a fusion module 103.

[0146] The network community segmentation module 101 is used to perform hybrid community segmentation on the network to obtain high-level communities and low-level sub-communities. The hybrid community segmentation includes a high-level community segmentation strategy and a low-level community segmentation strategy. The high-level community segmentation strategy uses a meta-path-based community segmentation algorithm to identify high-level communities, and the low-level community segmentation strategy uses the label propagation LPA algorithm to identify low-level communities.

[0147] The attack path planning module 102 is used to construct a high-level-low-level deep reinforcement learning framework, which includes a high-level community reinforcement learning model and a low-level community reinforcement learning model. The high-level community reinforcement learning model is used to optimize the global attack direction, and the low-level community reinforcement learning model is used to learn the optimal local vulnerability exploitation strategy to balance the attack success rate and defense cost.

[0148] The fusion module 103 is used to fuse the global attack direction output by the high-level community reinforcement learning model with the vulnerability exploitation selection result output by the low-level community reinforcement learning model to generate and output a complete attack path.

[0149] It should be noted that the attack path calculation system based on hierarchical reinforcement learning of the present invention corresponds one-to-one with the attack path calculation method based on hierarchical reinforcement learning of the present invention. The technical features and beneficial effects described in the embodiments of the attack path calculation method based on hierarchical reinforcement learning described above are applicable to the embodiments of the attack path calculation method based on hierarchical reinforcement learning. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0150] Furthermore, in the above embodiments of the attack path calculation system based on hierarchical reinforcement learning, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the attack path calculation system based on hierarchical reinforcement learning is divided into different program modules to complete all or part of the functions described above.

[0151] Please see Figure 3In one embodiment, an electronic device is provided for implementing an attack path calculation method based on hierarchical reinforcement learning. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as an attack path calculation method program 203 based on hierarchical reinforcement learning.

[0152] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, SmartMediaCard (SMC), SecureDigital (SD) card, FlashCard, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the attack path calculation method program 203 based on hierarchical reinforcement learning, but also to temporarily store data that has been output or will be output.

[0153] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0154] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0155] The attack path calculation method program 203 based on hierarchical reinforcement learning stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0156] The network is divided into hybrid communities to obtain high-level communities and low-level sub-communities. The hybrid community division includes a high-level community division strategy and a low-level community division strategy. The high-level community division strategy uses a meta-path-based community division algorithm to identify high-level communities, and the low-level community division strategy uses the label propagation LPA algorithm to identify low-level communities.

[0157] A high-level-low-level deep reinforcement learning framework is constructed, which includes a high-level community reinforcement learning model and a low-level community reinforcement learning model. The high-level community reinforcement learning model is used to optimize the global attack direction, and the low-level community reinforcement learning model is used to learn the optimal local vulnerability exploitation strategy to balance the attack success rate and defense cost.

[0158] By combining the global attack direction output by the high-level community reinforcement learning model with the vulnerability exploitation selection result output by the low-level community reinforcement learning model, a complete attack path is generated and output.

[0159] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0160] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An attack path calculation method based on hierarchical reinforcement learning, characterized in that, Includes the following steps: The network is divided into hybrid communities to obtain high-level communities and low-level sub-communities. The hybrid community division includes a high-level community division strategy and a low-level community division strategy. The high-level community division strategy uses a meta-path-based community division algorithm to identify high-level communities, and the low-level community division strategy uses the label propagation LPA algorithm to identify low-level communities. A high-level-low-level deep reinforcement learning framework is constructed, which includes a high-level community reinforcement learning model and a low-level community reinforcement learning model. The high-level community reinforcement learning model is used to optimize the global attack direction, and the low-level community reinforcement learning model is used to learn the optimal local vulnerability exploitation strategy to balance the attack success rate and defense cost. By combining the global attack direction output by the high-level community reinforcement learning model with the vulnerability exploitation selection result output by the low-level community reinforcement learning model, a complete attack path is generated and output.

2. The attack path calculation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The community partitioning algorithm for the meta-path performs the following steps: Based on the application scenario and network structure, at least one meta-path is defined to capture typical attacker behaviors; the meta-path is a predefined sequence of relationships connecting two nodes in a heterogeneous network. Calculate the number of paths between any two nodes in the test network along each of the meta-paths, and use the number of paths as a quantitative indicator of the similarity between nodes. A homogeneous graph is constructed based on the similarity between nodes. The nodes in the homogeneous graph are of the same type, and the edge weight is the meta-path similarity of the corresponding node in the original heterogeneous network. The edge weights of the homogeneous graph are set by combining CVSS scores, asset importance, and network isolation. The weighted homogeneous graph is then input into the Louvain algorithm, which iteratively optimizes the network modularity to identify high-level communities.

3. The attack path calculation method based on hierarchical reinforcement learning according to claim 2, characterized in that, The Louvain algorithm performs the following steps: Local optimization phase: During initialization, each node in the network is treated as an independent community. For each node i in the network, it is moved to the community j where its neighboring node is located. After the move, the gain of network modularity is calculated. If there are multiple neighboring communities that can make the modularity gain positive, node i will be moved to the neighboring community that can bring the maximum positive gain. Repeating the iteration until no node moves in the network can increase the modularity gain; Community aggregation phase: Each community identified in the local optimization phase is folded into a super node, the edge weights between super nodes and the self-loop weights of super nodes are updated, and the local optimization phase is repeated on the newly constructed super node network until a stable high-level community partitioning result is obtained.

4. The attack path calculation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The label propagation LPA algorithm performs the following steps: Assign a unique label to each node in the subgraph of each high-level community obtained by dividing the high-level community; The label of each node is updated iteratively in a random order. The node label is updated to the label that appears most frequently among its neighboring nodes. If there are multiple highest frequency labels, one is randomly selected until all node labels are stable or the preset maximum number of iterations is reached. Nodes with the same label are grouped into the same underlying sub-community, and the results of the underlying sub-community division within the higher-level community are output.

5. The attack path calculation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The high-level community reinforcement learning model performs the following steps: A first Actor-Critic architecture is constructed based on a first Actor network and a first Critic network. The first Actor network takes the current community state as input and outputs a deterministic attack action. The first Critic network takes the current community state and the attack action output by the first Actor network as input and outputs the Q-value of the state-action pair. A first-target Actor network and a first-target Critic network are introduced, and the time difference target value is calculated through the target network to stabilize the training process. Establish an experience replay buffer to store experience tuples (state, action, reward, next state) generated by the agent's interaction with the community environment. During training, randomly select experience tuples to update network parameters. Gaussian noise is added to the output action of the first Actor network to realize the exploration mechanism. The noise variance is reduced as the training process progresses, so that the strategy transitions from exploration to exploitation. Using the current community ID, the set of attacked communities, and remaining resources as state inputs, the system outputs a continuous attack strategy at the community level through the first Actor network, thereby optimizing the global attack direction.

6. The attack path calculation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The underlying community reinforcement learning model performs the following steps: A second Actor-Critic architecture is constructed based on a second Actor network and a second Critic network. The second Actor network takes the local state of the underlying sub-community as input and outputs the action probability distribution; the second Critic network takes the local state of the underlying sub-community as input and outputs the state value. The alternative objective function for PPO is defined as follows: ； in, The advantage function measures how much better a particular action is than the average; ϵ is a constant used to define the clipping range; the clip function calculates the probability ratio. It is limited to the range [1-ϵ, 1+ϵ]. The current node ID, vulnerability information, node connection attributes, defense mechanism information, and local path history are used as state inputs, and the vulnerability exploitation method selection, target node selection, and attack operation execution are used as the action space. Design a reward function that gives positive rewards for successful vulnerability exploitation, node penetration, and achievement of local sub-goals, and negative rewards for being detected by the defense system, attack failure, excessive resource consumption, and repeated attacks. Collect empirical data to calculate the advantage function, update the parameters of the second Actor network and the second Critic network through the gradient ascent method, and learn the optimal local vulnerability exploitation strategy.

7. The attack path calculation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The attack path fusion is specifically as follows: The community attack sequence output by the high-level community reinforcement learning model and the node attack path output by the low-level community reinforcement learning model are concatenated in the order of attack logic to form a complete attack path from the starting node to the target node. The complete attack path includes global community attack strategies and details of local vulnerability exploitation.

8. An attack path calculation system based on hierarchical reinforcement learning, characterized in that, The attack path calculation method based on hierarchical reinforcement learning, applied to any one of claims 1-7, includes a network community partitioning module, an attack path planning module, and a fusion module; The network community segmentation module is used to perform hybrid community segmentation on the network to obtain high-level communities and low-level sub-communities. The hybrid community segmentation includes a high-level community segmentation strategy and a low-level community segmentation strategy. The high-level community segmentation strategy uses a meta-path-based community segmentation algorithm to identify high-level communities, and the low-level community segmentation strategy uses the label propagation LPA algorithm to identify low-level communities. The attack path planning module is used to construct a high-level-low-level deep reinforcement learning framework, which includes a high-level community reinforcement learning model and a low-level community reinforcement learning model. The high-level community reinforcement learning model is used to optimize the global attack direction, and the low-level community reinforcement learning model is used to learn the optimal local vulnerability exploitation strategy to balance the attack success rate and defense cost. The fusion module is used to fuse the global attack direction output by the high-level community reinforcement learning model with the vulnerability exploitation selection result output by the low-level community reinforcement learning model to generate and output a complete attack path.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the attack path calculation method based on hierarchical reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the attack path calculation method based on hierarchical reinforcement learning as described in any one of claims 1-7.

Images