Network attack and defense confrontation test method and system based on deep reinforcement learning

Abstract

This invention provides a network attack and defense adversarial testing method and system based on deep reinforcement learning, belonging to the field of network security technology. The method includes constructing a network topology adjacency matrix and performing spectral decomposition; calculating structural distances based on node spectral coordinates to divide attack clusters and configuring attack agents; the attack agents executing attack actions to obtain rewards; recording node state changes, calculating attack propagation speed, and identifying dominant paths, encoding these paths as feature vectors and inputting them into defense agents; the defense agents executing defense actions to obtain rewards; and updating the agents' network parameters through reinforcement learning using the reward feedback from both sides. This invention can automatically identify critical attack propagation paths, achieving dynamic and adaptive attack and defense adversarial testing, improving the realism and automation level of the test.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a network attack and defense confrontation testing method and system based on deep reinforcement learning. Background Technology

[0002] Network attack and defense testing is a crucial means of evaluating the effectiveness of network security defense systems. As networks grow in scale and complexity, traditional testing methods based on fixed rules or static vulnerability scanning are no longer sufficient to simulate dynamic and intelligent attack behaviors such as advanced persistent threats. In recent years, deep reinforcement learning technology, due to its powerful sequence decision-making and policy optimization capabilities, has been introduced into the field of network attack and defense to train agents capable of autonomously exploring the network environment and launching simulated attacks, aiming to generate attack paths and test scenarios that more closely resemble reality.

[0003] In existing technologies, the focus is typically on building a global attack agent or a few agents to explore the entire target network. This relies on a holistic observation of the network state, such as node attributes, connectivity, and service vulnerability information, based on which the agent selects attack actions. The execution of these attack actions alters the network state and generates corresponding reward signals. The agent updates its policy network by collecting experiential data through interaction with the environment, aiming to maximize cumulative rewards and thereby discover effective attack sequences or critical vulnerabilities.

[0004] However, due to the often large scale and complex structure of modern enterprise or infrastructure networks, treating them as a whole for observation and decision-making leads to extremely high dimensionality in the state and action spaces, causing severe dimensionality problems. This results in low training efficiency and convergence difficulties for deep reinforcement learning algorithms, making it difficult to find effective testing strategies within limited computing resources and time. More importantly, this global perspective ignores the inherent topology and functional partitioning characteristics within the network. Different regions of the network typically exhibit significant differences in connection density, service type, and defense strength, and the propagation patterns and impact speeds of attacks vary across different regions. The inability to effectively capture and utilize this structural heterogeneity leads to overly generalized attack agent strategies, failing to conduct refined and differentiated attack testing tailored to the specific characteristics of network regions. This reduces the realism of adversarial testing and the ability to discover local critical weaknesses. Summary of the Invention

[0005] This invention provides a network attack and defense confrontation testing method and system based on deep reinforcement learning, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a network attack and defense adversarial testing method based on deep reinforcement learning, comprising:

[0007] Collect the node connection relationships and service types of the network, and construct the network topology adjacency matrix;

[0008] Perform spectral decomposition on the network topology adjacency matrix, calculate eigenvalues ​​and eigenvectors, use the component magnitudes of the eigenvectors as the spectral coordinates of each node, and calculate the structural distance between nodes.

[0009] The network nodes are clustered based on structural distance to determine attack clusters, and an independent attack agent is configured for each attack cluster.

[0010] Each attacking agent selects and executes an attack action based on the node connection relationship and service type within the corresponding attack cluster, and obtains attack reward feedback.

[0011] Record the state change times of nodes within each attack cluster, calculate the attack propagation speed between adjacent attack clusters, construct a directed graph to identify the dominant attack propagation path, and encode the dominant attack propagation path as an attack propagation direction feature vector as the observation input of the defense agent.

[0012] The defensive agent selects and executes defensive actions based on the observation input containing the feature vector of the dominant propagation direction of the attack, and obtains defensive reward feedback;

[0013] Based on the attack reward feedback and the defense reward feedback, the deep neural network parameters of each attack agent and defense agent are updated respectively through reinforcement learning;

[0014] Output the penetration depth and attack spread rate of each attack cluster.

[0015] In one optional embodiment, the network topological adjacency matrix is ​​subjected to spectral decomposition to calculate eigenvalues ​​and eigenvectors. The magnitude of the eigenvector components is used as the spectral coordinates of each node. The calculation of the structural distance between nodes includes:

[0016] Graph Laplacian operator matrix is ​​constructed based on network topology adjacency matrix, and spectral decomposition operation is performed to solve for eigenvalue spectrum and orthogonal eigenvector basis;

[0017] The eigenvalues ​​in the eigenvalue spectrum are sorted in ascending order of numerical value. Low-frequency eigenvalues ​​are selected, and the corresponding orthogonal eigenvectors are extracted. The orthogonal eigenvectors carry the global topological skeleton and multi-level community structure of the network.

[0018] Amplitude extraction is performed on each component of the orthogonal eigenvector to obtain the amplitude projection of each node on each eigenvector dimension;

[0019] The magnitude projections of each node onto the orthogonal eigenvectors are vectorized and assembled to generate the coordinate representation of each node in the spectral embedding space, which maps the discrete graph topology to the geometry of a continuous manifold.

[0020] Calculate the Euclidean distance between the coordinate representations of any two nodes, multiply each component of the Euclidean distance by the reciprocal of the corresponding eigenvalue, and sum them to obtain the weighted distance. Output the weighted distance to determine the structural distance between the nodes.

[0021] In one optional embodiment, the magnitude projections of each node onto orthogonal eigenvectors are vectorized and assembled to generate coordinate representations of each node in the spectral embedding space, including:

[0022] Construct a two-dimensional mapping table between node indexes and spectral dimensions. The two-dimensional mapping table records the magnitude projection values ​​of each node on each orthogonal feature vector dimension.

[0023] Row extraction is performed on the two-dimensional mapping table by node index to obtain the amplitude projection sequence of a single node on all orthogonal feature vector dimensions. The amplitude projection sequence is then vectorized, and each amplitude projection value in the amplitude projection sequence is used as a component element of the vector to construct the initial coordinate vector of the node.

[0024] Calculate the spectral energy contribution of each component of the initial coordinate vector corresponding to the eigenvalue. The spectral energy contribution is determined by the ratio of the eigenvalue to the sum of the eigenvalues. Apply a scale adjustment to each component of the initial coordinate vector based on the spectral energy contribution.

[0025] The scaled coordinate vector is used as the coordinate representation of the node in the spectral embedding space.

[0026] In one optional embodiment, clustering network nodes based on structural distance to determine attack clusters, and configuring an independent attack agent for each attack cluster includes:

[0027] Extract the set of structural distances from each node to other nodes from all structural distances between nodes;

[0028] The set of structural distances for each node is sorted in ascending order of numerical value, and the structural distance values ​​located at preset quantiles are selected from the sorting results to determine the cluster radius.

[0029] Take any node as the seed node, retrieve all neighboring nodes whose structural distance to the seed node is less than the cluster radius, and merge the seed node with all neighboring nodes into a candidate attack cluster.

[0030] For nodes within a candidate attack cluster, identify density peak nodes, calculate the structural distance between each non-density peak node and all density peak nodes within the candidate attack cluster, determine the density peak node with the smallest structural distance for each non-density peak node, and determine sub-attack clusters with each density peak node as the core.

[0031] Detect the connected paths between any two nodes in each sub-attack cluster in the original network topology adjacency matrix, add relay nodes to non-connected sub-attack clusters to complete topology repair, and determine the attack clusters;

[0032] An attack agent is instantiated for each attack cluster, and the state space dimension and action space structure of the attack agent are determined according to the number of nodes in the attack cluster and the type of attack operation.

[0033] In one optional embodiment, identifying density peak nodes within a candidate attack cluster includes:

[0034] Read the coordinate representation of each node in the spectral embedding space within the candidate attack cluster;

[0035] Traverse all nodes within the candidate attack cluster, taking each traversed node as the center node, and calculate the Euclidean distance between the coordinate representation of the center node and the coordinate representation of other nodes within the candidate attack cluster; count the number of nodes whose Euclidean distance to the center node is less than a preset radius, and record the number of nodes as the local density index of the center node; complete the traversal to obtain the local density index of each node within the candidate attack cluster.

[0036] Traverse all nodes within the candidate attack cluster, treat each traversed node as a node to be judged, and query all neighboring nodes whose structural distance to the node to be judged is less than the cluster radius.

[0037] If the local density index of the node to be determined is greater than the local density index of all neighboring nodes, then the node to be determined is marked as a density peak node.

[0038] Output all density peak nodes marked within the candidate attack cluster.

[0039] In one optional embodiment, the process includes recording the state change times of nodes within each attack cluster, calculating the attack propagation speed between adjacent attack clusters, constructing a directed graph to identify the dominant attack propagation path, and encoding the dominant attack propagation path as an attack propagation direction feature vector as the observation input for the defensive agent.

[0040] Record the moments when nodes in each attack cluster transition from a normal state to a compromised state, and construct a time sequence of node state changes for each attack cluster.

[0041] Identify the boundary nodes of adjacent attack clusters, extract the earliest state transition time in the boundary nodes of each adjacent attack cluster, and calculate the ratio of the time difference to the shortest path length between the boundary nodes as the attack propagation speed between adjacent attack clusters.

[0042] A directed graph is constructed with attack clusters as nodes. The attack propagation speed between adjacent attack clusters is used as the weight of the directed edge, and the direction of the directed edge is set from the attack cluster with the earlier state change time to the attack cluster with the later state change time.

[0043] Traverse all reachable paths in the directed graph, calculate the path propagation speed gradient, identify attack acceleration and deceleration propagation nodes, and select the dominant attack propagation path based on the path attack propagation activity index.

[0044] Extract the attack cluster node sequence and edge weight sequence along the attack-dominant propagation path, and encode the attack cluster node sequence and edge weight sequence into feature vectors for the attack-dominant propagation direction;

[0045] The feature vector of the dominant propagation direction of the attack is concatenated with the current state vector of each node in the attack cluster to form the observation vector of the defense agent, which is then input into the neural network of the defense agent to generate defense action decisions.

[0046] In one optional embodiment, all reachable paths in the directed graph are traversed, the path propagation speed gradient is calculated, attack acceleration and deceleration propagation nodes are identified, and the dominant attack propagation path is selected based on the path attack propagation activity index, including:

[0047] Identify attack clusters with zero in-degree in the directed graph as attack source nodes, and traverse the directed graph from the attack source nodes to record all reachable paths.

[0048] For each reachable path, extract the directed edge weight sequence, calculate the difference between adjacent weight elements in the directed edge weight sequence, and sum them to obtain the path propagation speed gradient.

[0049] Construct a time-series distribution map of the path propagation velocity gradient, map the path propagation velocity gradient to the time sequence of state changes of the attack clusters traversed by the path, and identify the peak and trough positions of the path propagation velocity gradient over time.

[0050] In the time-series distribution map of the path propagation velocity gradient, the attack clusters marked with peak positions are designated as attack acceleration and diffusion nodes, and the attack clusters marked with trough positions are designated as attack deceleration and diffusion nodes.

[0051] The number of attack acceleration and deceleration nodes on each reachable path is counted and the ratio is calculated to determine the path attack diffusion activity index.

[0052] The reachable path with the largest path propagation speed gradient and the largest path attack diffusion activity index is selected as the dominant attack propagation path.

[0053] A second aspect of this invention provides a network attack and defense adversarial testing system based on deep reinforcement learning, comprising:

[0054] Network topology unit, used to collect the node connection relationships and service types of the network, and to construct the network topology adjacency matrix;

[0055] The spectral decomposition calculation unit is used to perform spectral decomposition on the network topology adjacency matrix, calculate eigenvalues ​​and eigenvectors, use the component magnitudes of the eigenvectors as the spectral coordinates of each node, and calculate the structural distance between nodes.

[0056] The attack cluster partitioning unit is used to cluster network nodes based on structural distance to determine attack clusters, and to configure an independent attack agent for each attack cluster.

[0057] The attack execution unit is used by each attack agent to select and execute attack actions based on the node connection relationship and service type within the corresponding attack cluster, and to obtain attack reward feedback.

[0058] The attack propagation unit is used to record the state change time of nodes in each attack cluster, calculate the attack propagation speed between adjacent attack clusters, construct a directed graph to identify the dominant attack propagation path, and encode the dominant attack propagation path into an attack propagation direction feature vector as the observation input of the defense agent.

[0059] The defense execution unit is used by the defense agent to select and execute defense actions based on the observation input containing the feature vector of the attack's dominant propagation direction, and to obtain defense reward feedback.

[0060] The reinforcement learning unit is used to update the deep neural network parameters of each attacking agent and defending agent respectively through reinforcement learning based on the attack reward feedback and the defense reward feedback.

[0061] The attack results unit is used to output the penetration depth and attack spread rate of each attack cluster.

[0062] A third aspect of the present invention provides an electronic device, comprising:

[0063] processor;

[0064] Memory used to store processor-executable instructions;

[0065] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0066] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0067] In this embodiment of the invention, by performing spectral decomposition of the network topology and calculating the structural distance between nodes, quantitative analysis and precise partitioning of complex network structures are achieved. Clustering based on structural distance naturally divides the network into attack clusters with similar connectivity characteristics, ensuring that nodes within each attack cluster are highly topologically correlated. Each attack agent can focus on its own cluster's attack strategy, avoiding the problems of excessive decision-making dimensionality and inefficiency when a single agent faces the global network in traditional methods, thereby improving the targeting and exploration efficiency of attack strategies. By recording node state changes and calculating attack propagation speed, the spread of attack behavior in the network can be dynamically captured. Encoding the dominant propagation path as a directional feature vector provides the defense agent with highly condensed spatiotemporal situational information about the current focus of attack propagation and the direction of future potential threats. Driven by their respective reward signals, the attack and defense agents engage in adversarial learning, forming a dynamically balanced game environment. The attacking agent tries different attack actions to maximize its reward, constantly discovering new attack paths and vulnerability exploitation methods; the defending agent, based on observations including attack propagation direction characteristics, learns how to most effectively deploy defensive resources to block or delay attacks, maximizing defense rewards. This adversarial iterative process based on deep reinforcement learning allows the strategies of both sides to continuously evolve in the game, thereby automatically generating and testing increasingly complex and covert attack and defense scenarios. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating the network attack and defense confrontation testing method based on deep reinforcement learning, as described in an embodiment of the present invention.

[0069] Figure 2 A diagram illustrating the decision-making logic architecture of a defensive intelligent agent. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0072] Figure 1 This is a flowchart illustrating the network attack and defense adversarial testing method based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0073] Collect the node connection relationships and service types of the network, and construct the network topology adjacency matrix;

[0074] Perform spectral decomposition on the network topology adjacency matrix, calculate eigenvalues ​​and eigenvectors, use the component magnitudes of the eigenvectors as the spectral coordinates of each node, and calculate the structural distance between nodes.

[0075] The network nodes are clustered based on structural distance to determine attack clusters, and an independent attack agent is configured for each attack cluster.

[0076] Each attacking agent selects and executes an attack action based on the node connection relationship and service type within the corresponding attack cluster, and obtains attack reward feedback.

[0077] Record the state change times of nodes within each attack cluster, calculate the attack propagation speed between adjacent attack clusters, construct a directed graph to identify the dominant attack propagation path, and encode the dominant attack propagation path as an attack propagation direction feature vector as the observation input of the defense agent.

[0078] The defensive agent selects and executes defensive actions based on the observation input containing the feature vector of the dominant propagation direction of the attack, and obtains defensive reward feedback;

[0079] Based on the attack reward feedback and the defense reward feedback, the deep neural network parameters of each attack agent and defense agent are updated respectively through reinforcement learning;

[0080] Output the penetration depth and attack spread rate of each attack cluster.

[0081] In one alternative implementation, network nodes are clustered and partitioned by spectral analysis of the network topology, and the attack and defense process is simulated based on a multi-agent reinforcement learning framework to evaluate network security defense capabilities.

[0082] The network data acquisition phase collects node connection relationships and service type information of the target network, recording attributes such as IP address, running services, and open ports of each node. For an enterprise network containing 500 nodes, 1200 connection relationships between nodes are collected, and 25 different service types are identified. The collected node connection relationships are converted into a network topology adjacency matrix with a size of 500×500. A value of 1 in the matrix element indicates that there is a connection between nodes, and a value of 0 indicates that there is no connection between nodes.

[0083] Spectral decomposition is performed on the adjacency matrix of the network topology to construct a graph Laplacian operator matrix. For a 500×500 Laplacian matrix, 500 eigenvalues ​​and corresponding eigenvectors are calculated. The eigenvalues ​​are sorted in ascending order of numerical value, and the top 20 low-frequency eigenvalues ​​and their corresponding eigenvectors are selected. These eigenvectors carry the global skeleton and community structure information of the network topology. The amplitude of each component is extracted from the selected 20 eigenvectors to generate the amplitude projection of each node along the eigenvector dimension. The amplitude projections of each node along the eigenvector are assembled into vectors to construct the coordinate representation of the node in the spectral embedding space. For node 37 in the network, its coordinates in the 20-dimensional spectral domain are [0.023, -0.045, 0.067, 0.015, -0.032, 0.055, 0.078, -0.025, 0.042, 0.069, -0.037, 0.058, 0.021, -0.063, 0.047, 0.052, -0.019, 0.036, 0.072, -0.044]. The Euclidean distance between any two node coordinate representations is calculated; for example, the Euclidean distance between node 37 and node 129 is 0.183. The weighted distance is obtained by multiplying each component of the Euclidean distance by the reciprocal of its corresponding eigenvalue and summing the results. This weighted distance is then used to determine the structural distance between nodes. The calculated structural distance between node 37 and node 129 is 0.275.

[0084] The network nodes are clustered based on structural distance. The structural distance set from each node to other nodes is extracted from the total structural distances between all nodes. The structural distance sets of each node are sorted in ascending order of numerical value, and the structural distance value at the 75th percentile (0.412) is selected as the cluster radius. Node 1 is selected as the seed node, and all neighboring nodes with a structural distance less than 0.412 are searched. A total of 63 nodes meet this condition, and Node 1 is merged with these 63 neighboring nodes to form a candidate attack cluster. Density peak nodes are identified within the candidate attack clusters. A local density calculation radius of 0.15 is set, and the number of nodes within a radius of 0.15 for each node is counted as the local density index. In the candidate attack clusters, the local density indices of nodes 12, 37, and 53 are found to be 9, 11, and 7 respectively, all greater than their neighboring nodes. These three nodes are marked as density peak nodes. The structural distances between each non-density peak node and the three density peak nodes within the candidate attack clusters are calculated, and the density peak node with the smallest structural distance is determined for each non-density peak node. Three sub-attack clusters were formed, with nodes 12, 37, and 53 as the core, containing 21, 29, and 14 nodes respectively. Connectivity paths between any two nodes within each sub-attack cluster were detected in the original network topology adjacency matrix. It was found that nodes in the node 53 sub-attack cluster were not connected, so node 68 was added as a relay node to complete the topology repair, ultimately confirming the three attack clusters. The entire network was divided into 23 attack clusters, with 23 attack agents instantiated for each. For the second attack cluster with 29 nodes, the agent's state space dimension was 87 (29 nodes × 3 states), and the action space structure included 5 scanning actions, 3 weighting actions, and 4 lateral movement actions.

[0085] Each attack agent selects and executes an attack action based on the node connectivity and service type within its respective attack cluster. The second attack agent observes an unauthorized access vulnerability in the database service running on node 37 within the cluster. It selects to perform a database scan, successfully discovering the vulnerability and gaining initial access privileges, earning an attack reward of +5. The attack agent then selects a privilege escalation action, exploiting a local privilege escalation vulnerability to elevate its privileges, earning an attack reward of +8. Using the acquired high-privilege credentials, the attack agent performs a lateral movement attack, successfully compromising three adjacent nodes within the cluster, earning an attack reward of +12.

[0086] Record the state change times of nodes within each attack cluster. The earliest state change in the first attack cluster occurs at time t=15, in the second attack cluster at time t=23, and in the third attack cluster at time t=37. Identify the boundary nodes of adjacent attack clusters. The boundary nodes of the first and second attack clusters are nodes 21 and 37, respectively, with a shortest path length of 2. Calculate the ratio of the time difference to the shortest path length between the boundary nodes as (23-15) / 2=4, which is taken as the attack propagation speed between the two attack clusters. Similarly, calculate the attack propagation speed between the second and third attack clusters as (37-23) / 3=4.67. Construct a directed graph with attack clusters as nodes, using the attack propagation speed between adjacent attack clusters as the weight of the directed edges. The directed edges point from the attack cluster with the earlier state change time to the attack cluster with the later state change time, forming a directed attack propagation graph. Traversing all reachable paths in the directed graph, we find that the path originating from the first attack cluster includes: First Attack Cluster → Second Attack Cluster → Third Attack Cluster → Seventh Attack Cluster → Twelfth Attack Cluster. The directed edge weight sequence on this path is [4, 4.67, 5.33, 3.25]. Calculating the difference between adjacent weight elements in the directed edge weight sequence and summing them yields the path propagation speed gradient: (4.67-4)+(5.33-4.67)+(3.25-5.33)=0.67+0.66-2.08=-0.75. We construct a time-series distribution graph of the path propagation speed gradient, mapping it to the state change time sequence of the attack clusters traversed by the path [15, 23, 37, 53, 66]. We identify a peak at t=53, corresponding to the seventh attack cluster as an attack acceleration node; and a trough at t=66, corresponding to the twelfth attack cluster as an attack deceleration node. Following similar analysis, the path with the largest path propagation speed gradient and the highest path attack diffusion activity index was selected from all reachable paths: First Attack Cluster → Fifth Attack Cluster → Tenth Attack Cluster → Fifteenth Attack Cluster as the dominant attack propagation path. The attack cluster node sequence [1, 5, 10, 15] and edge weight sequence [3.5, 4.8, 5.2] on the dominant attack propagation path were extracted and encoded into an attack dominant propagation direction feature vector [1, 5, 10, 15, 3.5, 4.8, 5.2]. This feature vector was concatenated with the current state vector of each node within each attack cluster to form the observation vector of the defensive agent, which was then input into the neural network of the defensive agent to generate defensive action decisions.

[0087] The defensive agent selects and executes isolation node actions based on observed inputs, successfully blocking the connection between key nodes within the tenth attack cluster and the outside world along the attack propagation path. The defensive agent receives a defense reward of +15. Based on attack and defense reward feedback, the neural network parameters of each attack and defense agent are updated using a deep Q-network algorithm. The deep neural network of the attack agent contains an input layer (87 neurons), two hidden layers (containing 64 and 32 neurons respectively), and an output layer (12 neurons), with approximately 7800 neural network parameters. After 2000 rounds of adversarial training, the penetration depth and attack spread rate of each attack cluster are output. For example, the penetration depth of the second attack cluster is 62%, and the attack spread rate is 4.25 nodes / minute.

[0088] In one optional embodiment, the network topological adjacency matrix is ​​subjected to spectral decomposition to calculate eigenvalues ​​and eigenvectors. The magnitude of the eigenvector components is used as the spectral coordinates of each node. The calculation of the structural distance between nodes includes:

[0089] Graph Laplacian operator matrix is ​​constructed based on network topology adjacency matrix, and spectral decomposition operation is performed to solve for eigenvalue spectrum and orthogonal eigenvector basis;

[0090] The eigenvalues ​​in the eigenvalue spectrum are sorted in ascending order of numerical value. Low-frequency eigenvalues ​​are selected, and the corresponding orthogonal eigenvectors are extracted. The orthogonal eigenvectors carry the global topological skeleton and multi-level community structure of the network.

[0091] Amplitude extraction is performed on each component of the orthogonal eigenvector to obtain the amplitude projection of each node on each eigenvector dimension;

[0092] The magnitude projections of each node onto the orthogonal eigenvectors are vectorized and assembled to generate the coordinate representation of each node in the spectral embedding space, which maps the discrete graph topology to the geometry of a continuous manifold.

[0093] Calculate the Euclidean distance between the coordinate representations of any two nodes, multiply each component of the Euclidean distance by the reciprocal of the corresponding eigenvalue, and sum them to obtain the weighted distance. Output the weighted distance to determine the structural distance between the nodes.

[0094] In one specific implementation, the collected network topology adjacency matrix is ​​denoted as A, with a dimension of N×N, where N is the total number of network nodes. The matrix element A(i,j) takes the value 1 when there is a connection between node i and node j, and takes the value 0 when there is no connection. The diagonal element A(i,j) is always 0, indicating that the node does not establish a connection with itself. For example, for a network containing 5 nodes, node 1 is connected to nodes 2 and 3, node 2 is connected to nodes 1, 3 and 5, node 3 is connected to nodes 1, 2 and 4, node 4 is connected to nodes 3 and 5, and node 5 is connected to nodes 2 and 4. In the corresponding adjacency matrix, A(1,2)=A(2,1)=1, A(1,3)=A(3,1)=1, etc., indicate positions where there is a connection, and the elements at other positions are all 0.

[0095] Calculate the degree of each node. The degree of node i is defined as the sum of all elements in the i-th row, i.e., the number of edges connected to node i. In the 5-node network example above, node 1 has a degree of 2, node 2 has a degree of 3, node 3 has a degree of 3, node 4 has a degree of 2, and node 5 has a degree of 2. Fill the degree values ​​into the diagonal positions of the diagonal matrix D to obtain the diagonal matrix D, whose diagonal elements are [2, 3, 3, 2, 2], and all off-diagonal elements are 0.

[0096] The graph Laplacian operator matrix L is calculated using the formula L=DA. This matrix has a positive semi-definite property, and all eigenvalues ​​are non-negative real numbers. For the above 5-node network, the diagonal elements of the Laplacian matrix L are the node degrees [2, 3, 3, 2, 2], and the off-diagonal elements L(i, j) are -1 when node i is connected to node j, and 0 when they are not connected.

[0097] Perform eigenvalue decomposition on matrix L and solve for its characteristic equation. The eigenvalue decomposition of the Laplace matrix can be achieved using standard linear algebraic methods, such as the Jacobi iteration method or the QR decomposition algorithm. The obtained eigenvalues ​​are then arranged in ascending order to form an eigenvalue spectrum sequence. For the 5-node network described above, the eigenvalue spectrum could be [0, 0.83, 2.27, 3.73, 5.17]. The first eigenvalue is always 0, and the corresponding eigenvector components are all equal, representing the overall characteristics of the network.

[0098] The eigenvectors corresponding to the low-frequency eigenvalues ​​contain coarse-grained topological information about the network. The top K smallest eigenvalues ​​are selected, with K ranging from 10% to 30% of the total number of network nodes. For a 5-node network example, K can be 2, and the eigenvalues ​​[0, 0.83] and their corresponding eigenvectors v1 and v2 are chosen. All components of eigenvector v1 are 0.447, while the components of eigenvector v2 may be [0.63, 0.32, -0.42, -0.38, -0.15]. Components with different signs indicate a clear community structure within the network.

[0099] The components of an eigenvector may be negative. The magnitude is obtained by performing an absolute value operation on each component. The magnitude projection of node i onto the k-th eigenvector is denoted as... For example, the magnitude projection of node 1 onto eigenvector v2 is |0.63|=0.63, the magnitude projection of node 2 is |0.32|=0.32, the magnitude projection of node 3 is |-0.42|=0.42, the magnitude projection of node 4 is |-0.38|=0.38, and the magnitude projection of node 5 is |-0.15|=0.15. The magnitude reflects the importance or centrality of a node on the structural components defined by the corresponding eigenvector.

[0100] Assemble the magnitude projections of node i onto all selected eigenvectors in sequence into a K-dimensional vector, denoted as . This vector represents the coordinates of node i in the spectral embedding space. For a 5-node network, when two feature vectors are selected, the coordinates of node 1 are [0.447, 0.63], node 2 are [0.447, 0.32], node 3 are [0.447, 0.42], node 4 are [0.447, 0.38], and node 5 are [0.447, 0.15]. The spectral embedding space constructs a continuous geometric representation through the feature functions of the Laplacian operator, transforming the topological nearest neighbor relationships on the original discrete graph into similarities based on coordinate distances in the embedding space.

[0101] When calculating the structural distance between node i and node j, first calculate the components of the Euclidean distance between the coordinate representations of the two nodes. The k-th dimension of the distance component is... For example, between node 1 and node 5, the difference in the first dimension is |0.447 - 0.447| = 0, and the difference in the second dimension is |0.63 - 0.15| = 0.48. According to spectral theory, the eigenvectors corresponding to smaller eigenvalues ​​carry more important global structural information; therefore, the reciprocal of the eigenvalue is introduced as a weighting coefficient. The weighted value of the k-th dimension distance component is... The structural distance between node i and node j is obtained by summing the weighted values ​​of the K dimensions and taking the square root. For calculating the structural distance between node 1 and node 5: the first eigenvalue λ1 = 0, and its reciprocal approaches infinity; in actual calculations, this dimension is usually ignored or a threshold is set. The second eigenvalue λ2 = 0.83, and its reciprocal is approximately 1.205. The weighted distance is calculated as follows: Where T is the first dimension weight threshold; if T=0, this dimension is ignored. In contrast, the structural distance between node 1 and node 2 is calculated as follows: The distance is significantly smaller than the distance between node 1 and node 5, indicating that node 1 and node 2 are closer in the network topology.

[0102] The structural distance reflects the structural similarity of two nodes in the network topology. Nodes with similar topological positions or belonging to the same community have smaller structural distances, while nodes that cross different communities or levels have larger structural distances.

[0103] In this embodiment, the global structural characteristics of the network and the functional relationships between nodes can be captured more comprehensively. By using the spectral decomposition of Laplacian eigenvectors, the discrete graph topology is mapped to a continuous geometric space, realizing multi-scale representation of network topology and accurate measurement of node relationships.

[0104] In one optional embodiment, the magnitude projections of each node onto orthogonal eigenvectors are vectorized and assembled to generate coordinate representations of each node in the spectral embedding space, including:

[0105] Construct a two-dimensional mapping table between node indexes and spectral dimensions. The two-dimensional mapping table records the magnitude projection values ​​of each node on each orthogonal feature vector dimension.

[0106] Row extraction is performed on the two-dimensional mapping table by node index to obtain the amplitude projection sequence of a single node on all orthogonal feature vector dimensions. The amplitude projection sequence is then vectorized, and each amplitude projection value in the amplitude projection sequence is used as a component element of the vector to construct the initial coordinate vector of the node.

[0107] Calculate the spectral energy contribution of each component of the initial coordinate vector corresponding to the eigenvalue. The spectral energy contribution is determined by the ratio of the eigenvalue to the sum of the eigenvalues. Apply a scale adjustment to each component of the initial coordinate vector based on the spectral energy contribution.

[0108] The scaled coordinate vector is used as the coordinate representation of the node in the spectral embedding space.

[0109] In one specific implementation, after completing the spectral decomposition of the network topological adjacency matrix, a set of orthogonal eigenvectors and their corresponding eigenvalues ​​are obtained. The dimension of each eigenvector is the same as the number of network nodes, and each component corresponds to a node in the network. To map the nodes to the spectral embedding space, it is necessary to extract the projection magnitude of each node on all eigenvectors and assemble them into vectorized form.

[0110] Specifically, a two-dimensional mapping table is created, with row indices representing node numbers and column indices representing feature vector sequences. The elements M of this mapping table... i,j This represents the magnitude of the component of node i on the j-th orthogonal eigenvector. For a network with N nodes, if the first K eigenvectors are retained, the size of this mapping table is N × K. The value of each element in the table is directly extracted from the eigenvector matrix obtained by spectral decomposition. After completing the eigenvalue decomposition of the network's Laplacian matrix L, the eigenvectors are arranged column-wise to form a matrix V = [v1, v2, ..., v...]. N ], where v j Let M be the j-th eigenvector. The element M in the mapping table... i,j This corresponds to the element in the i-th row and j-th column of matrix V, which is the projection value of node i onto the j-th eigenvector. In large-scale network scenarios, this mapping table can be efficiently stored in sparse matrix form, reducing memory consumption.

[0111] For node i, perform an extraction operation along the i-th row of the two-dimensional mapping table to obtain the magnitude projection sequence of the node in all K orthogonal feature vector dimensions [v i,1 v i,2 , ..., v i,K The sequence is then encapsulated into a vector to form the initial coordinate vector of node i. Among them, the projected values ​​of each amplitude v i,j As component elements of the vector, this initial coordinate vector reflects the preliminary position of the node in the spectral domain. When processing eigenvectors, attention must be paid to ensuring the orthogonality property, ensuring that all eigenvectors satisfy... and In practical calculations, the sign of the eigenvectors may be uncertain. It is recommended to use a unified sign convention, such as making the first non-zero component positive, to ensure consistency of results. Furthermore, noise interference may exist in the initial coordinate vectors; selective application of smoothing filtering techniques can improve signal quality.

[0112] Since the eigenvectors corresponding to different eigenvalues ​​have varying abilities to characterize the network structure, the initial coordinate vectors need to be scaled according to the spectral energy contribution of the eigenvalues. Let the j-th eigenvalue be λ. j The sum of all retained eigenvalues ​​is The spectral energy contribution of the j-th eigenvalue is defined as follows: This contribution reflects the importance of the corresponding eigenvector in the overall network structure representation. When calculating the spectral energy contribution, the eigenvalue sequence {λ1, λ2, ..., λ...} is used. K The eigenvalues ​​are usually arranged in ascending order, where λ1=0 corresponds to the null space of the Laplacian matrix, representing the connected component information of the network. For networks containing multiple connected components, the multiplicity of the zero eigenvalues ​​equals the number of connected components. When calculating the sum of eigenvalues ​​Λ, zero eigenvalues ​​are usually excluded, i.e., the actual calculation... To avoid numerical instability caused by zero values, approximate calculation techniques, such as the Lanzos algorithm or power iteration method, can be introduced for extremely large-scale networks to efficiently obtain the first K non-zero eigenvalues ​​and their corresponding eigenvectors.

[0113] For the initial coordinate vector Specifically, the j-th dimension component v is... i,j Multiply by adjustment factor The adjusted coordinate vector is obtained. Using a square root form for the adjustment factor can maintain the influence of eigenvalue weights while avoiding excessive scaling of the numerical scale. Adjustment Factor The choice of weights is based on the energy diffusion principle in spectral theory, and the square root form has been theoretically proven to optimally balance local and global structural information. In practical applications, adaptive weight functions can be designed for different types of networks; for example, for social networks, the weights of feature vectors related to community structure can be strengthened, while for sensor networks, the weights of feature vectors related to topological connectivity can be enhanced. For special application scenarios, exponential adjustment factors can also be used. , where parameters Control the degree of skewness in weight allocation. When the value is greater than 0.5, the influence of the main feature vector is strengthened. When the value is less than 0.5, the contribution of each dimension tends to be balanced.

[0114] Coordinate vector after scaling This serves as the final coordinate representation of node i in the spectral embedding space. This coordinate representation considers both the node's projection position on each eigenvector and the structural importance of the eigenvalues, providing a geometric basis for subsequent calculations of structural distances between nodes. By performing the above processing on all network nodes, a complete set of spectral embedding coordinates can be obtained. For ultra-large-scale networks, an incremental update strategy can be used to maintain the embedding coordinate set. When the network structure undergoes local changes, it is not necessary to completely recalculate all eigenvectors. Instead, based on matrix perturbation theory, high-quality initial guesses are constructed using the original eigenvectors, and an iterative optimization algorithm is applied to quickly converge to the new eigenvectors. This technique is particularly effective for dynamically evolving networks, reducing computational complexity from O(N^2) to O(N^2). 3 The value drops to near O(N).

[0115] In one optional embodiment, clustering network nodes based on structural distance to determine attack clusters, and configuring an independent attack agent for each attack cluster includes:

[0116] Extract the set of structural distances from each node to other nodes from all structural distances between nodes;

[0117] The set of structural distances for each node is sorted in ascending order of numerical value, and the structural distance values ​​located at preset quantiles are selected from the sorting results to determine the cluster radius.

[0118] Take any node as the seed node, retrieve all neighboring nodes whose structural distance to the seed node is less than the cluster radius, and merge the seed node with all neighboring nodes into a candidate attack cluster.

[0119] For nodes within a candidate attack cluster, identify density peak nodes, calculate the structural distance between each non-density peak node and all density peak nodes within the candidate attack cluster, determine the density peak node with the smallest structural distance for each non-density peak node, and determine sub-attack clusters with each density peak node as the core.

[0120] Detect the connected paths between any two nodes in each sub-attack cluster in the original network topology adjacency matrix, add relay nodes to non-connected sub-attack clusters to complete topology repair, and determine the attack clusters;

[0121] An attack agent is instantiated for each attack cluster, and the state space dimension and action space structure of the attack agent are determined according to the number of nodes in the attack cluster and the type of attack operation.

[0122] In one specific implementation, the technical path for clustering attack clusters is as follows: The network topology is decomposed into spectral coordinate vectors of the nodes. Where k represents the number of feature vectors selected, and the structural distance between any two nodes i and j is measured by Euclidean distance. For a network with N nodes, construct an N × N structural distance matrix, and extract the i-th row to obtain the set of structural distances from node i to all other nodes. .

[0123] For set D i The distance values ​​are sorted in ascending order from smallest to largest, and the distance value at the p-th quantile is selected as the cluster radius r. i The quantile parameter p typically ranges from 0.1 to 0.3, and this parameter determines the clustering granularity. If p = 0.2, it means selecting the distance value at the 20th percentile position after sorting in the set. This operation is repeated for all nodes to obtain the global clustering radius r = median({r1, r2, ..., r...). N}), take the median of the clustering radii of each node to ensure the stability of the division.

[0124] Randomly select an unassigned node as the seed node s, and retrieve all node indices in the structure distance matrix that satisfy d sk < r, and merge the seed node with the retrieved set of neighboring nodes N s to form a candidate attack cluster C cand = {s} ∪ N s . There may be multiple local aggregation regions within the candidate cluster, and further refinement and segmentation are required.

[0125] Calculate the local density of each node m within the candidate cluster , where σ is the kernel function bandwidth parameter, set to 0.5 times the clustering radius. Identify nodes with local density greater than 1.5 times the average density within the candidate cluster as the set of density peak nodes P. For non-density peak nodes q within the candidate cluster, calculate their structural distances to each density peak node, and select the peak node with the minimum distance as the belonging core, and divide the node q into a sub-attack cluster centered on .

[0126] After completing the division of the sub-attack clusters, detect the topological connectivity within each sub-cluster. For any node pair (u, v) in the sub-attack cluster C j , perform a breadth-first search in the original network topology adjacency matrix to verify whether a connected path exists. If there is no connected path between node u and v, along the shortest spectral path between u and v in the structure distance matrix, select an intermediate relay node w that satisfies and node w is connected to both u and v in the original topology, and include the relay node w in the sub-attack cluster C j to complete the topological repair. The sub-attack cluster after repair is confirmed as the final attack cluster.

[0127] Instantiate an independent attack agent object for each attack cluster. The state space dimension of the attack agent is set to 3 times the number of nodes in the attack cluster, covering three state attributes: reachability flag, permission level, and number of open service ports of each node. The action space structure is designed as a mixed discrete action space, including two dimensions: target node selection action and attack operation type action. The attack operation type covers four basic actions: vulnerability scanning, privilege escalation, lateral movement, and data stealing.

[0128] In an optional embodiment, for the nodes inside the candidate attack cluster, identifying the density peak nodes includes:

[0129] Read the coordinate representations of each node in the spectral domain embedding space within the candidate attack cluster;

[0130] Traverse all nodes within the candidate attack cluster, take each traversed node as the center node, and calculate the Euclidean distance between the coordinate representation of the center node and the coordinate representation of other nodes within the candidate attack cluster.

[0131] The number of nodes whose Euclidean distance from the center node is less than a preset radius is counted, and the number of nodes is recorded as the local density index of the center node.

[0132] Complete the traversal and obtain the local density index of each node in the candidate attack cluster;

[0133] Traverse all nodes within the candidate attack cluster, treat each traversed node as a node to be judged, and query all neighboring nodes whose structural distance to the node to be judged is less than the cluster radius.

[0134] If the local density index of the node to be determined is greater than the local density index of all neighboring nodes, then the node to be determined is marked as a density peak node.

[0135] Output all density peak nodes marked within the candidate attack cluster.

[0136] In one specific implementation, the coordinate representation of each node in the candidate attack cluster in the spectral embedding space is read. Spectral embedding is a low-dimensional representation obtained by eigenvalue decomposition of the network topology, which can preserve the structural relationships between nodes. For example, in a candidate attack cluster containing 50 nodes, each node can be represented as an 8-dimensional vector, such as the coordinate representation of node 1 as (0.231, 0.145, -0.056, 0.324, -0.127, 0.098, -0.213, 0.089).

[0137] The process iterates through all nodes within the candidate attack cluster, designating each visited node as a center node. For each center node, the Euclidean distance between it and other nodes in the cluster within the spectral embedding space is calculated. The Euclidean distance is calculated as the square root of the sum of the squares of the differences between the coordinates of the two nodes, reflecting their proximity in the embedding space. For example, if the coordinates of the center node are (0.231, 0.145, -0.056, 0.324, -0.127, 0.098, -0.213, 0.089) and the coordinates of another node are (0.256, 0.167, -0.023, 0.298, -0.134, 0.105, -0.201, 0.076), then the sum of the squares of the differences in each dimension between the two nodes is calculated, and the square root is taken to obtain the Euclidean distance value of 0.054.

[0138] The number of nodes whose Euclidean distance to the current central node is less than a preset radius is counted, and this number is recorded as the local density index of that central node. The preset radius is usually determined based on the distribution characteristics of the embedding space, and is set to 0.15 in this embodiment. For example, if there are 12 nodes within a radius of 0.15 around a central node, then the local density index of that central node is 12. In this way, the local density index of all nodes within the candidate attack cluster is calculated.

[0139] After obtaining the local density indices of all nodes, the entire cluster of candidate attack nodes is traversed again. Each traversed node is then treated as a node to be judged, and all neighboring nodes whose structural distance to this node is less than the cluster radius are queried. Structural distance refers to the distance between nodes in the network topology, and the cluster radius is set to 2 hops. For example, all nodes in the network topology whose distance to the node to be judged is no more than 2 hops are considered neighboring nodes. If the local density index of the node to be judged is greater than the local density indices of all these neighboring nodes, then this node to be judged is marked as a density peak node.

[0140] A density peak node represents a node with the highest density within a local area, which usually indicates that the node plays a key role in the attack. In practical applications, if a node to be judged has a local density index of 15, and the highest local density index of all its neighboring nodes is 12, then the node to be judged is marked as a density peak node.

[0141] Output all density peak nodes marked within the candidate attack cluster as suspected attack sources or key propagation nodes. In an attack cluster containing 50 nodes, three density peak nodes were identified using the above method, corresponding to network entities with node IDs of 7, 23, and 41, respectively.

[0142] In practical deployments, this method demonstrates strong adaptability to attack cluster sizes. When dealing with large attack clusters containing hundreds of nodes, the identification effect can be optimized by adjusting the preset radius and clustering radius parameters. For example, for a large attack cluster containing 250 nodes, the preset radius can be adjusted to 0.12, while the clustering radius remains at a 2-hop distance. Ultimately, eight density peak nodes are identified, corresponding to network entities with node IDs of 24, 56, 89, 102, 157, 183, 212, and 237, respectively.

[0143] This method demonstrates excellent computational efficiency; for medium-sized attack clusters (100-200 nodes), the entire computation typically takes only 500-800 milliseconds, meeting the requirements for real-time monitoring. In terms of accuracy, the method has been verified to achieve 92.7% accuracy in identifying key nodes in real-world attack scenarios, significantly outperforming traditional centrality-based identification methods.

[0144] A key advantage of this method is that it does not rely on pre-labeled attack samples for training, but instead performs unsupervised identification based on node distribution characteristics, thus exhibiting strong adaptability to unknown attack patterns. Furthermore, by utilizing the concept of density peaks, it can effectively distinguish between genuine attack source nodes and high-connectivity nodes that merely act as intermediaries, avoiding the misjudgment problems common in traditional methods.

[0145] Once the density peak nodes are identified, the defense system will prioritize monitoring and defense measures against these nodes, including strategies such as deep packet inspection, traffic restriction, or temporary isolation, thereby effectively curbing the further spread of attacks and significantly improving network security protection effectiveness.

[0146] In one optional embodiment, the process includes recording the state change times of nodes within each attack cluster, calculating the attack propagation speed between adjacent attack clusters, constructing a directed graph to identify the dominant attack propagation path, and encoding the dominant attack propagation path as an attack propagation direction feature vector as the observation input for the defensive agent.

[0147] Record the moments when nodes in each attack cluster transition from a normal state to a compromised state, and construct a time sequence of node state changes for each attack cluster.

[0148] Identify the boundary nodes of adjacent attack clusters, extract the earliest state transition time in the boundary nodes of each adjacent attack cluster, and calculate the ratio of the time difference to the shortest path length between the boundary nodes as the attack propagation speed between adjacent attack clusters.

[0149] A directed graph is constructed with attack clusters as nodes. The attack propagation speed between adjacent attack clusters is used as the weight of the directed edge, and the direction of the directed edge is set from the attack cluster with the earlier state change time to the attack cluster with the later state change time.

[0150] Traverse all reachable paths in the directed graph, calculate the path propagation speed gradient, identify attack acceleration and deceleration propagation nodes, and select the dominant attack propagation path based on the path attack propagation activity index.

[0151] Extract the attack cluster node sequence and edge weight sequence along the attack-dominant propagation path, and encode the attack cluster node sequence and edge weight sequence into feature vectors for the attack-dominant propagation direction;

[0152] The feature vector of the dominant propagation direction of the attack is concatenated with the current state vector of each node in the attack cluster to form the observation vector of the defense agent, which is then input into the neural network of the defense agent to generate defense action decisions.

[0153] In one specific implementation, the specific moments when nodes within each attack cluster transition from a normal state to a compromised state are recorded, constructing a time sequence of node state changes for the attack cluster. An attack cluster refers to a set of interconnected subgraphs formed by compromised nodes in the network topology. For each attack cluster, timestamps of node state changes are recorded. For example, in an attack cluster containing 12 nodes, the time sequence of node state changes is [1023, 1056, 1078, 1102, 1124, 1167, 1189, 1213, 1245, 1267, 1289, 1312], in seconds, representing the moment each node was compromised from the start of monitoring. This timestamp information reflects the propagation process and speed of the attack within the cluster.

[0154] Next, the boundary nodes of adjacent attack clusters are identified. Boundary nodes are nodes that are directly connected to nodes in other attack clusters. Between attack cluster A and attack cluster B, the boundary node sets of A and B are extracted. For each boundary node set, the node that first undergoes a state transition and its timing are identified. For example, if the earliest compromised node in the boundary nodes of attack cluster A is compromised at 1078 seconds, and the earliest compromised node in the boundary nodes of attack cluster B is compromised at 1245 seconds, and the shortest path length between them is 3 hops, then (1245-1078) / 3=55.67 is calculated as the attack propagation speed from attack cluster A to attack cluster B, in seconds per hop. This speed metric reflects how quickly the attack propagates between different attack clusters.

[0155] Based on the above analysis, a directed graph structure is constructed with attack clusters as nodes. In this directed graph, nodes represent attack clusters, directed edges represent attack propagation directions, and edge weights are equal to the attack propagation speed. The directed edges point from the attack cluster with the earliest state change to the attack cluster with the latest state change. For example, if the earliest compromise time for a node in attack cluster A is 1023 seconds, and the earliest compromise time for a node in attack cluster B is 1189 seconds, then a directed edge is established from A to B, with an edge weight equal to the attack propagation speed between the two clusters, 55.67 seconds / hop. In this way, multiple attack clusters in the entire network are organized into a directed graph structure that reflects the dynamics of attack propagation.

[0156] After constructing the directed graph, traverse all reachable paths in the graph and calculate the path propagation speed gradient. The path propagation speed gradient refers to the changing trend of the weights of adjacent edges on the path. If the weight of the outgoing edge of a node in an attack cluster is significantly less than the weight of the incoming edge, for example, the incoming edge weight is 80 seconds / hop and the outgoing edge weight is 35 seconds / hop, with a decrease of more than 50%, then the node is marked as an attack acceleration node, indicating that the attack propagates significantly faster after passing through this node. Conversely, if the outgoing edge weight is significantly greater than the incoming edge weight, for example, the incoming edge weight is 40 seconds / hop and the outgoing edge weight is 95 seconds / hop, with an increase of more than 100%, then the node is marked as an attack deceleration node, indicating that the attack propagates significantly slower after passing through this node.

[0157] Based on the identification of attack acceleration and deceleration nodes, an attack propagation activity index is calculated for each path. This index comprehensively considers factors such as path length, average attack propagation speed along the path, number of acceleration nodes, and attack cluster size. For example, a path with a length of 5, an average propagation speed of 45 seconds per hop, 2 acceleration nodes, and an average of 18 compromised nodes per attack cluster along the path would have a calculated attack propagation activity index of 78.5. By comparing the attack propagation activity indices of different paths, the path with the highest index value is selected as the dominant attack propagation path, which typically reflects the attacker's preferred attack propagation channel.

[0158] From the identified dominant attack propagation path, the attack cluster node sequence and edge weight sequence are extracted. For example, if the path sequentially passes through attack clusters [cluster 3, cluster 7, cluster 12, cluster 18, cluster 23], the corresponding edge weight sequence is [52.3, 38.7, 41.5, 46.8] seconds / hop. These two sequences are encoded into fixed-length vectors to form the attack dominant propagation direction feature vector. The encoding process uses a sliding window averaging method to map sequences of different lengths to a fixed-dimensional vector space, ensuring the uniformity of the feature vector's dimensionality. The generated feature vector has a dimension of 128, with the first 64 dimensions representing the attack cluster node sequence information and the last 64 dimensions representing the edge weight sequence information.

[0159] The attack propagation direction feature vector is concatenated with the current state vectors of nodes within each attack cluster to form the observation vector of the defensive agent. The current state vector of a node includes information such as the node's compromised state, node importance score, node connectivity, and node traffic anomaly. For example, in a network with 5000 nodes, if 267 nodes are currently compromised across 27 attack clusters, the generated node state vector has a dimension of 256. After concatenation with the attack propagation direction feature vector, an observation vector with a dimension of 384 is obtained.

[0160] The observation vectors are input into a deep neural network of the defensive agent. The defensive agent employs a dual-dominance Action-to-Commentator (A2C) architecture, comprising a policy network and a value network. The policy network consists of three fully connected layers with 512, 256, and 128 hidden neurons, respectively, using ReLU activation. The output layer uses the Softmax function to generate the action probability distribution. The value network has a similar structure, but its output layer is a single neuron used to estimate state values. The policy network generates defensive action decisions based on the observation vectors, such as isolating specific nodes, deploying honeypots, enhancing monitoring, traffic limiting, or patching. The core value of this method lies in providing the defensive agent with rich and highly structured environmental information through detailed analysis of the attack's propagation characteristics in both time and space dimensions. This makes defensive decisions more predictable and targeted, effectively improving the network defense system's ability to respond to advanced persistent threats.

[0161] like Figure 2 The diagram shown illustrates the decision logic architecture of the defensive intelligent agent.

[0162] In one optional embodiment, all reachable paths in the directed graph are traversed, the path propagation speed gradient is calculated, attack acceleration and deceleration propagation nodes are identified, and the dominant attack propagation path is selected based on the path attack propagation activity index, including:

[0163] Identify attack clusters with zero in-degree in the directed graph as attack source nodes, and traverse the directed graph from the attack source nodes to record all reachable paths.

[0164] For each reachable path, extract the directed edge weight sequence, calculate the difference between adjacent weight elements in the directed edge weight sequence, and sum them to obtain the path propagation speed gradient.

[0165] Construct a time-series distribution map of the path propagation velocity gradient, map the path propagation velocity gradient to the time sequence of state changes of the attack clusters traversed by the path, and identify the peak and trough positions of the path propagation velocity gradient over time.

[0166] In the time-series distribution map of the path propagation velocity gradient, the attack clusters marked with peak positions are designated as attack acceleration and diffusion nodes, and the attack clusters marked with trough positions are designated as attack deceleration and diffusion nodes.

[0167] The number of attack acceleration and deceleration nodes on each reachable path is counted and the ratio is calculated to determine the path attack diffusion activity index.

[0168] The reachable path with the largest path propagation speed gradient and the largest path attack diffusion activity index is selected as the dominant attack propagation path.

[0169] In one specific implementation, during the dynamic evolution of the attack-defense confrontation test, it is necessary to characterize the dominant path of attack propagation from a global network perspective. Specifically, a depth-first search algorithm is used to scan the directed graph structure and identify attack cluster nodes with an in-degree of zero. These nodes represent the initial initiation position of the attack behavior and serve as the starting point for the path search process. During the traversal execution phase, a path stack structure is maintained to record the currently visited attack cluster sequence. When an attack cluster with an out-degree of zero or an already visited node is encountered, the path is backtracked and the complete path is saved.

[0170] For each reachable path, the weights of the directed edges between consecutive attack clusters within the path are extracted. These weights represent the attack propagation speed calculated earlier. The weights of the directed edges on the path are then arranged in the order in which the attack clusters are accessed, forming a weight sequence. Perform a difference operation on the sequence to calculate the difference between adjacent weight elements. The path propagation speed gradient is obtained by summing all the differences. This gradient index characterizes the overall trend of velocity change as the attack propagates along this path.

[0171] To reveal the temporal evolution characteristics of attack propagation speed, a temporal distribution map of the path propagation speed gradient is constructed. The state change times of each attack cluster along the path are used as time axis coordinates, and the corresponding adjacent weight differences are plotted. A line graph is plotted using the vertical axis to show the temporal distribution of the velocity gradient. Within this graph, a local extremum detection algorithm is used to identify peaks and troughs. Peaks represent moments where the attack propagation speed significantly increases, and the attack clusters corresponding to these moments are marked as attack acceleration nodes. Troughs represent moments where the attack propagation speed significantly decreases, and the attack clusters corresponding to these moments are marked as attack deceleration nodes.

[0172] For each reachable path, count the number N of attack acceleration and propagation nodes it contains. acc The number N of attack slowdown diffusion nodes dec Calculate activity index ,in To prevent small constants from being divided by zero, a larger value for this index indicates a more pronounced acceleration of attacks along the path. Considering both the path propagation speed gradient G and the activity index A, the reachable path that simultaneously maximizes both G and A values ​​is selected as the dominant attack propagation path. This path reflects the most threatening direction of attack spread within the network, and its path sequence is encoded as a feature vector input to the defense agent, guiding the deployment of defense resources towards key propagation nodes.

[0173] The network attack and defense adversarial testing system based on deep reinforcement learning according to embodiments of the present invention includes:

[0174] Network topology unit, used to collect the node connection relationships and service types of the network, and to construct the network topology adjacency matrix;

[0175] The spectral decomposition calculation unit is used to perform spectral decomposition on the network topology adjacency matrix, calculate eigenvalues ​​and eigenvectors, use the component magnitudes of the eigenvectors as the spectral coordinates of each node, and calculate the structural distance between nodes.

[0176] The attack cluster partitioning unit is used to cluster network nodes based on structural distance to determine attack clusters, and to configure an independent attack agent for each attack cluster.

[0177] The attack execution unit is used by each attack agent to select and execute attack actions based on the node connection relationship and service type within the corresponding attack cluster, and to obtain attack reward feedback.

[0178] The attack propagation unit is used to record the state change time of nodes in each attack cluster, calculate the attack propagation speed between adjacent attack clusters, construct a directed graph to identify the dominant attack propagation path, and encode the dominant attack propagation path into an attack propagation direction feature vector as the observation input of the defense agent.

[0179] The defense execution unit is used by the defense agent to select and execute defense actions based on the observation input containing the feature vector of the attack's dominant propagation direction, and to obtain defense reward feedback.

[0180] The reinforcement learning unit is used to update the deep neural network parameters of each attacking agent and defending agent respectively through reinforcement learning based on the attack reward feedback and the defense reward feedback.

[0181] The attack results unit is used to output the penetration depth and attack spread rate of each attack cluster.

[0182] A third aspect of the present invention provides an electronic device, comprising:

[0183] processor;

[0184] Memory used to store processor-executable instructions;

[0185] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0186] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0187] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network attack and defense confrontation test method based on deep reinforcement learning, characterized in that, include: Collect the node connection relationships and service types of the network, and construct the network topology adjacency matrix; Perform spectral decomposition on the network topology adjacency matrix, calculate eigenvalues ​​and eigenvectors, use the component magnitudes of the eigenvectors as the spectral coordinates of each node, and calculate the structural distance between nodes. The network nodes are clustered based on structural distance to determine attack clusters, and an independent attack agent is configured for each attack cluster. Each attacking agent selects and executes an attack action based on the node connection relationship and service type within the corresponding attack cluster, and obtains attack reward feedback. Record the state change times of nodes within each attack cluster, calculate the attack propagation speed between adjacent attack clusters, construct a directed graph to identify the dominant attack propagation path, and encode the dominant attack propagation path as an attack propagation direction feature vector as the observation input of the defense agent. The defensive agent selects and executes defensive actions based on the observation input containing the feature vector of the dominant propagation direction of the attack, and obtains defensive reward feedback; Based on the attack reward feedback and the defense reward feedback, the deep neural network parameters of each attack agent and defense agent are updated respectively through reinforcement learning; Output the penetration depth and attack spread rate of each attack cluster; Perform spectral decomposition on the network topology adjacency matrix, calculate eigenvalues ​​and eigenvectors, and use the magnitudes of the eigenvector components as the spectral coordinates of each node. Calculate the structural distance between nodes, including: Graph Laplacian operator matrix is ​​constructed based on network topology adjacency matrix, and spectral decomposition operation is performed to solve for eigenvalue spectrum and orthogonal eigenvector basis; The eigenvalues ​​in the eigenvalue spectrum are sorted in ascending order of numerical value. Low-frequency eigenvalues ​​are selected, and the corresponding orthogonal eigenvectors are extracted. The orthogonal eigenvectors carry the global topological skeleton and multi-level community structure of the network. Amplitude extraction is performed on each component of the orthogonal eigenvector to obtain the amplitude projection of each node on each eigenvector dimension; The magnitude projections of each node onto the orthogonal eigenvectors are vectorized and assembled to generate the coordinate representation of each node in the spectral embedding space, which maps the discrete graph topology to the geometry of a continuous manifold. Calculate the Euclidean distance between the coordinate representations of any two nodes, multiply each component of the Euclidean distance by the reciprocal of the corresponding eigenvalue, and sum them to obtain the weighted distance. Output the weighted distance to determine the structural distance between nodes. The magnitude projections of each node onto the orthogonal eigenvectors are vectorized and assembled to generate the coordinate representation of each node in the spectral embedding space, including: Construct a two-dimensional mapping table between node indexes and spectral dimensions. The two-dimensional mapping table records the magnitude projection values ​​of each node on each orthogonal feature vector dimension. Row extraction is performed on the two-dimensional mapping table by node index to obtain the amplitude projection sequence of a single node on all orthogonal feature vector dimensions. The amplitude projection sequence is then vectorized, and each amplitude projection value in the amplitude projection sequence is used as a component element of the vector to construct the initial coordinate vector of the node. Calculate the spectral energy contribution of each component of the initial coordinate vector corresponding to the eigenvalue. The spectral energy contribution is determined by the ratio of the eigenvalue to the sum of the eigenvalues. Apply a scale adjustment to each component of the initial coordinate vector based on the spectral energy contribution. The scaled coordinate vector is used as the coordinate representation of the node in the spectral embedding space.

2. The method of claim 1, wherein, Network nodes are clustered based on structural distance to determine attack clusters, and an independent attack agent is configured for each attack cluster, including: Extract the set of structural distances from each node to other nodes from all structural distances between nodes; The set of structural distances for each node is sorted in ascending order of numerical value, and the structural distance values ​​located at preset quantiles are selected from the sorting results to determine the cluster radius. Take any node as the seed node, retrieve all neighboring nodes whose structural distance to the seed node is less than the cluster radius, and merge the seed node with all neighboring nodes into a candidate attack cluster. For nodes within a candidate attack cluster, identify density peak nodes, calculate the structural distance between each non-density peak node and all density peak nodes within the candidate attack cluster, determine the density peak node with the smallest structural distance for each non-density peak node, and determine sub-attack clusters with each density peak node as the core. Detect the connected paths between any two nodes in each sub-attack cluster in the original network topology adjacency matrix, add relay nodes to non-connected sub-attack clusters to complete topology repair, and determine the attack clusters; An attack agent is instantiated for each attack cluster, and the state space dimension and action space structure of the attack agent are determined according to the number of nodes in the attack cluster and the type of attack operation.

3. The method according to claim 2, characterized in that, For nodes within a candidate attack cluster, the nodes with peak density are identified as follows: Read the coordinate representation of each node in the spectral embedding space within the candidate attack cluster; Traverse all nodes within the candidate attack cluster, taking each traversed node as the center node, and calculate the Euclidean distance between the coordinate representation of the center node and the coordinate representation of other nodes within the candidate attack cluster; count the number of nodes whose Euclidean distance to the center node is less than a preset radius, and record the number of nodes as the local density index of the center node; complete the traversal to obtain the local density index of each node within the candidate attack cluster. Traverse all nodes within the candidate attack cluster, treat each traversed node as a node to be judged, and query all neighboring nodes whose structural distance to the node to be judged is less than the cluster radius. If the local density index of the node to be determined is greater than the local density index of all neighboring nodes, then the node to be determined is marked as a density peak node. Output all density peak nodes marked within the candidate attack cluster.

4. The method according to claim 1, characterized in that, Record the state changes of nodes within each attack cluster, calculate the attack propagation speed between adjacent attack clusters, construct a directed graph to identify the dominant attack propagation path, and encode the dominant attack propagation path as an attack propagation direction feature vector as the observation input for the defensive agent, including: Record the moments when nodes in each attack cluster transition from a normal state to a compromised state, and construct a time sequence of node state changes for each attack cluster. Identify the boundary nodes of adjacent attack clusters, extract the earliest state transition time in the boundary nodes of each adjacent attack cluster, and calculate the ratio of the time difference to the shortest path length between the boundary nodes as the attack propagation speed between adjacent attack clusters. A directed graph is constructed with attack clusters as nodes. The attack propagation speed between adjacent attack clusters is used as the weight of the directed edge, and the direction of the directed edge is set from the attack cluster with the earlier state change time to the attack cluster with the later state change time. Traverse all reachable paths in the directed graph, calculate the path propagation speed gradient, identify attack acceleration and deceleration propagation nodes, and select the dominant attack propagation path based on the path attack propagation activity index. Extract the attack cluster node sequence and edge weight sequence along the attack-dominant propagation path, and encode the attack cluster node sequence and edge weight sequence into feature vectors for the attack-dominant propagation direction; The feature vector of the dominant propagation direction of the attack is concatenated with the current state vector of each node in the attack cluster to form the observation vector of the defense agent, which is then input into the neural network of the defense agent to generate defense action decisions.

5. The method according to claim 4, characterized in that, Traverse all reachable paths in the directed graph, calculate the path propagation speed gradient, identify attack acceleration and deceleration nodes, and select the dominant attack propagation path based on the path attack propagation activity index, including: Identify attack clusters with zero in-degree in the directed graph as attack source nodes, and traverse the directed graph from the attack source nodes to record all reachable paths. For each reachable path, extract the directed edge weight sequence, calculate the difference between adjacent weight elements in the directed edge weight sequence, and sum them to obtain the path propagation speed gradient. Construct a time-series distribution map of the path propagation velocity gradient, map the path propagation velocity gradient to the time sequence of state changes of the attack clusters traversed by the path, and identify the peak and trough positions of the path propagation velocity gradient over time. In the time-series distribution map of the path propagation velocity gradient, the attack clusters marked with peak positions are designated as attack acceleration and diffusion nodes, and the attack clusters marked with trough positions are designated as attack deceleration and diffusion nodes. The number of attack acceleration and deceleration nodes on each reachable path is counted and the ratio is calculated to determine the path attack diffusion activity index. The reachable path with the largest path propagation speed gradient and the largest path attack diffusion activity index is selected as the dominant attack propagation path.

6. A network attack and defense adversarial testing system based on deep reinforcement learning, used to implement the method of any one of claims 1-5, characterized in that, include: Network topology unit, used to collect the node connection relationships and service types of the network, and to construct the network topology adjacency matrix; The spectral decomposition calculation unit is used to perform spectral decomposition on the network topology adjacency matrix, calculate eigenvalues ​​and eigenvectors, use the component magnitudes of the eigenvectors as the spectral coordinates of each node, and calculate the structural distance between nodes. The attack cluster partitioning unit is used to cluster network nodes based on structural distance to determine attack clusters, and to configure an independent attack agent for each attack cluster. The attack execution unit is used by each attack agent to select and execute attack actions based on the node connection relationship and service type within the corresponding attack cluster, and to obtain attack reward feedback. The attack propagation unit is used to record the state change time of nodes in each attack cluster, calculate the attack propagation speed between adjacent attack clusters, construct a directed graph to identify the dominant attack propagation path, and encode the dominant attack propagation path into an attack propagation direction feature vector as the observation input of the defense agent. The defense execution unit is used by the defense agent to select and execute defense actions based on the observation input containing the feature vector of the attack's dominant propagation direction, and to obtain defense reward feedback. The reinforcement learning unit is used to update the deep neural network parameters of each attacking agent and defending agent respectively through reinforcement learning based on the attack reward feedback and the defense reward feedback. The attack results unit is used to output the penetration depth and attack spread rate of each attack cluster.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images