Attack fault tree security risk assessment system based on deep reinforcement learning

Abstract

The invention provides an attack fault tree security risk assessment system based on deep reinforcement learning, and relates to the technical field of information security. The system specifically comprises a network asset data fingerprint scanner module, a deep reinforcement learning network model generation and training module and an automatic penetration test tool set module. The invention aims to determine the optimal attack path from the system attack graph by using deep reinforcement learning. By combining the screened attack paths with various resources and time costs encountered in the actual penetration test, the failure rates of various security risks in the attack fault tree, the success probabilities of various attacks and other indexes can be determined. And finally, establishing an attack fault tree of a power system scene in the embodiment, comparing the attack lethality and the detection capability level of each attack path in a game phase, and qualitatively and quantitatively analyzing the final security capability level of the target system and the damage degree caused by attacks.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to an attack fault tree security risk assessment system based on deep reinforcement learning. Background Art

[0002] To defend against various cyberattacks targeting industrial control systems, many security companies and teams use penetration testing to explore system vulnerabilities. While penetration testing has mature tools and frameworks, some systems and application scenarios still rely on specialized expertise for targeted attacks. As cyberattacks become increasingly complex and dynamic, the need for highly realistic cyberattack simulations is becoming increasingly urgent. These simulations are crucial for predicting evolving threats and developing effective defense strategies to protect complex systems. Traditional system security approaches, which rely on predefined rules and signature mechanisms, are often insufficient to address these evolving threats.

[0003] Compared to traditional attack graph-based analysis techniques, the Attack Fault Tree (AFT) method plays a more significant role in analyzing potential risk events associated with system failures. By applying AFT analysis, security practitioners can gain a deeper understanding of the relationship between attack subtrees and the fundamental component failures in the main fault tree. To assess the probability of failure caused by cyberattacks, it is essential to accurately enumerate the possible attack paths against the target system network. However, traditional AFTs, manually constructed based on expert experience, fail to fully capture the various attack paths or methods in real-world scenarios. While numerous companies and research institutions have conducted extensive research on the security and reliability of industrial control systems, current research in this field often focuses on a single aspect: either security analysis or reliability analysis in industrial control system design, without effectively integrating the two. Furthermore, most existing AFT methodologies rely on professional experience or manual analysis rather than automated reasoning. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides an attack fault tree security risk assessment system based on deep reinforcement learning; it aims to propose an attack fault tree analysis framework based on deep reinforcement learning and Bayesian probability theory to optimize attack detection efficiency and characterize the impact of various attacks on the power system.

[0005] An attack fault tree security risk assessment system based on deep reinforcement learning includes: a network asset data fingerprint scanner module, a deep reinforcement learning network model generation and training module, and an automated penetration testing tool collection module.

[0006] The network asset data fingerprint scanner module uses scanner software to actively or passively scan the target system and its network to obtain raw network data, and then performs data cleaning, feature extraction, protocol behavior modeling and pattern matching analysis on the scan results to construct network fingerprint information of the target system and target application. In combination with the communication relationship, port mapping and routing response information between multiple hosts, the logical topology structure of the target network is reconstructed to clarify the connection relationship between each node, the core asset location and the security boundary division; and the CVE vulnerability information and network topology structure obtained in the scanning phase are used as input, and the attack graph generation tool MulVAL is used to generate an attack graph file containing the attack path. The attack graph is optimized and pruned in combination with the Bayesian prior probability to generate a Bayesian attack tree. After applying depth-first traversal to the Bayesian attack tree, a profit matrix of all attack paths from the starting point to the attack target is obtained, namely, a Q-value matrix.

[0007] The deep reinforcement learning network model generation and training module includes an agent, an environment, a state space, an attack action set, a replay buffer, a current Q-value network, and a target Q-value network;

[0008] The agent observes the environment to obtain current state information and then performs corresponding attack actions; the environment generates a new state according to the agent's execution of the action, and the attack benefit generated during the state update depends on the attack action used and is described using a Markov decision process; the transmission samples are processed by using experience replay, and the transmission samples obtained by the interaction between the agent and the environment are stored in the replay buffer at each moment; during the training phase, batch samples are randomly selected and the stochastic gradient descent algorithm is used to update the current Q-value network parameters, and after the training phase, the current Q-value network parameters are assigned to the target network, and then the network parameters are updated by minimizing the difference between the current Q-value and the target Q-value in combination with the loss function to generate the final deep reinforcement learning network model.

[0009] The automated penetration testing tool set module includes Metasploit, Auto-Pres, SSH-KeyFinder, Nikto, Hydra, sqlmap, and John the Ripper. By inputting the payoff matrix obtained from the network asset data fingerprint scanner module into a deep reinforcement learning network model, the optimal attack path corresponding to the predicted result is obtained. All attack actions included in the attack path are translated into commands that can be understood by the penetration testing tool. The penetration testing tool is then called to perform vulnerability exploitation, privilege escalation, and information collection operations on a real target system. The results of these operations are received as feedback to evaluate the success of the current attack and determine newly exposed assets or vulnerabilities in the target system, thereby determining the direction of the next stage of attack path selection, including whether to continue to penetrate the core assets of the target system, whether to move laterally to associated hosts, or whether to use newly obtained credentials to launch deeper attack behaviors.

[0010] The network asset data fingerprint scanner module is used to perform the following steps:

[0011] Step A1: Scan assets within the target network using active scanning technology to obtain first scanning data. Scanner software actively sends constructed, specific probe packets to hosts and ports in the target network. The specific probe packets are TCP SYN packets, UDP probe packets, ICMP echo request packets, HTTP request packets, or other protocol handshake packets to stimulate the target device to respond. The scanner then extracts key information from the response packets returned by the target host, including response characteristics of each network layer protocol, service version information, port status, packet retransmission delay, TTL, and window size, thereby obtaining network attributes and behavioral characteristics of the target asset. The active scanner software sends Internet Control Message Protocol (ICMP) and Address Resolution Protocol (ARP) to the network layer of the target network to detect the liveness of hosts in the target network, and sends Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) probe packets to the transport layer of the target network to identify the liveness status of hosts in the target network, open ports, service types and service versions corresponding to the ports, operating system types and versions, service response behavior characteristics, the presence and behavior patterns of firewalls or intrusion detection systems, and device fingerprinting information based on port return characteristics. Send the short network management protocol SNMP to the application layer to detect operating system and application service information.

[0012] Step A2: Scanning network traffic logs of terminal devices and network devices within the target network using a network traffic analysis method to obtain second scan data; wherein the information obtained from the terminal device scan includes device name, device type, device address, operating system, open ports, running services, service version, MAC address, manufacturer information, device response characteristics (e.g., TTL, window size), patch update status, host name, network interface information, security configuration status (e.g., firewall, antivirus software), account and login mechanism; and the information obtained from the network device scan includes one or more items of VLAN interface, routing information, hub information, and switch information.

[0013] The network traffic analysis method deploys detectors at network nodes to collect network traffic logs in devices, performs traffic data field analysis, analyzes the port field, flag field, window size, sequence number and other information in TCP response data packets to obtain port status, port corresponding service type and service behavior characteristics; obtains device operating system information based on the TTL field in the IP response data packet, and obtains the target host's IP address, subnet mask, gateway address and DNS server address, lease time, host name, vendor identifier and broadcast address by analyzing other fields in the Dynamic Host Configuration Protocol DHCP response data except the transaction ID and message type, thereby comprehensively characterizing the network configuration environment and operation characteristics of the target device.

[0014] Step A3: Extract fingerprint features from the first scan data and the second scan data, match them with existing asset fingerprint features in the fingerprint library, identify and obtain asset information within the target network as the first network asset information; then use a cyberspace search engine to scan the external network to obtain the second network asset information;

[0015] The second network asset information is specifically obtained by scanning external network assets through a cyberspace search engine to obtain the type and version of each network device, security device, server, terminal device, Internet of Things device, virtualization device, industrial control system device, wireless access device, and storage device connected to the external network, including the geographical or logical location, open ports and the corresponding service type and version, operating system type and kernel version, SSL / TLS certificate information, device manufacturer and model, MAC address, device fingerprint characteristics, uptime, service response characteristics, vulnerability information, public network reachability, and possible unauthorized access risk information. The cyberspace search engine is used to detect surviving hosts and scan surviving ports. By sending network requests and observing responses, the host is determined to be alive, and the open port number and port information on the host are obtained. The information obtained by the cyberspace search engine is organized and analyzed, including standardization of the obtained IP address, port number, service type, service version, TLS certificate information, response header characteristics, device fingerprint information, asset ownership determination, service fingerprint identification, geographic location mapping, threat label association, vulnerability matching analysis, and access control status judgment operations to obtain the second network asset information.

[0016] Step A4: Use the first network asset information and the second network asset information as the environment description of the target system and write a MulVAL input file. MulVAL uses the Datalog language as the model language. After converting the vulnerability scanning results of the vulnerability scanner in the first network asset information and the second network asset information, the configuration information and topology information of the network nodes, the network management policy provided by the network administrator, the device access control list, the port mapping rules, the routing policy, the user identity and permission allocation, the deployed security protection policy, the detected abnormal behavior pattern, and the network log and security audit data into fact inputs in the Datalog language, MulVAL's deduction engine performs logical reasoning and continuously expands the intermediate conditions that the attacker needs to meet to reach the target state, thereby automatically deducing possible attack paths and their dependencies, and finally generating a formalized attack tree structure.

[0017] The derivation process uses the attack target as the root node, starting from the attacker's initial privilege state, and working backwards along the system configuration and vulnerability exploitation logic to form a complete attack tree. Each node represents a step the attacker needs to complete the attack, and each path represents a feasible attack route. Finally, this attack tree structure is input into the Graphviz visualization tool to generate an attack graph that intuitively displays the attack process and path dependencies.

[0018] Step A5: After the attack graph is drawn, the initial information contained in the attack graph, including identified vulnerabilities, network topology, and the security configuration status of each host, is used as the starting node of the entire attack graph. A Bayesian attack tree model is constructed based on inference rules based on existing expert knowledge. The Bayesian attack tree model estimates the attack success probability for each target node. Each node in the attack graph has two types of attack success probabilities: intrinsic probability and cumulative probability. The intrinsic probability represents the inherent probability of successfully compromising the node if all required conditions are met, calculated using the CVSS base score. The cumulative probability, the final calculated probability, represents the overall probability of an attacker successfully reaching the target node and executing the attack within the given attack graph, derived from the Bayesian network. By applying forward or backward updates, the cumulative probability of attack success is recursively calculated for each node along each attack path from the starting node to the target node. Circular paths are removed from the attack graph, ultimately resulting in an optimized Bayesian attack tree.

[0019] Step A6: On the resulting Bayesian attack tree, all potential attack paths to the target node are enumerated using a depth-first traversal algorithm. The attack payoff and cumulative attack success probability of all nodes on each attack path are calculated. A simplified attack matrix is ​​then created by combining the attack payoff and cumulative probability. The first column of the simplified attack matrix represents the attack score of the starting node of each attack path, the middle columns represent the attack scores of each step in the attack process, and the last column represents the score of the target node. The simplified attack matrix is ​​provided as input to the deep reinforcement learning network.

[0020] The deep reinforcement learning network model generation and training module is used to build a deep reinforcement learning network model and train the deep reinforcement learning network model;

[0021] The construction of the deep reinforcement learning network model is as follows: the Bayesian attack tree is modeled as a sequential decision problem, which is regarded as a Markov decision process: M = (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the state transition matrix, R is the reward function, and γ≤1 is a regulation factor; the Markov decision process includes:

[0022] State space: The state space of the environment consists of all intermediate attack-fault trees generated after possible actions;

[0023] Action space: Given an intermediate attack-fault tree, the agent's goal is to increase the height of the Bayesian attack tree by injecting or connecting new nodes; therefore, the action space is n-discrete, where n represents the number of potential events;

[0024] Terminal: Once the target event occurs, the Markov decision process terminates.

[0025] Reward: The amount of money earned by executing each attack step, as measured by CVSS Base and Exploitability scores;

[0026] Because the action space is discrete, deep reinforcement learning is used to make decisions based on a Markov decision process. At each time step t, the agent stores the transmission samples obtained from interacting with the environment in a replay buffer. During each training phase, a batch of transmission samples is randomly selected and input into the deep reinforcement learning current Q-value network. The network parameters θ are updated using a stochastic gradient descent algorithm. The goal of the deep reinforcement learning network is to construct an algorithm that satisfies the Bellman optimality equation, as shown in the following equation:

[0027]

[0028] Where, Indicates that under the policy π, the agent is in state Take action The expected cumulative return that can be obtained, that is, the Q value, Indicates that the agent is in state Take action The immediate reward obtained, γ is the decay factor used to weigh the importance of future rewards, Indicates the next state Consider all possible actions , the maximum Q value that can be obtained, that is, the estimate of the optimal return Q value in the future, represents the current state at time step t, The current action taken at time step t, Indicates that the status Execute an action The next state reached after or Indicates the next state One of all possible actions taken in .

[0029] Training a deep reinforcement learning network model specifically includes the following steps:

[0030] Step B1: Calculate the attack benefit obtained by the attacker at time t:

[0031]

[0032] Where, It represents the cumulative discounted return starting from time step t, which is the sum of all rewards obtained from the current moment in the future, attenuated by the discount factor. represents the immediate reward obtained at time step t+1, γ is the decay factor, Indicates the degree of discount on the future reward of step k; It is the sum of all future rewards after being gradually discounted over time, that is, the discounted cumulative return.

[0033] By introducing the decay factor γ, at each time step t, the system is and attack actions Calculating immediate attack benefits and accumulate subsequent benefits in the form of discounts to form cumulative discount returns The return value is used to update the action value function , reflecting the long-term expected reward from taking a particular attack action in the current state. During training, the agent continuously updates its Q-value and network weights through a reinforcement learning algorithm, ultimately leading to convergence. During the testing or execution phase, the system selects the action with the highest Q-value in each state and constructs an attack path with the highest expected cumulative attack reward. This achieves a dynamic balance between short-term rewards and long-term effects, ultimately determining the optimal set of attack actions.

[0034] Step B2: By combining the current state s and the selected action α, the expected cumulative attack benefit Q value is updated according to the following formula:

[0035]

[0036] In the formula Indicates that under the policy π, the agent is in state Take action The expected cumulative return that can be obtained, that is, the Q value, E[•] mathematical expectation symbol, represents the average value of all possible reward sequences under the strategy π and the environment transition probability distribution, The discounted cumulative reward from time step t+1 to all future time steps, γ is the decay factor, Represents the immediate reward obtained at time step t+1+k. Indicates that the current state is s, that is, the current environmental state under the conditions. The current action is a, which is the current action performed by the agent. The policy π represents the probability distribution or behavior rule of the agent in each state.

[0037] make Represents the Q value output by the current value network of deep reinforcement learning, represents the parameters of the current value network, represents the Q value output by the deep reinforcement learning target value network, Represents the parameters of the target value network.

[0038] Step B3: Based on the Q value in step B2, use a greedy strategy to select an action Make Take the maximum value:

[0039]

[0040] In the formula At time step t, the agent chooses the optimal action, that is, the action that maximizes the action-value function in the current state. Is a mathematical operator, which means to find the function value The largest action a. Under a given policy π, the agent is in state Take action The expected cumulative return that can be obtained after represents the current state at time step t, and the policy π represents the probability distribution or behavior rule for the agent to choose an action in each state.

[0041] Step B4: Calculate the target Q value using the following formula:

[0042]

[0043] In the formula represents the target value used to update the Q-value network in the i-th iteration, r represents the immediate reward returned by the environment after the agent takes an action in the current state, γ is the decay factor, Indicates the next state In the example, the target Q value network parameters are used. Calculate all possible actions The Q value of the state is calculated and the maximum value is selected, which represents the maximum expected reward that can be obtained from the next state. Indicates the next state that the environment transitions to after executing the action in the current state. Indicates that the status All candidate actions that can be taken in are used to find the maximum Q value. Indicates the use of target network parameters Estimated action-value function. represents the parameter set of the target Q network in the i-th iteration.

[0044] By updating the parameters θ of the current value network in real time during each round of training, and copying the parameters θ to the deep reinforcement learning target Q value network after N rounds of iteration, the network parameters are adjusted by minimizing the mean square error between the current Q value and the target network Q value. The error function is expressed as:

[0045]

[0046] Where, represents the loss function of the deep reinforcement learning Q-value network in the i-th iteration, It represents the expectation of the experience sample of state s, action a, reward r, and next state s′. represents the target value used to update the Q-value network in the i-th iteration. Indicates the current Q value network parameters used The predicted Q value for state s and action a. Represents the parameter set of the current Q value network in the i-th iteration. Represents the squared difference between the predicted value and the target value.

[0047] The gradient descent calculation formula corresponding to the error function is as follows:

[0048]

[0049] Where, Represents the loss function About the current Q network parameters The gradient of , which is used for back propagation optimization of neural networks; It represents the expectation of the experience sample of state s, action a, reward r, and next state s′. represents the target value used to update the Q-value network in the i-th iteration. Indicates the current Q value network parameters used The predicted Q value for state s and action a. represents the parameter set of the current Q value network in the i-th iteration, Indicates the difference between the current sample predicted Q value and the target Q value, Indicates the current Q value network's parameters The gradient derivative of .

[0050] The automated penetration testing tool set module is used to perform the following steps:

[0051] Step C1: Build the penetration testing tool into a wrapper and provide a corresponding penetration testing case construction template. Based on the optimal attack path and optimal attack action set output by the deep reinforcement learning network model as the description input, the input is converted into an instruction sequence that can be executed by the penetration tool in the expected execution order. By running the instruction sequence and receiving the returned attack execution results, it is used to decide how to continue the given attack path.

[0052] Step C2: For the identified optimal attack path, the Bayesian attack tree is structurally simplified and merged to obtain an attack subtree model, including: path node labeling, redundant branch pruning, logic gate merging, path folding, indicator aggregation, and attack subgraph reconstruction. The fault tree analysis model (FTA) is constructed by introducing the system fault tree analysis method to characterize the system-level failure logic. Then, the attack subtree model is inserted into the safety-sensitive intermediate failure event node in the fault tree analysis model (FTA). The attack subtree is connected to the main function fault tree constructed based on the functional structure of the tested system through standard logic gates to form an integrated attack fault tree analysis model.

[0053] Step C3: Combined with expert experience or the failure probability of system-related components in public documents, the Monte Carlo statistical analysis method is used to calculate the target system's overall cumulative failure probability (CFR), mean time between failures (MTBF), mean time to repair (MTTR), and mean time between failures (MTBF), and conduct a quantitative assessment of system safety.

[0054] Step C4: For the estimation of the instantaneous failure rate of the fault event node caused by the information security attack in the system attack fault tree, the statistical time automaton method is used to simulate the finite automatic state machine of attackers with different capability levels and the system failure behavior caused by the attack event under different threat detection granularity conditions. The finite automatic state machine corresponding to the attack event is connected with the other basic fault event activity state machines through the logic gate of the parent node in the attack fault tree to obtain the overall statistical time automaton model of the system. The overall statistical time automaton model of the system is converted to construct a metalanguage model that can be run in the Uppaal SMC reliability analysis software, and simulation verification is carried out on the Uppaal SMC reliability analysis software to obtain the overall failure rate, mean time to failure, mean time to repair (MTTR), and mean time between failures (MTBF) of the target network information system under evaluation in various scenarios with different information security protection levels.

[0055] The beneficial effects of adopting the above technical solution are:

[0056] The present invention provides an attack fault tree security risk assessment system based on deep reinforcement learning. The present invention utilizes a deep reinforcement learning network and calculates attack reward scores based on Bayesian cumulative probability to improve the accuracy and effectiveness of attack path analysis. By automatically constructing an attack fault tree (AFT), a comprehensive trade-off between the security and reliability of the target system can be achieved. The AFT is modeled in combination with the statistical time automaton method of Uppaal SMC, making it possible to evaluate various accidental risks and malicious damage scenarios of the target system in a real environment. Over time, probability estimates of attack scenarios can be dynamically generated, providing real-time data support for subsequent management decisions. It can also calculate the expected cost, time, and losses caused by different adversaries, helping analysts better understand the potential impact of attacks. By comparing the attack costs in AS-IS and WHAT-IF scenarios, key protection measures can be identified and prioritized, thereby enhancing the security capabilities of the target system. Based on the results of quantitative statistical analysis, more targeted protection measures can be formulated to more effectively resist potential threat attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the functional modules of the attack fault tree security risk assessment system of the present invention;

[0058] Figure 2 Schematic diagram of the communication network topology of a power system;

[0059] Figure 3 Schematic diagram of a power system attack fault tree;

[0060] Figure 4 Attack the basic component failure diagram in the fault tree;

[0061] Figure 5 Schematic diagram of the basic attack steps in the attack fault tree. DETAILED DESCRIPTION

[0062] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0063] An attack fault tree security risk assessment system based on deep reinforcement learning. The functional modules in this embodiment are designed as follows: Figure 1 As shown, the system consists of three modules, including: network asset data fingerprint scanner module, deep reinforcement learning network model generation and training module, and automated penetration testing tool collection module.

[0064] The network asset data fingerprint scanner module uses scanner software such as NMAP or MASSCAN to actively or passively scan the target system and its network, obtains raw network data including host survival status, open ports, service type and version, operating system characteristics, response time, etc., and then performs data cleaning, feature extraction, protocol behavior modeling and pattern matching analysis on the scan results to construct network fingerprint information of the target system and target application, including service fingerprint, operating system fingerprint, device manufacturer and model identification, etc.; combines the communication relationship between multiple hosts, port mapping and routing response information, reconstructs the logical topology structure of the target network, clarifies the connection relationship between each node, the core asset location and the security boundary division; and combines the CVE vulnerability information and network topology structure obtained in the scanning phase as input, applies the attack graph generation tool MulVAL to generate an attack graph file containing the attack path, and optimizes and prunes the attack graph in combination with the Bayesian prior probability to generate a Bayesian attack tree. After applying depth-first traversal to the Bayesian attack tree, a profit matrix of all attack paths from the starting point to the attack target is obtained, namely the Q-value matrix.

[0065] The deep reinforcement learning network model generation and training module includes an agent, an environment, a state space, an attack action set, a replay buffer, a current Q-value network, and a target Q-value network;

[0066] The agent observes the environment to obtain current state information and then performs corresponding attack actions; the environment generates a new state according to the agent's execution of the action, and the attack benefit generated during the state update depends on the attack action used and is described using a Markov decision process; the Markov decision process can be replaced by a deep reinforcement learning network model; the transmission samples are processed by using experience replay, and the transmission samples obtained by the interaction between the agent and the environment are stored in the replay buffer at each moment; during the training phase, batch samples are randomly selected and the current Q-value network parameters are updated using the stochastic gradient descent algorithm, and after the training phase, the current Q-value network parameters are assigned to the target network, and then the network parameters are updated by minimizing the difference between the current Q-value and the target Q-value in combination with the loss function to generate the final deep reinforcement learning network model.

[0067] The automated penetration testing tool set module includes Metasploit, Auto-Pres, SSH-KeyFinder, Nikto, Hydra, sqlmap, and John the Ripper. By inputting the payoff matrix obtained from the network asset data fingerprint scanner module into a deep reinforcement learning network model, the optimal attack path corresponding to the predicted result is obtained. All attack actions included in the attack path are translated into commands that can be understood by the penetration testing tool. The penetration testing tool is then called to perform vulnerability exploitation, privilege escalation, and information collection operations on a real target system. The results of these operations are received as feedback to evaluate the success of the current attack and determine newly exposed assets or vulnerabilities in the target system, thereby determining the direction of the next stage of attack path selection, including whether to continue to penetrate the core assets of the target system, whether to move laterally to associated hosts, or whether to use newly obtained credentials to launch deeper attack behaviors.

[0068] The network asset data fingerprint scanner module is used to perform the following steps: In this embodiment, tools such as NMAP and MASSCAN are used as target system asset fingerprint scanners. The target system is scanned by setting various scanning options, and the MulVAL tool is used for knowledge reasoning to generate an attack graph. The specific implementation process is as follows.

[0069] Step A1: Use active scanning technology to scan the assets in the target network to obtain the first scanning data. Use NMAP or MASSCAN scanner software to actively send constructed specific probe data packets to the hosts and ports in the target network. The specific probe data packets are TCP SYN packets, UDP probe packets, ICMP echo request packets (Ping), HTTP request packets, or other protocol handshake packets to stimulate the target device to respond. The scanner then extracts key information from the response data packets returned by the target host, including the response characteristics of each network layer protocol (such as TCP three-way handshake response, ICMP type code), service version information, port status, packet retransmission delay, TTL (Time To Live), and window size, thereby obtaining the network attributes and behavioral characteristics of the target assets. The active scanner software sends Internet Control Message Protocol (ICMP) and Address Resolution Protocol (ARP) to the network layer of the target network to detect the liveness of the hosts in the target network, and sends Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) probe data packets to the transport layer of the target network to identify the liveness status of the hosts in the target network, open ports, the service type and service version corresponding to the port, the operating system type and version, and service response behavior characteristics (such as TCP The system can also detect the presence and behavior of firewalls or intrusion detection systems (e.g., window size, TTL value), device fingerprinting based on port return signatures, and send SNMP (Short Network Management Protocol) to the application layer to detect operating system and application service information.

[0070] Step A2: Scanning network traffic logs of terminal devices and network devices within the target network using a network traffic analysis method to obtain second scan data; wherein the information obtained from the terminal device scan includes device name, device type, device address, operating system, open ports, running services, service version, MAC address, manufacturer information, device response characteristics (e.g., TTL, window size), patch update status, host name, network interface information, security configuration status (e.g., firewall, antivirus software), account and login mechanism; and the information obtained from the network device scan includes one or more items of VLAN interface, routing information, hub information, and switch information.

[0071] The network traffic analysis method deploys detectors at network nodes to collect network traffic logs in devices, performs traffic data field analysis, and analyzes information such as port fields, flag fields (such as SYN, ACK), window size, and sequence number in TCP response data packets to obtain port status, the service type corresponding to the port, and service behavior characteristics; obtains device operating system information based on the TTL field in the IP response data packet, and obtains the IP address, subnet mask, gateway address, DNS server address, lease time, host name, vendor identifier, and broadcast address of the target host by analyzing other fields in the Dynamic Host Configuration Protocol DHCP response data except the transaction ID and message type, thereby comprehensively characterizing the network configuration environment and operation characteristics of the target device.

[0072] Step A3: Extract fingerprint features from the first scan data and the second scan data, match them with existing asset fingerprint features in the fingerprint library, identify and obtain asset information within the target network as the first network asset information; then use a cyberspace search engine to scan the external network to obtain the second network asset information as a supplement to the first network asset information;

[0073] The second network asset information is specifically obtained by scanning external network assets through existing public cyberspace search engines Fofa and ZoomEye to obtain the type and version of each network device, security device, server, terminal device, Internet of Things device, virtualization device, industrial control system device, wireless access device, and storage device connected to the external network, their geographical or logical location, open ports and the service type and version corresponding to the ports, operating system type and kernel version, SSL / TLS certificate information, device manufacturer and model, MAC address, device fingerprint characteristics, running time, service response characteristics (such as HTTP Banner, SSH welcome information), vulnerability information (such as associated CVE number), public network reachability, and possible unauthorized access risk information, and use cyberspace search engines to detect surviving hosts and scan surviving ports, and determine whether the host is alive by sending network requests and observing responses, and obtain the open port number and port information on the host. The information obtained by the cyberspace search engine is organized and analyzed, including standardization of the obtained IP addresses, port numbers, service types, service versions, TLS certificate information, response header features, device fingerprint information, asset ownership determination, service fingerprint identification, geographic location mapping, threat label association, vulnerability matching analysis, and access control status judgment operations to obtain the second network asset information.

[0074] Step A4: Use the first and second network asset information as the target system's environment description and compile a MulVAL input file. MulVAL uses Datalog as its model language (including vulnerability descriptions, rule descriptions, configuration descriptions, and permission systems). After converting the first and second network asset information into Datalog fact inputs, including vulnerability scan results from vulnerability scanners such as Nessus or OVAL, network node configuration and topology information, network management policies provided by the network administrator, device access control lists (ACLs), port mapping rules, routing policies, user identities and permission assignments, deployed security policies, detected abnormal behavior patterns, and network log and security audit data, MulVAL's inference engine performs logical reasoning based on these facts and predefined Datalog security rules, continuously developing the intermediate conditions that an attacker must meet to reach the target state. This automatically derives possible attack paths and their dependencies, ultimately generating a formalized attack tree structure.

[0075] The derivation process uses the attack target as the root node, starting from the attacker's initial privilege state and working backwards along the system configuration and vulnerability exploitation logic. This captures semantic relationships such as operating system privilege changes, vulnerability exploitation chains, and network access paths, gradually forming a complete attack tree. Each node represents a step required by the attacker to complete the attack, and each path represents a feasible attack route. Ultimately, this attack tree structure is input into the Graphviz visualization tool to generate an attack graph that intuitively displays the attack process and path dependencies. This helps network defenders understand the attack path, identify critical vulnerabilities and high-risk nodes, and thus support more efficient security hardening and policy deployment.

[0076] Step A5: After the attack graph is drawn, the initial information contained in the attack graph, including identified vulnerabilities in the network (such as service vulnerabilities, misconfigurations, and unpatched system components), network topology (including subnet divisions, routing paths, firewalls, and access control policies), and the security configuration status of each host (including operating system version, open ports, installed services, and user permissions), is used as the starting node of the entire attack graph. These initial conditions provide a factual basis for attack path deduction. A Bayesian attack tree model is constructed based on existing expert knowledge as inference rules. The Bayesian attack tree model estimates the corresponding attack success probability for each target node. Each node in the attack graph has two types of attack success probabilities: intrinsic probability and cumulative probability. The intrinsic probability represents the inherent probability of successfully compromising the current node if all required conditions are met, calculated using the CVSS base score. The cumulative probability, the final calculated probability, represents the overall probability of an attacker successfully reaching the target node and executing the attack in a given attack graph, derived from the Bayesian network. By applying the forward update or backward update method to recursively calculate the cumulative probability of attack success of each node on each attack path from the start node to the target node, and removing the cycle path from the attack graph, we finally get the optimized Bayesian attack tree.

[0077] Step A6: Using the resulting Bayesian attack tree, a depth-first traversal algorithm is used to enumerate all potential attack paths to the target node. For each attack path, the attack payoff and cumulative attack success probability of all nodes on it are calculated. A simplified attack matrix is ​​then created by combining the attack payoff and cumulative probability. The first column of the simplified attack matrix represents the attack score of the starting node of each attack path, the middle columns represent the attack scores of each step in the attack process, and the last column represents the score of the target node. This simplified attack matrix is ​​provided as input to a deep reinforcement learning network to train a model capable of evaluating the total score of all potential attack paths and finding the attack path with the highest score.

[0078] The deep reinforcement learning network model generation and training module is used to build a deep reinforcement learning network model and train the deep reinforcement learning network model;

[0079] The specific steps in building a deep reinforcement learning network model are: modeling the Bayesian attack tree as a sequential decision problem, which is considered a Markov decision process: M = (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the state transition matrix, R is the reward function, and γ≤1 is a regulation factor used to discount the reward based on the elapsed time. The Markov decision process includes:

[0080] State space: The state space of the environment consists of all intermediate attack-fault trees generated after possible actions;

[0081] Action space: Given an intermediate attack-fault tree, the agent's goal is to increase the height of the Bayesian attack tree by injecting or connecting new nodes; therefore, the action space is n-discrete, where n represents the number of potential events;

[0082] Terminal: Once the target event occurs, the Markov decision process terminates.

[0083] Reward: The amount of money earned by executing each attack step, as measured by CVSS Base and Exploitability scores;

[0084] Because the action space is discrete, deep reinforcement learning is used to make decisions based on a Markov decision process. At each time step t, the agent stores the transmission samples obtained from interacting with the environment in a replay buffer. During each training phase, a batch of transmission samples is randomly selected and input into the deep reinforcement learning current Q-value network. The network parameters θ are updated using a stochastic gradient descent algorithm. The goal of the deep reinforcement learning network is to construct an algorithm that satisfies the Bellman optimality equation, as shown in the following equation:

[0085]

[0086] Where, Indicates that under the policy π, the agent is in state Take action The expected cumulative return that can be obtained, that is, the Q value, Indicates that the agent is in state Take action The immediate reward obtained, γ is the decay factor used to weigh the importance of future rewards, Indicates the next state Consider all possible actions , the maximum Q value that can be obtained, that is, the estimate of the optimal return Q value in the future, represents the current state at time step t, The current action taken at time step t, Indicates that the status Execute an action The next state reached after or Indicates the next state One of all possible actions taken in .

[0087] Training a deep reinforcement learning network model specifically includes the following steps:

[0088] Step B1: Calculate the attack benefit obtained by the attacker at time t:

[0089]

[0090] Where, It represents the cumulative discounted return starting from time step t, which is the sum of all rewards obtained from the current moment in the future, attenuated by the discount factor. represents the immediate reward obtained at time step t+1, γ is the decay factor, Indicates the degree of discount on the future reward of the kth step; as k increases, the impact of future rewards gradually decreases; It is the sum of all future rewards after being gradually discounted over time, that is, the discounted cumulative return.

[0091] By introducing the decay factor γ, at each time step t, the system is and attack actions Calculating immediate attack benefits and accumulate subsequent benefits in the form of discounts to form cumulative discount returns This reward value is used to update the action value function , reflecting the long-term expected reward from taking a particular attack action in the current state. During training, the agent continuously updates its Q-value and network weights through a reinforcement learning algorithm, ultimately leading to convergence. During the testing or execution phase, the system selects the action with the highest Q-value in each state and constructs an attack path with the highest expected cumulative attack reward. This achieves a dynamic balance between short-term rewards and long-term effects, ultimately determining the optimal set of attack actions.

[0092] Step B2: By combining the current state s and the selected action a, the expected cumulative attack benefit Q value is updated according to the following formula:

[0093]

[0094] In the formula Indicates that under the policy π, the agent is in state Take action The expected cumulative return that can be obtained, that is, the Q value, E[•] mathematical expectation symbol, represents the average value of all possible reward sequences under the strategy π and the environment transition probability distribution, The discounted cumulative reward from time step t+1 to all future time steps, γ is the decay factor, Represents the immediate reward obtained at time step t+1+k. Indicates that the current state is s, that is, the current environmental state under the conditions. The current action is a, which is the current action performed by the agent. The policy π represents the probability distribution or behavior rule of the agent in each state.

[0095] make Represents the Q value output by the current value network of deep reinforcement learning, represents the parameters of the current value network, represents the Q value output by the deep reinforcement learning target value network, Represents the parameters of the target value network.

[0096] Step B3: Based on the Q value in step B2, use a greedy strategy to select an action Make Take the maximum value:

[0097]

[0098] In the formula At time step t, the agent chooses the optimal action, that is, the action that maximizes the action-value function in the current state. Is a mathematical operator, which means to find the function value The largest action a. Under a given policy π, the agent is in state Take action The expected cumulative return that can be obtained after represents the current state at time step t, and the policy π represents the probability distribution or behavior rule for the agent to choose an action in each state.

[0099] Step B4: Calculate the target Q value using the following formula:

[0100]

[0101] In the formula represents the target value used to update the Q-value network in the i-th iteration, r represents the immediate reward returned by the environment after the agent takes an action in the current state, γ is the decay factor, Indicates the next state In the example, the target Q value network parameters are used. Calculate all possible actions The Q value of the state is calculated and the maximum value is selected, which represents the maximum expected reward that can be obtained from the next state. Indicates the next state that the environment transitions to after executing the action in the current state. Indicates that the status All candidate actions that can be taken in are used to find the maximum Q value. Indicates the use of target network parameters Estimated action-value function. represents the parameter set of the target Q network in the i-th iteration.

[0102] By updating the parameters θ of the current value network in real time during each round of training, and copying the parameters θ to the deep reinforcement learning target Q value network after N rounds of iteration, the network parameters are adjusted by minimizing the mean square error between the current Q value and the target network Q value. The error function is expressed as:

[0103]

[0104] Where, represents the loss function of the deep reinforcement learning Q-value network in the i-th iteration, It represents the expectation of the experience sample of state s, action a, reward r, and next state s′. represents the target value used to update the Q-value network in the i-th iteration. Indicates the current Q value network parameters used The predicted Q value for state s and action a. Represents the parameter set of the current Q value network in the i-th iteration. Represents the squared difference between the predicted value and the target value.

[0105] The gradient descent calculation formula corresponding to the error function is as follows:

[0106]

[0107] Where, Represents the loss function About the current Q network parameters The gradient of , which is used for back propagation optimization of neural networks; It represents the expectation of the experience sample of state s, action a, reward r, and next state s′. represents the target value used to update the Q-value network in the i-th iteration. Indicates the current Q value network parameters used The predicted Q value for state s and action a. represents the parameter set of the current Q value network in the i-th iteration, Indicates the difference between the current sample predicted Q value and the target Q value, Indicates the current Q value network's parameters The gradient derivative of .

[0108] The automated penetration testing tool set module is used to perform the following steps:

[0109] Step C1: Build the penetration testing tool into a wrapper and provide a corresponding penetration testing case construction template. Based on the optimal attack path and optimal attack action set output by the deep reinforcement learning network model as the description input, the input is converted into an instruction sequence that can be executed by the penetration tool in the expected execution order. By running the instruction sequence and receiving the returned attack execution results, it is used to decide how to continue the given attack path.

[0110] Step C2: For the identified optimal attack path, perform structural simplification and merging optimization operations on the Bayesian attack tree to obtain an attack subtree model, including: path node labeling, redundant branch pruning, logic gate merging, path folding, indicator aggregation, and attack subgraph reconstruction;

[0111] By introducing a system fault tree analysis method to construct a fault tree analysis (FTA) model, the system-level failure logic is characterized. Attack subtree models are then inserted into the FTA model for security-sensitive intermediate failure event nodes, enabling it to reflect the mechanisms by which typical intrusion behaviors, vulnerability exploits, or attack chains induce functional failures. The attack subtrees are connected to the main functional fault tree constructed based on the functional structure of the system under test through standard logic gates (such as AND, OR, and FDEP), forming an integrated attack fault tree analysis model.

[0112] The model supports joint probabilistic simulation and risk path identification at the system level, thereby achieving unified modeling of both functional safety and network security.

[0113] Step C3: Combined with expert experience or the failure probability of system-related components in public documents, the Monte Carlo statistical analysis method is used to calculate the target system's overall cumulative failure probability (CFR), mean time between failures (MTBF), mean time to repair (MTTR), and mean time between failures (MTBF). Furthermore, the system safety is quantitatively evaluated using reliability metrics specific to the target system's reliability research field.

[0114] Step C4: To estimate the instantaneous failure rate of the fault event nodes caused by information security attacks in the system attack fault tree, the statistical timed automata method is used to simulate attackers with different capability levels and the finite automatic state machine of the system failure behavior caused by attack events under different threat detection granularity conditions. The finite automatic state machine corresponding to the attack event is connected with the other basic fault event activity state machines through the logic gates (such as AND, OR, SAND, PAND, FDEP, etc.) of the parent node in the attack fault tree to obtain the overall statistical timed automata model of the system. The overall statistical timed automata model of the system is converted to construct a metalanguage model that can be run in the Uppaal SMC reliability analysis software. Simulation verification is carried out on the Uppaal SMC reliability analysis software to obtain the overall failure rate, mean time to failure, mean time to repair (MTTR), and mean time between failures (MTBF) of the target network information system under various scenarios with different information security protection levels.

[0115] In this embodiment, the power system attack fault tree security risk assessment process is used as an example. Through a deep reinforcement learning-based attack fault tree security risk assessment system, accurate identification and hazard assessment of industrial control system information security and functional safety vulnerability risks are achieved.

[0116] The implementation process includes the following steps:

[0117] Step 1: By applying the network asset data fingerprint scanner module in the invention content to run steps 1-4, taking the network communication scenario for the power system as an example, service port scanning, POC and CVE vulnerability detection, network topology tomography and mapping can be constructed as follows: Figure 2 The following diagram shows a topology diagram of a smart grid power generation application scenario. The IEM represents the intelligent energy control unit (IEM) for the power generation system at each site, and the IFM represents the intelligent fault management system (IFM) that monitors and manages each IEM, isolating and recovering from IEM failures. Table 1 shows an example of the vulnerabilities identified after vulnerability testing for this system and their corresponding operational services.

[0118] Table 1 Example of vulnerability detection results for a power system

[0119]

[0120] Step 2: After completing the derivation and construction of the attack path between the hosts in the power system scenario by applying the network asset data fingerprint scanner module in the invention content and running steps 5-6, the attack benefit score is calculated for each node on the attack path according to the following process:

[0121] (1) The starting node reward value is 0.01, and the target node reward value is 100;

[0122] (2) For each node that exploits a vulnerability, the following formula is used to calculate the score value as the reward score;

[0123]

[0124] (3) For each node that executes code or accesses a file, a reward score of 1.5 is assigned to it;

[0125] (4) For other nodes in the attack tree, a reward score of 0 is given, and a penalty score of -1 is given if there is no path between the two nodes.

[0126] Step 3: Based on step 2, the intrinsic probability and cumulative probability of each node are calculated by combining the Bayesian conditional probability formula. The intrinsic probability can be calculated by the following formula:

[0127]

[0128] AV, AC, PR, and UI are all exploitability score indicators in the CVSS. AV represents the attack vector; a higher score indicates a more remote location from which an attacker can exploit the vulnerability; AC represents attack complexity; a higher score indicates a lower level of attack complexity; PR represents authentication; a higher score indicates a lower level of required privileges; and UI represents user interaction; a higher score indicates a lower level of required user interaction.

[0129] The nodes where multiple attack paths converge in the attack tree can be used to calculate their corresponding cumulative probabilities according to the corresponding logical operations of AND and OR, respectively, using the following formulas. P(S) represents the cumulative success probability of attacking the target node S, and p(S) represents the intrinsic success probability of the attack on the target node S. P(e i ) represents the i-th successor node e of the target node S in the current attack tree i The cumulative attack success probability.

[0130] AND form:

[0131] OR form:

[0132] After recursively traversing the attack tree to obtain the intrinsic probability and cumulative probability of all nodes, the probability of each node is multiplied by the attack reward obtained in step 2 to obtain the final node attack benefit and construct the simplified attack matrix M. 、 、 、 They represent the jth attack path (assuming that there are n attack paths between the starting node S and the target node T in the attack tree, which are represented by p1,…p n ) The starting node S on the attack tree, the node P0 reached in the attack tree after executing the first attack action, and the node P reached in the attack tree after the i-th attack action. i , and the target node T are calculated using the above formula to obtain the node attack benefit score.

[0133]

[0134] Step 4: Construct and train a deep reinforcement learning network model according to steps 1-4 in the deep reinforcement learning network model generation module in the invention content, and input the matrix M in the above step 3 into the model to obtain an output. The output corresponds to the optimal attack path from the attack starting point of the system to the target machine (IEM control station). All associated parent nodes on the attack path are combined with the node set itself to form an attack tree as the attack subtree for the target machine in the current system.

[0135] Step 5: Iteratively execute steps 1 to 4 above for all attack target machines in the target system. Add the attack subtree set obtained after running the above steps to the corresponding component failure event node set in the system fault tree. Based on the dependency relationship of the attack events, adjust the subtree structure and depth of other nodes in the system fault tree except the top event node, and recalculate the cumulative failure rate corresponding to each layer of nodes.

[0136] The attack fault tree of the power system in the embodiment of the present invention is shown in FIG. Figure 3 ,The gray color in the figure represents the attack subtree.

[0137] Step 6: For a Basic Component Failure (BCF) event, Figure 4 Modeling is performed in the manner shown. For each BCF event, the failure event occurrence point (Start), the failure process (Failing), and the failure occurrence point (Failed) are set. The failure rate λ and the loss d caused by the failure event are set according to expert experience, and the activation trigger condition (activate[id]?) is configured.

[0138] When the activation message from the top event is received, the invalidation process is executed, and a invalidation message (disrupt[id]!) is published after a fault occurs.

[0139] Step 7: After completing the modeling of basic component failure events, continue as follows Figure 5 As shown in the figure, the basic attack steps (BAS) are modeled for component failure events caused by information security risks. Figure 5The conditional probability expectation of detecting the attack behavior in the basic attack step in the attack fault tree described in , where w1 represents the probability expectation of the attack not being detected, and w2 represents the probability expectation of the attack being detected by the system.

[0140] When the attack begins, the attacker remains in a standby state until the cumulative attack cost reaches the set threshold f, and then proceeds to the next stage of attack. Escape detection, otherwise the detection attack probability is .

[0141] Step 8: If the system does not detect an attack, the attacker's current state will switch to "activated" or "ongoing." If the attacker is in the "ongoing" state branch, the system calculates the time required for the attack using the probability exponent λ and increases the attack cost by a daily increment v. If the attacker is currently in the "activated" state, in addition to measuring the attack time and cost using the same exponential probabilities λ and v, the attacker will then move back to the "attack execution" state.

[0142] Step 9: Refer to Figure 5 The conditional probability expectation of successful attack execution in the basic attack steps in the attack fault tree described in , where p represents the probability expectation that the attack can be successfully executed, and conversely q represents the probability expectation that the attack fails to be successfully executed.

[0143] Therefore, the probability that the attacker successfully implements the BAS is p / (p+q), and the total loss of the system due to the failure is d. The probability that the attack fails is q / (p+q).

[0144] The probability is primarily determined by the attacker's capabilities, estimated based on the CVSS vulnerability score and exploit difficulty. Table 2 shows an example of the attack path and configuration description of basic BAS variables in the attack fault tree model constructed for power systems.

[0145] Table 2 Attack path and basic attack step (BAS) variable configuration example

[0146]

[0147] Step 10: By modeling various common logic gates, basic component faults (BCFs), and basic attack steps (BAS) in the attack fault tree in UPPAAL SMC and combining them through model declarations, after completing the association between each event node, run the randomized process simulation to start the collaborative event flow simulation verification analysis.

[0148] Step 11: To simulate the impact of an attack, run the simulation in both the AS-IS and WHAT-IF scenarios. First, in the AS-IS scenario, remove the detection feature to establish a baseline for successful attacks using the configuration file. Then, change the WHAT-IF scenario configuration, set w1 and w2, and rerun the scenario with a high detection rate. By running the attack in different scenarios, you can determine the effectiveness of the detection mechanism and closely simulate real-world attack detection.

[0149] From the above steps, it can be seen that the present invention combines the attack path based on deep reinforcement learning algorithm and attack fault tree analysis technology, explores the attack path of potential vulnerabilities in the system by applying a scanner, and determines the route for conducting penetration testing from the attacker's perspective. And by applying the deep reinforcement learning algorithm to train the model, the best attack route is found. According to the expected execution order, it is converted into an instruction sequence that can be executed by the penetration tool. By running the instruction sequence and receiving the returned attack execution results, it is used to decide how to continue the given attack path. Finally, by constructing an attack fault tree model for the target system, a quantitative and qualitative evaluation of the reliability and security of the system is established, which provides an important basis for improving the overall security protection level of the system.

[0150] The above description is merely an illustration of the preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. An attack fault tree security risk assessment system based on deep reinforcement learning, characterized by: include: Network asset data fingerprint scanner module, deep reinforcement learning network model generation and training module, and automated penetration testing tool collection module; The network asset data fingerprint scanner module uses scanner software to actively or passively scan the target system and its network to obtain raw network data, and then performs data cleaning, feature extraction, protocol behavior modeling and pattern matching analysis on the scan results to construct network fingerprint information of the target system and target application. In combination with the communication relationship between multiple hosts, port mapping and routing response information, the module reconstructs the logical topology of the target network, clarifies the connection relationship between each node, the location of core assets and the security boundary division; and combines the CVE vulnerability information and network topology obtained in the scanning phase as input, applies the attack graph generation tool MulVAL to generate an attack graph file containing the attack path, and optimizes and prunes the attack graph in combination with the Bayesian prior probability to generate a Bayesian attack tree. After applying depth-first traversal to the Bayesian attack tree, a profit matrix of all attack paths from the starting point to the attack target is obtained, i.e., a Q-value matrix. The deep reinforcement learning network model generation and training module includes an agent, an environment, a state space, an attack action set, a replay buffer, a current Q-value network and a target Q-value network; wherein the agent observes the environment to obtain current state information and then performs corresponding attack actions; the environment generates a new state according to the agent's execution of the action, and the attack benefit generated during the state update depends on the attack action used and is described using a Markov decision process; the transmission samples are processed by using experience replay, and the transmission samples obtained by the interaction between the agent and the environment are stored in the replay buffer at each moment; during the training phase, batch samples are randomly selected and the current Q-value network parameters are updated using the stochastic gradient descent algorithm, and the current Q-value network parameters are assigned to the target network after the training phase, and then the network parameters are updated by minimizing the difference between the current Q-value and the target Q-value in combination with the loss function to generate the final deep reinforcement learning network model; The automated penetration testing tool set modules include Metasploit, Auto-Pres, SSH-KeyFinder, Nikto, Hydra, sqlmap, and John the Ripper. By inputting the payoff matrix obtained from the network asset data fingerprint scanner module into a deep reinforcement learning network model, the most preferred attack path corresponding to the prediction result is obtained; all attack actions included in the attack path are translated into commands that can be understood by the penetration testing tool, and the penetration testing tool is called to perform vulnerability exploitation, privilege escalation, and information collection operations on the real target system. The results of these operations are received as feedback to evaluate whether the current attack is successful and determine newly exposed assets or vulnerabilities in the target system, thereby determining the direction of the next stage of attack path selection, including whether to continue to penetrate the core assets of the target system, whether to move laterally to associated hosts, or whether to use newly obtained credentials to launch deeper attack behaviors.

2. The attack fault tree security risk assessment system based on deep reinforcement learning according to claim 1 is characterized in that: The network asset data fingerprint scanner module is used to perform the following steps: Step A1: Use active scanning technology to scan assets within the target network to obtain first scanning data. The scanner software actively sends a constructed specific probe data packet to the host and port in the target network. The specific probe data packet is a TCP SYN packet, UDP probe packet, ICMP echo request packet, HTTP request packet, or other protocol handshake packet, which stimulates the target device to respond. The scanner then extracts key information from the response data packet returned by the target host, including response characteristics of each network layer protocol, service version information, port status, packet retransmission delay, TTL, and window size, thereby obtaining the network attributes and behavioral characteristics of the target asset; The active scanner software sends Internet Control Message Protocol (ICMP) and Address Resolution Protocol (ARP) to the network layer of the target network to detect the survival of hosts in the target network, and sends Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) detection packets to the transport layer of the target network to identify the survival status of hosts in the target network, open ports, service types and service versions corresponding to the ports, operating system types and versions, service response behavior characteristics, the existence and behavior patterns of firewalls or intrusion detection systems, and device fingerprint identification information based on port return characteristics; it sends a short network management protocol (SNMP) to the application layer to detect operating system and application service information; Step A2: Scanning network traffic logs of terminal devices and network devices within the target network using a network traffic analysis method to obtain second scan data; wherein the information obtained from the terminal device scan includes device name, device type, device address, operating system, open ports, running services, service version, MAC address, manufacturer information, device response characteristics, patch update status, host name, network interface information, security configuration status, account and login mechanism; and the information obtained from the network device scan includes one or more items of information including VLAN interface, routing information, hub information, and switch information. Step A3: Extract fingerprint features from the first scan data and the second scan data, match them with existing asset fingerprint features in the fingerprint library, identify and obtain asset information within the target network as the first network asset information; then use a cyberspace search engine to scan the external network to obtain the second network asset information; Step A4: Using the first network asset information and the second network asset information as the target system's environment description, a MulVAL input file is written. MulVAL uses the Datalog language as a model language. After converting the vulnerability scanner's vulnerability scan results, network node configuration information and topology information, network management policies provided by the network administrator, device access control lists, port mapping rules, routing policies, user identities and permission allocations, deployed security protection policies, detected abnormal behavior patterns, and network logs and security audit data in the first network asset information and the second network asset information into Datalog language fact inputs, MulVAL's deduction engine performs logical reasoning, continuously developing intermediate conditions that the attacker must meet to reach the target state, thereby automatically deducing possible attack paths and their dependencies, and ultimately generating a formalized attack tree structure. The derivation process uses the attack target as the root node, starting from the attacker's initial privilege state, and working backwards along the system configuration and vulnerability exploitation logic to form a complete attack tree. Each node represents a step required for the attacker to complete the attack, and each path represents a feasible attack route. Finally, the attack tree structure is input into the Graphviz visualization tool to generate an attack graph that intuitively displays the attack process and path dependencies. Step A5: After the attack graph is drawn, the initial information contained in the attack graph, including the specific vulnerabilities identified in the network, the network topology, and the security configuration status of each host, is used as the starting node of the entire attack graph; A Bayesian attack tree model is constructed based on existing expert knowledge as inference rules. In the Bayesian attack tree model, the corresponding attack success probability is estimated for each attack target node. Each node in the attack graph has two types of attack success probabilities: intrinsic probability and cumulative probability; The intrinsic probability represents the inherent likelihood of successfully compromising the current node when all required conditions are met, calculated using the CVSS base score. The cumulative probability, the final calculated probability, represents the overall likelihood of an attacker successfully reaching the target node and executing the attack in a given attack graph, derived using a Bayesian network. By applying forward update or backward update to recursively calculate the cumulative probability of attack success for each node on each attack path from the start node to the target node, and remove the path containing cycles from the attack graph, we finally get the optimized Bayesian attack tree. Step A6: On the resulting Bayesian attack tree, all potential attack paths to the target node are enumerated using a depth-first traversal algorithm. The attack payoff and cumulative attack success probability of all nodes on each attack path are calculated. A simplified attack matrix is ​​then created by combining the attack payoff and cumulative probability. The first column of the simplified attack matrix represents the attack score of the starting node of each attack path, the middle columns represent the attack scores of each step in the attack process, and the last column represents the score of the target node. The simplified attack matrix is ​​provided as input to the deep reinforcement learning network. The deep reinforcement learning network model generation and training module is used to construct a deep reinforcement learning network model and train the deep reinforcement learning network model.

3. The attack fault tree security risk assessment system based on deep reinforcement learning according to claim 2 is characterized in that: The network traffic analysis method described in step A2 performs flow data field analysis by deploying detectors at network nodes to collect network traffic logs in the device, analyzing the port field, flag field, window size, sequence number and other information in the TCP response data packet to obtain the port status, the service type corresponding to the port and the service behavior characteristics; The device operating system information is obtained based on the TTL field in the IP response data packet. By analyzing other fields in the Dynamic Host Configuration Protocol DHCP response data except the transaction ID and message type, the IP address, subnet mask, gateway address and DNS server address, lease time, host name, vendor identifier, and broadcast address of the target host are obtained, thereby comprehensively characterizing the network configuration environment and operating characteristics of the target device.

4. The attack fault tree security risk assessment system based on deep reinforcement learning according to claim 2 is characterized in that: In step A3, the second network asset information is specifically obtained by scanning external network assets through a cyberspace search engine to obtain the type and version of each network device, security device, server, terminal device, Internet of Things device, virtualization device, industrial control system device, wireless access device, and storage device connected to the external network, the geographical or logical location, open ports and the service type and version corresponding to the port, operating system type and kernel version, SSL / TLS certificate information, device manufacturer and model, MAC address, device fingerprint characteristics, running time, service response characteristics, vulnerability information, public network reachability, and possible unauthorized access risk information. The cyberspace search engine is used to detect surviving hosts and scan surviving ports. By sending network requests and observing responses, it is determined whether the host is alive, and the open port number and port information on the host are obtained. The information obtained by the cyberspace search engine is sorted and analyzed, including standardization of the obtained IP address, port number, service type, service version, TLS certificate information, response header characteristics, device fingerprint information, asset ownership determination, service fingerprint identification, geographic location mapping, threat label association, vulnerability matching analysis, and access control status judgment operations to obtain the second network asset information.

5. The attack fault tree security risk assessment system based on deep reinforcement learning according to claim 2 is characterized in that: The construction of the deep reinforcement learning network model described in step A6 is specifically as follows: the Bayesian attack tree is modeled as a sequential decision problem, which is regarded as a Markov decision process: M = (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the state transition matrix, R is the reward function, and γ≤1 is a regulation factor. The Markov decision process includes: State space: The state space of the environment consists of all intermediate attack-fault trees generated after possible actions; Action space: Given an intermediate attack-fault tree, the agent's goal is to increase the height of the Bayesian attack tree by injecting or connecting new nodes; therefore, the action space is n-discrete, where n represents the number of potential events; Terminal: Once the target event occurs, the Markov decision process terminates; Reward: The amount of money earned by executing each attack step, as measured by CVSS Base and Exploitability scores; Due to the discrete nature of the action space, deep reinforcement learning is used to make decisions based on the Markov decision process. At each time step t, the agent stores the transmission samples obtained by interacting with the environment in the replay buffer. At each training stage, a batch of transmission samples are randomly selected and input into the deep reinforcement learning current Q value network, and the stochastic gradient descent algorithm is used to update the network parameters θ. The goal of the deep reinforcement learning network is to construct a model that satisfies the Bellman optimality equation, as shown in the following formula: Where, Indicates that under the policy π, the agent is in state Take action The expected cumulative return that can be obtained, that is, the Q value, Indicates that the agent is in state Take action The immediate reward obtained, γ is the decay factor used to weigh the importance of future rewards, Indicates the next state Consider all possible actions , the maximum Q value that can be obtained, that is, the estimate of the optimal return Q value in the future, represents the current state at time step t, The current action taken at time step t, Indicates that the status Execute an action The next state reached after or Indicates the next state One of all possible actions taken in .

6. The attack fault tree security risk assessment system based on deep reinforcement learning according to claim 2 is characterized in that: The training of the deep reinforcement learning network model described in step A6 specifically includes the following steps: Step B1: Calculate the attack benefit obtained by the attacker at time t: Where, represents the cumulative discounted return starting from time step t, which is the sum of all rewards obtained from the current moment in the future, attenuated by the discount factor; represents the immediate reward obtained at time step t+1, γ is the decay factor, Indicates the degree of discount on the future reward of step k; It is the sum of all future rewards after being gradually discounted over time, that is, the discounted cumulative return; By introducing the decay factor γ, at each time step t, the system is and attack actions Calculating immediate attack benefits and accumulate subsequent benefits in the form of discounts to form cumulative discount returns ; This reward value is used to update the action value function , reflecting the long-term expected benefit of taking a certain attack action in the current state; during the training process, the agent continuously updates the Q value and network weight through the reinforcement learning algorithm, and eventually tends to converge; during the testing or execution phase, the system selects the action with the highest Q value in each state and constructs an attack path with the maximum expected cumulative attack benefit, thereby achieving a dynamic balance between short-term benefits and long-term effects and determining the optimal attack action set; Step B2: By combining the current state s and the selected action α, the expected cumulative attack benefit Q value is updated according to the following formula: Where Indicates that under the policy π, the agent is in state Take action The expected cumulative return that can be obtained, that is, the Q value, E[•] mathematical expectation symbol, represents the average value of all possible reward sequences under the strategy π and the environment transition probability distribution, The discounted cumulative reward from time step t+1 to all future time steps, γ is the decay factor, Represents the immediate reward obtained at time step t+1+k; Indicates that the current state is s, that is, the current state of the environment under the conditions; The current action is a, which is the current operation performed by the agent. The strategy π represents the probability distribution or behavior rule of the agent in each state. make Represents the Q value output by the current value network of deep reinforcement learning, represents the parameters of the current value network, represents the Q value output by the deep reinforcement learning target value network, Represents the parameters of the target value network; Step B3: Based on the Q value in step B2, use a greedy strategy to select an action Make Take the maximum value: Where At time step t, the agent chooses the optimal action, that is, the action that maximizes the action-value function in the current state; Is a mathematical operator, which means to find the function value The largest action a; Under a given policy π, the agent is in state Take action The expected cumulative return that can be obtained after represents the current state at time step t, and the policy π represents the probability distribution or behavior rule of the agent's choice of action in each state; Step B4: Calculate the target Q value using the following formula: Where represents the target value used to update the Q-value network in the i-th iteration, r represents the immediate reward returned by the environment after the agent takes an action in the current state, γ is the decay factor, Indicates the next state In the example, the target Q value network parameters are used. Calculate all possible actions The Q value of , and select the maximum value, which represents the maximum expected reward that can be obtained from the next state; Indicates the next state the environment transitions to after executing the action in the current state; Indicates that the status All candidate actions that can be taken in are used to find the maximum Q value; Indicates the use of target network parameters Estimated action-value function; represents the parameter set of the target Q network in the i-th iteration; By updating the parameters θ of the current value network in real time during each round of training, and copying the parameters θ to the deep reinforcement learning target Q value network after N rounds of iteration, the network parameters are adjusted by minimizing the mean square error between the current Q value and the target network Q value. The error function is expressed as: Where, represents the loss function of the deep reinforcement learning Q-value network in the i-th iteration, Represents the expectation of the experience sample of state s, action a, reward r, and next state s′; represents the target value used to update the Q value network in the i-th iteration; Indicates the current Q value network parameters used The predicted Q value for state s and action a; Represents the parameter set of the current Q value network in the i-th iteration; Represents the squared difference between the predicted value and the target value; The gradient descent calculation formula corresponding to the error function is as follows: Where, Represents the loss function About the current Q network parameters The gradient of , which is used for back propagation optimization of neural networks; Represents the expectation of the experience sample of state s, action a, reward r, and next state s′; represents the target value used to update the Q value network in the i-th iteration; Indicates the current Q value network parameters used The predicted Q value for state s and action a; represents the parameter set of the current Q value network in the i-th iteration, Indicates the difference between the current sample predicted Q value and the target Q value, Indicates the current Q value network's parameters The gradient derivative of .

7. The attack fault tree security risk assessment system based on deep reinforcement learning according to claim 1 is characterized in that: The automated penetration testing tool set module is used to perform the following steps: Step C1: Build the penetration testing tool into a wrapper and provide a corresponding penetration testing case construction template. Based on the optimal attack path and optimal attack action set output by the deep reinforcement learning network model as description input, the input is converted into an instruction sequence that can be executed by the penetration tool in the expected execution order. By running the instruction sequence and receiving the returned attack execution results, it is used to decide how to continue the given attack path. Step C2: For the identified optimal attack path, the Bayesian attack tree is structurally simplified and merged to obtain an attack subtree model, including: path node labeling, redundant branch pruning, logic gate merging, path folding, indicator aggregation, and attack subgraph reconstruction. The fault tree analysis model (FTA) is constructed by introducing the system fault tree analysis method to characterize the system-level failure logic. Then, the attack subtree model is inserted into the safety-sensitive intermediate failure event node in the fault tree analysis model (FTA). The attack subtree is connected to the main function fault tree constructed based on the functional structure of the tested system through standard logic gates to form an integrated attack fault tree analysis model. Step C3: Combined with expert experience or the failure probabilities of system-related components in public documents, the Monte Carlo statistical analysis method is used to calculate the target system's overall cumulative failure probability (CFR), mean time between failures (MTBF), mean time to repair (MTTR), and mean time between failures (MTBF), thereby quantitatively assessing system safety. Step C4: For the estimation of the instantaneous failure rate of the fault event node caused by the information security attack in the system attack fault tree, the statistical time automaton method is used to simulate the finite automatic state machine of attackers with different capability levels and the system failure behavior caused by the attack event under different threat detection granularity conditions. The finite automatic state machine corresponding to the attack event is connected with the other basic fault event activity state machines through the logic gate of the parent node in the attack fault tree to obtain the overall statistical time automaton model of the system. The overall statistical time automaton model of the system is converted to construct a metalanguage model that can be run in the Uppaal SMC reliability analysis software, and simulation verification is carried out on the Uppaal SMC reliability analysis software to obtain the overall failure rate, mean time to failure, mean time to repair (MTTR), and mean time between failures (MTBF) of the target network information system under evaluation in various scenarios with different information security protection levels.

Images