A domain penetration testing method based on adaptive exploit

Abstract

The application discloses a domain penetration testing method based on adaptive vulnerability, and relates to the technical field of information security.The application trains an intelligent agent to perform dynamic analysis according to the dynamic changes of a domain environment and the vulnerability conditions in the domain, constructs a domain penetration testing model, and achieves the purpose of adaptively utilizing the existing vulnerabilities in the domain to generate a penetration testing path that meets the target and maximizes the cumulative reward, so that the security status of the domain environment can be evaluated, and security maintenance personnel can conveniently perform security maintenance.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a domain penetration testing method based on adaptive vulnerability exploitation. Background Technology

[0002] With the rapid development of information technology and the widespread adoption of the internet, the digital society has become central to our lives. However, this digital transformation has also brought increasingly severe cybersecurity challenges. Traditional network defense methods are proving inadequate in the face of constantly evolving attack methods, forcing the security field to seek more innovative and efficient solutions. In recent years, with the rise of artificial intelligence and automation technologies, cyber attackers are no longer relying on traditional manual methods but have turned to more intelligent and covert attack approaches. This trend necessitates a re-examination of current cybersecurity strategies and the search for innovative countermeasures to protect the security of the digital society.

[0003] Intelligent domain penetration testing automates the entire process from environmental information gathering and data analysis to penetration path planning, providing penetration testers with feasible solutions and identifying vulnerabilities in the domain environment for security maintenance personnel to patch. Artificial intelligence and machine learning technologies play a crucial role in this framework. These technologies help the system discover patterns and trends from massive amounts of data, automatically identify potential cybersecurity threats, and provide corresponding countermeasures. Compared to traditional penetration testing methods, intelligent domain penetration testing frameworks offer higher levels of automation and more accurate attack paths, providing security teams with more powerful tools to address increasingly complex cyberattacks. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] This invention is a domain penetration testing method based on adaptive vulnerability exploitation. It adaptively selects vulnerabilities for exploitation according to changes in the vulnerability situation and domain environment, and finally generates a penetration testing path to achieve the penetration testing goal, thereby providing assistance to security maintenance personnel and maintaining the security of the domain environment.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: Step 1: Passively and actively probe the target domain to collect domain environment information, and construct a structured representation of the collected raw information to form a domain environment model, generating a baseline network topology and a candidate set of exploitable points; Step 2: Based on the baseline network topology and the candidate set of exploitable points, establish a multi-dimensional feature representation for each potential vulnerability, and calculate the vulnerability success rate and exploitation cost using a probabilistic method; Based on the multi-dimensional feature representation and the vulnerability success rate and exploitation cost, define the environment space, state space, action space, and reward function required for the reinforcement learning deep Q-network; Step 3: Based on the environment space, state space, action space, and reward function, construct a reinforcement learning agent. The system is trained to find the optimal exploit path and build a domain penetration testing model. During training, an adversarial simulation sandbox is introduced to verify the success rate and detection risk of candidate penetration paths. Strategies with high detection costs or high risk of misoperation are eliminated until the strategy meets performance and security constraints, forming a deployable penetration testing model. Step 4: The deployable penetration testing model is deployed at a controlled starting point in the target domain. During actual execution, environmental changes are monitored in real time, and the optimal exploit action is adaptively selected and executed according to the current state. The execution results are fed back in real time for online updates, and model rollback is triggered when the detection rate or risk exceeds a preset threshold to maintain the security boundary.

[0008] As a preferred embodiment of the present invention, step 1 specifically includes: Starting with any controlled host within the domain, the penetration test collects information about the domain, including the domain name, names of ordinary hosts, domain controller name, server names, IP addresses of all hosts, port information of all hosts, and exploitable vulnerabilities. The controlled host communicates with other hosts within the domain to collect login user information and high-privilege user information. The collected information is then organized into structured records, with each host, service, and user treated as an attributed record. Attributes include operating system, port list, service version, candidate vulnerabilities, and last update time. The communication relationships between hosts are represented as a network. Network topology diagram; after data cleaning, a baseline network topology diagram and a candidate list of exploitable points are output. The baseline network topology diagram shows the connections between hosts, and the candidate list of exploitable points includes vulnerabilities or weaknesses that can be tried on each host and their sources. Based on the formatted information, penetration testing is carried out. Based on the obtained port, service, and exploitable vulnerability information, possible penetration testing paths are searched. Referring to the vulnerability database, the discovered exploitable vulnerabilities are analyzed. Exploitable vulnerabilities are screened by combining the success rate of vulnerability exploitation and the expected reward after successful vulnerability exploitation, and the optimal vulnerability exploitation path is determined. Among them, the exploitable vulnerabilities include system logic vulnerabilities, service configuration vulnerabilities, and information disclosure vulnerabilities.

[0009] As a preferred embodiment of the present invention, step 2 specifically includes: The collected environmental information is abstracted and classified, and the state space, action space and reward function required for the reinforcement learning deep Q network are defined. Among them, when defining the state space, the exploitable vulnerabilities existing in the domain are classified, and whether the vulnerability is exploitable is encoded as a 0 / 1 state. Each potential vulnerability is described by a vector containing several attributes, including vulnerability number, corresponding host and service version, whether there is publicly available exploit code, network reachability, and potential privilege escalation. A probabilistic method is used to combine the attribute information to calculate the expected success rate and expected cost of the vulnerability. The calculation results are output in the form of expected value and uncertainty. The environment space includes host information, service information, vulnerability information, network topology, and user information. Host information includes the host's IP address, operating system type and version, and host role. Service information includes the services running on the host, their versions, and a list of open ports. Vulnerability information includes a list of vulnerabilities on each host, the exploit success rate of each vulnerability, and the expected reward. Network topology includes connectivity information between hosts and a network topology diagram. User information includes a list of logged-in users on the host and their permission levels. The information in the environment space is abstracted and encoded, including host status encoding, vulnerability status encoding, and service status encoding. Host status encoding represents the existence of each host in binary, vulnerability status encoding represents the existence of each vulnerability in binary, and service status encoding represents the presence of each service vulnerability in binary. The existence state of the task is represented in binary; the action space includes scanning actions, information gathering actions, vulnerability exploitation actions, privilege escalation actions, and lateral movement actions; among them, scanning actions refer to scanning the open ports and running services of a specific host; information gathering actions refer to collecting user information and network connection information of the target host; vulnerability exploitation actions refer to attempting to exploit specific vulnerabilities in a specific host and service; privilege escalation actions refer to attempting to escalate the privileges of the current host by exploiting known vulnerabilities; lateral movement actions refer to attempting to acquire the privileges of other hosts as new controlled hosts; a layered or multi-layered intelligent agent architecture is adopted, using the global attack graph or topology information as the decision basis for the upper-layer strategy, the upper layer selects several valuable sub-targets or sub-strategies, and then the lower-layer strategy executes specific scanning, exploitation, or lateral movement operations within a smaller subset of actions; The reward function considers rewards for successful vulnerability exploitation, information gathering, privilege escalation, and penalties. Successful vulnerability exploitation awards a positive reward for successfully exploiting a vulnerability; information gathering awards a positive reward for collecting useful information; privilege escalation awards a positive reward for successfully escalating privileges; and penalties award a negative reward for executing ineffective actions or failing to exploit the vulnerability. The reward function also considers the stability and reliability of the vulnerability exploitation results, incorporating the vulnerability success rate assessment and fluctuations into the reward calculation. The action selection is optimized using a Softmax strategy, calculating the probability of each action being selected based on its Q-value. Finally, the reward function is defined as the cumulative reward generated by the set of actions that complete the change of control, minus the total reward value generated by all actions.

[0010] As a preferred embodiment of the present invention, step 3 specifically includes: A reinforcement learning agent is initialized using a deep Q-network algorithm. The Q-network is defined and initialized, its structure and parameters are set, and the agent's learning rate, discount factor, and other training hyperparameters are configured. Through multiple interactions with the simulated domain environment, quadruples of state, action, reward, and next state data are collected. In the early stages of training, the agent randomly selects actions to explore the environment. The collected training data is stored in an experience replay buffer. Training data is extracted from the buffer using mini-batch random sampling to calculate the Q-value of the current action and the target Q-value. The weights of the Q-network are updated according to the Bellman equation, where the target Q-value is the current reward plus a discount to the maximum Q-value of the next state. During training, candidate penetration paths are validated in an isolation sandbox. This sandbox simulates the target domain's operating environment and common defenses, used to test each candidate penetration path to estimate its success rate and risk of detection. If a path shows a high detection probability or may cause serious side effects in the sandbox, it is marked as high-risk. Risks are identified and eliminated from the candidate set or downgraded to require manual review; the results of sandbox experiments are used as training samples to feed back into the experience replay, helping the agent learn strategies to avoid high detection costs; the detection probabilities and misoperation costs estimated in the sandbox are mathematically incorporated into the learning objective as an additional penalty term for the total reward or as a constraint condition that must be met; a term representing risk cost is introduced into the value function or objective function, or constraints are imposed on detection probabilities above a threshold during policy optimization, so that the optimization process simultaneously balances benefits and safety costs; the detection and destruction risks of the strategy on different paths are periodically estimated using sandbox experiments, and the estimates are used to update constraint parameters or risk term weights, so that the strategy can maximize task rewards and meet the preset safety acceptability standards when converging; a Softmax strategy is used to select actions, and by adjusting the temperature parameter, exploration and exploitation are balanced to avoid getting trapped in local optima; the probability of each action being selected is calculated using the Softmax strategy, and actions are selected based on this probability; Repeat the steps of training data collection, experience replay, and Q-value update until the predetermined number of training iterations or convergence conditions are reached; monitor the cumulative reward changes during the training process to determine the learning effect of the agent; after sufficient training, form a deployable penetration testing model.

[0011] As a preferred embodiment of the present invention, step 3 specifically includes: real-time monitoring of the domain environment, continuously monitoring the dynamic changes of the target domain environment, including information on host status, service status, network topology changes and the emergence of new vulnerabilities; periodically or when significant changes are detected, updating the environment space and state space to ensure the accuracy of the environmental information perceived by the agent; Based on the latest environmental information obtained from real-time monitoring, the state space is dynamically adjusted, and new state information, including newly added hosts, services, and vulnerabilities, is re-encoded. In a dynamically changing environment, a Softmax strategy is used for action selection, calculating the probability of selecting each action based on the latest Q-value and current temperature parameters. Temperature parameters are adjusted in a timely manner according to environmental changes to balance the agent's exploration and exploitation behaviors. Based on the current state and action space, the agent selects the best exploitable vulnerability using the Softmax strategy and immediately executes the exploit action. By successfully exploiting vulnerabilities to obtain more environmental information and permissions, the agent's state space information is further enhanced. After executing the exploit action, the agent immediately calculates and receives a reward based on a predefined reward function. The system employs a reward function that considers real-time environmental information, including exploit success rate, privilege escalation status, and information gathering status. Based on the immediate reward feedback, the agent updates the Q-value of each action in the current state, optimizing its behavioral strategy. Through multiple iterations and optimizations, the results of each execution are fed back in real-time for online strategy updates and vulnerability success rate estimation. When the detection rate or risk exceeds a preset threshold, the model is immediately rolled back or switched to secure mode to maintain the security boundary. Through online feedback and rollback mechanisms, the agent continuously adjusts its strategy in a dynamic environment, gradually generating penetration testing paths that meet penetration testing objectives and maximize cumulative rewards. The agent is highly adaptive, capable of flexibly adjusting its strategy in a real-time environment to cope with various unforeseen circumstances and the emergence of new vulnerabilities.

[0012] As a preferred embodiment of the present invention, the binary representation of the host status code, the vulnerability status code, and the service status code is as follows: for the host status code, if the host exists, the code is 1, otherwise it is 0; for the vulnerability status code, if the vulnerability exists, the code is 1, otherwise it is 0; for the service status code, if the service exists, the code is 1, otherwise it is 0.

[0013] As a preferred embodiment of the present invention, the upper-layer strategy of the hierarchical or multi-layer intelligent agent architecture filters sub-targets or sub-strategies based on the global attack graph or topology information, and the lower-layer strategy performs scanning, vulnerability exploitation or lateral movement operations within a subset of actions to reduce the number of actions and reduce learning complexity.

[0014] In a preferred embodiment of the present invention, the temperature parameter of the Softmax strategy is adjustable to control the degree of exploration in action selection.

[0015] In a preferred embodiment of the present invention, the experience replay buffer uses a small-batch random sampling method to extract training data.

[0016] In a preferred embodiment of the present invention, the trigger condition for model rollback is that the detection rate or risk exceeds a preset threshold, and the preset threshold is dynamically adjusted according to the security policy of the domain environment.

[0017] The beneficial effects of this invention are as follows: Based on the dynamic changes in the domain environment and the vulnerability situation within the domain, this invention trains an intelligent agent to perform dynamic analysis. By constructing a domain penetration testing model, it achieves the goal of adaptively exploiting vulnerabilities within the domain and generating penetration testing paths that meet the objectives and maximize cumulative rewards. This allows for the assessment of the security status of the domain environment, facilitating security maintenance personnel to perform security maintenance.

[0018] Compared to traditional manual information gathering methods, the automated information gathering method of this invention significantly reduces the workload of penetration testers and enables more timely acquisition of key information about the domain environment. Furthermore, the acquired information comprehensively reflects the state of the entire domain environment. This comprehensive information gathering helps penetration testers fully understand the composition and operation of the domain environment, providing a solid foundation for subsequent penetration testing. The reinforcement learning DQN algorithm employed in this invention can autonomously generate penetration testing strategies through interactive learning with the environment. It can flexibly adjust and optimize penetration testing strategies based on dynamic changes in the domain environment and attack scenarios, thereby improving the accuracy and effectiveness of the tests.

[0019] The DQN algorithm possesses strong generalization capabilities, adapting to various complex domain environments and security attack scenarios. It can handle different types of vulnerabilities, changing network topologies, and unknown attack methods, providing broad applicability for various penetration testing tasks. Furthermore, the DQN algorithm has the ability to continuously learn and evolve, constantly optimizing its penetration testing strategies based on ongoing feedback and experience. Through continuous learning and improvement, the agent can adapt to ever-changing domain environments and continuously optimize its penetration testing capabilities.

[0020] This invention defines the observation space and reward function by treating the host and domain as independent observation dimensions, then integrating them into a unified whole to fully reflect the overall domain environment. Compared to traditional single-perspective methods, this approach is more comprehensive and detailed. By integrating information from the host and domain environments, the correlations and interactions within the domain environment can be understood more accurately, significantly improving the precision and accuracy of penetration testing. Simultaneously, a well-designed reward function can effectively guide the agent to identify security vulnerabilities and weaknesses during penetration testing, thereby optimizing the penetration testing strategy. This multi-dimensional, integrated approach not only realistically reflects the domain environment but also provides a more comprehensive assessment of the effectiveness and potential risks of penetration testing. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 The diagram shows the flowchart of this invention from scanning the domain environment to training the agent to building the domain penetration testing model.

[0022] Figure 2 This is a structural diagram of the parameters related to the agent training algorithm in this invention.

[0023] Figure 3 This is a structural diagram of the intelligent agent constructing domain penetration testing model in this invention.

[0024] Figure 4 This is a structural diagram of how the intelligent agent dynamically analyzes the domain environment and adaptively utilizes vulnerabilities to generate penetration testing paths in this invention. Detailed Implementation

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0026] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0027] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0028] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a domain penetration testing method based on adaptive vulnerability exploitation, comprising: Step 1: Passively and actively probe the target domain to collect domain environment information, construct a structured representation of the collected raw information, form a domain environment model, and generate a baseline network topology map and a candidate set of usable points.

[0029] In this embodiment of the invention, step S1 specifically includes: (1) Use any controlled host in the domain as the starting point for penetration testing. Based on this, collect information within the domain and obtain relevant information, including the domain name, the names of ordinary hosts in the domain, the domain controller name, the names of servers in the domain, the IP information of each host in the domain, the port information of each host in the domain, and the exploitable vulnerability information of the hosts in the domain.

[0030] (2) Communicate with other hosts in the domain through the controlled host to collect login user information, high-privilege user information, etc. of other hosts in the domain.

[0031] (3) Organize the collected information into a standardized record: each host, each service, and each user is recorded as an attributed record, with key fields clearly stated in the attributes (such as OS, port list, service version, candidate vulnerabilities, and last update time); at the same time, the communication between hosts is represented by a network diagram. After simple data cleaning, two types of results are output: one is a baseline network topology diagram (showing the hosts and their connections), and the other is a list of candidate exploitable points (vulnerabilities or weaknesses that can be tried on each host and their sources). These structured results can be directly used by subsequent algorithms.

[0032] (4) Based on the formatted information, conduct penetration testing. Using the obtained port, service, and exploitable vulnerability information, identify potential penetration testing paths. Referencing existing vulnerability databases, analyze the discovered exploitable vulnerabilities. Filter exploitable vulnerabilities based on their success rate and expected rewards, and determine the optimal exploitation path. Exploitable vulnerabilities include system logic vulnerabilities, service configuration vulnerabilities, and information disclosure vulnerabilities.

[0033] Step 2: Based on the baseline network topology graph and the candidate set of exploitable points, establish a multi-dimensional feature representation for each potential vulnerability, and use a probabilistic method to calculate the vulnerability success rate and exploitation cost; based on the multi-dimensional feature representation and the vulnerability success rate and exploitation cost, define the environment space, state space, action space and reward function required for the reinforcement learning deep Q network.

[0034] Specifically, based on the domain environment network topology mapped in step 1 and the acquired domain environment information, a multi-dimensional feature representation is established for each potential vulnerability, and a probabilistic method is used to calculate the vulnerability success rate and exploitation cost. The collected vulnerability information is organized, and the environment space, state space, action space, and reward function required for the reinforcement learning deep Q-Network (DQN) are defined, such as... Figure 2 As shown, it specifically includes: Step 2-1: Collect information about the target domain environment network, abstract and classify the collected environmental information, and define the state space, action space and reward function required for the reinforcement learning deep Q network based on the sorted environmental information.

[0035] Define a state space and categorize the exploitable vulnerabilities collected within the domain, such as known vulnerabilities like MS14-068, CVE-2021-42287, MS17-010, CVE-2019-0708, and MS08-067. Based on the compiled vulnerability information, encode the exploitability of a vulnerability as a 0 / 1 state for subsequent agent training.

[0036] Step 2-2: Describe each potential vulnerability using a vector containing several attributes (e.g., vulnerability number, corresponding host and service version, whether there is publicly available exploit code, network reachability, potential privilege escalation, etc.). Then, use a probabilistic method to combine this information to calculate the expected success rate and expected cost of the vulnerability. The calculation results are output in the form of expected value and uncertainty, serving as the basis for vulnerability ranking and subsequent reinforcement learning input.

[0037] Steps 2-3: Define the environment space. In this invention, the environment space refers to the state of the domain environment perceived by the reinforcement learning agent during penetration testing. The environment space mainly consists of host information, service information, vulnerability information, network topology, and user information.

[0038] Host information includes: the host's IP address, operating system type and version, and host role (e.g., domain controller, regular host, server, etc.). Service information includes: services running on the host and their versions, and a list of open ports. Vulnerability information includes: a list of vulnerabilities present on each host (e.g., MS14-068, CVE-2021-42287, etc.), the exploit success rate and expected reward for each vulnerability. Network topology includes: connectivity information between hosts and a network topology diagram.

[0039] User information includes: a list of users logged in on the host and their permission levels.

[0040] When defining the environment space, the above information needs to be abstracted and encoded so that it can be input into the reinforcement learning algorithm. The existence status of each host, the existence of each vulnerability, and the existence status of each service are encoded into binary representations, with 1 for existence and 0 for non-existence.

[0041] Steps 2-4: Define the action space. The action space refers to the set of actions an agent can choose in each state. In a penetration testing system, actions mainly include scanning actions, information gathering actions, vulnerability exploitation actions, privilege escalation actions, and lateral movement actions.

[0042] Scanning refers to scanning open ports and running services on a specific host. Information gathering refers to collecting user information and network connection information of the target host.

[0043] Exploitation actions refer to attempts to exploit specific actions on a particular host and service. Privilege escalation actions refer to attempts to escalate privileges on the current host by exploiting a known vulnerability.

[0044] Lateral movement refers to attempting to gain access to other hosts and become a new controlled host when there are no more exploitable points on the current host.

[0045] To effectively control the scale of actions and improve the generalization ability of strategies in actual deployment, a layered or multi-layered intelligent agent architecture is adopted: the global attack graph or topology information is used as the decision basis for the upper-layer strategy, the upper layer selects several valuable sub-targets or sub-strategies, and the lower-layer strategy executes specific scanning, exploitation or lateral movement operations within a smaller subset of actions.

[0046] By coupling the attack graph with the hierarchical strategy in this way, the number of actions to be considered at each step can be significantly reduced, the learning complexity can be decreased, and the adaptability to different network environments can be improved.

[0047] Steps 2-5: Define the reward function. The reward function refers to the feedback signal obtained by the agent after performing a certain action. The design of the reward function directly affects the agent's behavioral strategy. In penetration testing systems, the reward function mainly considers the following aspects: successful vulnerability exploitation reward: a high positive reward is given if a vulnerability is successfully exploited; information gathering reward: a certain positive reward is given for collecting useful information (such as high-privilege user information); privilege escalation reward: a high positive reward is given for successfully escalating privileges; and penalty measures: a negative reward is given for performing ineffective actions or failing to exploit vulnerabilities.

[0048] In the specific design, it is also necessary to comprehensively consider the stability and reliability of the vulnerability exploitation results, and include the vulnerability success rate assessment results and its fluctuations into the reward calculation, so that the agent is more inclined to choose the path with a high success probability and stable results when updating the policy. For cases with greater uncertainty in the results, the agent can be guided to perform additional verification or exploration actions to gradually reduce risks and improve subsequent judgments.

[0049] To further balance exploration and exploitation, a Softmax strategy is incorporated to optimize action selection. The Softmax strategy calculates the probability of selecting each action based on its Q-value (i.e., expected reward). Through the Softmax strategy, the agent can select actions probabilistically based on the Q-value of each action in the current environment, rather than always choosing the action with the highest Q-value. This avoids the agent getting trapped in local optima, allowing for the exploration of more action space in the early stages.

[0050] By comprehensively considering the reward function and the Softmax policy, the agent can select appropriate actions based on the current state in a dynamically changing domain environment. By continuously adjusting the policy, it can achieve the optimal penetration testing path and maximize the cumulative reward.

[0051] The final reward function is defined as the cumulative reward generated by the set of actions that change control, minus the reward value generated by all actions.

[0052] Step 3: Based on the environment space, state space, action space, and reward function, construct a reinforcement learning agent and train it to find the best vulnerability exploitation path and build a domain penetration testing model. During the training process, an adversarial simulation sandbox is introduced to verify the success rate and detection risk of candidate penetration paths. Strategies with high detection costs or high risk of misoperation are screened out until the strategies meet the performance and security constraints to form a deployable penetration testing model.

[0053] Specifically, based on step 2, a reinforcement learning agent is constructed and trained to find the optimal vulnerability exploitation path, and a domain penetration testing model is built, such as... Figure 3 As shown.

[0054] An adversarial simulation sandbox is introduced during training to model defense responses. The success rate and detection risk of candidate penetration paths are verified within the sandbox, and strategies with high detection costs or high risk of misoperation are eliminated until the strategies meet performance and security constraints, forming a deployable penetration testing model. Specifically, based on a defined environment space, state space, action space, and reward function, a reinforcement learning agent is initialized using a Deep Q-Network (DQN) algorithm. The Q-Network is defined and initialized, and its structure and parameters are set. The learning rate, discount factor, and other training hyperparameters of the agent are configured.

[0055] In the initial stages of training, the agent randomly selects actions to comprehensively explore the environment. The collected training data is stored in an experience replay buffer. Training data is extracted from this buffer through mini-batch random sampling to break data correlations and improve training effectiveness. For each mini-batch, the Q-value of the current action and the target Q-value are calculated.

[0056] The target Q-value is the current reward plus a discount to the maximum Q-value for the next state. During training, the strategy is validated in an isolated sandbox before being applied to the real environment: the sandbox simulates the operating environment of the target domain and common defenses (such as detection alerts, response delays, etc.) to experiment with each candidate penetration path to estimate the success rate and the risk of detection; if a path shows a high detection probability or may cause serious side effects in the sandbox, it is marked as high-risk and removed from the candidate set or downgraded to require manual review; the test results in the sandbox are also fed back as training samples to experience replay, helping the agent learn strategies to avoid high detection costs, thereby improving the security and usability of the final model in the real environment.

[0057] In order to enable the strategy to have an inherent "safety awareness" from the training stage, the detection probability and the cost of misoperation estimated in the sandbox are incorporated into the learning objective in mathematical form: it can be used as an additional penalty for the total reward, or as a constraint that must be met.

[0058] Introducing a term representing risk cost into the value function or objective function, or imposing constraints on detection probabilities above a threshold during policy optimization, allows the optimization process to simultaneously balance benefits and security costs.

[0059] This constrained learning mechanism is closely integrated with sandbox validation: the detection and destruction risks of the policy on different paths are estimated periodically using sandbox experiments, and these estimates are used to update the constraint parameters or risk item weights, so that the policy can maximize the task reward and meet the preset safety acceptability criteria when it converges.

[0060] This design ensures that high-risk paths are not eliminated only after training, but are actively suppressed or transformed into candidates requiring additional verification during the learning process, thus more thoroughly incorporating safety into the agent's decision-making process.

[0061] Subsequently, a Softmax strategy is used to select actions. The probability of each action being selected is calculated using the Softmax strategy, and actions are selected based on this probability to ensure that the agent can explore different vulnerability exploitation paths.

[0062] Repeat the steps of training data collection, experience replay, and Q-value update until the predetermined number of training iterations or convergence criteria are reached. Monitor the cumulative reward changes during training to determine the agent's learning performance and ensure it can effectively select the optimal exploit path.

[0063] A well-trained agent can adaptively select the best exploitable vulnerability and generate a penetration testing path that meets the penetration testing objectives and maximizes cumulative rewards. Deploying the trained agent in a real-world penetration testing system allows it to adaptively adjust penetration testing strategies based on dynamic changes in the target domain environment, achieving a comprehensive penetration testing evaluation of the target domain.

[0064] Step 4: Deploy the deployable penetration testing model at a controlled starting point in the target domain. During actual execution, monitor environmental changes in real time, adaptively select and execute the optimal vulnerability exploitation action according to the current state, and transmit the execution results back in real time for online updates. When the detection rate or risk exceeds a preset threshold, trigger the model rollback to maintain the security boundary.

[0065] Specifically, based on step 3, according to the dynamic changes in the domain environment, the agent adaptively selects the best exploitable vulnerability and exploits it to generate a penetration testing path that satisfies the penetration testing objective and maximizes the cumulative reward, such as... Figure 4 As shown, the execution results are fed back in real time for online updates: immediate model rollback is triggered to maintain security boundaries, thereby continuously generating penetration test paths that maximize cumulative expected returns and satisfy detection requirements in a dynamic domain environment. Specifically: Real-time monitoring of the domain environment, continuously monitoring the dynamic changes of the target domain environment, including host status, service status, network topology changes, and the emergence of new vulnerabilities.

[0066] The environment space and state space are updated periodically or when significant changes are detected to ensure that the environmental information perceived by the agent is always accurate. The state space is dynamically adjusted based on the latest environmental information obtained from real-time monitoring to ensure that the agent can promptly perceive changes in the environment.

[0067] New state information, such as newly added hosts, services, and vulnerabilities, is re-encoded to ensure the integrity and accuracy of the state space.

[0068] In dynamically changing environments, the Softmax strategy continues to be used for action selection, calculating the probability of selecting each action based on the latest Q-value and the current temperature parameter. The temperature parameter is adjusted in a timely manner according to environmental changes to balance the agent's exploration and exploitation behaviors, ensuring that the agent can still select the optimal action in new environments.

[0069] Based on its current state and action space, the agent selects the best exploitable vulnerability using a Softmax strategy and immediately executes the exploit action. By successfully exploiting the vulnerability, the agent gains more environmental information and permissions, further enhancing its state space information.

[0070] After executing the exploit, the agent immediately calculates and receives a reward based on a predefined reward function. This reward function considers real-time environmental information, such as the exploit success rate, privilege escalation status, and information gathering progress, ensuring the agent can adjust its strategy promptly.

[0071] The agent updates the Q-value of each action in the current state based on the reward value from real-time feedback, optimizing its behavioral strategy. This continuous iteration of the Q-value update process ensures that the agent's strategy is constantly optimized in the dynamic environment, gradually approaching the optimal vulnerability exploitation path.

[0072] Through multiple iterations and optimizations, the results of each execution are fed back in real time for online updates of strategies and vulnerability success rate estimates. When the detection rate or risk exceeds a preset threshold, the model is immediately rolled back or switched to secure mode to maintain the security boundary. Through this online feedback and rollback mechanism, the agent continuously adjusts its strategies in a dynamic environment and accumulates a large amount of penetration testing experience, gradually generating a penetration testing path that meets the penetration testing objectives and maximizes cumulative rewards.

[0073] This path includes each step of the exploitation action and its corresponding environmental state changes, which can effectively assess and exploit all potential vulnerabilities in the target domain environment.

[0074] Through the above steps, the agent can adaptively select and exploit the best available vulnerability in a dynamically changing domain environment, and ultimately generate a penetration test path that meets the penetration test objectives and maximizes the cumulative reward, thereby achieving a comprehensive penetration test evaluation of the target domain environment and effectively ensuring the security and stability of the domain environment.

[0075] The domain penetration testing method based on adaptive vulnerability exploitation proposed in this invention uses a deep Q-network of reinforcement learning as the core algorithm of the agent to perform penetration testing analysis on dynamically changing domain environments, construct a domain penetration testing model, and realize adaptive generation of penetration testing paths. This method can effectively assess the security status of the domain environment and ensure its security and stability.

[0076] The system also includes one or more processors and memory.

[0077] The memory is used to store operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of the domain penetration testing method based on adaptive vulnerability exploitation described in the foregoing embodiments, especially... Figure 1 The flowchart of the method is shown.

[0078] Other aspects disclosed in the embodiments of the present invention also propose a computer-readable medium for storing software including instructions executable by one or more computers, which, upon execution, cause the one or more computers to perform operations including the flow of the domain penetration testing method based on adaptive vulnerability exploitation of the foregoing embodiments, particularly... Figure 1 The flowchart of the method is shown.

[0079] It should be recognized that embodiments of the present invention may be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium.

[0080] The method can be implemented using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program in the computer program, wherein the storage medium is configured such that the computer operates in a specific and predefined manner.

[0081] Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system; however, if required, the program can be implemented in assembly or machine language.

[0082] In any case, the language can be either compiled or interpreted.

[0083] Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit.

[0084] The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0085] Furthermore, the method can be implemented in any suitable computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices.

[0086] Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether portable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein.

[0087] Furthermore, machine-readable code, or parts thereof, can be transmitted via wired or wireless networks.

[0088] When such media includes instructions or programs that combine with a microprocessor or other data processor to implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media.

[0089] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A domain penetration testing method based on adaptive vulnerability exploitation, characterized in that, include: Step 1: Passively and actively probe the target domain, collect domain environment information, construct a structured representation from the collected raw information, form a domain environment model, and generate a baseline network topology map and a candidate set of usable points; Step 2: Based on the baseline network topology and the candidate set of exploitable points, establish a multi-dimensional feature representation for each potential vulnerability, and use a probabilistic method to calculate the vulnerability success rate and exploitation cost; Based on the multidimensional feature representation and the vulnerability success rate and exploitation cost, the environment space, state space, action space, and reward function required for the reinforcement learning deep Q-network are defined. Step 2 includes: describing each potential vulnerability with a vector containing several attributes, including vulnerability number, corresponding host and service version, whether there is publicly available exploit code, network reachability, and possible privilege escalation; using a probabilistic method to combine the attribute information to calculate the expected success rate and expected cost of the vulnerability, and outputting the calculation results in the form of expected value and uncertainty; the reward function also comprehensively considers the stability and reliability of the vulnerability exploitation results, and incorporates the vulnerability success rate evaluation results and fluctuations into the reward calculation. Step 3: Based on the aforementioned environment space, state space, action space, and reward function, construct and train a reinforcement learning agent to find the optimal vulnerability exploitation path and build a domain penetration testing model. During training, an adversarial simulation sandbox is introduced to verify the success rate and detection risk of candidate penetration paths. The detection probability and misoperation cost estimated in the sandbox are mathematically incorporated into the learning objective as an additional penalty term for the total reward or as a constraint condition that must be met. A term representing risk cost is introduced into the value function or objective function, or a constraint is imposed on the detection probability above a threshold during policy optimization, so that the optimization process simultaneously balances the benefits and security costs. The detection and destruction risks of the policy on different paths are periodically estimated using sandbox experiments, and the estimated values ​​are used to update the constraint parameters or risk term weights to filter out policies with high detection costs or high misoperation risks until the policy meets the performance and security constraints, forming a deployable penetration testing model. Step 4: Deploy the deployable penetration testing model at a controlled starting point in the target domain. During actual execution, monitor environmental changes in real time, adaptively select and execute the optimal vulnerability exploitation action according to the current state. The execution results are fed back in real time for online updates, and the model is rolled back to maintain the security boundary when the detection rate or risk exceeds a preset threshold. The preset threshold is dynamically adjusted according to the security policy of the domain environment.

2. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 1, characterized in that, Step 1 specifically includes: Using any controlled host within the domain as the starting point for penetration testing, information is collected within the domain, including the domain name, the names of ordinary hosts within the domain, the domain controller name, the names of servers within the domain, the IP information of each host within the domain, the port information of each host within the domain, and the exploitable vulnerabilities of the hosts within the domain. The controlled host communicates with other hosts within the domain to collect login user information and high-privilege user information of other hosts within the domain. The collected information is organized into a standardized record, with each host, service, and user treated as an attributed record. Attributes include operating system, port list, service version, candidate vulnerabilities, and last update time. The communication relationships between hosts are represented by lines to form a network diagram. After data cleaning, a baseline network topology diagram and a candidate list of exploitable points are output. The baseline network topology diagram shows the hosts and their connections, and the candidate list of exploitable points includes vulnerabilities or weaknesses that can be tried on each host and their sources. Based on the formatted information, conduct penetration testing. Based on the obtained port, service and exploitable vulnerability information, find possible penetration testing paths. Refer to the vulnerability database to analyze the discovered exploitable vulnerabilities. Filter exploitable vulnerabilities by combining the vulnerability exploitation success rate and the expected reward after successful vulnerability exploitation, and analyze to find the best vulnerability exploitation path. The exploitable vulnerabilities include system logic vulnerabilities, service configuration vulnerabilities, and information disclosure vulnerabilities.

3. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 1, characterized in that, Step 2 specifically includes: The collected environmental information is abstracted and classified, and the state space, action space and reward function required for the reinforcement learning deep Q network are defined. Among them, when defining the state space, the exploitable vulnerabilities existing in the domain are classified, and whether the vulnerability is exploitable is encoded as a 0 / 1 state. The environment space includes host information, service information, vulnerability information, network topology, and user information. Host information includes the host's IP address, operating system type and version, and host role. Service information includes the services running on the host, their versions, and a list of open ports. Vulnerability information includes a list of vulnerabilities on each host, the exploit success rate of each vulnerability, and the expected reward. Network topology includes connectivity information between hosts and a network topology diagram. User information includes a list of logged-in users on the host and their permission levels. The information in the environment space is abstracted and encoded, including host status encoding, vulnerability status encoding, and service status encoding. Host status encoding represents the existence status of each host in binary, vulnerability status encoding represents the existence of each vulnerability in binary, and service status encoding represents the existence status of each service in binary. The action space includes scanning actions, information gathering actions, vulnerability exploitation actions, privilege escalation actions, and lateral movement actions. Among them, scanning actions refer to scanning open ports and running services of a specific host; information gathering actions refer to collecting user information and network connection information of the target host; vulnerability exploitation actions refer to attempting to exploit specific vulnerabilities in a specific host and service; privilege escalation actions refer to attempting to escalate the privileges of the current host by exploiting known vulnerabilities; and lateral movement actions refer to attempting to acquire the privileges of other hosts as new controlled hosts. A layered or multi-layered intelligent agent architecture is adopted, using the global attack graph or topology information as the basis for decision-making for upper-layer strategies. The upper layer selects several valuable sub-targets or sub-strategies, and then the lower-layer strategies execute specific scanning, exploitation, or lateral movement operations within a smaller subset of actions. The reward function considers rewards for successful vulnerability exploitation, rewards for information gathering, rewards for privilege escalation, and penalties. Specifically, the reward for successful vulnerability exploitation is a positive reward for successfully exploiting a vulnerability; the reward for information gathering is a positive reward for collecting useful information; the reward for privilege escalation is a positive reward for successfully escalating privileges; and the penalty is a negative reward for performing an ineffective action or failing to exploit a vulnerability. The action selection is optimized by combining the Softmax strategy. The probability of each action being selected is calculated based on the Q value of each action. The final reward function is defined as the cumulative reward generated by the set of actions that change control, minus the reward value generated by all actions.

4. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 1, characterized in that, Step 3 specifically includes: Initialize the reinforcement learning agent using the deep Q-network algorithm. Define and initialize the Q-network, set the network structure and parameters, and configure the agent's learning rate, discount factor, and other training hyperparameters. Through multiple interactions with the simulated domain environment, quadruple data of state, action, reward, and next state are collected. In the early stage of training, the agent randomly selects actions to explore the environment. The collected training data is stored in the experience replay buffer. Training data is extracted from the buffer through mini-batch random sampling. The Q value of the current action and the target Q value are calculated. The weights of the Q network are updated according to the Bellman equation, where the target Q value is the current reward plus the discount value of the maximum Q value of the next state. During training, candidate penetration paths are validated in an isolation sandbox. The isolation sandbox simulates the operating environment and common defenses of the target domain and is used to test each candidate penetration path to estimate the success rate and the risk of detection. If a path shows a high detection probability or may cause serious side effects in the sandbox, it is marked as high risk and removed from the candidate set or downgraded to require manual review. The test results in the sandbox are used as training samples to feed back into the experience replay, helping the agent learn strategies to avoid high detection costs. The detection probability and cost of misoperation estimated in the sandbox are incorporated into the learning objective in mathematical form as an additional penalty term for the total reward or as a constraint condition that must be met; a term representing risk cost is introduced into the value function or objective function, or a constraint is imposed on the detection probability above a threshold during policy optimization, so that the optimization process weighs the benefits and safety costs simultaneously; the detection and destruction risks of the policy on different paths are estimated periodically using sandbox experiments, and the estimates are used to update the constraint parameters or risk term weights, so that the policy can maximize the task reward and meet the preset safety acceptability standard when it converges; The softmax strategy is used to select actions. By adjusting the temperature parameter, the exploration and utilization are balanced to avoid getting trapped in local optima. The probability of each action being selected is calculated using the softmax strategy, and the action is selected based on this probability. Repeat the steps of training data collection, experience playback, and Q-value update until the predetermined number of training iterations or convergence criteria are reached; monitor the cumulative reward changes during the training process to determine the learning performance of the agent; After thorough training, a deployable penetration testing model is formed.

5. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 1, characterized in that, Step 3 specifically includes: Real-time monitoring of the domain environment, continuously monitoring the dynamic changes of the target domain environment, including host status, service status, network topology changes and information on the emergence of new vulnerabilities; periodically or when significant changes are detected, updating the environment space and state space to ensure the accuracy of the environmental information perceived by the agent; Based on the latest environmental information obtained from real-time monitoring, the state space is dynamically adjusted and new state information is recoded, including newly added hosts, services, and vulnerabilities. In a dynamically changing environment, the Softmax strategy is used for action selection. The probability of selecting each action is calculated based on the latest Q value and the current temperature parameter. The temperature parameter is adjusted in a timely manner according to changes in the environment to balance the exploration and exploitation behavior of the agent. Based on its current state and action space, the agent selects the best exploitable vulnerability using the Softmax strategy and immediately executes the exploit action. By successfully exploiting the vulnerability, the agent gains more environmental information and permissions, further enhancing its state space information. After executing the exploit action, the agent immediately calculates and receives feedback rewards according to the predefined reward function. The reward function takes into account real-time environmental information, including the exploit success rate, privilege escalation status, and information collection status. Based on the immediate feedback reward value, the agent updates the Q value of each action in the current state and optimizes its behavior strategy. Through multiple iterations and optimizations, the results of each execution are fed back in real time for online updates of strategies and vulnerability success rate estimates. When the detection rate or risk exceeds a preset threshold, the model is immediately rolled back or switched to secure mode to maintain the security boundary. Through online feedback and rollback mechanisms, the agent continuously adjusts its strategies in a dynamic environment, gradually generating penetration testing paths that meet penetration testing objectives and maximize cumulative rewards. Intelligent agents are highly adaptive and can flexibly adjust their strategies in real-time environments to cope with various emergencies and the emergence of new vulnerabilities.

6. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 3, characterized in that, The binary representations of host status codes, vulnerability status codes, and service status codes are as follows: For host status codes, the code is 1 if the host exists, and 0 otherwise; for vulnerability status codes, the code is 1 if the vulnerability exists, and 0 otherwise; for service status codes, the code is 1 if the service exists, and 0 otherwise.

7. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 3, characterized in that, The upper-layer strategy of the hierarchical or multi-layered intelligent agent architecture filters sub-targets or sub-strategies based on the global attack graph or topology information, while the lower-layer strategy performs scanning, vulnerability exploitation, or lateral movement operations within a subset of actions to reduce the number of actions and lower learning complexity.

8. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 3, characterized in that, The temperature parameter of the Softmax strategy can be adjusted to control the degree of exploration in action selection.

9. The domain penetration testing method based on adaptive vulnerability exploitation according to claim 4, characterized in that, The experience replay buffer uses a small-batch random sampling method to extract training data.

Images