Reinforcement learning-based windows domain information scanning and penetration testing path planning method

Abstract

The application discloses a Windows domain information scanning and penetration testing path planning method based on reinforcement learning, which comprises the following steps: detecting information of a target Windows domain environment, mapping a network topology of the target Windows domain environment, establishing a connection with a target machine by using an LDAP protocol and the like, and collecting relevant information in the domain; collecting and sorting the detected relevant information and abstracting the same, using a Markov decision process to solve a multi-stage sequence decision problem, realizing environment simulation, constructing a reinforcement learning intelligent agent, and generating a penetration testing path by interacting with the simulated environment for training, adjusting according to a reward strategy, finding an optimal penetration testing path by accumulating rewards, constructing a Windows domain penetration testing model, and finally generating the optimal penetration testing path. The application realizes the collection of information in the Windows domain and the planning of the penetration testing path, and finally achieves the evaluation and maintenance of the security of the Windows domain environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine learning and information security, and specifically relates to a Windows domain information scanning and penetration testing path planning method based on reinforcement learning. BACKGROUND

[0002] With the continuous improvement of enterprise informatization, the Windows domain environment has become the mainstream choice for enterprise network architecture due to its convenient centralized management characteristics. However, the complex domain trust relationship, multi-level permission structure and diversified service configuration make the Windows domain environment face serious security challenges. Penetration testing, as a core means of actively discovering security vulnerabilities, its efficiency and intelligence directly affect the construction effect of the security protection system.

[0003] Traditional Windows domain information scanning tools often need to perform specific attack actions to obtain information, which is relatively large and easy to be discovered by firewalls when facing high version operating systems. Traditional penetration testing methods mainly rely on manual detection and experience judgment by security experts, which has problems such as long testing period, high labor cost and difficult to guarantee testing coverage. Although existing automatic tools such as Metasploit, Cobalt Strike, etc. can achieve partial automation testing through pre-set attack modules, but their essence still adopts linear scanning strategy based on rule library, lacking dynamic adaptation ability to complex network environment. The specific performance is: (1) It is easy to fall into local optimization when encountering multi-hop attack path, and cannot effectively identify combined attack vectors across nodes; (2) It lacks self-determination ability when facing dynamic changes of network topology and real-time updates of vulnerability information; (3) It lacks depth correlation analysis of key elements such as group policy objects and active directory hierarchical relationship specific to Windows domain.

[0004] In recent years, some researches have tried to apply machine learning technology to the field of penetration testing, but there are still significant limitations in practical application: supervised learning methods rely on a large amount of labeled data for model training, but the acquisition of real attack scene data is limited by laws and ethics; traditional reinforcement learning algorithms based on Q-learning face dimension disaster problem when dealing with high-dimensional state space, especially in scenarios involving hundreds of domain nodes, the algorithm convergence speed decreases sharply. In addition, existing methods generally lack specific design for Windows domain environment, which is specifically manifested in: (1) The state representation of Windows-specific attack surfaces such as Kerberos authentication and NTLM hash transmission is not fully considered; (2) The permission transfer relationship between domain controllers, member servers and client workstations is not well modeled; (3) There is a lack of dynamic weight evaluation mechanism for key risk points such as group policy preferences and horizontal movement paths.

[0005] The present application relates to a penetration testing system capable of deeply integrating Windows domain environment characteristics and achieving intelligent path planning through autonomous reinforcement learning, to solve the key problems of low path planning efficiency, incomplete attack surface coverage and weak environment adaptability existing in the prior art. SUMMARY

[0006] In order to solve the above problems, the purpose of the present application is to provide a Windows domain information scanning and penetration testing path planning method based on reinforcement learning, to obtain Windows domain environment information with minimum action and static, abstract domain penetration testing as a Markov decision process, train reinforcement learning agents, build a domain penetration testing model, find the optimal penetration testing path by realizing the penetration target and satisfying the maximum cumulative reward, and find the weak link of the domain environment, evaluate the security status of the domain environment.

[0007] In order to achieve the above purpose, the present application is realized by the following technical scheme:

[0008] A Windows domain information scanning and penetration testing path planning method based on reinforcement learning, comprising the following steps:

[0009] Step 1: Detect and scan the target Windows domain, establish a connection with the target host using the LDAP protocol, query the target domain environment information by setting filtering conditions, detect related vulnerabilities of ADCS, delegation, domain controller and Exchange server, and collect information including domain host information, login user information, related service configuration information, live IP information and group policy information, etc.

[0010] Step 2: On the basis of the related information collected in step 1, classify, organize and store the collected environment information and potential vulnerability information, and combine the detected environment information and potential vulnerability information to retain valuable information and screen out worthless information.

[0011] Step 3: Based on step 2, the Windows domain penetration testing process is abstracted as a Markov decision process, and the state space, action space and reward function are defined. The state space is the combination of the state of the host and the overall Windows domain. The state of the host includes login user permissions, host operating system, current control type and special service type, etc. The state of the domain includes whether it is a root domain, current control type, whether there is a non-restricted delegation user, whether there is a dangerous group policy, whether there is a user that does not require Kerberos pre-authentication, and whether there is a user with SPN, etc. The action space includes a series of attack actions on the host, including privilege escalation, vulnerability exploitation and group policy attack, etc. The reward function is based on the current policy to take action, and the immediate reward of the current action is obtained after executing any action;

[0012] Step 4: Based on step 3, build a reinforcement learning agent and train it by interacting with the simulated environment to generate a penetration testing path. Adjust according to the reward policy, accumulate rewards to find the optimal penetration testing path, build a Windows domain penetration testing model, and finally generate the optimal penetration testing path.

[0013] Further, in step 1, an LDAP connection is established through the LDAP protocol, the target domain environment is probed, and the domain information is collected, specifically:

[0014] Under the premise of obtaining the credentials of any host in the target Windows domain, in order to reduce the scanning action and avoid being detected by the target host firewall, an LDAP connection is established through the LDAP protocol, and a filter is set to collect relevant information and exploitable vulnerability information in the domain. The collected domain-related information includes domain name, domain normal host name, domain controller name, domain server name, domain user information, user group information, domain trust relationship information, etc. The probed domain vulnerability information includes related vulnerabilities for ADCS, domain controller, delegation and Exchange server.

[0015] Further, in step 2, based on step 1, the collected environment information and potential vulnerability information are classified, sorted and stored, and valuable information is retained and worthless information is filtered out, specifically:

[0016] The acquired relevant information is processed, combined with the detected possible available vulnerability information and the acquired domain host and other related information, to analyze whether the potential vulnerability has utilization conditions, vulnerability utilization success rate, and the like, to analyze whether the user information, service information and other conditions required for vulnerability utilization exist, and if they exist, the corresponding available vulnerability is stored in the dictionary, and if they do not exist, the vulnerability information is deleted, thereby retaining valuable vulnerability information and domain host and other related information. Finally, a yaml file is generated according to the processed information, which is used for subsequent agent training.

[0017] Further, in step 3, on the basis of step 2, the Windows domain penetration testing process is abstracted as a Markov decision process, and the state space, action space and reward function are defined, specifically:

[0018] Step 3-1: Abstract the Windows domain penetration testing process as a Markov decision process. The Markov decision process is a framework for solving multi-stage sequential decision problems. It is a mathematical model based on state transition probability and reward, used to describe the case where an agent takes a series of decisions in a random environment. The Markov decision process is composed of four tuples, representing the action space, state space, reward function and state transition probability function.

[0019] Step 3-2: Define the action space. Each action in the action space is composed of a three-tuple (type, target, action). Type is 0, representing the action target object as a domain host, and type is 1, representing the action target object as a Windows domain. Target represents the serial number of the target host or Windows domain. Action is the action serial number corresponding to each attack action. The attack actions for the host include privilege escalation, unconstrained delegation, eternal blue, MS14-068, MS08-067, Exchange Relay, Exchange Proxy, MSSQL CLR, Hash Dump, DCSYNC. The attack actions for the domain include CVE-2020-1472, CVE-2021-42287, group policy attack, AS_REP Roasting, Kerberoasting, SIDHistory.

[0020] Step 3-3: Define the state space. The state space is composed of the state space of the domain host and the state space of the Windows domain.

[0021] The state space of the host is described by a two-dimensional array, which respectively represents the host object in the domain as the test target and a certain feature corresponding to the description of the host. The features include identity, current control type, whether to obtain operating system information, special service type, whether the current login user contains local administrator group members, whether there is a high-privilege process in the domain, and whether the high-privilege credential in the domain is obtained. The value of the identity is 0, which represents that the host is a normal domain host, 1 represents that the host is a normal domain server, 2 represents that the host is a normal domain controller, and 3 represents that the host is a root domain controller. The value of the current control type is 0, which represents that the control right of the host has not been obtained, 1 represents that the normal domain user permission of the host is obtained, and 2 represents that the SYSTEM control permission of the host is obtained. The value of the special service type is 0, which represents that the host does not exist special service, 1 represents that the host exists MSSQL database service, and 2 represents that the host exists Exchange mail service.

[0022] The state space of the domain is also described by a two-dimensional array, which respectively represents the domain object as the test target and a certain feature corresponding to the description of the domain. The features include whether it is a root domain, current control type, whether any high-privilege credential of the domain has been obtained, whether there is a non-restricted delegation account, whether there is a dangerous group policy file, whether there is a user who does not need Kerberos pre-authentication, and whether there is a user who has SPN. The state space of the domain is described using 0 / 1 coding, which is used to represent the value of whether the item is found.

[0023] Step 3-4: Define the reward function. The reinforcement learning agent observes the state in the environment, and takes action based on the current policy. After executing any action, the agent will get the immediate reward of the current action according to the reward function. The goal of the agent is to maximize the cumulative reward in the future, so the setting of the reward function can be used to guide the agent to learn the optimal policy. The reward is set in several parts, including the reward setting for the change of host control, the reward setting for the change of domain control, and the reward setting for the collection of credentials.

[0024] Further, in step 4, based on step 3, a reinforcement learning agent is constructed to find the optimal penetration testing path, which is specifically:

[0025] On the basis of the defined state space, action space and reward function, a deep Q network algorithm is used, a Q network is defined and initialized, an environment state is coded into a numerical vector, including current host information, network topology, domain control state and the like. The neural network structure and its parameters are set, including the input layer, hidden layer and output layer of the core architecture, an action mask is set, and the Q value of illegal action is set to negative infinity to filter invalid actions. A double network mechanism is used to update the parameters in real time in the training network for action selection and current Q value calculation. The parameters are synchronized from the training network to the target network at regular intervals for target Q value calculation to stabilize the training process. The synchronization frequency is set. During training, actions are selected according to the greedy algorithm, and during deployment, the action with the maximum Q value is selected.

[0026] During the training process, a simulated Windows domain environment is first constructed, and the training network and the target network are initialized. Secondly, the hyperparameters are set, including the discount factor, the experience replay buffer capacity, the batch size, the initial exploration rate, the decay rate, the learning rate and the like. In a single training cycle, the environment is interacted in a loop, the action is selected and the experience is stored, and the network is updated, the target Q value is calculated, and the like, and finally the training is terminated according to the set termination condition.

[0027] After sufficient training, the agent can select exploit actions based on the current state and action space, obtain more environment information and permissions through successful exploitation of vulnerabilities, and further improve the state space information. After selecting the exploit action, the agent calculates and accepts the reward according to the reward function. The agent updates the Q value of each action under the current state according to the feedback reward value, and optimizes its behavior strategy. In the process of continuously iterating the Q value, the strategy of the agent is continuously optimized, and gradually approaches the optimal penetration testing path.

[0028] Through the above steps, the agent can finally generate a penetration testing path that meets the penetration testing target and maximizes the cumulative reward, realizes comprehensive penetration testing evaluation of the target Windows domain environment with the smallest dynamic and static, and finally achieves the purpose of evaluating and maintaining the security of the Windows domain environment.

[0029] The present application obtains the environment information in the Windows domain with the smallest dynamic and static, abstracts the domain penetration testing into a Markov decision process, trains the reinforcement learning agent, constructs the domain penetration testing model, finds the optimal penetration testing path by realizing the penetration target and satisfying the cumulative reward maximization, and thus finds the weak link of the domain environment and evaluates the security status of the domain environment.

[0030] Compared with the prior art, the present application has the following advantages:

[0031] (1) Using the LDAP protocol to establish an LDAP connection with the target host, and collecting information in the domain by setting a filter. The current common information collection method for domain penetration testing mainly relies on obtaining control of a domain host and confirming the live hosts in the domain by communicating with other domain hosts. This method is prone to be discovered by defense means due to the large domain activity. The method using the LDAP protocol only needs the communication credentials of a domain host without complete control, so it can obtain domain information with minimal activity. The subsequent information collected is used to train the agent, and the whole process can be carried out without being discovered by defense means, which has good concealment.

[0032] (2) Compared with the traditional rule-based static scanning method, this dynamic scanning method implemented through a deep reinforcement learning framework can more timely collect key information of the domain environment and reduce the efficiency reduction caused by manual methods. The deep reinforcement Q network can fully consider various complex domain environments and penetration testing scenarios, can be compatible with new Windows network forms such as Azure AD hybrid architecture and multi-domain trust forest, and can timely respond to changes in network topology, greatly improving the efficiency while ensuring wide applicability.

[0033] (3) When defining the state space, the method fully considers the modeling method for Windows domain characteristics, encodes Kerberos tickets, GPO policies, AD topology and other elements into a reinforcement learning state vector from the dimensions of host state and domain state, fully reflects the status of the entire domain, solves the problem of insufficient extraction of domain environment characteristics in the prior art, and greatly improves the penetration success rate in a real domain environment. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The flowchart of the present application from scanning the domain environment to training the agent to constructing the domain penetration testing model to planning the penetration path.

[0035] Figure 2 The flowchart of the present application for collecting target domain information.

[0036] Figure 3 The structure diagram of information collection, analysis and screening in the present application.

[0037] Figure 4 The structure diagram of the parameters required for constructing the agent in the present application.

[0038] Figure 5 The flowchart of the agent planning the penetration path in the present application. DETAILED DESCRIPTION

[0039] The application will be described in greater detail with reference to the accompanying drawings, in which, for purposes of clarity and for illustration, a number of practical details are shown. It is understood that the practical details can be varied and are not intended to limit the application.

[0040] The application discloses a Windows domain information scanning and penetration testing path planning method based on reinforcement learning, which aims to collect information and plan penetration testing paths for a target Windows domain with minimum actions. The method comprises three modules, namely a domain environment detection module, a domain information processing module and a penetration path planning module. The main steps include: detecting information of the target Windows domain environment, mapping the network topology, connecting with the target machine by using an LDAP protocol to reduce actions during information scanning and avoid being discovered by the firewall of the target host, collecting domain-related information, including domain host information, domain-related service configuration information, domain live IP information and possible exploitable vulnerability information, collecting and sorting the detected information and abstracting the information, using a Markov decision process to solve a multi-stage sequence decision problem, abstracting the Windows domain penetration testing process into a Markov decision process, realizing environment simulation, defining a state space, an action space, a reward function and a state transition probability required for training a deep reinforcement learning DQN network according to the abstracted information, designing a domain penetration testing model based on reinforcement learning and training the model, evaluating behaviors by interacting with the simulation environment according to the collected domain information of the target Windows domain environment, adjusting the strategy according to the reward, gradually reaching an optimal strategy, finally realizing the penetration testing target and giving an optimal penetration testing path. The application adopts a deep reinforcement learning Q network, models the Windows domain penetration testing scene as a Markov decision process, collects and analyzes domain information, constructs a Windows domain penetration testing model based on reinforcement learning, realizes the collection of Windows domain information and the planning of penetration testing paths, and finally achieves the purpose of evaluating and maintaining the security of the Windows domain environment.

[0041] Specifically, a Windows domain information scanning and penetration testing path planning method based on reinforcement learning, as shown in Figure 1 The method comprises the following steps:

[0042] Step 1: probe and scan the target Windows domain, establish a connection with the target host using the LDAP protocol, query the target domain environment information by setting filter conditions, probe related vulnerabilities against ADCS, delegation, domain controller and Exchange server, and collect information including domain host information, login user information, related service configuration information, live IP information and group policy information, etc.

[0043] Step 2: Based on the information collected in step 1, classify, organize and store the collected environment information and potential vulnerability information, and retain valuable information and filter out worthless information based on the detected environment information and potential vulnerability information.

[0044] Step 3: Based on step 2, abstract the Windows domain penetration testing process as a Markov decision process, define state space, action space and reward function. The state space is the state combination of the domain host and the overall Windows domain. The state of the domain host includes login user permission, host operating system, current control type and special service type, etc. The state of the domain includes whether it is a root domain, current control type, whether there is a non-constrained delegation user, whether there is a dangerous group policy, whether there is a user who does not need Kerberos pre-authentication, and whether there is a user who has an SPN, etc. The action space includes a series of attack actions on the host, including privilege escalation, vulnerability exploitation and group policy attack, etc. The reward function is based on the current policy to take action, and the immediate reward of the current action is obtained after executing any action;

[0045] Step 4: Based on step 3, build a reinforcement learning agent and train it by interacting with the simulated environment to generate a penetration testing path, adjust it according to the reward policy, accumulate rewards to find the optimal penetration testing path, build a Windows domain penetration testing model, and finally generate the optimal penetration testing path.

[0046] As shown in Figure 2 Step 1 is specifically:

[0047] Under the premise of obtaining the credentials of any host in the target Windows domain, in order to reduce the scanning action and avoid being detected by the target host firewall, an LDAP connection is established through the LDAP protocol, and a filter is set to collect domain-related information and domain exploitable vulnerability information. The collected domain-related information includes domain name, domain host name, domain controller name, domain server name, domain user information, user group information, domain trust relationship information, etc. The domain vulnerability information detected includes related vulnerabilities against ADCS, domain controller, delegation and Exchange server.

[0048] As Figure 3 shown, step 2 is specifically:

[0049] The obtained relevant information is processed, combined with the detected possible available vulnerability information and the obtained domain host and other related information, to analyze whether the potential vulnerability has utilization conditions, vulnerability utilization success rate, etc. The user information, service information and other conditions required for vulnerability exploitation are analyzed. If the conditions exist, the corresponding available vulnerability is stored in the dictionary. If the conditions do not exist, the vulnerability information is deleted, thereby retaining valuable vulnerability information and domain host and other related information. Finally, a yaml file is generated according to the processed information for subsequent agent training.

[0050] As Figure 4 shown, step 3 is specifically:

[0051] Step 3-1: Abstract the Windows domain penetration testing process as a Markov decision process. The Markov decision process is a framework for solving multi-stage sequence decision problems. It is a mathematical model based on state transition probability and reward, used to describe the case where an agent takes a series of decisions in a random environment. The Markov decision process is composed of four tuples, representing the action space, state space, reward function and state transition probability function.

[0052] Step 3-2: Define the action space. Each action in the action space is composed of a three-tuple (type, target, action). Type is 0, representing the action target object as a domain host, and type is 1, representing the action target object as a Windows domain. Target represents the serial number of the target host or Windows domain. Action is the action serial number corresponding to each attack action. The attack actions for the host include privilege escalation, unconstrained delegation, eternal blue, MS14-068, MS08-067, Exchange Relay, Exchange Proxy, MSSQL CLR, Hash Dump, DCSYNC. The attack actions for the domain include CVE-2020-1472, CVE-2021-42287, group policy attack, AS_REP Roasting, Kerberoasting, SIDHistory.

[0053] Step 3-3: Define the state space. The state space is composed of the state space of the domain host and the state space of the Windows domain.

[0054] The state space of a host is represented by a two-dimensional array, which represents the host object as the target of the test and a certain feature corresponding to the host. The features include identity, current control type, whether to obtain operating system information, special service type, whether the current login user contains a local administrator group member, whether there is a high-privilege process in the domain, and whether a high-privilege credential in the domain is obtained. The value of the identity is 0, which represents that the host is a normal domain host, 1 represents that the host is a normal domain server, 2 represents that the host is a normal domain controller, and 3 represents that the host is a root domain controller. The value of the current control type is 0, which represents that the control right of the host has not been obtained, 1 represents that the normal domain user right of the host is obtained, and 2 represents that the SYSTEM control right of the host is obtained. The value of the special service type is 0, which represents that the host does not exist special service, 1 represents that the host exists MSSQL database service, and 2 represents that the host exists Exchange mail service.

[0055] The state space of a domain is also represented by a two-dimensional array, which represents the domain object as the target of the test and a certain feature corresponding to the domain. The features include whether it is a root domain, a current control type, whether any high-privilege credential of the domain has been obtained, whether there is a non-restricted delegation account, whether there is a dangerous group policy file, whether there is a user who does not need Kerberos pre-authentication, and whether there is a user who has an SPN. The state space of the domain is represented by 0 / 1 coding, which is used to represent the value of whether the item is found.

[0056] Step 3-4: Define the reward function. The reinforcement learning agent observes the state in the environment, and takes action based on the current policy. After executing any action, the agent will obtain the immediate reward of the current action according to the reward function. The goal of the agent is to maximize the cumulative reward in the future, so the setting of the reward function can be used to guide the agent to learn the optimal policy. The reward is set in several parts, including the reward setting for the change of the control right of the host, the reward setting for the change of the control right of the domain, and the reward setting for the collection of the credential.

[0057] As shown in Figure 5 , step 4 is specifically:

[0058] On the basis of the defined state space, action space and reward function, a deep Q network algorithm is used to define and initialize the Q network, encode the environment state into a numerical vector, including current host information, network topology, domain control state and other information. The neural network structure and its parameters are set, including the input layer, hidden layer and output layer of the core architecture, the action mask is set, and the Q value of illegal action is set to negative infinity to filter invalid actions. A double network mechanism is used to update the parameters in real time in the training network for action selection and current Q value calculation. In the target network, the parameters are synchronized from the training network at regular intervals for target Q value calculation to stabilize the training process. The synchronization frequency is set. During training, actions are selected according to the greedy algorithm, and the action with the maximum Q value is selected during deployment.

[0059] During the training process, the Windows domain environment is first constructed, and the training network and target network are initialized. Second, the hyperparameters are set, including the discount factor, the experience replay buffer capacity, the batch size, the initial exploration rate, the decay rate, the learning rate and other parameters. In a single training cycle, the environment is interacted in a loop, the action is selected and the experience is stored, and the network is updated, the target Q value is calculated, and finally the training is terminated according to the set termination condition.

[0060] After sufficient training, the agent can select exploit actions based on the current state and action space, obtain more environment information and permissions by successfully exploiting vulnerabilities, and further improve the state space information. After selecting exploit actions, the agent calculates and accepts rewards according to the reward function. The agent updates the Q value of each action under the current state according to the feedback reward value, and optimizes its behavior strategy. In the process of continuously iterating the Q value, the agent's strategy is continuously optimized, gradually approaching the optimal penetration testing path.

[0061] Through the above steps, the agent can finally generate a penetration testing path that meets the penetration testing goal and maximizes the cumulative reward, achieving comprehensive penetration testing evaluation of the target Windows domain environment with minimal dynamic and static, and ultimately achieving the purpose of evaluating and maintaining the security of the Windows domain environment.

[0062] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for Windows domain information scanning and penetration testing path planning based on reinforcement learning, characterized in that, Comprise the following steps: Step 1: probe and scan the target Windows domain, establish a connection with the target host using the LDAP protocol, query the target domain environment information by setting the filtering conditions, probe the domain vulnerability information, scan and collect the host information, login user information, related service configuration information, live IP information and group policy information in the domain; Step 2: based on the collected Windows domain related information, classify, organize and store the collected environment information and potential vulnerability information, and combine the detected environment information and potential vulnerability information to retain valuable information and exclude worthless information; Step 3: abstract the Windows domain penetration testing process into a Markov decision process, define the state space, action space and reward function; the state space is the state combination of the host in the domain and the whole Windows domain, the state of the host in the domain includes the login user permission, the host operating system, the current control type and the special service type, the state of the domain includes whether it is a root domain, the current control type, whether there is a non-constrained delegation user, whether there is a dangerous group policy, whether there is a user who does not need Kerberos pre-authentication and whether there is a user who has an SPN; The action space includes a series of attack actions for the host, including privilege escalation, vulnerability exploitation and group policy attack; The reward function is based on the current policy to take action, and the immediate reward of the current action is obtained according to the reward function after executing any action; Step 4: build a reinforcement learning agent and train it by interacting with the simulated environment to generate a penetration testing path, adjust it according to the reward strategy, find the optimal penetration testing path by accumulating rewards, build a Windows domain penetration testing model, and finally generate the optimal penetration testing path; Step 4 is specifically: On the basis of the defined state space, action space and reward function, the deep Q network algorithm is used, the Q network is defined and initialized, the environment state is coded into a numerical vector, including the current host information, network topology and domain control state information; set the neural network structure and its parameters, including the input layer, hidden layer and output layer of the core architecture, set the action mask, set the Q value of illegal action to negative infinity to filter invalid actions; adopt double network mechanism, update parameters in training network in real time, used for selecting action and calculating current Q value; in the target network, parameters are synchronized from the training network regularly, used for calculating target Q value, stabilizing the training process, and setting the synchronization frequency, selecting the action with the maximum Q value in the training according to the greedy algorithm, and selecting the action with the maximum Q value in the deployment; In the training process, first, build a simulated Windows domain environment, initialize the training network and the target network; second, set the hyperparameters, including the discount factor, the experience replay buffer capacity, the batch size, the initial exploration rate, the decay rate and the learning rate; in a single training cycle, interact with the environment in a loop, select the executed action and store the experience, and update the network, calculate the target Q value, and finally terminate the training according to the set termination condition; After being fully trained, the intelligent agent selects a vulnerability exploitation action based on the current state and the action space, obtains more environment information and permissions by successfully exploiting the vulnerability, and further improves the state space information; after selecting the vulnerability exploitation action, the intelligent agent calculates and accepts the reward according to the reward function, and updates the Q value of each action under the current state according to the feedback reward value, and optimizes its behavior strategy; In the process of continuously iterating the Q value, the strategy of the intelligent agent is continuously optimized, and gradually approaches the optimal penetration testing path, and the intelligent agent finally generates a penetration testing path that meets the penetration testing target and maximizes the cumulative reward.

2. The method of claim 1, wherein the method is based on reinforcement learning. The step 1 is specifically: On the premise of obtaining the credentials of any host in the target Windows domain, an LDAP connection is established through the LDAP protocol, and a filter is set to collect the relevant information in the domain and the exploitable vulnerability information in the domain; The collected domain-related information includes domain name, domain normal host name, domain controller name, domain server name, domain user information, user group information, domain trust relationship information, and the detected domain vulnerability information includes related vulnerabilities of ADCS, domain controller, delegation and Exchange server.

3. The method of claim 1, wherein the method is based on reinforcement learning. The step 2 is specifically: The obtained relevant information is processed, combined with the detected possible exploitable vulnerability information and the obtained domain host related information, to analyze whether the potential vulnerability has exploitation conditions and the vulnerability exploitation success rate, and analyze whether the user information and service information conditions required for vulnerability exploitation exist, if they exist, the corresponding exploitable vulnerability is stored in the dictionary, if they do not exist, the vulnerability information is deleted, thereby retaining valuable vulnerability information and domain host related information, and finally generating a yaml file according to the processed information for subsequent intelligent agent training.

4. The method of claim 1, wherein the method is based on reinforcement learning. Step 3 is specifically: Step 3-1: Abstract the Windows domain penetration testing process as a Markov decision process, which is a framework for solving multi-stage sequence decision problems, it is a mathematical model based on state transition probability and reward, used to describe the case that an intelligent agent takes a series of decisions in a random environment, the Markov decision process is composed of four tuples, representing the action space, state space, reward function and state transition probability function; Step 3-2: Define the action space; each action in the action space is composed of a triple type, target, action, type is 0, representing the action target object as a domain host, type is 1, representing the action target object as a Windows domain; target represents the serial number of the target host or Windows domain; action is the action serial number corresponding to each attack action, including privilege escalation, unconstrained delegation, eternal blue, MS14-068, MS08-067, Exchange Relay, Exchange Proxy, MSSQL CLR, Hash Dump, DCSYNC for host attack actions; attack actions for domains include CVE-2020-1472, CVE-2021-42287, group policy attack, AS_REP Roasting, Kerberoasting, SIDHistory; Step 3-3: Define the state space; the state space is composed of the state space of the domain host and the state space of the Windows domain; The state space of the host is composed of a two-dimensional array, representing the domain host object as the test target and the corresponding description of the host, and the characteristics include identity, current control type, whether the operating system information is obtained, special service type, whether the current login user contains local administrator group members, whether there is a high-privilege process in the domain, and whether the high-privilege credential in the domain is obtained; The value of identity is 0, representing that the host is a normal domain host, 1 representing that the host is a normal domain server, 2 representing that the host is a normal domain controller, and 3 representing that the host is a root domain controller; The value of the current control type is 0, representing that the control of the host has not been obtained, 1 representing that the normal domain user permission of the host is obtained, and 2 representing that the SYSTEM control permission of the host is obtained; The value of the special service type is 0, representing that the host does not exist special service, 1 representing that the host exists MSSQL database service, and 2 representing that the host exists Exchange mail service; The state space of the domain is composed of a two-dimensional array, representing the domain object as the test target and the corresponding description of the domain, and the characteristics include whether it is a root domain, the current control type, whether it has obtained any high-privilege credential of the domain, whether there is an unconstrained delegation account, whether there is a dangerous group policy file, whether there is a user without Kerberos pre-authentication, and whether there is a user with SPN; The state space of the domain uses 0 / 1 coding to represent whether the item is found or not; Step 3-4: Define the reward function; The reinforcement learning agent observes the state in the environment, and takes action based on the current policy, and obtains the immediate reward of the current action according to the reward function after executing any action, and the goal of the agent is to maximize the cumulative reward in the future, so the setting of the reward function is used to guide the agent to learn the optimal policy, and the reward is set in several parts, including the reward setting for the change of the host control right, the reward setting for the change of the domain control right and the reward setting for the credential collection.

Images