Intelligent penetration test system based on deep reinforcement learning and large language model hierarchical decision

Abstract

The invention provides an intelligent penetration testing system based on deep reinforcement learning and large language model hierarchical decision, and belongs to the technical field of penetration testing. The system comprises a macroscopic action decision model, an action library, a state information encoder, a target selector, a downstream task processing model and a tool library, wherein the target selector selects a to-be-tested target according to a detection result of a target network environment; the macroscopic action decision model uses a deep reinforcement learning algorithm to receive a state vector of a target environment, and the state vector is determined according to ATTamp; the CK framework selects and outputs a penetration test task to be completed in the next step from the action library, and the output action serves as a downstream task target and is issued to a downstream task processing model; and the downstream task processing model is a large language model, autonomously generates a solution according to the task description and calls the penetration test tool in the tool library for execution, a feedback result after task execution is observed and recorded by the large language model, and whether the downstream task is completed or not is judged by the large language model.

Description

Technical Field

[0001] The present invention belongs to the technical field of penetration testing, and in particular relates to an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model. Background Art

[0002] Reinforcement learning is a machine learning method that is mainly used to solve problems involving decision making and sequential action selection. Relying on the interaction between the agent and the environment, reinforcement learning can maximize long-term rewards in a dynamic, trial-and-error manner. Deep reinforcement learning combines the advantages of deep learning and reinforcement learning. Through the powerful feature extraction and representation capabilities of deep learning, combined with the decision-making and optimization mechanisms of reinforcement learning, it can learn effective strategies in complex, high-dimensional environments. At present, deep reinforcement learning has demonstrated its outstanding application potential in many fields such as autonomous driving, robot control, recommendation systems, and video games.

[0003] The Large Language Model (LLM) uses a self-supervised pre-training method to automatically learn knowledge from a large amount of general domain unlabeled data, stores knowledge with large model parameters, and has the ability to handle dozens of text tasks. At present, the large language model has general cognitive capabilities. The intelligent agent built on this basis can interact with humans through text, vision, and voice signals, and can use human tools to handle complex tasks. It has been used in many fields such as government affairs and business for intelligent empowerment.

[0004] Penetration testing is an important means to improve information security protection capabilities. It comprehensively detects possible security flaws in the target system, network architecture or application from the attacker's perspective. This test not only involves technical vulnerability scanning and exploitation, but also includes the evaluation of the organization's security strategy. Its core purpose is to simulate real attack scenarios and reveal potential attack paths, thereby helping organizations to promptly discover and fix security vulnerabilities and strengthen the stability of the security defense system. Through this comprehensive penetration test, enterprises can not only verify the effectiveness of existing security measures, but also test their emergency response capabilities when security incidents occur, ensuring that critical business data and important assets can be protected to the maximum extent when facing real threats.

[0005] Existing intelligent penetration testing platforms are designed to train intelligent agents to find the optimal penetration testing path in a specific network environment. Intelligent penetration testing systems represented by AutoPentest combine attack graphs and deep reinforcement learning algorithms. Their training process relies on prior knowledge of the target network environment, making it difficult to apply to unknown network environments. In addition, the process of building attack graphs is complex, making it difficult to achieve end-to-end learning and training in a real network environment. DeepExploit is an open source penetration testing system based on deep reinforcement learning. It uses Metasploit for tool calls at the bottom layer and uses reinforcement learning technology to improve penetration testing efficiency. However, its state space definition has defects, and the trained model is difficult to apply to other targets. PENTESTGPT is an intelligent penetration testing framework built around a large language model. It can autonomously plan and record the penetration testing process and give suggestions for the next step, but it cannot achieve automated penetration testing. Information input and specific execution of actions still require manual implementation by the user. AUTOATTACKER is an intelligent penetration testing system with a two-layer structure. The upper layer uses a large language model to make decisions and gives action suggestions in text form. The lower layer uses retrieval augmentation generation technology (RAG) to generate specific action sequences based on the action suggestions. This solution alleviates the problem of unstable action execution of large language model agents, but it still relies on manual notification of the action goals at each stage of the penetration test and cannot meet the testing tasks in a multi-layer network environment. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model.

[0007] The present invention proposes an intelligent penetration testing system based on deep reinforcement learning and large language model hierarchical decision-making, including: a macro action decision model, an action library, a state information encoder, a target selector, a downstream task processing model, and a tool library. Among them: the target selector selects the target to be tested according to the detection result of the target network environment; the macro action decision model uses a deep reinforcement learning algorithm to receive the state vector of the target environment, and selects and outputs the penetration test task to be completed in the next step from the action library according to the ATT&CK framework, and the output action is published to the downstream task processing model as the downstream task target; the downstream task processing model is a large language model, which autonomously generates a solution according to the task description and calls the penetration test tool in the tool library for execution. The feedback results after the task is executed are observed and recorded by the large language model, and it determines whether the downstream task is completed; If completed, the action execution result is input into the state information encoder and converted into an updated state vector and reward signal; If not completed, the feedback results will be input into the large language model to support the next round of solution generation until the downstream task is completed or the maximum number of interactions is reached.

[0008] Furthermore, the workflow of the system specifically includes: Step 101: Execute target network survival detection and initialize the target queue to be tested; Step 102: Determine whether the queue is empty; wherein: Determine whether the current target queue to be tested is empty. If it is empty, the system ends the operation, otherwise it goes to step 103; Step 103: The target selector selects a target host and initializes the state space; Step 104: Determine whether the maximum number of interactions has been reached; wherein: According to the set threshold, determine whether the number of actions currently performed on the target has reached the maximum number of interactions. If the maximum number of interactions has been reached, proceed to step 102; otherwise, proceed to step 105; Step 105: Execute macro action decision based on deep reinforcement learning algorithm; Step 106: The large language model outputs a micro decision plan based on the task description; Step 107: Call related tools to perform actions based on the output of the large language model, and record feedback information; Step 108: Determine whether the task is completed; If completed, proceed to step 111, otherwise proceed to step 109; Step 109: Determine again whether the maximum number of interactions has been reached; wherein: According to the set threshold, it is determined whether the number of actions currently performed on the target has reached the maximum number of interactions. If the maximum number of interactions has been reached, the process proceeds to step 111; otherwise, the process proceeds to step 110; Step 110: Integrate the feedback information into the task description; Integrate the feedback information into the task description and proceed to step 106 as a basis for the large language model to further complete downstream tasks; Step 111: Obtain action execution result feedback for updating the target state vector in step 112; Step 112: Update the target host state space; Step 113: Determine whether to obtain the target host authority; wherein: Determine whether to obtain the target host authority according to the execution result of the action. If the authority has been obtained, proceed to step 115; otherwise, proceed to step 114; Step 114: The deep reinforcement learning model performs strategy update; Step 115: Update the target queue to be tested; After obtaining the target host authority, the system continues to use survivability scanning on the controlled host to discover surviving nodes connected to the host, and adds the newly discovered surviving nodes to the target queue to be tested.

[0009] Further, in step 101: during the system startup phase, a list of target hosts to be tested is initialized; based on the provided network address range, a network survivability scan is performed using the ICMP protocol and the ARP protocol to identify active hosts; after successful identification, the IP address of the active host is included in the queue of target hosts to be tested.

[0010] Furthermore, in step 103: the target selector adopts a depth-first search strategy to select a target IP from the queue to be tested, and then initializes the state space of the target; the state space covers the access control status of the target host, the set of open ports, the list of active services, the type of operating system, Web fingerprint data, and the sequence of operations performed on the target; the state space is defined as a one-dimensional array of fixed length, and all parameters are set to 0 during the initialization phase.

[0011] Further, in step 105: the encoded state vector of the state space of the target host is input into the macro action decision model, and the macro action of the next step is output; the macro action decision model adopts a reinforcement learning algorithm, and the reinforcement learning algorithm uses a discrete action space. The output of the macro action decision model is the selection probability of each action, where the action corresponding to the maximum probability is the macro action to be executed in the next step.

[0012] Further, in step 106: the macro action is input into the large language model as the downstream task goal, the system generates task description prompt words by using prompt engineering, and inputs the task description prompt words into the large language model, guiding the large language model to output the solution of the downstream task based on the penetration testing tool in the tool library.

[0013] Further, in step 107: relevant penetration testing tools are called according to the solution output by the large language model, and the obtained feedback information is recorded, including the type of tool currently used, whether the execution is successful, and the target machine feedback information.

[0014] Further, in step 111: Get the feedback results of the large language model. The feedback results are in JSON format, and the various information contained in it is stored in text form, including target open port information and vulnerability test task execution result information; At the same time, the system performs the calculation of reward signals at this stage. Reward signals refer to the immediate feedback from the environment to the reinforcement learning algorithm for performing a certain action, which is a factor used to guide behavior during the reinforcement learning training process. pass Compute the reward signal, where r Indicates reward, is at the time step t Actions taken, is at the time step t The feedback information obtained, Represents a reward signal; Through the feedback information obtained Positive value and action The difference between the costs of the two is the final reward value, which is calculated as follows: ,in, Indicates the positive value obtained after obtaining feedback information. Indicates execution of the current action a the cost; The result feedback of the action execution is used in step 112 to update the target state vector.

[0015] Furthermore, in step 112: the current state vector is updated according to the feedback information of the action execution result; the embedding model of the state space is trained using the bidirectional pre-trained encoder RoBERTa based on the Transformer structure, and the original text information of the action feedback is input into RoBERTa to output a dense vector of fixed length. The dense vector reflects the semantic relationship between the corresponding texts in the vector space, so that the reinforcement learning algorithm describes the differences between different environments.

[0016] Further, in step 114: The DDQN algorithm is used to train the macro action decision model, which uses two neural networks with the same structure but different parameters, one is an online network for selecting actions, and the other is a target network for evaluating actions; When updating the strategy, experience samples are randomly drawn from the experience replay pool; for each experience sample, the online network is used to select the next state The action with the highest Q value , , argmax Represents the maximum value index, Q online represents the Q value predicted by the online network; use the target network to evaluate Take action Q value, target Q value Q target The calculation formula is:

[0017] in, It is a discount factor used to weigh immediate rewards against future rewards. Done indicates whether the task is completed. If it is completed, it takes 1, and if it is not completed, it takes 0. For each sample, the Q value predicted by the online network Q online and target Q value Q target The difference between them is used to calculate the loss function L:

[0018] Where N represents the total step length; The gradient of the loss function with respect to the online network parameters is calculated by the back-propagation algorithm, and the parameters of the online network are updated using gradient descent to reduce the difference between the predicted Q value and the target Q value.

[0019] In summary, the present invention performs state space representation based on text embedding technology, performs coarse-grained decision-making based on deep reinforcement learning, performs fine-grained decision-making based on a large language model and realizes automatic calling of penetration testing tools, and can complete automated penetration testing tasks in diversified target machine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of system composition according to an embodiment of the present invention; Figure 2 4 is a system working flow chart according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] The present invention provides an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model. Figure 1As shown, the system includes a macro action decision model, an action library, a state information encoder, an action selector, a downstream task processing model and a tool library. Among them, the target selector selects the target to be tested according to the detection results of the target network environment. The macro action decision model uses a deep reinforcement learning algorithm, which can receive the state vector of the target environment, and select and output the next penetration test task to be completed from the action library according to the ATT&CK framework. The output action is published to the downstream task processing model as the downstream task target. The downstream task processing model is a large language model, which can autonomously generate solutions according to the task description and call the penetration testing tool for execution, such as Figure 1 The feedback after the task is completed is observed and recorded by the large language model (such as Figure 1 The observation part in the middle) determines whether the downstream task is completed. If completed, the action execution result is input into the encoder and converted into an updated state vector and reward signal. If not completed, the feedback result is input into the large language model to support the next round of solution generation until the downstream task is completed or the maximum number of interactions is reached.

[0024] The operation process of the system includes the following steps (such as Figure 2 shown): Step 101: Detect the target network viability and initialize the target queue to be tested.

[0025] During the system startup phase, the list of target hosts to be tested is initialized. Based on the provided network address range, network survivability scans are performed using ICMP and ARP protocols to identify active hosts. After successful identification, the IP addresses of active hosts are added to the queue of target hosts to be tested.

[0026] Step 102: Determine whether the queue is empty.

[0027] It is determined whether the current target queue to be tested is empty. If it is empty, the system ends the operation, otherwise it goes to step 103.

[0028] Step 103: The target selector selects a target host and initializes the state space.

[0029] The target selector uses a depth-first search strategy to select a target IP from the queue to be tested, and then initializes the state space of the target. The state space covers the target host's access control status, open port set, active service list, operating system type, Web fingerprint data, and the sequence of operations performed on the target. The state space is defined as a one-dimensional array of fixed length, and all parameters are set to 0 during the initialization phase.

[0030] Step 104: Determine whether the maximum number of interactions has been reached.

[0031] According to the set threshold, it is determined whether the number of actions currently performed on the target has reached the maximum number. If it has reached the maximum number, the process proceeds to step 102, otherwise, the process proceeds to step 105.

[0032] Step 105: Macro action decision based on deep reinforcement learning algorithm.

[0033] The encoded state vector of the target host's state space is input into the macro action decision model (reinforcement learning model) and outputs the next macro action. In this system, the reinforcement learning algorithm uses a discrete action space, and the model outputs the selection probability of each action, where the action with the highest corresponding probability is the macro action to be executed next. The reinforcement learning algorithm makes coarse-grained decisions, and the action it outputs is a penetration testing method in the ATT&CK framework.

[0034] Step 106: The large language model outputs a micro decision plan based on the task description.

[0035] Macro-actions are input into the large language model as downstream task goals. The system uses prompt engineering methods to generate task description prompt words and inputs them into the large language model, guiding the large language model to output solutions to downstream tasks based on existing penetration testing tools in the tool library.

[0036] Step 107: Based on the output of the large language model, relevant tools are called to perform actions and feedback information is recorded.

[0037] According to the solution output by the large language model, relevant penetration testing tools are called and the feedback information obtained is recorded, including the type of tool currently used, whether it is successfully executed, and the information returned by the target machine.

[0038] Step 108: Determine whether the task is completed.

[0039] The large language model determines whether the current downstream task is completed based on the recorded feedback information. If it is completed, the process proceeds to step 111 , otherwise, the process proceeds to step 109 .

[0040] Step 109: Determine whether the maximum number of interactions has been reached.

[0041] According to the set threshold, it is determined whether the number of actions currently performed on the target has reached the maximum number. If the maximum number has been reached, the process proceeds to step 111, otherwise, the process proceeds to step 110.

[0042] Step 110: The feedback information is integrated into the task description.

[0043] The feedback information contains various information about the execution of the action, which is integrated into the task description through the prompt engineering method and enters step 106, serving as the basis for the large language model to further complete the downstream tasks. Step 111: Obtain action execution result feedback.

[0044] After the action is executed, the feedback result organized by the large language model will be obtained. The feedback result itself is in JSON format, and the various information contained in it is in the form of text, such as the target open port information, vulnerability test task execution result information, etc. At the same time, the system will also calculate the reward signal at this stage. The reward signal is the immediate feedback of the environment on the execution of an action by the reinforcement learning model, and is the key information used to guide behavior during the reinforcement learning training process. Use the formula Compute the reward signal, where r is the function for calculating the reward, is the time step t Actions taken, is the feedback information obtained. In this system, the feedback information obtained Value and Action The final reward signal is obtained by subtracting the cost of itself. The specific reward function is: ,in, Indicates the positive value obtained after obtaining the target host information. Indicates execution of an action a The result feedback of the action execution is used in step 112 to update the target state vector.

[0045] Step 112: Update the target host state space The system updates the current state vector based on the feedback information of the action execution results. The system uses a bidirectional pre-trained encoder RoBERTa based on the Transformer structure to train the embedding model of the state space. The original text information of the action feedback is input into RoBERTa to output a dense vector of fixed length. These vectors can reflect the semantic relationship between the corresponding texts to a certain extent in the vector space, so that the reinforcement learning model can understand the differences between different environments, thereby improving its generalization ability.

[0046] Step 113: Determine whether to obtain the target host authority Determine whether to obtain the target host authority based on the execution result of the action. If the authority has been obtained, proceed to step 115; otherwise, proceed to step 114.

[0047] Step 114: Deep reinforcement learning model performs policy update The system uses the Double DQN (DDQN) algorithm to train the macro action decision model. DDQN is an algorithm used to solve the overestimation problem in reinforcement learning. It is an improved version of the Deep Q-Network (DQN) algorithm. DDQN uses two neural networks with the same structure but different parameters: one for selecting actions (online network) and the other for evaluating these actions (target network). When updating the strategy, a batch of experience samples (mini-batch) must be randomly drawn from the experience replay pool. For each sample, the online network is used to select the next state. The action with the highest Q value is , then, use the target network to evaluate Take this action The target Q value is calculated as ,in, is a discount factor that weighs immediate rewards against future rewards. For each sample, the difference between the Q value predicted by the online network and the target Q value is calculated and the formula is used Calculate the loss function. Finally, the gradient of the loss function with respect to the online network parameters is calculated through the back propagation algorithm and the parameters of the online network are updated using gradient descent to reduce the difference between the predicted Q value and the target Q value.

[0048] Step 115: Update the target queue to be tested After obtaining the target host authority, the system continues to use survivability scanning on the controlled host to discover surviving nodes connected to the host, and adds the newly discovered surviving nodes to the target queue to be tested.

[0049] In summary, the present invention constructs an intelligent penetration testing framework with hybrid decision-making of deep reinforcement learning and large language models, which can independently complete penetration testing tasks, use deep reinforcement learning algorithms to complete upstream coarse-grained decisions, and use large language models to complete fine-grained execution of downstream tasks. In terms of deep reinforcement learning algorithms, the present invention uses RoBERTa's text embedding model as an encoder to encode environmental state information stored in text form into a one-dimensional vector input into a macro action selection model (reinforcement learning model), and performs learning and training for selected test targets based on the macro action selection model of the DDQN algorithm. In terms of large language model decision-making, the present invention completes the processing of downstream tasks by constructing an automated framework of plan-observation-action, and guides the large language model to call various penetration testing tools in the tool library through the method of prompt engineering to assist in the completion of downstream tasks. The present invention uses the RoBERTa encoder to represent environmental information, which can reflect its corresponding semantic relationship, so that the macro action selection model can make similar decisions for similar target environments, and can realize strategy migration in multiple scenarios. The introduction of a large language model makes it easier for it to use rich prior knowledge to call various tools to solve downstream tasks. It not only greatly reduces the action space of reinforcement learning, but also enhances the processing capability of tasks related to semantic understanding, and effectively solves the problem that models learned and trained in simulated environments are difficult to apply to real network scenarios.

[0050] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, several variations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the attached claims.

Claims

1. An intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of large language models, characterized in that: The system includes: a macro action decision model, an action library, a state information encoder, a target selector, a downstream task processing model, and a tool library; wherein: The target selector selects the target to be tested according to the detection result of the target network environment; The macro action decision model uses a deep reinforcement learning algorithm to receive the state vector of the target environment, and selects and outputs the next penetration test task to be completed from the action library according to the ATT&CK framework. The output action is published to the downstream task processing model as the downstream task target; The downstream task processing model is a large language model, which autonomously generates solutions based on the task description and calls the penetration testing tools in the tool library to execute them. The feedback results after the task is executed are observed and recorded by the large language model, which determines whether the downstream task is completed; If completed, the action execution result is input into the state information encoder and converted into an updated state vector and reward signal; If not completed, the feedback results will be input into the large language model to support the next round of solution generation until the downstream task is completed or the maximum number of interactions is reached.

2. According to claim 1, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: The workflow of the system specifically includes: Step 101: Execute target network survival detection and initialize the target queue to be tested; Step 102: Determine whether the queue is empty; wherein: Determine whether the current target queue to be tested is empty. If it is empty, the system ends the operation, otherwise it goes to step 103; Step 103: The target selector selects a target host and initializes the state space; Step 104: Determine whether the maximum number of interactions has been reached; wherein: According to the set threshold, determine whether the number of actions currently performed on the target has reached the maximum number of interactions. If the maximum number of interactions has been reached, proceed to step 102; otherwise, proceed to step 105; Step 105: Execute macro action decision based on deep reinforcement learning algorithm; Step 106: The large language model outputs a micro decision plan based on the task description; Step 107: Call related tools to perform actions based on the output of the large language model, and record feedback information; Step 108: Determine whether the task is completed; If completed, proceed to step 111, otherwise proceed to step 109; Step 109: Determine again whether the maximum number of interactions has been reached; wherein: According to the set threshold, it is determined whether the number of actions currently performed on the target has reached the maximum number of interactions. If the maximum number of interactions has been reached, the process proceeds to step 111; otherwise, the process proceeds to step 110; Step 110: Integrate the feedback information into the task description; Integrate the feedback information into the task description and proceed to step 106 as a basis for the large language model to further complete downstream tasks; Step 111: Obtain action execution result feedback for updating the target state vector in step 112; Step 112: Update the target host state space; Step 113: Determine whether to obtain the target host authority; wherein: Determine whether to obtain the target host authority according to the execution result of the action. If the authority has been obtained, proceed to step 115; otherwise, proceed to step 114; Step 114: The deep reinforcement learning model performs strategy update; Step 115: Update the target queue to be tested; After obtaining the target host authority, the system continues to use survivability scanning on the controlled host to discover surviving nodes connected to the host, and adds the newly discovered surviving nodes to the target queue to be tested.

3. According to claim 2, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: In step 101: during the system startup phase, a list of target hosts to be tested is initialized; based on the provided network address range, a network survivability scan is performed using the ICMP protocol and the ARP protocol to identify active hosts; After successful identification, the IP address of the active host is included in the queue of target hosts to be tested.

4. According to claim 3, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: In step 103: the target selector adopts a depth-first search strategy to select a target IP from the queue to be tested, and then initializes the state space of the target; the state space covers the access control state of the target host, the set of open ports, the list of active services, the type of operating system, the Web fingerprint data, and the sequence of operations executed for the target; The state space is defined as a one-dimensional array of fixed length, and all parameters are set to 0 during the initialization phase.

5. According to claim 4, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: In step 105: the encoded state vector of the state space of the target host is input into the macro action decision model, and the next macro action is output; the macro action decision model adopts a reinforcement learning algorithm, and the reinforcement learning algorithm uses a discrete action space. The output of the macro action decision model is the selection probability of each action, where the action corresponding to the maximum probability is the macro action to be executed in the next step.

6. According to claim 5, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: In step 106: the macro action is input as the downstream task goal into the large language model, the system generates task description prompt words by using prompt engineering, and inputs the task description prompt words into the large language model, guiding the large language model to output the solution of the downstream task based on the penetration testing tool in the tool library.

7. According to claim 6, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: In step 107: the relevant penetration testing tool is called according to the solution output by the large language model, and the feedback information obtained is recorded, including the type of tool currently used, whether the execution is successful, and the target machine feedback information.

8. According to claim 7, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: In step 111: Get the feedback results of the large language model. The feedback results are in JSON format, and the various information contained in it is stored in text form, including target open port information and vulnerability test task execution result information; At the same time, the system performs the calculation of reward signals at this stage. Reward signals refer to the immediate feedback from the environment to the reinforcement learning algorithm for performing a certain action, which is a factor used to guide behavior during the reinforcement learning training process. pass Compute the reward signal, where r Indicates reward, is at the time step t Actions taken, is at the time step t The feedback information obtained, Represents a reward signal; Through the feedback information obtained Positive value and action The difference between the costs of the two is the final reward value, which is calculated as follows: ,in, Indicates the positive value obtained after obtaining feedback information. Indicates execution of the current action a the cost; The result feedback of the action execution is used in step 112 to update the target state vector.

9. According to claim 8, an intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model is characterized in that: In step 112: the current state vector is updated according to the feedback information of the action execution result; the embedding model of the state space is trained using the bidirectional pre-trained encoder RoBERTa based on the Transformer structure, and the original text information of the action feedback is input into RoBERTa to output a dense vector of fixed length. The dense vector reflects the semantic relationship between the corresponding texts in the vector space, so that the reinforcement learning algorithm can describe the differences between different environments.

10. The intelligent penetration testing system based on deep reinforcement learning and hierarchical decision-making of a large language model according to claim 9 is characterized in that: In step 114: The DDQN algorithm is used to train the macro action decision model, which uses two neural networks with the same structure but different parameters, one is an online network for selecting actions, and the other is a target network for evaluating actions; When updating the strategy, experience samples are randomly drawn from the experience replay pool; for each experience sample, the online network is used to select the next state The action with the highest Q value , , argmax Represents the maximum value index, Q online represents the Q value predicted by the online network; Use the target network to evaluate Take action Q value, target Q value Q target The calculation formula is: ， in, It is a discount factor used to weigh immediate rewards against future rewards. Done indicates whether the task is completed. If it is completed, it takes 1, and if it is not completed, it takes 0. For each sample, the Q value predicted by the online network Q online and target Q value Q target The difference between them is used to calculate the loss function L: ， Where N represents the total step length; The gradient of the loss function with respect to the online network parameters is calculated by the back-propagation algorithm, and the parameters of the online network are updated using gradient descent to reduce the difference between the predicted Q value and the target Q value.

Images