Penetration path planning method, system and device based on llm and dpo
By combining LLM and DPO, the problems of data scarcity and reward design mismatch in automated penetration testing are solved, generating efficient and covert penetration paths that conform to expert experience and improving the penetration testing capabilities of the agent.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-04-21
- Publication Date
- 2026-07-14
AI Technical Summary
Existing automated penetration testing methods have shortcomings in knowledge structure extraction and policy preference alignment, making it difficult for agents to fully learn from expert experience. Furthermore, the reward design is complex and prone to mismatch, making it difficult to generate efficient and covert penetration paths.
An automated information extraction pipeline based on a large language model (LLM) is constructed to transform unstructured text into structured penetration graph trajectories, and a policy is trained using a graph neural network (GNN). Combined with the Direct Preference Optimization (DPO) algorithm, the policy network is optimized to conform to the implicit preferences of human experts.
It achieves high-fidelity extraction of penetration map trajectories from unstructured text, generates optimal penetration paths that meet the needs of real-world scenarios, and improves the agent's state reasoning and path planning capabilities in complex network environments.
Smart Images
Figure CN122394887A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of network security and automated penetration testing technology, and in particular to penetration path planning methods, systems and devices based on LLM and DPO. Background Technology
[0002] With the rapid development of artificial intelligence and big data technologies, the global cybersecurity situation is becoming increasingly severe. The surge in the number of vulnerabilities and the frequent occurrence of advanced persistent threats (APT) attacks have made it difficult for traditional passive defense systems that rely on perimeter protection to cope with increasingly complex and ever-changing attack methods.
[0003] As a proactive defense method, penetration testing can proactively identify potential security risks in a system by simulating the strategies and behaviors of real attackers. However, the traditional manual penetration testing model, which relies on security experts, suffers from problems such as high cost, long testing cycle, limited coverage, and difficulty in standardizing and reproducing the process.
[0004] To improve testing efficiency and scalability, automated penetration testing technology has gradually become a research hotspot. In recent years, academia has attempted to introduce reinforcement learning into penetration path planning tasks, modeling the penetration process as a Markov decision process, allowing the agent to autonomously learn attack strategies through interaction with the environment. Nevertheless, existing reinforcement learning-based penetration testing methods still face the following two key technical bottlenecks in practical deployment: First, there is a lack of high-quality structured training data. Although a large amount of penetration testing expert knowledge exists in unstructured texts such as CTF competition reports and vulnerability analysis articles, such texts are mostly narrative descriptions and cannot accurately reflect the dynamic evolution of states, complex dependencies between entities, and causal exploitation relationships during penetration testing. Existing methods lack effective means to transform unstructured attack logs into machine-understandable, structured, dynamic attack graph trajectories, making it difficult for agents to fully learn and utilize massive amounts of expert experience.
[0005] Secondly, designing a reward mechanism for reinforcement learning is difficult. Penetration testing tasks inherently present two major challenges: sparse rewards and implicit preferences. On the one hand, successful penetration often requires executing long sequences of actions to obtain the final reward, resulting in a lack of effective feedback for the agent in most exploration steps and low learning efficiency. On the other hand, the implicit preferences of human experts in making decisions, such as attack efficiency, operational stealth, and step logic, are difficult to precisely formalize into a reward function. Imperfect reward design can easily lead to "reward hacking," where the agent exploits the flaws in the reward function to obtain high scores, deviating from the fundamental goal of generating the optimal penetration path that is actually efficient and stealthy.
[0006] In summary, existing automated penetration testing methods still have significant shortcomings in both knowledge structure extraction and strategy preference alignment, which limits their reliable application in real-world high-risk network environments. Therefore, there is an urgent need for an intelligent penetration path planning method that can automatically extract expert knowledge and align with implicit human decision-making preferences. Summary of the Invention
[0007] The purpose of this invention is to provide a penetration path planning method, system, and device based on LLM and DPO, in order to solve two core problems in existing automated penetration testing technologies: the difficulty in effectively utilizing unstructured expert knowledge and the difficulty in aligning reinforcement learning strategies with the implicit preferences of human experts in terms of attack efficiency and stealth.
[0008] To achieve the above objectives, this invention provides a penetration path planning method based on LLM and DPO, comprising the following steps: Step S1: Construct an automated information extraction pipeline based on the Large Language Model (LLM) to transform unstructured cybersecurity text into structured dynamic penetration map trajectories and store them in an expert penetration trajectory set. Step S2: Construct a policy network based on graph neural network (GNN), and use the expert penetration trajectory set for supervised fine-tuning of SFT training to obtain a reference policy; Step S3: Construct a trajectory preference dataset and optimize the DPO algorithm based on direct preferences. Using the reference policy as a benchmark, optimize and align the policy network to obtain the target policy that generates the optimal penetration path that conforms to implicit preferences.
[0009] Preferably, step S1 specifically includes: Step S11: Define the dynamic graph ontology, including node type, edge type, and dynamic evolution mechanism; Step S12: Perform semantic-driven attack step temporal segmentation on unstructured network security text to obtain an atomic text fragment sequence; Step S13: Based on the current graph state context, extract the graph incremental update element from each text fragment; Step S14: Perform multidimensional consistency verification and correction on the extraction results; Step S15: Update the graph state according to the extracted incremental update elements, and serialize the state-action pairs into trajectories and store them in the expert penetration trajectory set.
[0010] Preferably, in step S1, the constructed dynamic graph ontology include: Node set Nodes containing host, port, service, vulnerability, user, credential, permission, and file type; edge set It includes directed edges representing structural dependencies, causal utilization, and information relationships; and a dynamic evolution mechanism that defines the dynamic graph state. It evolves incrementally as the penetration action is executed. Node attribute set Edge attribute set .
[0011] Step S2 specifically includes: Construct a graph neural network policy network that takes the dynamic graph state as input and the node selection probability distribution in the graph as output; The graph neural network strategy network is trained under supervision using a set of expert penetration trajectories to obtain a reference strategy with basic expert decision-making capabilities, which serves as the initial penetration path planning strategy.
[0012] Preferably, in step S2, the graph neural network policy network includes: The graph feature encoding layer is used to receive the dynamic graph state, which is processed by a graph attention network to generate node embedding vectors that incorporate global context information. The policy decoding head is used to map the node embedding vectors to scalar values, normalize them using the Softmax function, and output the probability distribution of selecting each node in the graph as the target of the next action.
[0013] Preferably, in step S2, the training loss function is supervised and fine-tuned. Optimization is performed using the negative log-likelihood loss function: ; in, This represents the expected value calculation for the distribution of the dataset; This represents the expert penetration trajectory set; This represents a set of state-action pairs sampled from the dataset. for The dynamic state at any given moment. The target node (attack action) selected by the expert in this state; This indicates that the graph neural network policy is in the input graph state. hour; The goal is to minimize this loss function so that the output distribution of the policy network approximates the expert behavior distribution, and a reference policy is obtained after training. .
[0014] Preferably, in step S3, the trajectory preference dataset ,in, For preferred trajectories, For non-preference trajectories, the construction method is as follows; Based on expert judgment: Multiple different penetration trajectories targeting the same infiltration target are extracted, and security experts judge their merits to form a preference pair. ; Reference strategy-based sampling: using high-fidelity expert penetration trajectories as preference trajectories. Using reference strategy In a simulation environment, different paths are generated as unpreferred trajectories through random sampling. .
[0015] Preferably, in step S3, the DPO algorithm is optimized based on direct preference, using the reference policy as the reference policy. Based on the baseline, the policy network is optimized and aligned to obtain the target policy that generates the optimal penetration path that conforms to implicit preferences, including: Step S31: Refer to the strategy Initialize the parameters to the optimal penetration strategy to be optimized. and from trajectory preference dataset Medium batch sampling preference ; Step S32: Calculate the reference strategy Total log probability of the trajectory The calculation formula is as follows: ; in, This represents a complete penetration test trajectory. Indicates the policy network in state Choose below The logarithmic probability; Step S33, Implicit Reward Estimation: Calculate the implicit reward of the trajectory. The calculation formula is as follows: ; in, Hyperparameters representing the degree of deviation from the control strategy; Step S34: Optimize the loss function by minimizing direct preferences. To update the optimal penetration strategy The parameters are calculated using the following formula: ; in, In the trajectory preference dataset Take the expected value. This represents the Sigmoid activation function, used to map reward differences to probability intervals. This represents the reward difference between the preferred trajectory and the non-preferred trajectory.
[0016] This invention also provides a penetration path planning system based on LLM trajectory extraction and DPO alignment, used to execute the penetration path planning method based on LLM and DPO as described above, including: The information extraction module is used to build an automated information extraction pipeline based on the Large Language Model (LLM), which transforms unstructured cybersecurity text into structured dynamic penetration map trajectories and stores them in an expert penetration trajectory set. The policy network construction and training module is used to construct a policy network based on a graph neural network (GNN), and to perform supervised fine-tuning of the SFT training using an expert penetration trajectory set to obtain a reference policy. The DPO alignment optimization module is used to construct a trajectory preference dataset and optimize the DPO algorithm based on direct preference. Using the reference policy as a benchmark, it optimizes and aligns the policy network to obtain the target policy that generates the optimal penetration path that conforms to implicit preferences. The dynamic graph ontology inspection module is used to define and maintain dynamic graph ontology. The dynamic graph ontology includes node type, edge type and dynamic evolution mechanism. The node set includes nodes of host, port, service, vulnerability, user, credential, permission and file type. The edge set includes directed edges representing structural dependencies, causal exploitation and information association. The consistency verification module is used to perform multi-dimensional consistency verification and correction on the information extraction results; The trajectory serialization and storage module is used to update the graph state based on the extracted incremental update elements and serialize the state-action pairs into trajectories, which are then stored in the expert penetration trajectory set. The graph feature encoding submodule is used to receive the dynamic graph state, process it using a graph attention network, and generate node embedding vectors that incorporate global context information. The policy decoding output submodule is used to map the node embedding vectors to scalar values, normalize them using the Softmax function, and output the probability distribution of selecting each node in the graph as the target of the next action. The preference dataset construction submodule is used to construct the trajectory preference dataset through two methods: expert adjudication and reference strategy sampling. The expert adjudication method extracts multiple different penetration trajectories for the same penetration target and the security experts adjudicate their merits to form preference pairs. The reference strategy sampling method uses high-fidelity expert penetration trajectories as preferred trajectories and uses the reference strategy to generate different paths as non-preferred trajectories in the simulation environment through random sampling. The implicit reward calculation module is used to calculate the implicit reward of the trajectory. The implicit reward is obtained by multiplying the total log probability difference between the trajectory of the policy to be optimized and the reference policy by the hyperparameter of the deviation of the control policy.
[0017] The present invention also provides a computer device including a memory and a processor, the memory being used to store instructions and the processor being used to execute the instructions to implement the method as described above.
[0018] Therefore, the beneficial technical effects of the penetration path planning method, system, and equipment based on LLM and DPO described above are as follows: (1) By constructing an automated information extraction pipeline based on a Large Language Model (LLM) and designing a complete process including intelligent segmentation, context-aware extraction, and multi-dimensional verification, this invention can extract structured dynamic penetration map trajectories from unstructured cybersecurity reports, CTF challenge analyses, and other texts with high fidelity. This directly transforms a large amount of unused expert experience into a high-quality dataset that is machine-readable and can be used for supervised training, fundamentally solving the data bottleneck in learning automated penetration testing strategies.
[0019] (2) By introducing the Direct Preference Optimization (DPO) algorithm and combining it with the constructed trajectory preference dataset (obtained through expert adjudication and reference policy sampling), this invention completely bypasses the step of manually designing complex and mismatch-prone explicit reward functions in the policy learning process. This method can learn directly from human experts' preference judgments on complete attack paths, enabling the optimized policy to internalize and weigh implicit standards that are difficult to formalize, such as attack efficiency and operational stealth, thereby generating an optimal path that better meets the needs of actual attack and defense scenarios.
[0020] (3) By formalizing the knowledge state of penetration testing into a dynamically evolving directed attribute graph, and innovatively defining the action space as the choice on the nodes of this dynamic graph, this invention is naturally adapted to the graph neural network (GNN) architecture. The policy network based on graph attention network (GAT) can automatically learn the dependency and exploitation relationship weights between key entities during the penetration process, thereby learning generalizable high-level attack logic and improving the agent's ability to perform state reasoning and path planning in complex and unknown network environments. Attached Figure Description
[0021] Figure 1 This is a penetration path planning method that integrates LLM graph trajectory extraction and DPO strategy alignment; Figure 2 This is a penetration map trajectory extraction method based on Large Language Model (LLM); Figure 3 Penetration test status diagram Evolution and trajectory Record example; Figure 4 This is a diagram of a policy network architecture based on a graph neural network. Figure 5Supervised fine-tuning (SFT) training based on behavior cloning; Figure 6 To construct a trajectory preference dataset; Figure 7 The optimal penetration strategy based on direct preference optimization train. Detailed Implementation
[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0023] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.
[0024] Example 1 This embodiment details the implementation process of an optimal penetration path planning method that integrates LLM graph trajectory extraction and DPO strategy alignment. For example... Figure 1 As shown, the overall process of this method consists of three cascaded core modules: a graph trajectory extraction module based on a large language model, a reference strategy construction module based on a graph neural network, and a penetration strategy alignment module based on direct preference optimization. The implementation methods of each module are described in detail below.
[0025] 1. Penetration map trajectory extraction module based on large language model.
[0026] This module aims to automatically and faithfully convert unstructured cybersecurity text (such as penetration test reports and CTF (Capture The Flag) solutions) into structured, dynamic penetration testing maps. The specific process is as follows: Figure 2 As shown.
[0027] Step 1: Construct the dynamic graph schema for penetration testing.
[0028] First, define a dynamic evolution graph model for formally representing the state of a penetration test. .
[0029] Node set This includes entity types such as target host, open port, running service, exploitable vulnerability, user account, credential, privilege, and file.
[0030] Each node has attributes , This represents a "node attribute collection": it records detailed information about each node. Example: For the "Host" node, attributes might include operating system type (Windows / Linux), IP address, etc.; for the "Service" node, attributes might include service name (nginx), version number (1.18.0), etc.
[0031] edge set Define the directed logical relationships between entities, mainly including: Structural dependencies: such as a host owning a port (HAS_PORT) and the port running a service (RUNS_SERVICE).
[0032] Causal exploitation relationship: such as service vulnerability (IS_VULNERABLE_TO), vulnerability-based privilege acquisition (GRANTS_SHELL_AS).
[0033] Information associations: such as user reading file (CAN_READ) and file inclusion credentials (CONTAINS).
[0034] Each edge can have attributes. , Represents a "set of edge attributes": such as the tools (nmap, msfconsole) required to connect the edge, the time taken for the action, or the risk weight.
[0035] Dynamic evolution mechanism: The penetration testing process is modeled as an incremental growth process of a graph. Initial state It contains only limited information (such as the target IP). As the attack progresses... The execution of (corresponding to a target node in the diagram) changes the dynamic graph state from... Evolved into For example, adding newly discovered nodes or edges, or updating node / edge attributes. Figure 3 This demonstrates a simplified example of graph state evolution and corresponding trajectory recording.
[0036] Step 2: Sequential segmentation of attack steps based on semantics.
[0037] The complete unstructured cybersecurity report is input into an intelligent report segmenter built on a large language model (such as GPT-4, ChatGLM, etc.). Designed prompts guide the model to identify the chain-of-thought in the report, decomposing the continuous text stream into a list of text fragments according to the attack sequence and atomic operations. Ensure each text fragment Focus on a single core action or key discovery. Indicates the time.
[0038] Step 3: Context-aware graph incremental extraction.
[0039] Design a context-aware information extractor (also based on LLM). For the current text segment to be processed... The extractor simultaneously receives its text content and the current graph state context maintained by the system. Using prompts containing graph ontology definitions and self-correction instructions, the extractor parses the text content and identifies the target node for this step. It also analyzes the graph structure changes caused by the action and generates a set of graph incremental update instructions.
[0040] Step 4: Multidimensional output verification and correction.
[0041] To suppress the "illusion" that LLM might produce, a validator is set up to perform three-dimensional consistency verification on the extraction results: Segmented semantic consistency verification: ensuring that the extracted structured information is faithful to the semantic consistency of the extracted information. The original description.
[0042] Graph ontology consistency check: Ensure that the types and connection rules of newly added nodes and edges conform to the graph ontology constraints predefined in step 1.
[0043] Graph state context consistency check: Ensures that the operation to be performed is within the current dynamic graph state. The following logic is feasible (e.g., the target node of the attack must already exist in the graph).
[0044] If the verification fails, the system triggers a feedback correction mechanism, sending the error message along with... , The data is then fed back to the extractor for reprocessing until it passes.
[0045] Step 5: Dynamic graph state management and trajectory serialization.
[0046] After successful verification, the system will update the current dynamic graph status. The snapshot and the action extracted in this step Pairing to form a single-step trajectory record Subsequently, the graph state is updated according to the incremental update instruction, resulting in... This is then passed as a new context to the next step. After processing all text fragments, the complete time-series trajectory is obtained. Serialize it into JSON Lines format and store it in the expert penetration trajectory set. .
[0047] 2. Reference strategy construction module based on graph neural network.
[0048] This module utilizes extracted expert trajectory data to train a policy network capable of understanding dynamic graph states and making basic attack decisions. Its network architecture is as follows: Figure 4 As shown, the training process is as follows: Figure 5 As shown.
[0049] Step 1: Construct an end-to-end graph neural network strategy architecture.
[0050] Design a dynamic graph state A policy network that takes the node selection probability distribution in the graph as input and outputs it.
[0051] Graph Encoder: Input processing: Receive dynamic graph status The attributes of various heterogeneous nodes are embedded and normalized to convert them into feature vectors of a unified dimension.
[0052] Graph Attention Network (GAT): This layer uses GAT as its backbone network. It aggregates neighbor information through message passing and assigns weights to different edges using an attention mechanism, thereby learning complex relationships between nodes (such as the dependency of a vulnerability on a specific service). The output of this layer is a node embedding vector rich in global context information.
[0053] Policy Head: Define a node-centric action space, where the agent's action is a choice. One of the nodes As the next target for attack or detection.
[0054] Each node of the GAT output is embedded and mapped to a scalar score using a multilayer perceptron (MLP) with shared parameters.
[0055] The scores of all nodes are normalized using the Softmax function, and the action probability distribution is output. , indicating in The probability of selecting each node as the next action.
[0056] Step 2: Supervised fine-tuning (SFT) training based on behavior clones.
[0057] Using expert penetration tracking sets The goal of supervised training of the policy network is to enable the network to mimic the attack sequences of experts.
[0058] Training data: Decompose the trajectory into state-action pairs .
[0059] Loss function: Negative log-likelihood loss is used. ; in, This represents the expected value calculation for the distribution of the dataset. This represents the expert penetration trajectory set. This represents a set of state-action pairs sampled from the dataset. This indicates that the graph neural network policy is applied to the input dynamic graph state. At that time, by minimizing this loss, the policy network can be made to function in a given dynamic graph state. The real actions of prediction experts Maximize the probability.
[0060] Output: After training converges, freeze the model parameters to obtain the reference policy. This strategy possesses the fundamental ability to replicate the successful paths of experts.
[0061] 3. Penetration strategy alignment module based on direct preference optimization.
[0062] This module aims to enable strategies beyond simple imitation, allowing them to learn and align with experts' implicit preferences in areas such as efficiency and concealment. It includes the construction of a preference dataset. Figure 6 ) and DPO training ( Figure 7 Two key parts.
[0063] Step 1: Constructing a trajectory preference dataset ,in For the preferred (better) trajectory, This is a non-preferred (suboptimal) trajectory. It is generated using two strategies: The generation strategy based on expert adjudication: Utilize a large language model pipeline to extract multiple different penetration trajectories targeting the same penetration objective (denoted as...). The quality of the trajectory will be determined by security experts, and the determination will be made accordingly. The advantages and disadvantages.
[0064] Based on reference strategy Sampling generation strategy: The extracted high-fidelity expert penetration trajectory is directly used as the "preference trajectory". Simultaneously, utilizing the pre-trained reference strategy Running in a controlled simulation environment, paths different from expert trajectories are generated using stochastic sampling methods such as temperature sampling, and these are labeled as "non-preferred trajectories". .
[0065] Finally, all collected preference pairs are standardized, and the output is a trajectory preference dataset in JSON Lines format. This provides a data foundation for subsequent training.
[0066] Step 2: Optimize strategies based on DPO.
[0067] The core idea of Direct Preference Optimization (DPO) is to bypass the explicit reward model and directly optimize the policy by comparing the preferences of trajectory pairs.
[0068] Initialization: Refer to the policy The parameters are copied to the optimal penetration strategy to be optimized. .
[0069] Trajectory probability calculation: For a trajectory ,Strategy The total log probability assigned to it is the sum of the log probabilities of each decision step: ; in, This represents a complete penetration test trajectory; Represents the policy network in the dynamic graph state Choose below The logarithmic probability.
[0070] Implicit reward definition: Calculating the implicit reward for each trajectory. The reward is defined as the trainable optimal penetration strategy. Relative reference strategy The logarithmic probability difference is calculated using the following formula: ; in, Hyperparameters that indicate the degree of deviation from the control strategy.
[0071] DPO Loss Function: Optimizes the loss function by minimizing direct preferences. To update the strategy parameters. The calculation formula is as follows: ; in, Indicates in the preference dataset Take the expected value; This represents the Sigmoid activation function, used to map reward differences to probability intervals; This represents the reward difference between the preferred trajectory and the non-preferred trajectory.
[0072] Only the optimal penetration strategy is updated using the gradient descent algorithm. The parameters enable the optimal penetration strategy. Learn to assign "preference trajectories" Compared to "non-preference trajectories" Higher implicit rewards. This process bypasses the dependence of traditional reinforcement learning on explicit reward models, enabling the optimal penetration strategy. It internalizes decision-making criteria that SFT cannot learn and that are difficult for human experts to formalize. The resulting strategy can not only reproduce successful penetration paths, but also weigh attack efficiency (choosing shorter paths), stealth (avoiding detection), or robustness (prioritizing high-success-rate vulnerabilities) among multiple feasible options, generating a more optimal penetration path that aligns with expert "intuition".
[0073] Example 2 A penetration path planning system based on LLM trajectory extraction and DPO alignment includes: The information extraction module is used to build an automated information extraction pipeline based on the Large Language Model (LLM), which transforms unstructured cybersecurity text into structured dynamic penetration map trajectories and stores them in an expert penetration trajectory set. The policy network construction and training module is used to construct a policy network based on a graph neural network (GNN), and to perform supervised fine-tuning of the SFT training using an expert penetration trajectory set to obtain a reference policy. The DPO alignment optimization module is used to construct a trajectory preference dataset and optimize the DPO algorithm based on direct preference. Using the reference policy as a benchmark, it optimizes and aligns the policy network to obtain the target policy that generates the optimal penetration path that conforms to implicit preferences. The dynamic graph ontology inspection module is used to define and maintain dynamic graph ontology. The dynamic graph ontology includes node type, edge type and dynamic evolution mechanism. The node set includes nodes of host, port, service, vulnerability, user, credential, permission and file type. The edge set includes directed edges representing structural dependencies, causal exploitation and information association. The consistency verification module is used to perform multi-dimensional consistency verification and correction on the information extraction results; The trajectory serialization and storage module is used to update the graph state based on the extracted incremental update elements and serialize the state-action pairs into trajectories, which are then stored in the expert penetration trajectory set. The graph feature encoding submodule is used to receive the dynamic graph state, process it using a graph attention network, and generate node embedding vectors that incorporate global context information. The policy decoding output submodule is used to map the node embedding vectors to scalar values, normalize them using the Softmax function, and output the probability distribution of selecting each node in the graph as the target of the next action. The preference dataset construction submodule is used to construct the trajectory preference dataset through two methods: expert adjudication and reference strategy sampling. The expert adjudication method extracts multiple different penetration trajectories for the same penetration target and the security experts adjudicate their merits to form preference pairs. The reference strategy sampling method uses high-fidelity expert penetration trajectories as preferred trajectories and uses the reference strategy to generate different paths as non-preferred trajectories in the simulation environment through random sampling. The implicit reward calculation module is used to calculate the implicit reward of the trajectory. The implicit reward is obtained by multiplying the total log probability difference between the trajectory of the policy to be optimized and the reference policy by the hyperparameter of the deviation of the control policy.
[0074] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0075] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0076] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0077] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.
[0078] Therefore, this invention adopts the above-mentioned penetration path planning method, system and equipment based on LLM and DPO. Through three core means, namely, LLM automatic pipeline extraction, GNN dynamic graph representation and decision framework and DPO implicit preference alignment, it systematically solves the three key problems of data scarcity, weak strategy generalization ability and reward design mismatch in automated penetration testing. Finally, it realizes intelligent, interpretable and expert experience-based automatic optimal penetration path planning.
[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A penetration path planning method based on LLM and DPO, characterized in that, Includes the following steps: Step S1: Construct an automated information extraction pipeline based on the Large Language Model (LLM) to transform unstructured cybersecurity text into structured dynamic penetration map trajectories and store them in an expert penetration trajectory set. Step S2: Construct a policy network based on graph neural network (GNN), and use the expert penetration trajectory set for supervised fine-tuning of SFT training to obtain a reference policy; Step S3: Construct a trajectory preference dataset and optimize the DPO algorithm based on direct preferences. Using the reference policy as a benchmark, optimize and align the policy network to obtain the target policy that generates the optimal penetration path that conforms to implicit preferences.
2. The penetration path planning method based on LLM and DPO according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Define the dynamic graph ontology, including node type, edge type, and dynamic evolution mechanism; Step S12: Perform semantic-driven attack step temporal segmentation on unstructured network security text to obtain an atomic text fragment sequence; Step S13: Based on the current graph state context, extract the graph incremental update element from each text fragment; Step S14: Perform multidimensional consistency verification and correction on the extraction results; Step S15: Update the graph state according to the extracted incremental update elements, and serialize the state-action pairs into trajectories and store them in the expert penetration trajectory set.
3. The penetration path planning method based on LLM and DPO according to claim 1, characterized in that, In step S1, the dynamic graph ontology is constructed. ,include: Node set Nodes containing host, port, service, vulnerability, user, credential, permission, and file type; edge set It includes directed edges representing structural dependencies, causal utilization, and information relationships; and a dynamic evolution mechanism that defines the dynamic graph state. It evolves incrementally as the penetration action is executed. Node attribute set Edge attribute set .
4. The penetration path planning method based on LLM and DPO according to claim 1, characterized in that, Step S2 specifically includes: Construct a graph neural network policy network that takes the dynamic graph state as input and the node selection probability distribution in the graph as output; The graph neural network strategy network is trained under supervision using a set of expert penetration trajectories to obtain a reference strategy with basic expert decision-making capabilities, which serves as the initial penetration path planning strategy.
5. The penetration path planning method based on LLM and DPO according to claim 4, characterized in that, In step S2, the graph neural network policy network includes: The graph feature encoding layer is used to receive the dynamic graph state, which is processed by a graph attention network to generate node embedding vectors that incorporate global context information. The policy decoding head is used to map the node embedding vectors to scalar values, normalize them using the Softmax function, and output the probability distribution of selecting each node in the graph as the target of the next action.
6. The penetration path planning method based on LLM and DPO according to claim 4, characterized in that, In step S2, supervised fine-tuning training uses the negative log-likelihood loss function. Optimize: ； in, This represents the expected value calculation for the distribution of the dataset; This represents the expert penetration trajectory set; This represents a set of state-action pairs sampled from the dataset. for The dynamic graph status input at any given moment. The target node label selected by the expert in this state; This indicates that the graph neural network policy is in the input graph state. hour; The goal is to minimize this loss function so that the output distribution of the policy network approximates the expert behavior distribution, and a reference policy is obtained after training. .
7. The penetration path planning method based on LLM and DPO according to claim 1, characterized in that, In step S3, the trajectory preference dataset ,in, For preferred trajectories, For non-preference trajectories, the construction method is as follows; Based on expert judgment: Multiple different penetration trajectories targeting the same infiltration target are extracted, and security experts judge their merits to form a preference pair. ; Reference strategy-based sampling: using high-fidelity expert penetration trajectories as preference trajectories. Using reference strategy In a simulation environment, different paths are generated as unpreferred trajectories through random sampling. .
8. The penetration path planning method based on LLM and DPO according to claim 7, characterized in that, In step S3, based on the Direct Preference Optimization (DPO) algorithm, the policy network is optimized and aligned using the reference policy as a benchmark to obtain the target policy that generates the optimal penetration path that conforms to implicit preferences, including: Step S31: Refer to the strategy Initialize the parameters to the policy network to be optimized. and from trajectory preference dataset Medium batch sampling preference ; Step S32: Calculate the total log probability of the reference strategy for the trajectory. The calculation formula is as follows: ； in, This represents a complete penetration test trajectory. Indicates the policy network in state Choose below The logarithmic probability; Step S33, Implicit Reward Estimation: Calculate the implicit reward of the trajectory. The calculation formula is as follows: ； in, Hyperparameters representing the degree of deviation from the control strategy; Step S34: Optimize the loss function by minimizing direct preferences. To update the policy network The parameters are calculated using the following formula: ； in, In the trajectory preference dataset Take the expected value. This represents the Sigmoid activation function, used to map reward differences to probability intervals. This represents the reward difference between the preferred trajectory and the non-preferred trajectory.
9. A penetration path planning system based on LLM trajectory extraction and DPO alignment, used to execute the penetration path planning method based on LLM and DPO as described in any one of claims 1-8, characterized in that, include: The information extraction module is used to build an automated information extraction pipeline based on the Large Language Model (LLM), which transforms unstructured cybersecurity text into structured dynamic penetration map trajectories and stores them in an expert penetration trajectory set. The policy network construction and training module is used to construct a policy network based on a graph neural network (GNN), and to perform supervised fine-tuning of the SFT training using an expert penetration trajectory set to obtain a reference policy. The DPO alignment optimization module is used to construct a trajectory preference dataset and optimize the DPO algorithm based on direct preference. Using the reference policy as a benchmark, it optimizes and aligns the policy network to obtain the target policy that generates the optimal penetration path that conforms to implicit preferences. The dynamic graph ontology inspection module is used to define and maintain dynamic graph ontology. The dynamic graph ontology includes node type, edge type and dynamic evolution mechanism. The node set includes nodes of host, port, service, vulnerability, user, credential, permission and file type. The edge set includes directed edges representing structural dependencies, causal exploitation and information association. The consistency verification module is used to perform multi-dimensional consistency verification and correction on the information extraction results; The trajectory serialization and storage module is used to update the graph state based on the extracted incremental update elements and serialize the state-action pairs into trajectories, which are then stored in the expert penetration trajectory set. The graph feature encoding submodule is used to receive the dynamic graph state, process it using a graph attention network, and generate node embedding vectors that incorporate global context information. The policy decoding output submodule is used to map the node embedding vectors to scalar values, normalize them using the Softmax function, and output the probability distribution of selecting each node in the graph as the target of the next action. The preference dataset construction submodule is used to construct the trajectory preference dataset through two methods: expert adjudication and reference strategy sampling. The expert adjudication method extracts multiple different penetration trajectories for the same penetration target and the security experts adjudicate their merits to form preference pairs. The reference strategy sampling method uses high-fidelity expert penetration trajectories as preferred trajectories and uses the reference strategy to generate different paths as non-preferred trajectories in the simulation environment through random sampling. The implicit reward calculation module is used to calculate the implicit reward of the trajectory. The implicit reward is obtained by multiplying the total log probability difference between the trajectory of the policy to be optimized and the reference policy by the hyperparameter of the deviation of the control policy.
10. A computer device, characterized in that, It includes a memory and a processor, the memory being used to store instructions and the processor being used to execute the instructions to implement the method as described in any one of claims 1 to 8.