LLM-based attack decision-making method and apparatus

By introducing a large language model (LLM) into a reinforcement learning model, the problem that existing technologies cannot understand unstructured text information is solved, achieving higher decision adaptability and accuracy, similar to the knowledge-driven reasoning decision-making of human experts.

CN123116282APending Publication Date: 2026-11-17CHINA RAILWAY INFORMATION TECH CENT +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611183237.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-05
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing automated penetration testing methods based on deep reinforcement learning cannot understand unstructured text information such as penetration test reports, service version fingerprints, and vulnerability descriptions. This results in an inability to make knowledge-based reasoning decisions like human experts, reducing the adaptability and accuracy of reasoning decisions to the actual situation.

Method used

By incorporating Large Language Models (LLMs) into reinforcement learning models, hierarchical state representation and cognition can enhance the understanding and reasoning ability of unstructured text information, enabling the model to make knowledge-driven reasoning decisions similar to those of human experts.

Benefits of technology

It improves the adaptability and accuracy of target attack decisions to the actual environment, and can understand unstructured text information such as penetration test reports, service version fingerprints and vulnerability descriptions, and make knowledge-based reasoning decisions like human experts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116282A_ABST
    Figure CN123116282A_ABST
Patent Text Reader

Abstract

This invention provides an attack decision-making method and apparatus based on LLM (Large Language Model). The method includes: inputting the initial scan result corresponding to the target to be scanned and the generated text summary prompts into a reinforcement learning model to obtain a first scan decision output by the reinforcement learning model; generating an initial attack decision based on the first scan decision; inputting the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain an intermediate attack decision output by the reinforcement learning model; and generating a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and thought logic text prompts. This invention incorporates a large language model into the reinforcement learning model, abandoning fixed-dimensional numerical state vectors and using LLM to increase understanding and reasoning capabilities, enabling the model to make knowledge-based reasoning decisions like human experts, thereby improving the adaptability and accuracy of the target attack decision to the actual situation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of automated penetration testing technology, and in particular to an attack decision-making method and apparatus based on LLM. Background Technology

[0002] Currently, the core process of automated penetration testing methods based on Deep Reinforcement Learning (DRL) involves abstracting the network environment into a finite state space (such as node permissions and vulnerability existence) and a finite action space (such as port scanning and vulnerability exploitation), and using a Deep Q-Network (DQN) or its variant, the Dual Deep Q-Network, as the decision-making agent. The agent learns through trial and error in a simulated environment, ultimately outputting an optimal sequence of actions—the attack path—from the attack origin to the target node.

[0003] However, existing methods can only process numerical vectors. Traditional DRL models lack semantic understanding and knowledge reasoning capabilities for unstructured text information. They cannot understand unstructured text information such as penetration test reports, service version fingerprints, and vulnerability descriptions. They are unable to flexibly cope with complex and ever-changing network environments like human experts, resulting in an inability to make knowledge-based reasoning decisions like human experts. This reduces the adaptability and accuracy of reasoning decisions to the actual situation. Summary of the Invention

[0004] This invention provides an attack decision-making method and apparatus based on LLM (Large Language Modeling) to address the shortcomings of existing technologies where intelligent agents can only process numerical vectors and cannot understand unstructured text information such as penetration test reports, service version fingerprints, and vulnerability descriptions. This results in agents being unable to make knowledge-based reasoning decisions like human experts, reducing the adaptability and accuracy of reasoning decisions to the actual situation. The invention integrates a large language model into the reinforcement learning model. Based on the hierarchical state representation and cognition of the large language model, it abandons fixed-dimensional numerical state vectors, enhancing the model's understanding and reasoning ability of unstructured text information. This enables the model to make knowledge-driven reasoning decisions similar to those of human experts, improving the adaptability and accuracy of target attack decisions to the actual environment.

[0005] This invention provides an attack decision-making method based on LLM, comprising the following steps: The initial scanning results and generated text summary prompts corresponding to the target to be scanned are input into the reinforcement learning model to obtain the first scanning decision output by the reinforcement learning model. The reinforcement learning model is trained on the large language network and deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scanning sample results corresponding to multiple scanning samples and the problem-solving reports corresponding to each scanning sample. Based on the first scanning decision, an initial attack decision is generated; The initial attack result corresponding to the initial attack decision is input into the reinforcement learning model to obtain the intermediate attack decision output by the reinforcement learning model. Based on the intermediate attack decision and thought logic text prompts, a target attack decision containing thought logic text and historical experience is generated.

[0006] According to an LLM-based attack decision method provided by the present invention, the step of generating an initial attack decision based on a first scanning decision includes: executing the first scanning decision to obtain an initial scanning result; and determining the initial attack decision based on attack clues in the initial scanning result.

[0007] According to an LLM-based attack decision method provided by the present invention, determining the initial attack decision based on attack clues in the initial scan results includes: if attack clues exist in the initial scan results, inputting the initial scan results into the reinforcement learning model to obtain the initial attack decision; if no attack clues exist in the initial scan results, continuing to input the initial scan results into the reinforcement learning model to obtain a scanning strategy, executing the scanning strategy, until attack clues exist in the scan results obtained by executing the scanning strategy, then inputting the scanning strategy with attack clues into the reinforcement learning model to obtain the initial attack decision.

[0008] According to an LLM-based attack decision method provided by the present invention, the step of generating a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and thought logic text prompts includes: executing the intermediate attack decision to obtain an initial attack result; and determining the target attack decision containing thought logic text and historical experience based on the initial attack result and the thought logic text prompts.

[0009] According to an LLM-based attack decision method provided by the present invention, the step of determining the target attack decision containing the thought logic text and historical experience based on the initial attack result and the thought logic text prompts includes: if the initial attack result is successful, inputting the initial attack result and the thought logic text prompts into the reinforcement learning model to obtain the target attack decision containing the thought logic text and referenced historical experience; if the initial attack result is unsuccessful, inputting the initial attack result and the thought logic text prompts into the reinforcement learning model to obtain a retry attack decision; and determining the target attack decision based on the attack result corresponding to the retry attack decision and the thought logic text prompts.

[0010] According to the present invention, an attack decision-making method based on LLM is provided, wherein the reinforcement learning model is trained based on the following steps: obtaining initial scan sample results corresponding to multiple scan samples and problem-solving reports corresponding to each scan sample; inputting the initial scan results and sample-generated text summaries into the large language network in the initial reinforcement learning model to obtain embedding vectors and sample state summary texts; inputting the embedding vectors into the deep reinforcement learning policy network in the initial reinforcement learning model to obtain latent action representations; and training the model parameters of the initial reinforcement learning model based on the latent action representations, the sample state summary texts, and the problem-solving reports to obtain the reinforcement learning model.

[0011] According to the present invention, an attack decision-making method based on LLM is provided, wherein training the model parameters of an initial reinforcement learning model based on the latent action representation, the sample state summary text, and the problem-solving report to obtain the reinforcement learning model includes: inputting the latent action representation, the sample state summary text, and the sample thought logic text into the large language network to obtain sample action planning text and sample attack decision; and training the model parameters of the initial reinforcement learning model based on the sample action planning text, the sample attack decision, and the problem-solving report to obtain the reinforcement learning model.

[0012] According to an LLM-based attack decision method provided by the present invention, the method for training the model parameters of an initial reinforcement learning model based on the sample action planning text, the sample attack decision, and the problem-solving report to obtain the reinforcement learning model includes: executing the sample attack decision to obtain a sample attack result; determining an extrinsic reward based on the sample attack result and the problem-solving report; inputting the sample state summary text, the sample action planning text, the sample attack result, and generated comment prompts into the large language network to obtain an intrinsic reward; and training the model parameters of the initial reinforcement learning model based on the extrinsic reward and the intrinsic reward to obtain the reinforcement learning model.

[0013] According to an LLM-based attack decision method provided by the present invention, the step of training the model parameters of the initial reinforcement learning model based on the extrinsic reward and the intrinsic reward to obtain the reinforcement learning model includes: determining a target reward based on the extrinsic reward and the intrinsic reward; and training the model parameters of the initial reinforcement learning model based on the target reward until the trained target reward converges to obtain the reinforcement learning model.

[0014] The present invention also provides an LLM-based attack decision-making apparatus, comprising the following modules: The first input module is used to input the initial scanning result and the generated text summary prompt words corresponding to the target to be scanned into the reinforcement learning model to obtain the first scanning decision output by the reinforcement learning model; the reinforcement learning model is trained on the large language network and deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scanning sample results corresponding to multiple scanning samples and the problem-solving reports corresponding to each scanning sample. The first generation module is used to generate an initial attack decision based on the first scanning decision; The second input module is used to input the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain the intermediate attack decision output by the reinforcement learning model. The second generation module is used to generate a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and thought logic text prompts.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the LLM-based attack decision method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the LLM-based attack decision method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the LLM-based attack decision method as described above.

[0018] The LLM-based attack decision-making method and apparatus provided by this invention inputs the initial scan result corresponding to the target to be scanned and the generated text summary prompts into a reinforcement learning model to obtain a first scan decision output by the reinforcement learning model. The reinforcement learning model is trained on a large language model and a deep reinforcement learning policy network based on the initial scan sample results corresponding to multiple scan samples and the solution reports corresponding to each scan sample. Based on the first scan decision, an initial attack decision is generated. The initial attack result corresponding to the initial attack decision is input into the reinforcement learning model to obtain an intermediate attack decision output by the reinforcement learning model. Based on the intermediate attack decision and the thought logic text prompts, a target attack decision containing thought logic text and historical experience is generated. In this way, a large language model (LLM) is integrated into the reinforcement learning model. Based on the hierarchical state representation and cognition of the large language model, the fixed-dimensional numerical state vector is abandoned, enhancing the model's understanding and reasoning ability of unstructured text information, enabling it to make knowledge-driven reasoning decisions similar to those of human experts, and improving the adaptability and accuracy of the target attack decision to the actual environment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the attack decision-making method based on LLM provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the attack decision system based on LLM provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the attack decision-making device based on LLM provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] The existing methods also have the following drawbacks: (1) State space explosion and poor generalization ability: The state is defined by a fixed N nodes and M vulnerabilities. When the network topology changes (N increases) or unknown vulnerabilities appear (M increases), the dimension of the state vector changes, and the model needs to be retrained, resulting in no generalization ability. (2) Rigid action space and non-scalable: Actions are hard-coded for specific vulnerability exploitation. When faced with 0-day vulnerabilities or undefined common vulnerabilities and disclosure numbers (CVEs), the action fails. Scanning actions (such as scanning adjacent networks) are coarse-grained and cannot adapt to complex network environments. (3) Difficulty in designing reward functions and inefficient exploration: The reward function (+10, -0.1, etc.) is highly dependent on expert experience and it is difficult to balance exploration and exploitation. In large networks, the agent is very likely to get trapped in local optima or invalid loops, and training convergence is extremely slow. (4) Lack of semantic understanding and reasoning ability: The agent can only process numerical vectors and cannot understand unstructured text information such as penetration test reports, service version fingerprints, and vulnerability descriptions, which makes it unable to make knowledge-based reasoning decisions like human experts. (5) Unexplainable decision-making process: DQN is a black box that only outputs Q-values ​​and cannot explain why this action was chosen, which is unacceptable under security audit and compliance requirements.

[0026] To address the aforementioned problems, this invention provides an attack decision-making method based on LLM. By incorporating a large language model into a reinforcement learning model, and leveraging the hierarchical state representation and cognition of the large language model, the fixed-dimensional numerical state vector is abandoned. The use of LLM enhances understanding and reasoning capabilities, enabling the model to understand unstructured textual information such as penetration test reports, service version fingerprints, and vulnerability descriptions. This allows the model to make knowledge-based reasoning decisions similar to those of human experts, improving the adaptability and accuracy of target attack decisions to the actual situation.

[0027] The following is combined Figures 1-2 The present invention describes an LLM-based attack decision method. The execution subject of this method can be an electronic device or an LLM-based attack decision device installed in the electronic device. The LLM-based attack decision device can be implemented by software, hardware or a combination of both.

[0028] Figure 1This is a flowchart illustrating the attack decision-making method based on LLM provided by the present invention, as follows: Figure 1 As shown, the method includes the following: Step 101: Input the initial scanning result and generated text summary prompts corresponding to the target to be scanned into the reinforcement learning model to obtain the first scanning decision output by the reinforcement learning model.

[0029] The reinforcement learning model is obtained by training the large language network and deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scan sample results corresponding to multiple scan samples and the problem-solving reports corresponding to each scan sample.

[0030] Here, the targets to be scanned may include ports, Internet Protocol (IP) addresses, domain names, host services, web paths, interfaces, etc.

[0031] Here, the initial scan results may include a list of nodes and their permissions, service fingerprints, vulnerability scan report snippets, network connections, etc.

[0032] Here, the text summary prompts can be determined according to customer needs and are used to generate intermediate text for the reinforcement learning model to output the first scan decision.

[0033] Here, the first scan decision is used to indicate the next scan target and scanning method. For example, the first scan decision is used to indicate whether to continue scanning the current port and how to scan the port.

[0034] Here, the solution report can include the solution logic, vulnerability principle, and attack path.

[0035] Step 102: Based on the first scanning decision, generate an initial attack decision.

[0036] Here, the initial attack decision can be determined directly based on the scan results corresponding to the first scan decision; or it can be determined based on the scan results of multiple scan strategies.

[0037] Furthermore, generating an initial attack decision based on the first scanning decision includes: executing the first scanning decision to obtain an initial scanning result; and determining the initial attack decision based on the attack clues in the initial scanning result.

[0038] Here, the first scanning decision can be executed through an Application Programming Interface (API) service-oriented interaction layer to obtain initial scan results. This API-based interaction layer encapsulates various penetration testing tools and target environments. For example, penetration testing tools may include network mappers (Nmap), exploit frameworks (Metasploit), and vulnerability scanners (Nuclei). Target environments may include vulnerability targets (Metasploitable) and vulnerability ranges (Vulhub). The target environment is a specially built security training and testing environment that simulates real network and system vulnerabilities, including virtual hosts, servers, websites, network devices, and various vulnerability scenarios.

[0039] Here, attack clues can include exploitable vulnerabilities, open services, or weak passwords.

[0040] Here, the initial attack decision can be output by the reinforcement learning model based on the initial scan results.

[0041] In this embodiment of the invention, attack clues in the initial scan results are used to accurately determine the initial attack decision, thereby improving the accuracy and success rate of the attack.

[0042] Furthermore, determining the initial attack decision based on the attack clues in the initial scan results includes: if attack clues exist in the initial scan results, inputting the initial scan results into the reinforcement learning model to obtain the initial attack decision; if no attack clues exist in the initial scan results, continuing to input the initial scan results into the reinforcement learning model to obtain a scanning strategy, executing the scanning strategy, until attack clues exist in the scan results obtained by executing the scanning strategy, then inputting the scanning strategy with attack clues into the reinforcement learning model to obtain the initial attack decision.

[0043] It's important to note that when attack clues are present in the initial scan results, it indicates the discovery of an attackable point. The reinforcement learning model can then output an attack decision based on these initial scan results and execute the attack on that point. If no attack clues are found in the initial scan results, the reinforcement learning model outputs a scanning decision and continues scanning. During this process, the scan results from each execution of a scanning decision need to be input into the reinforcement learning model to optimize the next scan decision, until attack clues appear in the scan results. In other words, the scan results after executing the scanning strategy before generating an attack decision must continuously interact with the reinforcement learning model until attack clues appear in the scan results before an attack decision can be obtained.

[0044] In this embodiment of the invention, when faced with a completely new network topology, unknown CVE vulnerabilities, or different operating systems, LLM's semantic understanding capability enables it to understand the new environment. When no attack clues are found in the scan results, the reinforcement learning model is repeatedly used to continuously output scan decisions, thereby improving the accuracy of the scan decisions.

[0045] Step 103: Input the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain the intermediate attack decision output by the reinforcement learning model.

[0046] It should be noted that if the initial attack successfully obtains the target, no intermediate attack decision is needed; if the initial attack fails to obtain the target, the reinforcement learning model optimizes the attack decision to obtain an intermediate attack decision.

[0047] It should be noted that after the initial attack fails, the reinforcement learning model can rescan the target to be scanned and output an intermediate attack decision based on the rescan result; or it can obtain an intermediate attack decision by simply optimizing the initial attack decision without rescanning.

[0048] Step 104: Based on the intermediate attack decision and the thought logic text prompts, generate a target attack decision that includes thought logic text and historical experience.

[0049] Here, logical thinking text prompts are used to prompt the reinforcement learning model to think when outputting attack decisions, and output the text of the thinking process.

[0050] Here, historical experience can be obtained by retrieving cases from a long-term memory of experience.

[0051] In this embodiment, a large language model is incorporated into the reinforcement learning model. Based on the hierarchical state representation and cognition of the large language model, the fixed-dimensional numerical state vector is abandoned. The LLM is used to increase the understanding and reasoning capabilities, enabling the model to understand unstructured text information such as penetration test reports, service version fingerprints, and vulnerability descriptions, and make knowledge-based reasoning decisions like human experts, thereby improving the adaptability and accuracy of target attack decisions to the actual situation.

[0052] Furthermore, the step of generating a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and the thought logic text prompts includes: executing the intermediate attack decision to obtain an initial attack result; and determining the target attack decision containing thought logic text and historical experience based on the initial attack result and the thought logic text prompts.

[0053] Here, the initial attack result can be either successful or unsuccessful. If the attack fails, the attack decision needs to be re-output. If the attack succeeds, the intermediate attack decision, along with the corresponding thought process text and historical experience, will be used as the target attack decision.

[0054] Here, the target attack decision may include intermediate attack decisions, or it may be an optimized version of the intermediate attack decisions.

[0055] In this embodiment of the invention, an LLM-based dynamic action space and tool invocation are implemented: actions are no longer hard-coded; the agent's action space is abstracted as invoking a penetration testing tool with accompanying natural language parameters. LLM is responsible for dynamically generating specific, executable commands. This allows the agent to utilize any existing tools and adaptively respond to unknown vulnerabilities.

[0056] Furthermore, determining the target attack decision containing the thought logic text and historical experience based on the initial attack result and the thought logic text prompt includes: if the initial attack result is successful, inputting the initial attack result and the thought logic text prompt into the reinforcement learning model to obtain the target attack decision containing the thought logic text and referenced historical experience; if the initial attack result is unsuccessful, inputting the initial attack result and the thought logic text prompt into the reinforcement learning model to obtain a retry attack decision, and determining the target attack decision based on the attack result corresponding to the retry attack decision and the thought logic text prompt.

[0057] It should be noted that if the initial attack result is successful, it means the attack on the attack point has been successful, and there is no need to further optimize the attack decision. At this time, the intermediate attack decision, the corresponding thought process text, and the referenced historical experience text are output as the target attack decision. If the initial attack result is unsuccessful, it means the attack on the attack point has failed. At this time, other attack points can be rescanned for attack, or the attack decision can be optimized and the attack on the attack point can be continued. In this process, each attack result needs to interact with the reinforcement learning model until the attack decision is successful. The attack decision corresponding to the successful attack, the thought process text corresponding to the attack decision, and the referenced historical experience text are output as the target attack decision.

[0058] It should be noted that when the attack result corresponding to the retry attack decision is successful, the retry attack decision is the target attack decision; when the attack result corresponding to the retry attack decision is unsuccessful, the reinforcement learning model continues to output the optimized attack decision, and the interaction between the attack result and the reinforcement learning model continues until the attack decision is successful, thus obtaining the target attack decision.

[0059] In this embodiment of the invention, the LLM-based dynamic action space and tool invocation mean that actions are no longer hard-coded. The agent's action space is abstracted as invoking a penetration testing tool with accompanying natural language parameters. LLM is responsible for dynamically generating specific, executable commands. This allows the agent to utilize any existing tools and adaptively respond to unknown vulnerabilities.

[0060] In this embodiment of the invention, during production environment deployment, at each decision step, the system not only outputs action commands but also simultaneously outputs the thought chain text generated by the LLM and referenced historical experience. Security analysts can interact with the system through a natural language interface (such as "Why did you decide to scan that port?"), and the system invokes the LLM to provide the answer.

[0061] In this embodiment of the invention, for ultra-large-scale networks, multiple reinforcement learning model agents can be deployed. One master LLM agent is responsible for global strategic planning (such as "segmenting the network and defeating enemies one by one"), while multiple executing LLM agents are responsible for tactical penetration within designated network segments. Asynchronous communication and coordination are achieved through a shared "blackboard" (a common state summary maintained by the LLM).

[0062] Furthermore, the reinforcement learning model is trained based on the following steps: obtaining initial scan sample results corresponding to multiple scan samples and problem-solving reports corresponding to each scan sample; inputting the initial scan results and sample-generated text summaries into the large language network in the initial reinforcement learning model to obtain embedding vectors and sample state summary texts; inputting the embedding vectors into the deep reinforcement learning policy network in the initial reinforcement learning model to obtain latent action representations; and training the model parameters of the initial reinforcement learning model based on the latent action representations, the sample state summary texts, and the problem-solving reports to obtain the reinforcement learning model.

[0063] Here, the initial scan sample result corresponding to the scan sample can be obtained from historical penetration test logs.

[0064] Here, the embedding vector is a fixed-dimensional semantic embedding vector, for example, through the hidden state of the penultimate layer of a large language model.

[0065] Here, the deep reinforcement learning policy network can be either a DRL or a Decision Transformer. The Decision Transformer treats reinforcement learning as a new paradigm for sequence modeling problems. It directly uses the historical sequence of states, actions, and rewards as input and generates the next action through autoregression. The Decision Transformer can naturally integrate Large Language Models (LLMs).

[0066] For example, the initial scan results are input into the LLM cognitive encoder, and the sample is given the prompt "Please summarize the following penetration test status into a concise text summary, highlighting the compromised hosts, high-value targets, critical vulnerabilities, and network topology." The LLM outputs the sample status summary text and the embedding vector embedding_t.

[0067] Here, latent action representation refers to all possible candidate actions that can be taken, abstractly represented in the form of vectors / feature vectors.

[0068] Potential actions refer to all candidate actions that can be chosen in the future but have not yet been executed, such as scanning ports, launching brute-force attacks, exploiting vulnerabilities, and rescanning.

[0069] The initial training of the reinforcement learning model's parameters essentially involves using the loss value between the attack decision predicted by the optimizer's gradient descent and the actual attack decision, until the loss value converges, thus obtaining the reinforcement learning model.

[0070] In this embodiment of the invention, the semantic understanding capability of LLM enables it to understand new environments without retraining the DRL model, and it has extremely strong generalization ability and zero-shot adaptation: facing completely new network topologies, unknown CVE vulnerabilities, or different operating systems.

[0071] Furthermore, the step of training the model parameters of the initial reinforcement learning model based on the latent action representation, the sample state summary text, and the problem-solving report to obtain the reinforcement learning model includes: inputting the latent action representation, the sample state summary text, and the sample thinking logic text into the large language network to obtain sample action planning text and sample attack decision; and training the model parameters of the initial reinforcement learning model based on the sample action planning text, the sample attack decision, and the problem-solving report to obtain the reinforcement learning model.

[0072] Here, the LLM action planner can output sample action planning text and sample attack decisions.

[0073] Here, the sample thinking logic text is used by the LLM action planner to think when outputting attack decisions and to retrieve similar successful cases in the long-term experience memory to obtain the sample action planning text.

[0074] For example, an LLM action planner receives potential action representations and sample state summary text, and retrieves similar success stories from its long-term experience memory. LLM generates detailed action plans from sample thought logic text, or thought chains (CoT), for example: Thoughts: The current status shows I have control of web-server-1. A scan revealed that db-server-2 has port 3306 open and is vulnerable to 'CVE-2012-2122' (a weak password vulnerability in MySQL). My goal is to gain root privileges on db-server-2. My previous successful experience was using Hydra to brute-force MySQL. Therefore, I should perform the following action: brute-force the weak password on db-server-2:3306.

[0075] Command: hydra -L users.txt -P passwords.txt mysql: / / db-server-2 -t 4 The above considerations constitute the sample action planning text, while the command represents the sample attack decision.

[0076] After outputting the sample action planning text, the model parameters of the initial reinforcement learning model are optimized using the sample action planning text, sample attack decisions, and problem-solving reports to obtain the reinforcement learning model.

[0077] In this embodiment of the invention, every step of the thought process and reflection during the decision-making process is recorded, which meets the requirements of security auditing and provides security experts with insights to improve attack strategies.

[0078] Furthermore, the step of training the model parameters of the initial reinforcement learning model based on the sample action planning text, the sample attack decision, and the problem-solving report to obtain the reinforcement learning model includes: executing the sample attack decision to obtain the sample attack result; determining the extrinsic reward based on the sample attack result and the problem-solving report; inputting the sample state summary text, the sample action planning text, the sample attack result, and the generated comment prompt words into the large language network to obtain the intrinsic reward; and training the model parameters of the initial reinforcement learning model based on the extrinsic reward and the intrinsic reward to obtain the reinforcement learning model.

[0079] Here, the external reward is determined by comparing the results of the sample attack with the results in the solution report. For example, if the goal (Flag) is achieved, the reward is +100; if not achieved, the reward is 0.

[0080] Then, the sample state summary text, sample action planning text, sample attack results, and generated comment prompts are input into the LLM to comment on the attack and provide an intrinsic reward.

[0081] For example, the API-based interaction layer parses commands and calls the corresponding tools to execute sample attack decisions, obtaining the sample attack result `result_t` (success / failure and output). The environment returns an extrinsic reward `r_ext` (e.g., +100 for achieving the target flag, 0 for failure). The sample state summary text, sample action planning text, and sample attack result are input into an LLM commentator, generating a comment prompt: "Evaluate this action. Did it provide new information? Did it advance the attack? Was it more efficient than a simple scan? Give a score between -1 and 1." The LLM commentator outputs an intrinsic reward `r_int` (e.g., +0.7, because new valid credentials were found).

[0082] In this embodiment of the invention, the LLM reflector generates an experience text from the decision attack, attack results, and thought logic: "After controlling the web-server, it is efficient to brute-force weak passwords on the db-server because it directly obtains access permissions and avoids complex vulnerability exploitation." Furthermore, the model parameters are trained based on extrinsic and intrinsic rewards to obtain the reinforcement learning model.

[0083] In this embodiment of the invention, the intrinsic reward generated by LLM provides dense and meaningful feedback signals, avoiding the sparse rewards and blind exploration problems of traditional reinforcement learning.

[0084] Furthermore, the step of training the model parameters of the initial reinforcement learning model based on the extrinsic reward and the intrinsic reward to obtain the reinforcement learning model includes: determining a target reward based on the extrinsic reward and the intrinsic reward; and training the model parameters of the initial reinforcement learning model based on the target reward until the trained target reward converges to obtain the reinforcement learning model.

[0085] Here, goal setting can be the sum of extrinsic and intrinsic rewards, or a weighted sum of the two.

[0086] Furthermore, the model parameters are adjusted using gradient descent, and attack decisions are inferred based on the adjusted model parameters to obtain a new target reward. The model parameters are then adjusted based on the new target reward for training until the target reward converges and stabilizes, resulting in a reinforcement learning model.

[0087] In this embodiment of the invention, LLM can understand and translate it into adjustments to constraints or intrinsic rewards during decision-making, thus achieving true intelligent collaboration.

[0088] The LLM-based attack decision-making device provided by the present invention will be described below. The LLM-based attack decision-making device described below can be referred to in correspondence with the LLM-based attack decision-making method described above.

[0089] Figure 2 This is a schematic diagram of the attack decision system based on LLM provided by the present invention, as shown below. Figure 2 As shown, the LLM-based attack decision system 200 includes an intelligence gathering unit 210, a target scanning unit 220, an attack execution unit 230, an attack success unit 240, and a reinforcement learning model 250. Specifically, the intelligence gathering unit determines the target to be scanned and sends it to the target scanning unit. The target scanning unit scans the target to be scanned, obtains a single-step scan result (i.e., the initial scan result), and inputs it into the reinforcement learning model. The reinforcement learning model outputs the initial attack decision to the target scanning unit. The target scanning decision sends the initial attack decision to the attack execution unit. The attack execution unit executes the initial attack decision, obtains a single-step exploit result (i.e., the initial attack result), and sends it to the reinforcement learning model. The reinforcement learning model outputs the next exploit decision (i.e., the attack decision) to the attack execution unit. When the attack execution unit successfully completes the attack, it sends the attack result to the attack success unit.

[0090] Figure 3 This is a schematic diagram of the attack decision-making device based on LLM provided by the present invention, as shown below. Figure 3 As shown, the LLM-based attack decision-making device 300 includes: The first input module 310 is used to input the initial scanning result and the generated text summary prompt words corresponding to the target to be scanned into the reinforcement learning model to obtain the first scanning decision output by the reinforcement learning model; the reinforcement learning model is trained on the large language network and deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scanning sample results corresponding to multiple scanning samples and the problem-solving reports corresponding to each scanning sample. The first generation module 320 is used to generate an initial attack decision based on the first scanning decision; The second input module 330 is used to input the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain the intermediate attack decision output by the reinforcement learning model. The second generation module 340 is used to generate a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and thought logic text prompts.

[0091] In this embodiment of the invention, the first generation module 320 is specifically used to: execute the first scanning decision to obtain an initial scanning result; and determine the initial attack decision based on the attack clues in the initial scanning result.

[0092] In this embodiment of the invention, the first generation module 320 is further specifically configured to: if there are attack clues in the initial scan results, input the initial scan results into the reinforcement learning model to obtain the initial attack decision; if there are no attack clues in the initial scan results, continue to input the initial scan results into the reinforcement learning model to obtain a scanning strategy, and execute the scanning strategy until there are attack clues in the scan results obtained by executing the scanning strategy, then input the scanning strategy with attack clues into the reinforcement learning model to obtain the initial attack decision.

[0093] In this embodiment of the invention, the second generation module 340 is specifically used to: execute the intermediate attack decision to obtain the initial attack result; and determine the target attack decision containing the thinking logic text and historical experience based on the initial attack result and the thinking logic text prompt words.

[0094] In this embodiment of the invention, the second generation module 340 is further specifically configured to: if the initial attack result is successful, input the initial attack result and the thought logic text prompt into the reinforcement learning model to obtain the target attack decision containing the thought logic text and referenced historical experience; if the initial attack result is unsuccessful, input the initial attack result and the thought logic text prompt into the reinforcement learning model to obtain a retry attack decision; and determine the target attack decision based on the attack result corresponding to the retry attack decision and the thought logic text prompt.

[0095] In this embodiment of the invention, the LLM-based attack decision-making device 300 further includes a training module, specifically configured to: acquire initial scan sample results corresponding to multiple scan samples and problem-solving reports corresponding to each scan sample; input each initial scan result and sample-generated text summary into the large language network in the initial reinforcement learning model to obtain embedding vectors and sample state summary text; input the embedding vectors into the deep reinforcement learning policy network in the initial reinforcement learning model to obtain latent action representations; and train the model parameters of the initial reinforcement learning model based on the latent action representations, the sample state summary text, and the problem-solving reports to obtain the reinforcement learning model.

[0096] In this embodiment of the invention, the training module is further specifically used to: input the latent action representation, the sample state summary text, and the sample thinking logic text into the large language network to obtain sample action planning text and sample attack decision; and train the model parameters of the initial reinforcement learning model based on the sample action planning text, the sample attack decision, and the problem-solving report to obtain the reinforcement learning model.

[0097] In this embodiment of the invention, the training module is further specifically configured to: execute the sample attack decision to obtain the sample attack result; determine the extrinsic reward based on the sample attack result and the problem-solving report; input the sample state summary text, the sample action planning text, the sample attack result, and the generated comment prompt words into the large language network to obtain the intrinsic reward; and train the model parameters of the initial reinforcement learning model based on the extrinsic reward and the intrinsic reward to obtain the reinforcement learning model.

[0098] In this embodiment of the invention, the training module is further specifically used for: determining a target reward based on the extrinsic reward and the intrinsic reward; and training the model parameters of the initial reinforcement learning model based on the target reward until the trained target reward converges, thereby obtaining the reinforcement learning model.

[0099] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an LLM-based attack decision method. This method includes: inputting the initial scan result corresponding to the target to be scanned and the generated text summary prompts into a reinforcement learning model to obtain a first scan decision output by the reinforcement learning model; the reinforcement learning model is trained on a large language network and a deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scan sample results corresponding to multiple scan samples and the problem-solving reports corresponding to each scan sample; generating an initial attack decision based on the first scan decision; inputting the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain an intermediate attack decision output by the reinforcement learning model; and generating a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and thought logic text prompts.

[0100] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the LLM-based attack decision method provided by the above methods. The method includes: inputting the initial scan result corresponding to the target to be scanned and the generated text summary prompts into a reinforcement learning model to obtain a first scan decision output by the reinforcement learning model; the reinforcement learning model is trained on the large language network and deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scan sample results corresponding to multiple scan samples and the problem-solving reports corresponding to each scan sample; generating an initial attack decision based on the first scan decision; inputting the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain an intermediate attack decision output by the reinforcement learning model; and generating a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and the thought logic text prompts.

[0102] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the LLM-based attack decision method provided by the above methods. The method includes: inputting an initial scan result corresponding to the target to be scanned and generated text summary prompts into a reinforcement learning model to obtain a first scan decision output by the reinforcement learning model; the reinforcement learning model is trained on a large language network and a deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scan sample results corresponding to multiple scan samples and the problem-solving reports corresponding to each scan sample; generating an initial attack decision based on the first scan decision; inputting the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain an intermediate attack decision output by the reinforcement learning model; and generating a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and thought logic text prompts.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An attack decision-making method based on LLM, characterized in that, include: The initial scanning result and the generated text summary prompt words corresponding to the target to be scanned are input into the reinforcement learning model to obtain the first scanning decision output by the reinforcement learning model. The reinforcement learning model is obtained by training the large language network and deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scan sample results corresponding to multiple scan samples and the problem-solving reports corresponding to each scan sample. Based on the first scanning decision, an initial attack decision is generated; The initial attack result corresponding to the initial attack decision is input into the reinforcement learning model to obtain the intermediate attack decision output by the reinforcement learning model. Based on the intermediate attack decision and thought logic text prompts, a target attack decision containing thought logic text and historical experience is generated.

2. The attack decision-making method based on LLM according to claim 1, characterized in that, The step of generating an initial attack decision based on the first scanning decision includes: Execute the first scanning decision to obtain the initial scanning result; The initial attack decision is determined based on the attack clues in the initial scan results.

3. The attack decision-making method based on LLM according to claim 2, characterized in that, The step of determining the initial attack decision based on the attack clues in the initial scan results includes: If attack clues are found in the initial scan results, the initial scan results are input into the reinforcement learning model to obtain the initial attack decision; If no attack clues are found in the initial scan results, the initial scan results are continued to be input into the reinforcement learning model to obtain a scanning strategy, and the scanning strategy is executed until attack clues are found in the scan results obtained by executing the scanning strategy. If attack clues are found, the scanning strategy with attack clues is input into the reinforcement learning model to obtain the initial attack decision.

4. The attack decision-making method based on LLM according to claim 1, characterized in that, The process of generating a target attack decision based on the intermediate attack decision and thought logic text prompts, including thought logic text and historical experience, includes: Execute the intermediate attack decision to obtain the initial attack result; Based on the initial attack results and the thought logic text prompts, a target attack decision containing thought logic text and historical experience is determined.

5. The attack decision-making method based on LLM according to claim 4, characterized in that, The step of determining the target attack decision containing the thought logic text and historical experience based on the initial attack result and the thought logic text prompts includes: If the initial attack is successful, the initial attack result and the thought logic text prompts are input into the reinforcement learning model to obtain the target attack decision containing the thought logic text and referenced historical experience. If the initial attack fails, the initial attack result and the thought logic text prompt are input into the reinforcement learning model to obtain a retry attack decision. Based on the attack result corresponding to the retry attack decision and the thought logic text prompt, the target attack decision is determined.

6. The attack decision method based on LLM according to any one of claims 1 to 5, characterized in that, The reinforcement learning model is trained based on the following steps: Obtain the initial scan sample results corresponding to multiple scan samples and the solution report corresponding to each scan sample; The initial scan results and sample-generated text summary prompts are input into the large language network in the initial reinforcement learning model to obtain the embedding vector and sample state summary text. The embedding vector is input into the deep reinforcement learning policy network in the initial reinforcement learning model to obtain the latent action representation; Based on the latent action representation, the sample state summary text, and the problem-solving report, the model parameters of the initial reinforcement learning model are trained to obtain the reinforcement learning model.

7. The attack decision method based on LLM according to claim 6, characterized in that, The step of training the model parameters of the initial reinforcement learning model based on the latent action representation, the sample state summary text, and the problem-solving report to obtain the reinforcement learning model includes: The potential action representation, the sample state summary text, and the sample thinking logic text are input into the large language network to obtain the sample action planning text and the sample attack decision. Based on the sample action planning text, the sample attack decision, and the problem-solving report, the model parameters of the initial reinforcement learning model are trained to obtain the reinforcement learning model.

8. The attack decision method based on LLM according to claim 7, characterized in that, The step of training the model parameters of the initial reinforcement learning model based on the sample action planning text, the sample attack decision, and the problem-solving report to obtain the reinforcement learning model includes: Execute the sample attack decision to obtain the sample attack result; Based on the sample attack results and the solution report, the external reward is determined; The sample state summary text, the sample action planning text, the sample attack result, and the generated comment prompts are input into the large language network to obtain the intrinsic reward. Based on the extrinsic reward and the intrinsic reward, the model parameters of the initial reinforcement learning model are trained to obtain the reinforcement learning model.

9. The attack decision-making method based on LLM according to claim 8, characterized in that, The step of training the model parameters of the initial reinforcement learning model based on the external reward and the internal reward to obtain the reinforcement learning model includes: Based on the extrinsic and intrinsic rewards, the target reward is determined; Based on the target reward, the model parameters of the initial reinforcement learning model are trained until the target reward converges, thus obtaining the reinforcement learning model.

10. An attack decision-making device based on LLM, characterized in that, include: The first input module is used to input the initial scanning result and the generated text summary prompt words corresponding to the target to be scanned into the reinforcement learning model to obtain the first scanning decision output by the reinforcement learning model; the reinforcement learning model is trained on the large language network and deep reinforcement learning policy network in the initial reinforcement learning model based on the initial scanning sample results corresponding to multiple scanning samples and the problem-solving reports corresponding to each scanning sample. The first generation module is used to generate an initial attack decision based on the first scanning decision; The second input module is used to input the initial attack result corresponding to the initial attack decision into the reinforcement learning model to obtain the intermediate attack decision output by the reinforcement learning model. The second generation module is used to generate a target attack decision containing thought logic text and historical experience based on the intermediate attack decision and thought logic text prompts.