A large language model driven penetration testing agent system and testing method

Abstract

The present invention provides a penetration test agent system and testing method driven by a large language model, including: a memory unit, which is composed of a penetration test knowledge base, contains data elements including vulnerability information, penetration methods, and tool information, and uses a knowledge graph to establish a vulnerability knowledge representation model; an intelligent unit, which is implemented by a large model and is used to generate tool call instructions, penetration test loads, and penetration test strategies based on the memory unit; a planning unit, which is used to split the penetration test task into a task planning state machine composed of controllable subtasks, and then guide the large language model to perform specific tasks. The present invention improves the level of automation and intelligence of penetration test tasks through the introduction of large models, knowledge bases, and task planning units; improves the completeness of knowledge retrieval, and ensures the interpretability and consistency of large model output; implements highly customized proxy behaviors for complex scenarios in the penetration test process, and ensures the execution process of penetration test tasks.

Description

Technical Field

[0001] The present invention relates to the field of penetration testing, and in particular to a penetration testing intelligent agent system and a testing method driven by a large language model. Background Art

[0002] Cyberspace security defense forms are becoming increasingly complex, and the offensive and defensive game for information system security is becoming more and more intense. If the vulnerabilities in the information system are exploited by attackers, it may cause serious consequences such as security failure, sensitive data leakage, and system damage. In 2017, Equifax, one of the three major credit rating agencies in the United States, suffered a large-scale data leak. Hackers used the unpatched Apache Struts OGNL injection vulnerability (CVE-2017-5638) in its system to achieve remote code execution and steal the highly sensitive data of nearly 147 million consumers. Using penetration testing to simulate attacks, reveal security vulnerabilities in the system, and promote the improvement of security defense capabilities is an important means to ensure the security of information systems.

[0003] However, penetration testing is a comprehensive application form of the integration of multiple cutting-edge technologies in the field of cyberspace security. It is the integration and expansion of various theories and technologies at the application level. It relies on the long-term focus and investment of security experts. It is a field with high technical difficulty and strong professional requirements. It is urgent to improve the intelligence and automation level of penetration testing. The attempts to intelligentize and automate penetration testing have never gotten rid of the human-in-the-loop model and rely on the professionalism and experience of the penetration test implementation subject. By introducing intelligent agents to assist human work, the transformation of the penetration testing model from human-in-the-loop to human-on-the-loop can be achieved. Summary of the invention

[0004] In response to the problems existing in the prior art, a large language model-driven penetration testing intelligent agent system and testing method are provided. Through the ability of the large model to process, learn and analyze penetration testing patterns, the intelligence and automation level of penetration testing is improved to reduce dependence on expert resources.

[0005] The first aspect of the present invention proposes a penetration testing agent system driven by a large language model, comprising:

[0006] The memory unit is composed of a penetration test knowledge base, which contains data elements such as vulnerability information, penetration methods, and tool information, and uses a knowledge graph to establish a vulnerability knowledge representation model;

[0007] The intelligent unit, implemented by the large model, is used to generate tool call instructions, penetration test payloads, and penetration test strategies based on the memory unit;

[0008] The planning unit is used to split the penetration test task into a task planning state machine consisting of controllable subtasks, and then guide the large language model to perform specific tasks.

[0009] As a preferred solution, in the memory unit, a knowledge graph is used to establish a unified vulnerability knowledge representation model, integrate penetration test related data, and integrate information across resource boundaries.

[0010] As a preferred solution, the planning unit includes a state representation module, a transfer condition module, a feedback optimization module and a tool calling module, wherein the state representation module is used to display the different stages of the penetration testing task and the achievement status of the stage goals; the transfer condition module is used to define the transfer conditions between states; the feedback optimization module is used to decide whether to feed back the step to the user based on preset thresholds of state results, task progress and number of attempts; the tool calling module is used to select the corresponding tool to perform a specific task according to the current task.

[0011] As a preferred solution, in the intelligent unit, the large model realizes domain capability enhancement based on the memory unit.

[0012] As a preferred solution, the domain capability enhancement includes: using the penetration testing knowledge base as the knowledge source of the large model; instilling the domain knowledge in the knowledge graph into the model parameters and representations through fine-tuning, and making full use of the domain knowledge in the knowledge graph when performing downstream tasks; by aligning the data in the knowledge graph with the large model, facts retrieved from the knowledge graph and implicit knowledge obtained from the large language model, and guiding the large language model to reason in the form of a mind map.

[0013] The second aspect of the present invention proposes a large language model driven penetration testing method, comprising:

[0014] Collect data from the Internet and build a penetration testing knowledge base;

[0015] Build a large language model, combine it with the penetration test knowledge base to complete domain enhancement, and complete retrieval enhancement generation by searching the penetration test knowledge base;

[0016] Build a task planning state machine and define task state machines for different types of penetration testing tasks;

[0017] Form an intelligent agent based on the penetration testing knowledge base, the domain enhancement big model, and the task planning state machine;

[0018] Complete penetration testing through intelligent agents.

[0019] As a preferred solution, the penetration testing knowledge base includes data elements such as vulnerability information, penetration methods, and tool information, and uses a knowledge graph to establish a vulnerability knowledge representation model.

[0020] As a preferred solution, the domain enhancement includes: using the penetration testing knowledge base as the knowledge source of the large model; instilling the domain knowledge in the knowledge graph into the model parameters and representations through fine-tuning, and making full use of the domain knowledge in the knowledge graph when performing downstream tasks; by aligning the data in the knowledge graph with the large model, facts retrieved from the knowledge graph and implicit knowledge obtained from the large language model, and guiding the large language model to reason in the form of a mind map.

[0021] As a preferred solution, the task state machine ensures the selection and calling of tools in each link through predefined rules.

[0022] As a preferred solution, the penetration test completed by the agent specifically includes:

[0023] The intelligent agent receives the penetration test target and prompt word sent by the user, detects the penetration test target, and obtains the detection result of the penetration test target;

[0024] The agent completes the penetration test task planning based on the detection results and provides feedback to the user;

[0025] After receiving the penetration test task planning confirmation information from the user, the agent generates the penetration test exploit load and instructions according to the task and sends them to the penetration test target;

[0026] The intelligent agent receives the exploitation purpose achievement signal from the penetration test feedback, and sends it to the user after completing the interpretation and evaluation.

[0027] Compared with the prior art, the beneficial effects of adopting the above technical solution are:

[0028] (1) The intelligent agent system proposed in the present invention improves the level of automation and intelligence of penetration testing tasks by introducing a large model, a knowledge base, and a task planning unit.

[0029] (2) Using knowledge graphs as the memory unit of intelligent agents supports the enhanced retrieval generation of large models, improves the completeness of knowledge retrieval, and ensures the interpretability and consistency of large model outputs.

[0030] (3) Using the task state machine as the planning unit of the intelligent agent, the tool selection and calling of key links are guaranteed through predefined rules. In view of the complex scenarios of the penetration test process, highly customized agent behavior is implemented to ensure the execution process of the penetration test task. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the penetration testing agent system driven by the large language model proposed in the present invention.

[0032] Figure 2 The figure is a flow chart of penetration testing in one embodiment of the present invention.

[0033] Figure 3 The figure is a schematic diagram of a web application penetration test provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be construed as limitations on the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0035] In recent years, the natural language understanding and content generation capabilities of large language models have empowered many vertical fields and are used to solve intelligence and automation problems in vertical fields. The embodiment of the present invention proposes a penetration testing agent system driven by a large language model, which improves the intelligence and automation level of penetration testing through the ability of the large model to process, learn and analyze penetration testing patterns, thereby reducing dependence on expert resources.

[0036] Penetration testing is a long-chain problem with a complex process. The main links include vulnerability scanning of information systems, generation and delivery of penetration test payloads, and the use of related penetration test tools. To realize the automation and intelligence of penetration testing, it is necessary to have the functions of task intent understanding, intelligent payload generation and automatic payload delivery. The present invention proposes a penetration testing intelligent agent system based on a large model. The large model is an artificial intelligence model based on deep learning. The advantage of the large model lies in multimodal semantic understanding and multimodal content generation. The intelligent agent system is a system that can perceive the environment and take actions based on the perceived information. The advantage of the intelligent agent lies in scene perception and scene interaction. By integrating the advantages of large models and intelligent agents, the problems of penetration test payload generation, payload delivery and task understanding in the penetration test process can be solved.

[0037] Please refer to Figure 1 ,The penetration testing agent system mainly includes a memory unit, an intelligence unit and a planning unit.

[0038] In this embodiment, the memory unit is implemented based on the penetration test knowledge base, which contains data elements including vulnerability information, penetration methods, and tool information, and uses the knowledge graph to establish a vulnerability knowledge representation model. Among them, the vulnerability information includes: Common Weakness Enumeration (CWE), Common Vulnerability Exposure (CVE) and other data related to vulnerability information; the penetration method includes the vulnerability exploitation database (ExploitDB), Common Attack Pattern Enumeration and Classification (CAPEC), and the attack matrix (ATT&CK); the tool information includes common network security testing tools, such as: network mapping tool Nmap, penetration testing tool Metasploit, network security testing tool Burpsuite, password blasting tool JohntheRipper, etc., functions, operation manuals, teaching manuals, instructions and other data of various network security testing tools. Collecting and representing through this series of data elements is the basis for enhancing the reasoning ability of large language models, making up for the lack of knowledge in the field of large language models, and ensuring that large language models generate reliable, explainable, accurate, and highly consistent results.

[0039] Furthermore, in the memory unit proposed in this embodiment, the knowledge graph technology is used to establish a unified vulnerability knowledge representation model, integrate massive penetration test related data, integrate information across resource boundaries, extract and store a large number of key knowledge entities and relationships, and provide a capability to integrate, manage and analyze multi-source heterogeneous data, thereby realizing the complementary advantages of graphs and large models and knowledge collaboration.

[0040] In the penetration testing agent system proposed in this embodiment, the memory unit plays a vital role in promoting situational understanding and long-term information retention. Knowledge includes short-term and long-term knowledge. Short-term knowledge acts as a dynamic repository of the agent's current behaviors and ideas, similar to its "training thoughts", responding to user queries in real time, allowing the agent to maintain a situational understanding of ongoing interactions, thereby achieving seamless and coherent communication, and enabling large models to acquire short-term knowledge through situational learning prompts. Long-term knowledge acts as a comprehensive log, providing the agent with the ability to retain and call information over a longer period of time, usually through external vector storage and rapid retrieval, or domain fine-tuning to enhance its domain long-term knowledge capabilities.

[0041] In this embodiment, the planning unit is a system process in which the agent analyzes user queries, collects relevant information, and formulates action strategies to provide the best suggestions or solutions. By splitting the given complex tasks, a task planning state machine consisting of controllable subtasks is formed to guide the specific tasks of the large language model, and a feedback loop is formed at key nodes to synchronize information to the user. If the task fails, manual intervention or feedback can be actively provided.

[0042] Specifically, the planning unit mainly includes a state representation module, a transfer condition module, a feedback optimization module, and a tool call module. Among them, the state representation module is used to display the different stages of the penetration test task and the achievement status of the stage goals. The transition condition module defines the transition conditions between states, and the transition function ,in Is an event set that describes the state transition conditions. If the current state is ,event occurs, then transfer to state ,Right now The feedback optimization module decides whether to hand over the step to a human according to the predefined thresholds of the status results, task progress, and number of attempts to improve the overall efficiency and accuracy of the task. The feedback function is defined ,in is in state The feedback value is the event of executing the action, the success rate, etc., and the definition To maximize cumulative feedback, ,in The tool calling module selects the appropriate tool to perform a specific task according to the current task. For example, before the task starts, it uses the network detection tool to obtain information such as ports and operating systems. During the payload delivery phase, it uses tools such as Burpsuite to intercept traffic and modify the payload to a malicious payload, thus achieving the effect of penetration testing.

[0043] The intelligent unit is the core of the intelligent system. It is the central coordination module responsible for managing the key logic and behavioral characteristics of the intelligent agent. It is a key part of generating tool call instructions, penetration test payloads and penetration test strategies. It can be regarded as the "intention understanding and content generation module" of the intelligent agent.

[0044] In this embodiment, the intelligent unit is implemented by a large model. In some professional fields or specific scenarios, the vertical field capabilities of the general large model are relatively weak. Therefore, in this embodiment, the field capabilities of the general large model are enhanced.

[0045] Specifically, the enhancement of domain capabilities mainly includes: combining a large amount of penetration test-related data captured in the graph structure in the memory unit, which can be used as a knowledge source for the large model. Through fine-tuning, the domain knowledge in the knowledge graph is instilled into the model parameters and representations, so that each step of the model in cognitive modeling can be affected by the context graph, thereby shaping complex reasoning in a tighter and more fine-grained manner, and making full use of the domain knowledge in the knowledge graph when performing downstream tasks. By aligning the data in the knowledge graph with the large model to improve the mastery of penetration test-related knowledge in the large language model, the generation and inference based on professional knowledge is realized, the facts retrieved from the knowledge graph and the implicit knowledge obtained from the large language model are used, and the large language model is guided to reason in the form of a mind map to solve the problem that the large language model is difficult to obtain knowledge updates in a timely manner and the effect is poor when dealing with time-sensitive issues.

[0046] The intelligent agent system proposed in the present invention improves the level of automation and intelligence of penetration test tasks by introducing large models, knowledge bases and task planning units. The knowledge graph is used as the memory unit of the intelligent agent to support the enhanced retrieval generation of the large model, improve the completeness of knowledge retrieval, and ensure the interpretability and consistency of the large model output. The task state machine is used as the planning unit of the intelligent agent, and the tool selection and calling of key links are guaranteed through predefined rules. For the complex scenarios of the penetration test process, highly customized agent behavior is implemented to ensure the execution process of the penetration test task.

[0047] Furthermore, the present invention also provides a penetration testing method for an intelligent agent system driven by the aforementioned large language model, comprising:

[0048] (1) Collect data from the Internet and build a penetration testing knowledge base;

[0049] (2) Build a domain-enhanced big model, combine it with the penetration testing knowledge base, and fine-tune the big model. The big model searches the penetration testing knowledge base to achieve retrieval-enhanced generation;

[0050] (3) Build a task planning state machine and define task state machines for different types of penetration testing tasks to ensure the consistency of the large model task execution process;

[0051] (4) Forming an intelligent agent based on the penetration testing knowledge base, the domain enhancement model, and the task planning state machine;

[0052] (5) After completing the construction of the intelligent agent system, the penetration test can be completed directly through the intelligent agent.

[0053] Please refer to Figure 2 , this embodiment provides a specific process of completing the penetration test through an intelligent agent:

[0054] ① The agent receives the penetration test target and prompt word sent by the user, detects the penetration test target, and obtains the detection result of the penetration test target;

[0055] ② The agent completes the penetration test task planning based on the detection results and provides feedback to the user;

[0056] ③After receiving the penetration test task planning confirmation information from the user, the agent generates the penetration test exploit load and instructions according to the task and sends them to the penetration test target;

[0057] ④ The intelligent agent receives the signal of achieving the purpose of exploitation from the penetration test feedback, and sends it to the user after completing the interpretation and evaluation.

[0058] The following takes the penetration test of the web application with IP address 10.0.2.5 as an example to illustrate how the penetration test agent works.

[0059] Please refer to Figure 3 First, the penetration test big model generates the task plan for the penetration test, and assigns tasks to the tool in combination with the penetration test strategy: For example, if the first step is to detect the target, a penetration test tool call instruction is generated: $nmap 10.0.2.5, and the task is conveyed to the tool call middleware, which actually calls the nmap tool. If the call is successful, the tool output is returned to the tool call big model. The tool call big model summarizes the output information of nmap: $ Only port 80 is open on 10.0.2.5. After that, by calling Nikto and the test script written specifically for the Flask service, it is determined whether the above vulnerabilities (misconfigured routes, template injection, and insecure deserialization) are triggered based on the running results of the test script collected by the middleware. If not triggered, the next test case is generated and the test script is called again.

[0060] The big model collects key information. Assuming that a template injection vulnerability is found, the penetration test big model updates the penetration test strategy and gives the next task. The tool calls the big model to generate the corresponding exp-{{''.__class__.mro()[1].__subclasses__()

[71] ('path / to / flag / file').read()}}. The middleware uses the tool to call the exp generated by the big model to test the 10.0.2.5 system, and finally successfully exploits the template injection vulnerability.

[0061] For those skilled in the art, the specific meanings of the above terms in the present invention can be understood in specific situations; the drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0062] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A penetration testing agent system driven by a large language model, characterized in that: include: The memory unit is composed of a penetration test knowledge base, which contains data elements such as vulnerability information, penetration methods, and tool information, and uses a knowledge graph to establish a vulnerability knowledge representation model; The intelligent unit, implemented by the large model, is used to generate tool call instructions, penetration test payloads, and penetration test strategies based on the memory unit; The planning unit is used to split the penetration test task into a task planning state machine consisting of controllable subtasks, thereby guiding the large language model to perform specific tasks; In the intelligent unit, the large model realizes domain capability enhancement based on the memory unit; the domain capability enhancement includes: using the penetration test knowledge base as the knowledge source of the large model; instilling the domain knowledge in the knowledge graph into the model parameters and representations through fine-tuning, and making full use of the domain knowledge in the knowledge graph when performing downstream tasks; aligning the data in the knowledge graph with the large model, facts retrieved from the knowledge graph and implicit knowledge obtained from the large language model, and guiding the large language model to reason in the form of a mind map; The planning unit includes a state representation module, a transfer condition module, a feedback optimization module and a tool calling module, wherein the state representation module is used to display the different stages of the penetration test task and the achievement status of the stage goals; the transfer condition module is used to define the transfer conditions between states; the feedback optimization module is used to decide whether to provide feedback to the user based on preset thresholds of state results, task progress and number of attempts; the tool calling module is used to select the corresponding tool to perform a specific task according to the current task.

2. The penetration testing agent system driven by a large language model according to claim 1, characterized in that: In the memory unit, a knowledge graph is used to establish a unified vulnerability knowledge representation model, integrate penetration test related data, and integrate information across resource boundaries.

3. A penetration testing method driven by a large language model, characterized in that: include: Collect data from the Internet and build a penetration testing knowledge base; Build a large language model, combine it with the penetration test knowledge base to complete domain enhancement, and complete retrieval enhancement generation by searching the penetration test knowledge base; The domain enhancement includes: using the penetration test knowledge base as the knowledge source of the big model; instilling the domain knowledge in the knowledge graph into the model parameters and representations through fine-tuning, and making full use of the domain knowledge in the knowledge graph when performing downstream tasks; aligning the data in the knowledge graph with the big model, facts retrieved from the knowledge graph and implicit knowledge obtained from the big language model, and guiding the big language model to reason in the form of a mind map; Build a task planning state machine and define task state machines for different types of penetration testing tasks; Form an intelligent agent based on the penetration testing knowledge base, the domain enhancement big model, and the task planning state machine; Complete penetration testing through intelligent agents; The penetration test knowledge base includes data elements such as vulnerability information, penetration methods, and tool information, and uses a knowledge graph to establish a vulnerability knowledge representation model; The penetration test completed by the agent specifically includes: The intelligent agent receives the penetration test target and prompt word sent by the user, detects the penetration test target, and obtains the detection result of the penetration test target; The agent completes the penetration test task planning based on the detection results and provides feedback to the user; After receiving the penetration test task planning confirmation information from the user, the agent generates the penetration test exploit load and instructions according to the task and sends them to the penetration test target; The intelligent agent receives the exploitation purpose achievement signal from the penetration test feedback, and sends it to the user after completing the interpretation and evaluation.

4. The large language model driven penetration testing method according to claim 2, characterized in that: The task state machine ensures the selection and calling of tools in each link through predefined rules.

Images