An automated penetration testing method and system based on a large language model

Abstract

This invention discloses an automated penetration testing method and system based on a large language model. It includes: acquiring the original text information of the test target and the penetration testing task; cleaning the information; predicting vulnerability types; generating vulnerability test cases using the large model; generating a penetration testing strategy, converting it into a test script and executing it automatically, generating execution results; determining whether the test is complete; if not, integrating the test records and regenerating the test strategy; if complete, generating a test report. The above steps are implemented using a large language model. This invention utilizes the capabilities of the large model that align with the reasoning, understanding, code generation, and basic security knowledge required in penetration testing, and improves the large model's ability to solve penetration testing sub-tasks through strategies such as the Prompt project. It constructs a system with a large model as the execution kernel capable of automated penetration testing, providing a new technical means for automated penetration testing.

Description

Technical Field

[0001] This invention relates to the field of penetration testing, and more particularly to an automated penetration testing method and system based on a large language model. Background Technology

[0002] Penetration testing is a crucial process for verifying the robustness and security of a system or website. Its main steps include information gathering, vulnerability scanning, vulnerability testing, vulnerability exploitation, and report generation. By conducting penetration testing on a target system or website, operations and maintenance personnel can mitigate risks in a timely manner, reduce the likelihood of system attacks, and developers can help fix code problems and improve code security.

[0003] To reduce the manual workload of security engineers and improve the efficiency of penetration testing, existing automated penetration testing tools use manually predefined rule sets. These rule sets form the basis for matching results with expected vulnerability test results in request-response packets. However, rule-based approaches lack the ability to autonomously reason and dynamically adjust testing strategies when faced with complex and evolving vulnerability types. Therefore, rule-based penetration testing often struggles to adapt to new scenarios and discover zero-day vulnerabilities, leading to high false positive rates when application behavior does not conform to predefined patterns.

[0004] The characteristics exhibited by large models reveal their suitability for addressing the challenges faced by automated penetration testing tools. However, the development of automated penetration testing tools using large models still faces numerous challenges:

[0005] (1) In manual testing, basic test cases undergo multiple iterations of optimization to generate correct test cases that can trigger vulnerabilities in the current testing environment. These optimized test cases are usually added to the rule set used by automated penetration testing tools. Although generating penetration testing test cases using large models seems intuitive, these test cases are usually quite basic and may not be suitable for the current testing environment. They often need to be iterated and adjusted multiple times, just like manual testing, to be optimized into correct test cases.

[0006] (2) Penetration testing covers a wide range of vulnerability types and exploitation methods. Penetration testing requires the use of diverse test cases for initial detection to identify as many potential vulnerabilities as possible, and then these basic test cases are further optimized. However, when generating basic test cases, large models typically generate common vulnerability test cases, such as SQL injection and XSS injection, resulting in a narrower range of vulnerabilities covered. Furthermore, large models tend to generate simpler and more commonly used test cases, while more complex and less frequently used test cases are difficult to generate directly. Therefore, it is necessary to change the traditional way large models generate basic test cases to generate test cases with broader vulnerability coverage and more diverse exploitation methods.

[0007] (3) During the output process of a large model, if the input information remains unchanged or the large model cannot detect changes in the information, the output content often becomes repetitive. Although this is a normal phenomenon, in the context of penetration testing, repeatedly generating test plans or test cases can hinder the testing process. Illusion can cause the large model to output useless or incorrect content. Both illusion and repetition can greatly reduce the effectiveness of the test content generated by the large model, hinder the penetration testing workflow, and ultimately lead to the failure of the penetration testing task.

[0008] The characteristics exhibited by the large model match the capabilities required by automated penetration testing tools. However, due to the above difficulties, there is a lack of relevant automated penetration testing tools. Therefore, it is important and necessary to design relevant exploitation methods and systems. Summary of the Invention

[0009] To address the aforementioned problems, this invention provides an automated penetration testing method and system based on a large language model.

[0010] The specific technical solution adopted in this invention is as follows:

[0011] A first aspect of this invention provides an automated testing method based on a large model, comprising the following steps:

[0012] 1) Obtain the address of the test target and the penetration test task; the test target is the system or website to be penetrated, and the address of the test target is an IP address or domain name;

[0013] 2) Crawl information about the test target based on its address, including the test target's source code and request headers;

[0014] 3) Send the information crawled in step 2) to the large language model. The large language model cleans the crawled information to obtain valid information.

[0015] 4) Determine whether the length of the information crawled in step 2) exceeds the input length limit of the large language model. If it exceeds the length limit, select the effective information after cleaning in step 3); otherwise, select the information crawled in step 2.

[0016] 5) Send the information selected in step 4) to the large language model, which then performs vulnerability type prediction.

[0017] 6) Send the vulnerability type predicted in step 5) and the information selected in step 4) to the large language model. The large language model generates vulnerability test cases and their usage instructions.

[0018] 7) Send the vulnerability test cases and their usage instructions generated in step 6) to the large language model, which will then output the test strategy.

[0019] 8) Send the test strategy generated in step 7) to the large language model, which then generates test scripts based on the test strategy;

[0020] 9) Execute the generated test script and obtain the execution results;

[0021] 10) Send the execution strategy of step 7), the test script of step 8), and the execution result of step 9) to the large language model, and the large language model integrates the sent content;

[0022] 11) Send the penetration testing task from step 1) and the integration result from step 10) to the large language model. The large language model determines whether the execution result completes the penetration testing task. If it is not completed, add the integration result from step 10) to the historical execution record, restart execution from step 7), and send the historical execution record to the large language model in step 7). If the result is complete, integrate the vulnerability test cases and their usage instructions generated in step 6) and the integration result from step 10) into a test report and output it.

[0023] In a second aspect, the present invention provides an automated testing system with a large model as its core, for implementing the above-mentioned automated testing method with a large model as its core, the system comprising an information collection module and an analysis and reasoning module;

[0024] The information collection module includes a crawler submodule, an information cleaning submodule, a vulnerability type prediction submodule, and a penetration aid knowledge submodule.

[0025] The crawler submodule is used to crawl information about the test target; the information cleaning submodule is used to clean up invalid information from the crawled information and retain valid information; the vulnerability type prediction submodule is used to predict the vulnerability type of the test target; and the penetration aid knowledge submodule is used to generate test cases and usage instructions.

[0026] The analysis and reasoning module includes a strategy generation submodule, a strategy execution submodule, a result integration submodule, and a historical execution record submodule.

[0027] The strategy generation submodule generates the test strategy for this test based on test cases, usage instructions, and historical execution records. The strategy execution submodule converts the test strategy into a test script and executes it. The result integration submodule integrates the penetration testing auxiliary knowledge, test strategy, test script, and execution results into the test report, compares the execution results with the penetration testing task, determines whether the test is completed, and outputs the test report. The historical execution record submodule extracts the test strategy, test script, test cases, and execution results for this test and generates historical execution records.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] This invention presents an automated testing method and system for penetration testing engineering, with a large model as its core. It proposes an automated penetration testing tool based on the large model and establishes a systematic and practical framework for solving how to apply the large model to the field of automated penetration testing. This invention also provides customized Prompt strategies for each sub-task of penetration testing, offering an effective method to improve the ability of the large model to complete each sub-task of penetration testing. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall structure of an automated testing system with a large model as its core, designed for penetration testing engineering.

[0031] Figure 2 This is a flowchart illustrating an automated testing method centered on a large model for penetration testing engineering.

[0032] Figure 3 This is a schematic diagram of the information collection module;

[0033] Figure 4 This is a schematic diagram of the analysis and reasoning module. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0035] like Figure 1 As shown, this invention proposes an automated testing system for penetration testing engineering, based on a large language model. It mainly consists of an information collection module and an analysis and reasoning module, comprising a total of eight sub-modules. This system is suitable for automated penetration testing.

[0036] The workflow of the entire automated penetration testing method is as follows: Figure 2 As shown, it includes the following steps:

[0037] (1) Enter the target address and penetration test task.

[0038] The test target is the system or website to be penetrated, and the address of the test target is an IP address or domain name;

[0039] The penetration testing task is a descriptive text, such as: "Detect whether the test target has security vulnerabilities", "Detect 5 security vulnerabilities in the test target", or "Detect whether the test target has security vulnerabilities of the backdoor attack type".

[0040] (2) Use Python's Request module to obtain the original text information of the test target, including the test target source code and request header information.

[0041] (3) Use large language models to extract information from the target text that is helpful for penetration testing.

[0042] In this embodiment, the large language model adopts the GPT model.

[0043] In this embodiment, there are two cases: the target original text exceeds the input length limit of the large language model and the target original text is within the input length limit of the large language model. If the target original text exceeds the limit, the target original text will be segmented and the large language model will be used to extract the effective information separately. If the length of the target original text does not exceed the limit, the large language model will be used directly to extract the effective information in the target original text.

[0044] (4) If the target information text exceeds the limit, select the valid information after segmentation processing by the large language model; if it does not exceed the limit, directly select the crawled target original text information.

[0045] (5) Send the information and penetration test task selected in step (4) to the big language model. The big model uses the selected information and penetration test task to predict the possible vulnerability types and output them in the form of an array. For example, if the target may have SQL injection vulnerability and command execution vulnerability, it will be output in the form of an array of ['SQL injection vulnerability', 'command execution vulnerability'].

[0046] (6) Send the penetration testing task from step (1), the vulnerability type predicted in step (5), and the information selected in step (4) to the large language model. The large language model describes the inference mode and output requirements. Using the large model, based on the input penetration testing task, vulnerability type, and selected target information, and according to the inference mode and output requirements, output vulnerability test cases and their usage instructions. The vulnerability test cases are a series of vulnerability exploitation parameters that can be input to the test target.

[0047] The reasoning mode and output requirements include: comprehensively considering the type of test function points, determining whether a filter exists and attempting to bypass it, ensuring that the output vulnerability test cases are syntactically correct, not being generated repeatedly, and generating usage methods and use cases for vulnerability test cases.

[0048] (7) Send the penetration test task from step (1), the vulnerability type predicted in step (5), the information selected in step (4), and the vulnerability test cases generated in step (6) to the big language model, describe the thinking pattern to the big language model, and use the big model to output the test ideas according to the thinking pattern for the input penetration test task, vulnerability type, target information, and vulnerability test cases.

[0049] In this embodiment, if it is the first time a test strategy is generated, no historical execution records are required; if it is not the first time a test strategy is generated, the historical execution records also need to be entered.

[0050] The thought process is as follows: if it is the initial stage of testing, the vulnerability test cases generated in step (6) are used for testing; if it is not the initial stage of testing, further testing is performed based on the test results that produce different echoes according to the historical execution records; when the test encounters a bottleneck, the vulnerability test cases generated in step (6) are used for testing again; avoid generating test strategies that are already in the historical execution records.

[0051] The testing strategy outlines which vulnerability test cases to use. Whether it's the initial stage of testing can be determined by the number of test results recorded in the historical execution log. Whether the test has encountered a bottleneck can be determined based on the historical execution log: if the most recent execution results are identical, then the test is considered to have encountered a bottleneck.

[0052] (8) Send the address of the test target and the penetration test task in step (1), the vulnerability type predicted in step (5), the information selected in step (4), and the test strategy generated in step (7) to the big language model, and use the big language model to generate a test script based on the input penetration test task, the predicted vulnerability type, the target information and the test strategy.

[0053] (9) Use Python's subprocess module to execute the script and output the execution result. The execution result is the output of the test target.

[0054] In this embodiment, there are two types: the current execution result exceeds the input length limit of the large language model and the result is within the length limit. If the current execution result exceeds the limit, the test script will be sent to the large language model after the test script is executed, so that the large language model can generate a test script without any test cases. Then, the difference between the two test results is compared, and the different text content is used as the output of the current test script. If the current execution result does not exceed the limit, the execution result is returned directly.

[0055] (10) Use large model analysis to integrate the execution results and add them to the historical execution record.

[0056] In this embodiment, a large language model is used to extract the core content of the testing strategy, the core code of the test script, and the content in the execution results that is helpful for further testing. The large language model is also used to analyze the expected execution results of the test script, integrating the extracted content and the expected execution results. If the expected execution results do not match the actual execution results, it is considered that a security vulnerability has been discovered. The meaning of the core content, core code, and content helpful for further testing can be automatically analyzed and determined by the large language model based on the testing strategy, test script, execution results, and penetration testing task.

[0057] (11) Use the large model to determine whether the penetration test task is completed based on the execution result. If it is completed, stop the test and output the test report. Otherwise, add the integration result of step (10) to the historical execution record, start the execution again from step (7), and send the historical execution record to the large language model in step (7).

[0058] In this embodiment, the test objective is the penetration testing task input in step (1), namely, "detecting whether the test target has a security vulnerability". Therefore, if a security vulnerability is found in this test, the penetration testing task is completed. The test report will integrate the vulnerability test cases and usage instructions from step (6) and the integration results from step (10).

[0059] In the above steps, except for steps (1), (2) and (4), the remaining steps are all implemented through the large language model, and each step is an independent dialogue.

[0060] In summary, a penetration test report based on this objective will be generated.

[0061] This embodiment also provides an automated penetration testing system based on a large language model as its kernel, which is used to implement the above embodiments. The terms "module," "unit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible.

[0062] The penetration testing system includes an information gathering module and an analysis and reasoning module;

[0063] The information collection module includes a crawler submodule, an information cleaning submodule, a vulnerability type prediction submodule, and a penetration aid knowledge submodule.

[0064] The crawler submodule is used to crawl information about the test target;

[0065] The information cleaning submodule is used to clean up invalid information from the crawled information and retain valid information.

[0066] The vulnerability type prediction submodule is used to predict the vulnerability type of the test target;

[0067] The Penetration Testing Assistance Knowledge Submodule is used to generate test cases and usage instructions;

[0068] The analysis and reasoning module includes a strategy generation submodule, a strategy execution submodule, a result integration submodule, and a historical execution record submodule.

[0069] The strategy generation submodule is used to generate the test strategy for this test based on the penetration test task, the crawled or cleaned information, the predicted vulnerability types, penetration aids, test cases and instructions, and historical execution records.

[0070] The strategy execution submodule is used to convert test strategies into test scripts and execute them;

[0071] The results integration submodule is used to integrate the auxiliary knowledge, testing strategies, test scripts and execution results of this penetration test as the content of the test report. It also compares the execution results with the penetration test task, decides whether to complete the test and outputs the test report.

[0072] The historical execution record submodule is used to extract the current test strategy, test script, test cases, and execution results to generate historical execution records.

[0073] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0074] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0075] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automated penetration testing method based on a large language model, characterized in that, Includes the following steps: 1) Obtain the address of the test target and the penetration testing task; 2) Crawl information about the test target based on its address, including the test target's source code and request headers; 3) Send the information crawled in step 2) to the large language model. The large language model cleans the crawled information to obtain valid information. 4) Determine whether the length of the information crawled in step 2) exceeds the input length limit of the large language model. If it exceeds the length limit, select the effective information after cleaning in step 3); otherwise, select the information crawled in step 2. 5) Send the information selected in step 4) to the large language model, which then performs vulnerability type prediction. 6) Send the vulnerability type predicted in step 5) and the information selected in step 4) to the large language model. The large language model generates vulnerability test cases and their usage instructions. 7) Send the vulnerability test cases and their usage instructions generated in step 6) to the large language model, and the large language model will output the test strategy; 8) Send the test strategy generated in step 7) to the large language model, which then generates test scripts based on the test strategy; 9) Execute the generated test script and obtain the execution results; 10) Send the execution strategy of step 7), the test script of step 8), and the execution result of step 9) to the large language model, and the large language model integrates the sent content; 11) Send the penetration testing task from step 1) and the integration result from step 10) to the large language model. The large language model determines whether the execution result completes the penetration testing task. If it is not completed, add the integration result from step 10) to the historical execution record, restart execution from step 7), and send the historical execution record to the large language model in step 7). If the result is complete, integrate the vulnerability test cases and their usage instructions generated in step 6) and the integration result from step 10) into a test report and output it.

2. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... Step 1) The test target is the system or website to be penetrated, and the address of the test target is an IP address or domain name; The penetration testing task is to detect whether the test target has any security vulnerabilities.

3. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... Step 3) specifically refers to: Determine whether the length of the crawled information exceeds the input length limit of the large language model; If the input length limit of the large language model is exceeded, the crawled information is segmented into multiple text segments, and the effective information in each text segment is extracted using the large language model. If the length limit is not exceeded, the effective information in the crawled information is extracted directly using the large language model; The valid information includes the functional parameters, functional description, request path, and response request header information of the test target.

4. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... Step 5) specifically refers to: The information and penetration testing task selected in step 4) are sent to the large language model, which then infers the potential security vulnerabilities of the test target and outputs the vulnerability types of the potential security vulnerabilities in the form of an array.

5. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... Step 6) specifically refers to: Send the penetration testing task from step 1), the vulnerability type predicted in step 5), and the information selected in step 4) to the large language model, and input the inference mode and output requirements into the large language model so that the large language model generates vulnerability test cases and their usage instructions according to the inference mode and output requirements. The reasoning mode and output requirements include: comprehensively considering the type of test function points, determining whether there is a filter and attempting to bypass it, ensuring that the output vulnerability test cases are syntactically correct, cannot be generated repeatedly, and generating usage methods and use cases for vulnerability test cases.

6. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... Step 7) specifically refers to: Send the penetration testing task from step 1), the vulnerability type predicted in step 5), the information selected in step 4), and the vulnerability test cases and their usage instructions generated in step 6) to the large language model. If this is not the first time a test strategy has been generated, also send the historical execution records to the large language model so that the large language model can generate the test strategy according to the thinking mode. The thought process is as follows: if it is the initial stage of testing, then use the vulnerability test cases generated in step 6) to perform testing; if it is not the initial stage of testing, then perform further testing based on the test results that produce different echoes according to the historical execution records; when the test encounters a bottleneck, then perform testing again based on the vulnerability test cases generated in step 6). Avoid generating test strategies that already exist in the historical execution records.

7. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... Step 8) specifically refers to: The address of the test target and the penetration test task in step 1), the predicted vulnerability type in step 5), the selected information in step 4), and the test strategy generated in step 7) are sent to the large language model, which then generates test scripts based on the test strategy.

8. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... Step 9) After obtaining the execution result, if the execution result exceeds the input length limit of the large language model, then perform the following steps: 9.1) Send the test script generated in step 8) to the large language model, so that the large language model can generate a test script without any test cases added; 9.2) Execute the test script without any test cases added; 9.3) Compare the execution results of the original test script with the execution results of the test script without any added test cases, and extract the differences between the two as the execution results to be used in subsequent steps.

9. The automated penetration testing method based on a large language model as its kernel, as described in claim 1, is characterized in that... The integration of the sent content described in step 10) specifically involves: using a large language model to extract the core content of this test strategy, the core code of the test script, and the content in the execution results that is helpful for further testing, and analyzing the expected execution results of the test script. The extracted content and expected execution results are the integration results.

10. An automated penetration testing system with a large language model as its core, characterized in that, The system for implementing the testing method of claim 1 includes an information collection module and an analysis and reasoning module; The information collection module includes a crawler submodule, an information cleaning submodule, a vulnerability type prediction submodule, and a penetration aid knowledge submodule. The crawler submodule is used to crawl information about the test target; The information cleaning submodule is used to clean up invalid information from the crawled information and retain valid information; the vulnerability type prediction submodule is used to predict the vulnerability type of the test target. The Penetration Testing Assistance Knowledge Submodule is used to generate test cases and usage instructions; The analysis and reasoning module includes a strategy generation submodule, a strategy execution submodule, a result integration submodule, and a historical execution record submodule. The strategy generation submodule is used to generate the test strategy for this test based on the test cases, usage instructions, and historical execution records. The strategy execution submodule is used to convert test strategies into test scripts and execute them; The results integration submodule is used to integrate the auxiliary knowledge, testing strategies, test scripts and execution results of this penetration test as the content of the test report, and compare the execution results with the penetration test task to determine whether the test is completed and output the test report. The historical execution record submodule is used to extract the current test strategy, test script, test cases, and execution results to generate historical execution records.

Images