Automatic penetration testing method and system with large language model as kernel

Abstract

The invention discloses an automatic penetration testing method and system with a large language model as a kernel. Comprising the steps of obtaining original text information of a test target and a penetration test task; cleaning information; performing vulnerability type prediction; generating a vulnerability test case by using the large model; generating a penetration test strategy, converting the penetration test strategy into a test script and automatically executing the test script to generate an execution result; and judging whether the test is completed or not, if not, integrating the test record and regenerating a test strategy, and if so, forming a test report. The steps are realized by adopting a large language model. According to the method, multiple capabilities of reasoning, understanding, code generation, basic safety knowledge reserve and the like required in fit and penetration test engineering in a large model are utilized, the capability of solving penetration test subtasks of the large model is improved through multiple strategies such as Prompt engineering, and a set of system which takes the large model as an execution kernel and can realize automatic penetration test is constructed. And a new technical means is provided for automatic penetration testing.

Description

Technical Field

[0001] The present invention relates to the field of penetration testing, and in particular to an automated penetration testing method and system with a large language model as the core. Background Art

[0002] Penetration testing is a key project to test the robustness and security of a system or website. Its steps are mainly divided into information collection, vulnerability scanning, vulnerability testing, vulnerability exploitation, and report generation. By performing penetration testing on the target system or website, it can help operation and maintenance personnel avoid risks in a timely manner, reduce the possibility of the system being attacked, and help developers fix code problems and improve code security.

[0003] To reduce the manual workload of security engineers and improve the efficiency of penetration testing, existing automated penetration testing tools use manually predefined rule sets. These rule sets are the basis for matching results with expected vulnerability test results in request-response packets. However, rule-based approaches lack the ability to autonomously reason and dynamically adjust test strategies when faced with complex and evolving vulnerability types. As a result, rule-based penetration testing often has difficulty adapting to new scenarios and discovering zero-day vulnerabilities, resulting in a high false positive rate when application behavior does not conform to predefined patterns.

[0004] Combining the characteristics of large models, it can be found that they are very suitable for solving the problems faced by automated penetration testing tools. However, the development of automated penetration testing tools using large models still faces many challenges:

[0005] (1) In manual testing, basic test cases are optimized through multiple iterations to generate correct test cases that can trigger vulnerabilities in the current test environment. These optimized test cases are usually added to the rule set used by automated penetration testing tools. Although it seems intuitive to use large models to generate penetration testing test cases, these test cases are usually relatively basic and may not be suitable for the current test environment. They often need to be adjusted through multiple iterations, just like manual testing, before they can be optimized into correct test cases.

[0006] (2) A wide range of vulnerability types and exploitation methods are covered in penetration testing. Penetration testing requires the use of a variety of test cases for preliminary detection to identify as many potential vulnerabilities as possible, and then further optimizes these basic test cases. However, when generating basic test cases, large models usually generate common vulnerability test cases, such as SQL injection and XSS injection, resulting in a narrow range of vulnerabilities covered. In addition, large models tend to generate simpler and more commonly used test cases, while more complex and less commonly used test cases are difficult to generate directly. Therefore, it is necessary to change the traditional way of generating basic test cases with large models to generate test cases with wider vulnerability coverage and more diverse exploitation methods.

[0007] (3) During the output process of the large model, if the input information remains unchanged or the large model cannot detect the change in information, the output content will often be repeated. Although this is a normal phenomenon, in the context of penetration testing, repeated generation of test plans or test cases may hinder the testing process. The hallucination phenomenon will cause the large model to output useless or incorrect content. Both hallucinations and repetitions will greatly reduce the effectiveness of the test content generated by the large model, hinder the workflow of the penetration test, and ultimately lead to the failure of the penetration test task.

[0008] The characteristics exhibited by the large model are consistent with the capabilities required by automated penetration testing tools. However, due to the above difficulties, there is a lack of relevant automated penetration testing tools. Therefore, it is important and necessary to design relevant utilization methods and utilization systems. Summary of the invention

[0009] In view of the above problems, the present invention provides an automated penetration testing method and system with a large language model as the core.

[0010] The specific technical solution adopted by the present invention is as follows:

[0011] The first aspect of the present invention provides an automated testing method with a large model as the core, comprising the following steps:

[0012] 1) Obtain the address of the test target and the penetration test task; the test target is the system or website to be penetrated, and the address of the test target is an IP address or a domain name;

[0013] 2) crawling the information of the test target according to the address of the test target, wherein the information includes the source code and request header of the test target;

[0014] 3) Sending the crawled information in step 2) to the large language model, the large language model performs information cleaning on the crawled information to obtain valid information;

[0015] 4) Determine whether the length of the information crawled in step 2) exceeds the input length limit of the large language model. If it exceeds the length limit, select the valid information after cleaning in step 3), otherwise select the information crawled in step 2);

[0016] 5) Sending the information selected in step 4) to the large language model, which predicts the vulnerability type;

[0017] 6) Sending the vulnerability type predicted in step 5) and the information selected in step 4) to the large language model, and the large language model generates vulnerability test cases and instructions for use thereof;

[0018] 7) Send the vulnerability test cases and their instructions generated in step 6) to the large language model, and the large language model will output the test strategy;

[0019] 8) sending the test strategy generated in step 7) to the large language model, and the large language model generates a test script based on the test strategy;

[0020] 9) Execute the generated test script to obtain the execution result;

[0021] 10) sending the execution strategy of step 7), the test script of step 8), and the execution result of step 9) to the large language model, and the large language model integrates the sent content;

[0022] 11) The penetration test task in step 1) and the integrated result of step 10) are sent to the large language model, and the large language model determines whether the execution result completes the penetration test task; if not, the integrated result of step 10) is added to the historical execution record, and the execution is restarted from step 7), and the historical execution record is sent to the large language model in step 7); if the judgment result is completed, the vulnerability test case generated in step 6) and its usage instructions and the integrated result of step 10) are integrated into a test report and output.

[0023] A second aspect of the present invention provides an automated testing system with a large model as the core, which is used to implement the automated testing method with a large model as the core, wherein the system includes an information collection module and an analysis and reasoning module;

[0024] The information collection module includes a crawler submodule, an information cleaning submodule, a vulnerability type prediction submodule and a penetration auxiliary knowledge submodule;

[0025] The crawler submodule is used to crawl the information of the test target; the information cleaning submodule is used to clean the invalid information in the crawled information and retain the valid information; the vulnerability type prediction submodule is used to predict the vulnerability type of the test target; the penetration auxiliary knowledge submodule is used to generate test cases and instructions for use;

[0026] The analysis and reasoning module includes a strategy generation submodule, a strategy execution submodule, a result integration submodule and a historical execution record submodule;

[0027] The strategy generation submodule is used to generate the test strategy for this test based on the test cases, instructions for use and historical execution records; the strategy execution submodule is used to convert the test strategy into a test script and execute it; the result integration submodule is used to integrate the penetration auxiliary knowledge, test strategy, test script and execution results as the content of the test report, and compare the execution results with the penetration test tasks to determine whether the test is completed in order to output the test report; the historical execution record submodule is used to extract the test strategy, test script, test case and execution results of this test to generate a historical execution record.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention provides an automated testing method and system with a big model as the core for penetration testing projects, proposes an automated penetration testing tool based on the big model, and establishes a systematic practical framework for solving how to apply the big model to the field of automated penetration testing, which is practical; the present invention provides a personalized Prompt strategy for each subtask of the penetration test, and provides an effective method for improving the ability of the big model in completing each subtask of the penetration test. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the overall structure of an automated testing system based on a large model for penetration testing projects;

[0031] Figure 2 This is a flowchart of an automated testing method based on a large model for penetration testing projects;

[0032] Figure 3 This is a schematic diagram of the information collection module;

[0033] Figure 4 Schematic diagram of the analysis and reasoning module. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0035] like Figure 1 As shown, the present invention proposes an automated testing system for penetration testing engineering with a large language model as the core, which is mainly divided into an information collection module and an analysis and reasoning module, and contains eight submodules in total. The system is suitable for automated penetration testing work.

[0036] The workflow of the entire automated penetration testing method is as follows: Figure 2 As shown, the following steps are included:

[0037] (1) Enter the test target address and penetration test task.

[0038] The test target is a system or website to be subjected to penetration testing, and the address of the test target is an IP address or a domain name;

[0039] The penetration test task is a descriptive text, for example: "Detect whether the test target has security vulnerabilities", "Detect 5 security vulnerabilities in the test target" or "Detect whether the test target has backdoor attack type security vulnerabilities.

[0040] (2) Use Python's Request module to obtain the original text information of the test target, including the test target source code and request header information.

[0041] (3) Use large language models to extract information from the target text that is helpful for penetration testing.

[0042] In this embodiment, the large language model adopts the GPT model.

[0043] In this embodiment, there are two situations: the target original text exceeds the input length limit of the large language model and the target original text is within the input length limit of the large language model. If the target original text exceeds the limit, the target original text will be segmented and the large language model will be used to extract valid information separately. If the target original text length does not exceed the limit, the large language model will be used directly to extract valid information in the target original text.

[0044] (4) If the target information text exceeds the limit, the valid information after segmentation processing by the large language model is selected. If it does not exceed the limit, the crawled target original text information is directly selected.

[0045] (5) The information and penetration test task selected in step (4) are sent to the large language model. The large model is used to predict the possible vulnerability types based on the selected information and penetration test task, and the predictions are output in the form of an array. For example, if the target may have a SQL injection vulnerability or a command execution vulnerability, the predictions are output in the form of an array of ['SQL injection vulnerability', 'command execution vulnerability'].

[0046] (6) Sending the penetration test task of step (1), the vulnerability type predicted by step (5) and the information selected by step (4) to the large language model, and describing the reasoning mode and output requirements by the large language model, using the large model to output the vulnerability test case and its usage instructions according to the input penetration test task, vulnerability type, selected target information and the reasoning mode and output requirements. The vulnerability test case is a series of vulnerability exploitation parameters that can be input to the test target.

[0047] The reasoning mode and output requirements include: comprehensively considering the type of test function points, determining whether there is a filter and trying to bypass it, ensuring that the output vulnerability test cases are grammatically correct and cannot be generated repeatedly, and generating usage methods and use cases for vulnerability test cases.

[0048] (7) The penetration test task of step (1), the vulnerability type predicted based on step (5), the information selected in step (4), and the vulnerability test case generated in step (6) are sent to the large language model, and the thinking mode is described to the large language model. The large model is used to output the test ideas for the input penetration test task, vulnerability type, target information, and vulnerability test case according to the thinking mode.

[0049] In this embodiment, if the test strategy is generated for the first time, no historical execution record is required. If the test strategy is not generated for the first time, the historical execution record needs to be input.

[0050] The thinking mode is: if it is the early stage of testing, use the vulnerability test case generated in step (6) for testing; if it is not the early stage of testing, further test the test results that produce different echoes according to the historical execution records; when the test encounters a bottleneck, test again according to the vulnerability test case generated in step (6); avoid generating test strategies that already exist in the historical execution records.

[0051] The test strategy is to use which vulnerability test cases for testing. Whether it is the initial stage of testing can be judged based on the number of test results recorded in the historical execution records. Whether the test encounters a bottleneck can be judged based on the historical execution records: if the recent execution results in the historical execution records are the same, it is considered that the test has encountered a bottleneck.

[0052] (8) The address of the test target and the penetration test task of step (1), the vulnerability type predicted in step (5), the information selected in step (4), and the test strategy generated in step (7) are sent to the large language model, and the large language model is used to generate a test script according to the input penetration test task, predicted vulnerability type, target information, and test strategy.

[0053] (9) Use Python's subprocess module to execute the script and output the execution result. The execution result is the output of the test target.

[0054] In this embodiment, the execution result is divided into two types: the execution result exceeds the input length limit of the large language model and the execution result is within the length limit. If the execution result exceeds the limit, the test script will be sent to the large language model after the test script is executed, so that the large language model generates a test script without adding any test cases, and then compares the difference between the test results of the two, and uses the different text content as the output of this test script. If the execution result does not exceed the limit, the execution result is directly returned.

[0055] (10) Use a large model to analyze and integrate execution results and add them to historical execution records.

[0056] In this embodiment, a large language model is used to extract the core content of this test strategy, the core code of the test script, and the content in the execution result that is helpful for further testing, and the large language model is used to analyze the expected execution result of the test script, and the integrated execution result includes the extracted content and the expected execution result. If the expected execution result does not match the actual execution result, it is considered to be a security vulnerability. The meaning of the core content, core code, and content that is helpful for further testing can be analyzed and judged by the large language model based on the test strategy, test script, execution results, and penetration test tasks.

[0057] (11) Use the large model to determine whether the penetration test task is completed based on the current execution result. If completed, stop the test and output the test report. Otherwise, add the integrated result of step (10) to the historical execution record, restart the execution from step (7), and send the historical execution record to the large language model in step (7).

[0058] In this embodiment, the test target is the penetration test task input in step (1), that is, "detect whether the test target has security vulnerabilities", so if this test finds security vulnerabilities, the penetration test task is completed. The test report will integrate the vulnerability test cases and instructions of step (6) and the integrated results of step (10).

[0059] In the above steps, except for steps (1), (2) and (4), the remaining steps are implemented through the large language model, and the execution of each step is an independent dialogue.

[0060] In summary, a penetration test report based on this target will be output eventually.

[0061] In this embodiment, an automated penetration testing system with a large language model as the core is also provided, and the system is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0062] The penetration testing system includes an information collection module and an analysis and reasoning module;

[0063] The information collection module includes a crawler submodule, an information cleaning submodule, a vulnerability type prediction submodule and a penetration auxiliary knowledge submodule;

[0064] The crawler submodule is used to crawl the information of the test target;

[0065] The information cleaning submodule is used to clean invalid information from the crawled information and retain valid information;

[0066] The vulnerability type prediction submodule is used to predict the vulnerability type of the test target;

[0067] The penetration-assisted knowledge submodule is used to generate test cases and usage instructions;

[0068] The analysis and reasoning module includes a strategy generation submodule, a strategy execution submodule, a result integration submodule and a historical execution record submodule;

[0069] The strategy generation submodule is used to generate the test strategy for this test based on the penetration test task, crawled information or cleaned information, predicted vulnerability type, penetration auxiliary knowledge, test cases and instructions, and historical execution records;

[0070] The strategy execution submodule is used to convert the test strategy into a test script and execute it;

[0071] The result integration submodule is used to integrate the penetration auxiliary knowledge, test strategy, test script and execution results as the content of the test report, and compare the execution results with the penetration test task to decide whether to complete the test and output the test report;

[0072] The historical execution record submodule is used to extract the test strategy, test script, test case and execution results of this test and generate the historical execution record.

[0073] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0074] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0075] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An automated penetration testing method based on a large language model, characterized in that: The following steps are involved: 1) Obtain the address of the test target and the penetration test task; 2) crawling the information of the test target according to the address of the test target, wherein the information includes the source code and request header of the test target; 3) Sending the crawled information in step 2) to the large language model, the large language model performs information cleaning on the crawled information to obtain valid information; 4) Determine whether the length of the information crawled in step 2) exceeds the input length limit of the large language model. If it exceeds the length limit, select the valid information after cleaning in step 3), otherwise select the information crawled in step 2); 5) Sending the information selected in step 4) to the large language model, which predicts the vulnerability type; 6) Sending the vulnerability type predicted in step 5) and the information selected in step 4) to the large language model, and the large language model generates vulnerability test cases and instructions for use thereof; 7) Send the vulnerability test cases and their instructions generated in step 6) to the large language model, and the large language model outputs the test strategy; 8) sending the test strategy generated in step 7) to the large language model, and the large language model generates a test script based on the test strategy; 9) Execute the generated test script to obtain the execution result; 10) sending the execution strategy of step 7), the test script of step 8), and the execution result of step 9) to the large language model, and the large language model integrates the sent content; 11) The penetration test task in step 1) and the integrated result of step 10) are sent to the large language model, and the large language model determines whether the execution result completes the penetration test task; if not, the integrated result of step 10) is added to the historical execution record, and the execution is restarted from step 7), and the historical execution record is sent to the large language model in step 7); if the judgment result is completed, the vulnerability test case generated in step 6) and its usage instructions and the integrated result of step 10) are integrated into a test report and output.

2. According to claim 1, an automated penetration testing method with a large language model as the core is characterized in that: The test target described in step 1) is a system or website to be subjected to penetration testing, and the address of the test target is an IP address or a domain name; The penetration test task is to detect whether there are security vulnerabilities in the test target.

3. The automated penetration testing method based on a large language model as the core according to claim 1, characterized in that: Step 3) is specifically: Determine whether the length of the crawled information exceeds the input length limit of the large language model; If the input length exceeds the limit of the large language model, the crawled information is divided into multiple text segments, and the large language model is used to extract valid information from each text segment. If the length limit is not exceeded, the large language model is used directly to extract valid information from the crawled information; The valid information includes the function parameters, function description, request path and response request header information of the test target.

4. The automated penetration testing method based on a large language model as the core according to claim 1, characterized in that: Step 5) is specifically: The information selected in step 4) and the penetration test task are sent to the large language model, so that the large language model infers the potential security vulnerabilities of the test target and outputs the vulnerability types of the potential security vulnerabilities in the form of an array.

5. The automated penetration testing method based on a large language model as the core according to claim 1, characterized in that: Step 6) is specifically as follows: Send the penetration test task of step 1), the vulnerability type predicted by step 5), and the information selected by step 4) to the large language model, and input the reasoning mode and output requirements to the large language model, so that the large language model generates vulnerability test cases and instructions for use thereof according to the reasoning mode and output requirements; The reasoning model and output requirements include: comprehensively considering the type of test function points, determining whether there are filters and trying to bypass them, ensuring that the output vulnerability test cases are grammatically correct and cannot be generated repeatedly, and generating usage methods and use cases for vulnerability test cases.

6. The automated penetration testing method based on a large language model as the core according to claim 1, characterized in that: Step 7) is specifically as follows: The penetration test task of step 1), the vulnerability type predicted based on step 5), the information selected in step 4), and the vulnerability test case generated in step 6) and its usage instructions are sent to the large language model. If this is not the first time to generate a test strategy, the historical execution record is also sent to the large language model, so that the large language model generates a test strategy according to the thinking mode; The thinking mode is: if it is the early stage of testing, then use the vulnerability test case generated in step 6) for testing; if it is not the early stage of testing, then further test the test results with different echoes according to the historical execution records; when the test encounters a bottleneck, test again according to the vulnerability test case generated in step 6); Avoid generating test strategies that already exist in historical execution records.

7. The automated penetration testing method based on a large language model as the core according to claim 1, characterized in that: Step 8) is specifically as follows: The address of the test target and the penetration test task of step 1), the vulnerability type predicted in step 5), the information selected in step 4), and the test strategy generated in step 7) are sent to the large language model, and the large language model generates a test script based on the test strategy.

8. The automated penetration testing method based on a large language model as the core according to claim 1, characterized in that: Step 9) After obtaining the execution result, if the execution result exceeds the input length limit of the large language model, execute the following steps: 9.1) Sending the test script generated in step 8) to the large language model, so that the large language model generates a test script without adding any test cases; 9.2) Execute the test script without adding any test cases; 9.3) Compare the execution results of the original test script with the execution results of the test script without adding any test cases, and extract the different contents between the two as the execution results adopted in the subsequent steps.

9. The automated penetration testing method based on a large language model as the core according to claim 1, characterized in that: The integration of the sent content described in step 10) is specifically as follows: using a large language model to extract the core content of this test strategy, the core code of the test script, and the content in the execution results that is helpful for further testing, and analyzing the expected execution results of the test script. The extracted content and expected execution results are the integration results.

10. An automated penetration testing system with a large language model as its core, characterized in that: For implementing the test method of claim 1, the system comprises an information collection module and an analysis and reasoning module; The information collection module includes a crawler submodule, an information cleaning submodule, a vulnerability type prediction submodule and a penetration auxiliary knowledge submodule; The crawler submodule is used to crawl the information of the test target; The information cleaning submodule is used to clean invalid information from the crawled information and retain valid information; the vulnerability type prediction submodule is used to predict the vulnerability type of the test target; The penetration-assisted knowledge submodule is used to generate test cases and usage instructions; The analysis and reasoning module includes a strategy generation submodule, a strategy execution submodule, a result integration submodule and a historical execution record submodule; The strategy generation submodule is used to generate the test strategy for this test based on the test cases, usage instructions and historical execution records; The strategy execution submodule is used to convert the test strategy into a test script and execute it; The result integration submodule is used to integrate the penetration auxiliary knowledge, test strategy, test script and execution results as the content of the test report, and compare the execution results with the penetration test task to determine whether the test is completed and output the test report; The historical execution record submodule is used to extract the test strategy, test script, test case and execution results of this test and generate the historical execution record.

Images