Penetration testing methods, products, equipment and media based on LLM and entity extraction

Abstract

The present application relates to the field of penetration testing technology, and in particular to a penetration testing method, product, device and medium based on LLM and entity extraction. The method comprises: obtaining the current vulnerability information input by the user, using LLM to generate a task plan corresponding to the current vulnerability information based on the current vulnerability information and a known vulnerability database; calling an API prompt file, inputting the task plan and the API prompt file into LLM to obtain an initial call dictionary; normalizing the initial call dictionary based on a preset API specification vocabulary to obtain a call dictionary; and calling the corresponding API based on the call dictionary to execute the task plan and obtain a penetration test result. The present application can flexibly handle new vulnerabilities and achieve efficient penetration testing.

Description

Technical Field

[0001] The present application relates to the field of penetration testing technology, and in particular to a penetration testing method, product, equipment and medium based on LLM and entity extraction. Background Art

[0002] As the number of IoT devices around the world has exploded, cybersecurity threats in the IoT field are also increasing. The types of security attacks and threats faced by enterprises and organizations are increasing, and their complexity is also increasing. Although traditional manual penetration testing methods are effective, they have many limitations. Automated penetration testing can significantly improve testing efficiency, shorten testing cycles, and enable testing to be conducted more frequently, thereby promptly discovering and fixing security vulnerabilities. In addition, automated penetration testing has a wider coverage, especially when facing large-scale networks and complex systems. It can quickly scan and test a wide range of targets through automated tools, improving the comprehensiveness of the test.

[0003] There are many difficulties in conducting automated penetration testing on the Internet of Things, especially in the field of video surveillance. First, there are many brands and models of IoT video surveillance devices, and the hardware and software architectures of different devices vary greatly, which poses a challenge to the compatibility and adaptability of automated testing tools. Secondly, device manufacturers continue to release firmware and software updates, and automated testing tools need to continuously track and adapt these updates to ensure the accuracy and reliability of test results. In addition, in the face of the complexity of unknown vulnerabilities, automated testing tools also need to have strong vulnerability identification and similarity matching capabilities. For unknown vulnerabilities, the tool must be able to find possible related vulnerabilities through similarity calculations, and provide relevant information to try to use the corresponding test scripts. For known vulnerabilities, the tool needs to quickly update and apply the corresponding vulnerability exploits to verify whether the device has the same or similar security risks. All of these require automated penetration testing tools to be highly intelligent and flexible to cope with ever-changing security threats and device environments.

[0004] In the prior art, there are already some automated penetration testing methods for the Internet of Things field, such as a static method that uses predefined rules and vulnerability databases to scan and identify known vulnerabilities in the system. This method usually relies on predefined rules and vulnerability databases for scanning and identification. This static method cannot flexibly adapt to newly emerging vulnerabilities, resulting in the inability to update and use new attack methods in a timely manner. Summary of the invention

[0005] In order to solve the problem that the penetration testing method in the prior art cannot adapt to new vulnerabilities, the present application provides a penetration testing method, product, equipment and medium based on LLM and entity extraction.

[0006] In the first aspect, the present application provides a penetration testing method based on LLM and entity extraction, which adopts the following technical solutions:

[0007] A penetration testing method based on LLM and entity extraction, including:

[0008] Obtain current vulnerability information input by the user, and use LLM to generate a task plan corresponding to the current vulnerability information based on the current vulnerability information and a known vulnerability database;

[0009] Retrieving an API prompt file, inputting the task plan and the API prompt file into the LLM, and obtaining an initial call dictionary;

[0010] Normalizing the initial call dictionary based on a preset API standard vocabulary to obtain a call dictionary;

[0011] Based on the calling dictionary, the corresponding API is called to execute the task planning and obtain the penetration test result.

[0012] By adopting the above technical solution, the current vulnerability information input by the user is obtained, and the current vulnerability information and the known vulnerability database are compared by LLM, which can effectively identify and handle unknown vulnerabilities, ensuring that each step in the test process can be flexibly updated and adapted to new vulnerabilities. The overall penetration test task is divided into reasonable subtasks through LLM and the known vulnerability database to complete the task planning, and the initial call dictionary containing the scheduling plan is generated based on the task planning and API prompt file. The customized preset API specification vocabulary unifies the vector representation of entity words in different documents, and accurately maps the parameter information in the initial call dictionary to the preset API specification vocabulary through word vector similarity. LLM is used for task planning and flexible assembly of custom APIs, and task planning is executed through the reflection mechanism, realizing efficient and automated penetration testing.

[0013] In a preferred example, the present application may be further configured as follows: the step of using LLM to generate a task plan corresponding to the current vulnerability information based on the current vulnerability information and a known vulnerability database includes:

[0014] Using the LLM to search the known vulnerability database for similar vulnerabilities matching the current vulnerability information and the EXP scripts of the similar vulnerabilities;

[0015] The current vulnerability information and the EXP script are spliced ​​to obtain an executable instance;

[0016] Retrieve a task prompt file, input the executable instance and the task prompt file into the LLM, and obtain a task plan corresponding to the current vulnerability information.

[0017] By adopting the above technical solution, similar vulnerabilities matching the current vulnerability information are searched from the known vulnerability database, and new attack modes and vulnerability information can be obtained dynamically and in real time. Based on the semantic understanding and logical reasoning capabilities of LLM, the key entity extraction of the current vulnerability information and the similarity calculation with the known vulnerability database can be realized more accurately and efficiently. The current vulnerability information and the EXP script are spliced ​​to obtain the executable script corresponding to the current vulnerability information. The task prompt file is used to guide the process of generating task plans from executable instances. Based on the executable instance and the task prompt file, a task plan that conforms to the current vulnerability information can be generated.

[0018] In a preferred example, the present application may be further configured as follows: the initial call dictionary is normalized based on a preset API standard vocabulary to obtain a call dictionary, including:

[0019] Reading the initial call dictionary into a parameter format corresponding to the preset API specification vocabulary;

[0020] Convert the initial call dictionary after adjusting the parameter format into a vector form;

[0021] Matching the initial call dictionary in vector form with the preset API specification vocabulary, performing a replacement operation on each successfully matched keyword in the initial call dictionary to obtain a call dictionary after replacement;

[0022] Wherein, for any successfully matched keyword in the initial call dictionary, the replacement operation is: determining a standard word that successfully matches the keyword in the preset API standard word list, and replacing the keyword with the standard word.

[0023] By adopting the above technical solution and pre-defining the API specification vocabulary, we can flexibly integrate and configure the newly added custom API, ensure that the test plan can respond to new vulnerabilities in a timely manner, and generate a specified call dictionary based on LLM to drive the specified API for testing. By uniformly representing the same entity, it ensures that the test data call dictionary generated by LLM strictly matches the actual API parameters, avoiding test errors caused by parameter mismatch and improving the flexibility and scalability of system functions.

[0024] In a preferred example, the present application may be further configured as follows: the task plan includes a plurality of subtasks having an execution order, and each subtask corresponds to an API prompt file;

[0025] The step of inputting the task plan and the API prompt file into the LLM to obtain an initial call dictionary includes:

[0026] Using the LLM to process the multiple subtasks in order of execution, and using the model output containing the API call sequence obtained after processing the last subtask as the initial call dictionary;

[0027] Wherein, any subtask among the multiple subtasks is used as a target subtask, and when the target subtask is the first subtask to be executed, the target subtask is processed by using the LLM, including: inputting the target subtask and the API prompt file corresponding to the target subtask into the LLM, and obtaining the model output corresponding to the target subtask;

[0028] When the target subtask is not the first subtask to be executed, the target subtask is processed using the LLM, including: inputting the target subtask, the API prompt file corresponding to the target subtask, and the model output corresponding to the previous task of the target subtask into the LLM to obtain the model output corresponding to the target subtask.

[0029] By adopting the above technical solution, task segmentation and subtask combination are adopted, and a call dictionary is used to define the function calls, parameters and output descriptions of each penetration testing task, flexible organization and accurate execution of tasks are ensured, and automated operations in different testing phases are supported. LLM is used for task planning and API call planning, and test task planning and API call sequence are intelligently generated, which improves the intelligence level and adaptability of penetration testing.

[0030] In a preferred example, the present application may be further configured as follows: the calling of the corresponding API based on the calling dictionary to execute the task planning and obtain the penetration test result includes:

[0031] Determine, based on the call dictionary, steps in the task plan that do not have dependencies under the same subtask, and submit the steps that do not have dependencies to a thread pool, wherein the task plan includes a scanning subtask, a vulnerability detection subtask, and a vulnerability exploitation subtask;

[0032] The scanning subtask, the vulnerability detection subtask and the vulnerability exploitation subtask are sequentially executed based on the calling dictionary and the corresponding API, wherein the thread pool processes the steps having no dependency in parallel.

[0033] By adopting the above technical solution, the scanning subtask, vulnerability detection subtask and vulnerability exploitation subtask are executed in sequence, ensuring the efficiency and comprehensiveness of the penetration test. The steps without dependencies are submitted to the thread pool for parallel processing, thereby improving the execution efficiency of the penetration test.

[0034] In a preferred example, the present application can be further configured as follows: the method further includes:

[0035] Obtain a threat intelligence document, and obtain a span in an encoded form based on the threat intelligence document;

[0036] Constructing a span multi-classifier, and using the span multi-classifier to determine the preset entity category to which each span belongs with the highest probability from multiple preset entity categories, and obtaining the categories and probabilities corresponding to each of the multiple spans;

[0037] Combining the code and probability of each span into a feature vector, and constructing a storage matrix based on the feature vectors corresponding to each of the plurality of spans;

[0038] Determine a fusion feature of each span based on the storage matrix;

[0039] Input the fusion features of each span into the multi-layer perceptron to obtain the category of each span;

[0040] The obtained categories of the multiple spans are stored as a preset API specification vocabulary.

[0041] By adopting the above technical solution, the document entity extraction method is used to obtain the unified representation of entities in the threat intelligence document and create a preset API keyword table to match API parameter information. This realizes the analysis and classification of vulnerability text data that is difficult to regularize, ensures strict matching of parameters during the test process, and can cope with the security challenges brought by ever-changing unknown vulnerabilities.

[0042] In a preferred example, the present application may be further configured as follows: determining the fusion feature of each span based on the storage matrix includes:

[0043] Taking any span among the multiple spans as a target span, taking the encoding of the target span as a local feature of the target span, and taking the category corresponding to the target span as a target category;

[0044] Determine a span set whose category is the target category from the multiple spans;

[0045] Based on the weighted average of each span feature vector in the span set, a global feature of the target span is obtained;

[0046] The local features and global features of the target span are balanced through a gate mechanism to obtain a fusion feature of the target span.

[0047] By adopting the above technical solution, for each span, its local features (i.e., encoding) are combined with the global features (the weighted average features of other spans under the same category), and the two are intelligently balanced through the gate mechanism to generate a more comprehensive and accurate fusion feature, which effectively improves the richness and accuracy of information representation.

[0048] In a second aspect, the present application provides a computer program product, which adopts the following technical solution:

[0049] A computer program product includes a computer program. When the computer program is executed by a processor, it implements the penetration testing method based on LLM and entity extraction as described in any one of the first aspects.

[0050] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0051] one or more processors;

[0052] Memory;

[0053] At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the penetration testing method based on LLM and entity extraction as described in any one of the first aspects.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0055] A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the penetration testing method based on LLM and entity extraction as described in any one of the first aspects.

[0056] In summary, this application includes the following beneficial technical effects:

[0057] This application obtains the current vulnerability information input by the user and uses LLM to compare the current vulnerability information with the known vulnerability database, so as to effectively identify and handle unknown vulnerabilities, ensure that each step in the test process can be flexibly updated and adapted to new vulnerabilities, and divide the overall penetration test task into reasonable subtasks through LLM and the known vulnerability database to complete task planning, and generate an initial call dictionary containing scheduling planning based on task planning and API prompt files. The customized preset API specification vocabulary unifies the vector representation of entity words in different documents, and accurately maps the parameter information in the initial call dictionary to the preset API specification vocabulary through word vector similarity. LLM is used for task planning and flexible assembly of custom APIs, and task planning is executed through the reflection mechanism, thereby realizing efficient and automated penetration testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of a penetration testing method based on LLM and entity extraction provided in an embodiment of the present application;

[0059] Figure 2is a workflow diagram of a known vulnerability database provided in an embodiment of the present application;

[0060] Figure 3 is a schematic diagram of the structure of a virtual module provided in an embodiment of the present application;

[0061] Figure 4 It is a module flow chart provided in the embodiment of the present application;

[0062] Figure 5 It is a backend function execution flow chart provided in an embodiment of the present application;

[0063] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following is combined with Figure 1 -Attached Figure 6 This application is described in further detail.

[0065] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0068] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0069] Related technologies also use deep reinforcement learning to train agents to plan attacks in simulated environments, but its application is limited to small-scale action spaces, which means that training and testing can usually only be performed in a limited network space and cannot cover more complex real environments. Existing automated penetration testing platforms are often difficult to update flexibly to deal with unknown vulnerabilities. Due to the timeliness and unknown nature of unknown vulnerability information, existing systems are usually unable to adapt quickly, or their information may be known, but due to changes in vulnerability conditions and the diversification of attack methods, the system also needs to have sufficient flexibility to adapt.

[0070] In addition, as new attack patterns and vulnerability intelligence are constantly released in text form, when using the Copilot method of large language model technology to generate specific information to drive API testing, it is necessary to ensure that the text parameters can strictly match the custom API, which means that the same entities in different documents need to have a unified representation. These problems highlight the shortcomings of existing systems in dealing with dynamic security threats and technology updates. Therefore, a highly adaptable and intelligent automated penetration testing method is urgently needed.

[0071] The present application embodiment provides a penetration testing method based on LLM and entity extraction, such as Figure 1 As shown, the method provided in the embodiment of the present application is executed by an electronic device, which can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. The method includes steps S101 to S104, wherein:

[0072] S101, obtaining the current vulnerability information input by the user, and using LLM to generate a task plan corresponding to the current vulnerability information based on the current vulnerability information and a known vulnerability database.

[0073] Specifically, the electronic device obtains the current vulnerability information input by the user, and the current vulnerability information is the natural language description information for a certain vulnerability. For example: Dahua IP camera ONVIF vulnerability, CVE number: CVE-2022-30563, CNNVD number: CNNVD-202206-2712, hazard level: high risk, vulnerability type: authentication bypass, collection time: 2022-06-28. Vulnerability introduction: Dahua IPC-HFW2XXX and others are products of Dahua, China. Dahua IPC-HFW2XXX is an IP camera. Dahua IPC-HDBW2XXX is a series of cameras. Dahua ASI7XXXX is a series of face recognition access controllers. There are security vulnerabilities in many Dahua products. Attackers can use this vulnerability to sniff request packets using a man-in-the-middle attack and successfully log in through ONVIF. When logging in, they can log in to the device by replaying the user's login packet. The following products and versions are affected: IPC-HDBW2XXX, IPC-HFW2XXX, ASI7XXXX, and official patches.

[0074] Furthermore, LLM can extract key entity information from the current vulnerability information input by the user. LLM (Large Language Model) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. LLM can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and is an important path to artificial intelligence.

[0075] For example, the entity information that can be extracted from the above natural language description information includes: "CVE-2022-30563" as the CVE number, "CNNVD-202206-2712" as the CNNVD number, "Authentication Bypass" as the vulnerability type, "Dahua Company of China" as the vulnerability manufacturer, "High Risk" as the vulnerability hazard level, "IPC-HDBW2XXX, IPC-HFW2XXX, ASI7XXXX" as the versions affected by the vulnerability, and "Exploiting this vulnerability when using a man-in-the-middle attack to sniff the request packet and successfully log in through ONVIF, you can log in to the device by replaying the user's login packet" as the vulnerability attack path.

[0076] The known vulnerability database is pre-built to store the device manufacturer information and corresponding vulnerability information disclosed on the Internet, including system IP address, component type, vulnerability type, attack path and other information. The vulnerability information in the known vulnerability database can be stored in the form of vectors. Optionally, the public vulnerability information is stored in the vector database chromadb as the known vulnerability database. The known vulnerability database also stores the EXP script corresponding to the vulnerability information. The EXP script is an action program that can exploit the known vulnerability to attack.

[0077] After receiving the current vulnerability information, the vulnerability information matching the current vulnerability information is searched from the known vulnerability database as a similar vulnerability, and the EXP script of the similar vulnerability is extracted from the known vulnerability database. The current vulnerability information and the EXP script are spliced ​​into an executable instance, and then the attack task corresponding to the current vulnerability information is planned through LLM.

[0078] Figure 2 The flowchart of the known vulnerability database provided by the embodiment of the present application is shown in FIG. Figure 2 The known vulnerability database stores the vulnerability information disclosed on the Internet. Specifically, the vulnerability text disclosed on the Internet is segmented and vectorized to obtain the known vulnerability information in vector form and store it in the vector database, thereby completing the construction of the known vulnerability database. The current vulnerability information used for input is the question text. The current vulnerability information is vectorized and matched with the known vulnerability database. The query result returned is the information in the known vulnerability database that can successfully match the current vulnerability information. The prompt text representing the current vulnerability information and the query result are combined into an executable instance and input into the LLM, so as to carry out subsequent task planning, and the returned answer is the task planning.

[0079] S102, retrieve the API prompt file, input the task plan and the API prompt file into the LLM, and obtain an initial call dictionary.

[0080] Specifically, the prompt file can be manually written and configured in the electronic device, including generation rules, available tools and real examples. The API prompt file is used to help LLM generate the initial call dictionary according to the task plan. The API prompt file contains the generation rules, available tools and real examples of the call dictionary based on the task plan. The generated initial call dictionary contains the API type and call order that need to be called to execute the planned task.

[0081] S103: normalize the initial call dictionary based on a preset API standard vocabulary to obtain a call dictionary.

[0082] Specifically, a preset API standard vocabulary is stored in an electronic device, which pre-stores standardized representations of keywords, and matches the initial call dictionary with the preset API standard vocabulary. For any successfully matched keyword in the initial call dictionary, the keyword is replaced with a standard word in the preset API standard vocabulary that matches the keyword. After replacing the successfully matched keywords in the initial call dictionary, a call dictionary is obtained.

[0083] S104: Call the corresponding API based on the calling dictionary to execute task planning and obtain the penetration test result.

[0084] The embodiment of the present application can effectively identify and handle unknown vulnerabilities by obtaining current vulnerability information input by the user and using LLM to compare the current vulnerability information with a known vulnerability database, thereby ensuring that each step in the test process can be flexibly updated and adapted to new vulnerabilities. The overall penetration test task is divided into reasonable subtasks through LLM and a known vulnerability database to complete task planning, and an initial call dictionary containing a scheduling plan is generated based on the task plan and API prompt file. The customized preset API specification vocabulary unifies the vector representation of entity words in different documents, and accurately maps the parameter information in the initial call dictionary to the preset API specification vocabulary through word vector similarity. LLM is used for task planning and flexible assembly of custom APIs, and task planning is executed through a reflection mechanism, thereby achieving efficient and automated penetration testing.

[0085] A possible implementation of the embodiment of the present application is to use LLM to generate a task plan corresponding to the current vulnerability information based on the current vulnerability information and a known vulnerability database, including:

[0086] Use LLM to search for similar vulnerabilities and EXP scripts that match the current vulnerability information from the known vulnerability database;

[0087] Combine the current vulnerability information and EXP script to get an executable instance;

[0088] Retrieve the task prompt file, input the executable instance and task prompt file into LLM, and obtain the task plan corresponding to the current vulnerability information.

[0089] In this embodiment, LLM can be fine-tuned in advance by technicians through vulnerability data to make the adjusted model more suitable for vulnerability information processing. The current vulnerability information and the vulnerability information in the known vulnerability database are input into LLM, and LLM can output the similarity between the current vulnerability information and each vulnerability information in the known vulnerability database. The vulnerability information with the highest similarity to the current vulnerability information and exceeding the preset similarity threshold can be selected as a similar vulnerability, and the vulnerability information with a similarity exceeding the preset similarity threshold can also be used as a similar vulnerability, wherein the preset similarity threshold can be set by technicians based on actual experience, and optionally, the preset similarity threshold can be set to 0.5. When using LLM to match the current vulnerability information with the known vulnerability database, the matching accuracy can be improved by index enhancement.

[0090] If the match is successful, the EXP script name of the similar vulnerability that successfully matches the current vulnerability information is called. If there is no similar vulnerability in the known vulnerability database whose similarity with the current vulnerability information exceeds the preset similarity threshold, the match is determined to be unsuccessful, and the current vulnerability information is marked as an unknown vulnerability, reminding the technicians to further process it.

[0091] The above matching process can be expressed as:

[0092] # Similarity matching output: The initial output shows the most similar vulnerabilities and their EXP scripts.

[0093] Most similar vulnerabilities:

[0094] ID: 3, Similarity: 0.8721, EXP Script: exp_network_device.py

[0095] ID: 5, Similarity: 0.8543, EXP Script: exp_buffer_overflow.py

[0096] ID: 7, Similarity: 0.8235, EXP Script: exp_firmware_exploit.py

[0097] # Handling unknown vulnerabilities: If LLM does not provide similar vulnerability information, the current vulnerability information will be marked as an unknown vulnerability and it will be stated that there is no existing EXP script.

[0098] No close match found, treating as a 0-day vulnerability.

[0099] ID: 0-day, Description: Critical buffer overflow in network devicefirmware, EXP Script: No existing EXP script

[0100] # Handling SQL injection vulnerabilities: Mark the vulnerabilities and EXP scripts related to SQL injection, and add SQL injection attacks to task planning.

[0101] For SQL Injection vulnerabilities:

[0102] ID: 8, Similarity: 0.9104, EXP Script: exp_sql_injection.py

[0103] ID: 12, Similarity: 0.8973, EXP Script: exp_sql_injection_v2.py

[0104] The task prompt file contains the generation rules, available tools, and real examples of generating task plans based on executable instances. A feasible example is to enter the Instruction command: "Scan IP 192.168.1.1, scan ports, detect possible vulnerabilities and try to perform SQL injection penetration testing, related vulnerabilities and corresponding EXP Script: exp_sql_injection.py", and a generated task plan example is as follows:

[0105] {

[0106] "scan_task": "Scan the open ports and services of the target system, obtain the port information and running service list of the target system.",

[0107] "vulnerability_detection_task": "Scan the target system for vulnerabilities, identify possible security vulnerabilities, and record vulnerability details.",

[0108] "exploit_test_task": "Try to exploit known vulnerabilities to attack the target system to verify the actual exploitation of the vulnerability. Similar ones are known: EXP Script: exp_sql_injection."

[0109] }

[0110] The embodiment of the present application can dynamically and real-time acquire new attack patterns and vulnerability information by searching similar vulnerabilities that match the current vulnerability information from a known vulnerability database. Based on the semantic understanding and logical reasoning capabilities of LLM, it can more accurately and efficiently extract key entities of the current vulnerability information and calculate the similarity with the known vulnerability database. The current vulnerability information and the EXP script are spliced ​​to obtain an executable script corresponding to the current vulnerability information. The task prompt file is used to guide the process of generating a task plan from an executable instance. Based on the executable instance and the task prompt file, a task plan that conforms to the current vulnerability information can be generated.

[0111] A possible implementation of the embodiment of the present application is to normalize the initial call dictionary based on a preset API standard vocabulary to obtain a call dictionary, including:

[0112] Read the initial call dictionary into the parameter format corresponding to the preset API specification vocabulary;

[0113] Convert the initial call dictionary after adjusting the parameter format into a vector form;

[0114] Match the initial call dictionary in vector form with the preset API specification vocabulary, perform a replacement operation on each successfully matched keyword in the initial call dictionary, and obtain a call dictionary after replacement;

[0115] For any successfully matched keyword in the initial call dictionary, the replacement operation is: determining a standard word that successfully matches the keyword in the preset API standard word list, and replacing the keyword with the standard word.

[0116] After determining the task plan corresponding to the current vulnerability information, load the corresponding API prompt file according to the task plan and input it into LLM. LLM generates the API call sequence to ensure that each test task is performed in the correct order. The generated API call sequence can be expressed as an initial call dictionary. For example, a SQL injection test task is generated, and the initial call dictionary contains the API type to be called and the corresponding call sequence, such as executing the scanning task first, then the vulnerability detection task, and finally the vulnerability exploitation task.

[0117] Among them, the design format of generating API prompt files is as follows:

[0118] #json

[0119] {

[0120] "Global prompt:": "Please follow the given instructions and use the given functions to complete the task step by step. At each step, you can only select one or more non-dependent functions from the following function libraries...",

[0121] "Function_Library:": "Load different custom function lists for different tasks. These functions can load and run specified exploit scripts",

[0122] "Url_Library:": "Optional, some functions call the local penetration url service, corresponding to the GET / POST sending function"

[0123] "Instruction:case1": "###Function Call: call_dict..." / / Actual function call process

[0124] "Instruction: Current task planning":

[0125] }

[0126] An example of generating an initial call dictionary is as follows:

[0127] #python

[0128] {

[0129] "step1": {

[0130] "url": "http: / / api.example.com / scan",

[0131] "function1": "perform_port_scan",

[0132] "arg1": ["Target system IP", "Scan port", "Service list"],

[0133] "output1": "port_scan_results",

[0134] "description1": "Scan the open ports and services of the target system, obtain the port information and running service list of the target system."

[0135] },

[0136] "step2": {

[0137] "url": "http: / / api.example.com / vulnerability_scan",

[0138] "function1": "run_vulnerability_scan",

[0139] "arg1": ["Target system IP", "Vulnerability scanning tool configuration"],

[0140] "output1": "vulnerability_scan_results",

[0141] "description2": "Scan the target system for vulnerabilities, identify possible security vulnerabilities, and record vulnerability details."

[0142] },

[0143] "step3": {

[0144] "url": "",

[0145] "function1": "attempt_exploit",

[0146] "arg1": ["Target system IP", "Vulnerability details", "Exploitation tool configuration"],

[0147] "output1": "exploit_test_results",

[0148] "description3": "Try to exploit known vulnerabilities to attack the target system to verify the actual exploitation of the vulnerability."

[0149] }

[0150] }

[0151] In this embodiment, the initial call dictionary can be read into the parameter format json corresponding to the preset API specification vocabulary through a regular expression, and then the initial call dictionary can be converted into a vector form through context prediction (CBOW) and skip-gram model (Skip-Gram). The method of matching the vector initial call dictionary and the preset API specification vocabulary can adopt cosine similarity matching.

[0152] For example, if the preset API specification vocabulary requires "target IP", "scan port", "penetration method" and "url" (send a get / post request to an existing internal penetration test service url) as parameters, it is necessary to ensure that the target IP and port information extracted from the current vulnerability information are consistent with the parameter format of the preset API specification vocabulary, and "penetration method" and "url" should be in the penetration method and accessible url keyword table. A feasible pseudo code for this function is as follows, using open source chromadb and pre-trained models:

[0153] # Define a list of proper nouns and a set of input words

[0154] terms = list (keyword list of existing API)

[0155] input_terms = input_list (entities extracted from the input: "target IP", "scan port", "penetration method", "url", etc.)

[0156] # Load the pre-trained model

[0157] model = SentenceTransformer('model_name.model')

[0158] # Vectorized functions

[0159] def vectorize(text):

[0160] return model.encode(text).tolist()

[0161] # Vectorize a list of proper nouns

[0162] term_vectors = [vectorize(term) for term in terms]

[0163] # Generate a list of unique IDs

[0164] term_ids = [f"term_{i}" for i in range(len(terms))]

[0165] # Create a ChromaDB client

[0166] client = chromadb.Client()

[0167] # Create a collection and add data

[0168] # collection = client.create_collection(name='terms')

[0169] collection.add(documents=terms, embeddings=term_vectors, ids=term_ids)

[0170] # Define a function to process a single input item

[0171] def find_nearest_term(input_term):

[0172] input_vector = vectorize(input_term)

[0173] results = collection.query(query_embeddings=[input_vector], n_results=1)

[0174] if results['documents']:

[0175] return results['documents'][0][0] # Returns the most similar items

[0176] else:

[0177] return None

[0178] # Process the set of input words and get the corresponding fuzzy matching results

[0179] matched_terms = {input_term: find_nearest_term(input_term) for input_term in input_terms}

[0180] The embodiments of the present application predefine the API specification vocabulary to flexibly integrate and configure newly added custom APIs, ensure that the test plan can respond to newly emerging vulnerabilities in a timely manner, generate a specified call dictionary based on LLM to drive the specified API for testing, and ensure that the test data call dictionary generated by LLM and the actual API parameters are strictly matched by uniformly representing the same entity, thereby avoiding test errors caused by parameter mismatch and improving the flexibility and scalability of system functions.

[0181] In a possible implementation of the embodiment of the present application, the task planning includes a plurality of subtasks having an execution order, and each subtask corresponds to an API prompt file;

[0182] Input the mission plan and API prompt files into LLM to obtain the initial call dictionary, including:

[0183] Use LLM to process multiple subtasks in order of execution, and use the model output containing the API call sequence obtained after processing the last subtask as the initial call dictionary;

[0184] Wherein, any subtask among the multiple subtasks is used as a target subtask, and when the target subtask is the first subtask to be executed, the target subtask is processed by using LLM, including: inputting the target subtask and the API prompt file corresponding to the target subtask into LLM, and obtaining the model output corresponding to the target subtask;

[0185] When the target subtask is not the first subtask to be executed, the target subtask is processed using LLM, including: inputting the target subtask, the API prompt file corresponding to the target subtask, and the model output corresponding to the previous task of the target subtask into LLM to obtain the model output corresponding to the target subtask.

[0186] In this embodiment, the task planning may include three subtasks: a scanning subtask, a vulnerability detection subtask, and a vulnerability exploitation subtask.

[0187] The embodiment of the present application adopts task segmentation and sub-task combination, and uses a call dictionary to define the function calls, parameters and output descriptions of each penetration testing task, ensuring flexible organization and accurate execution of tasks, and supporting automated operations in different testing phases. It uses LLM for task planning and API call planning, and intelligently generates test task planning and API call sequence, thereby improving the intelligence level and adaptability of penetration testing.

[0188] A possible implementation of the embodiment of the present application is to call the corresponding API based on the calling dictionary to execute the task planning and obtain the penetration test result, including:

[0189] Based on the call dictionary, determine the steps without dependencies under the same subtask in the task plan, and submit the steps without dependencies to the thread pool, the task plan includes scanning subtasks, vulnerability detection subtasks and vulnerability exploitation subtasks;

[0190] Based on the calling dictionary, the corresponding API is called to sequentially execute the scanning subtask, the vulnerability detection subtask and the vulnerability exploitation subtask, wherein the thread pool processes the steps without dependencies in parallel.

[0191] In this embodiment, the electronic device performs penetration testing by calling the corresponding API according to the call dictionary reflection. Specifically, according to the functions and parameters defined in the call dictionary, the corresponding API is automatically called to perform scanning subtasks, vulnerability detection subtasks, and vulnerability exploitation subtasks. For the steps of the same subtask that have no dependencies, multiple parse_and_exe(args,...,function_name) are submitted in parallel using a thread pool to speed up the task execution. For example, for a scanning task, the module will first call the 'perform_port_scan' function of step1 and pass in the corresponding parameters to automatically perform a port scan. A pseudocode implementation is as follows:

[0192] #python

[0193] def parse_and_exe(call_dict, result_buffer, parallel_step: str='1',func_name):

[0194] """

[0195] Parse the input and call the corresponding function to get the result.

[0196] :param call_dict: dict, containing function call parameters, function name and output

[0197] :param result_buffer: dict, stores intermediate results

[0198] :param parallel_step: str, parallel step count

[0199] :param func_name: determines which function to call, such as perform_port_scan

[0200] :return: Returns the result of the function execution and stores the corresponding result in result_buffer

[0201] """

[0202] # Get the calling parameter list

[0203] arg_list = call_dict.get('arg' + parallel_step, [])

[0204] # Replace the reference in the parameter and extract the result stored in result_buffer

[0205] replace_arg_list = [

[0206] result_buffer[item][0]if isinstance(item, str) and ('result'in item or 'input' in item) else item

[0207] for item in arg_list ]

[0209] # Get function name, output key and description

[0210] func_name = call_dict.get('function' + parallel_step)

[0211] output_key = call_dict.get('output' + parallel_step)

[0212] description = call_dict.get('description' + parallel_step, '')

[0213] # Execute the function and get the result

[0214] func = globals().get(func_name) # Get the function object from the global scope

[0215] if func is None:

[0216] raise ValueError(f"Function {func_name} not found")

[0217] result = func(*replace_arg_list)

[0218] # Store the results in result_buffer

[0219] result_buffer[output_key] = (result, description)

[0220] return result_buffer

[0221] After the penetration test is completed, filter and extract key information returned by the function call or service interface from the API response, such as scan results, vulnerability details, etc., and prepare for further analysis. For example, extract the open port information in the port scan results and organize it into structured data.

[0222] Furthermore, the data returned by the API is asynchronously output to the outputBuffer, providing the complete call results of the automated penetration test. This step outputs the test results in an appropriate format for security personnel to analyze and make decisions. For example, the scan results, vulnerability details, and the results of the exploit test are asynchronously output to the buffer, all processes are recorded, and the final output is organized.

[0223] The embodiment of the present application ensures the efficiency and comprehensiveness of the penetration test by executing the scanning subtask, vulnerability detection subtask and vulnerability exploitation subtask in sequence, submits the steps without dependencies to the thread pool for parallel processing, and improves the execution efficiency of the penetration test.

[0224] In a possible implementation manner of the embodiment of the present application, the method further includes:

[0225] Obtain a threat intelligence document, and obtain a span in an encoded form based on the threat intelligence document;

[0226] Construct a span multi-classifier, and use the span multi-classifier to determine the preset entity category with the highest probability that each span belongs to from multiple preset entity categories, and obtain the categories and probabilities corresponding to each of the multiple spans;

[0227] The code and probability of each span are combined into a feature vector, and a storage matrix is ​​constructed based on the feature vectors corresponding to each of the multiple spans;

[0228] Determine the fusion features of each span based on the storage matrix;

[0229] Input the fusion features of each span into the multi-layer perceptron to obtain the category of each span;

[0230] The obtained categories of the multiple spans are stored as a preset API specification vocabulary.

[0231] The embodiment of the present application adopts a document entity extraction method to obtain a unified representation of entities in a threat intelligence document and to create a preset API keyword table for matching API parameter information, thereby realizing the analysis and classification of vulnerability text data that is difficult to regularize, ensuring strict matching of parameters during the testing process, and being able to cope with the security challenges brought about by ever-changing unknown vulnerabilities.

[0232] A possible implementation of the embodiment of the present application is to determine the fusion feature of each span based on the storage matrix, including:

[0233] Taking any span among the multiple spans as the target span, taking the encoding of the target span as the local feature of the target span, and taking the category corresponding to the target span as the target category;

[0234] Determine a span set whose category is the target category from the multiple spans;

[0235] Based on the weighted average of each span feature vector in the span set, the global feature of the target span is obtained;

[0236] The local and global features of the target span are balanced through the gate mechanism to obtain the fused features of the target span.

[0237] In this embodiment, the threat intelligence document is used as the input text segment, and each word is first represented as a word embedding vector e i , word embedding is a real number vector of fixed length, which represents the semantic information of a word. These word embeddings can be generated by a word embedding method (such as the word embedding layer in BERT). For example, the input text fragment "Apple was founded by Steve Jobs" is segmented by BERT's WordPiece tokenizer to obtain ["Apple", "by", "Steve", "·", "Jobs", "founded"], and each token gets the corresponding word embedding vector e1, e2, e3, ..., e6, where e i €R d , d is the dimension of word embedding (for example, BERT’s word embedding dimension is 768).

[0238] These words are embedded into the language model through pre-training (such as BERT). i is further processed to obtain the context-aware representation h i This step is achieved through the model's multiple self-attention layers (Transformer layers), so that the representation of each word depends not only on itself, but also on the context of the entire sentence. i =BERT(e i ), h i Contains the word e i Semantic information in the context of the entire sentence. For each word embedding e i Encode through BERT to obtain the context-related vector representation h i .

[0239] In document-level tasks, we focus on a span, such as a continuous segment from word i to word j. To get the representation of this span, we can aggregate the context encodings of multiple words in this span. For example, the span S=[i,j] may represent an entity in the text, such as "Apple" or "Steve Jobs". Assume that the span S corresponds to multiple words in the text, such as "Steve Jobs" corresponds to S=[3,5].

[0240] In order to get the representation of the entire span h s , can be represented by the context of the words in the span h, h i+1 ,…,h j Aggregating: h s =AGG({h i ,h i+1 ,…,h j}). AGG is an aggregation method that can integrate these vectors into a feature vector representing the entire span using methods such as average or maximum.

[0241] For example, if the span S = [3,5] corresponds to "Steve Jobs"; the context encoding of "Steve" is h3 = [0.6, 0.8, 0.3], and the context encoding of "Jobs" is h5 = [0.5, 0.7, 0.4]. Through aggregation, the representation of the span "Steve Jobs" is obtained: h s =[max(0.6,0.5),max(0.8,0.7),max(0.3,0.4)]=[0.6,0.8,0.4]. Or use average as the aggregation method: h s =[(0.6+0.5) / 2, (0.8+0.7) / 2, (0.3+0.4) / 2]=[0.55, 0.75, 0.35].

[0242] Summary from word embedding i To span h s The process is as follows: 1. Word embedding (e i : Segment the text into words and map each word into a vector of fixed dimension. 2. Context Encoding (h i ): Through the pre-trained BERT model, combined with the context information of the entire sentence, the context representation of each word is obtained. 3. Span representation (h s ): By aggregating the contextual representations of words within a span, a vector representation of the entire span is obtained. This process completes the encoding of text fragments into vector representations that can be used for subsequent tasks (such as entity classification).

[0243] Use a pre-trained language model (such as BERT or LSTM) to encode each word and get the context representation of each word. For span S=[i,j], its encoding representation is still: h s =AGG({h i ,h i+1 ,…,h j}), h s The feature of the entire span is represented by aggregating (e.g., taking the average or maximum value) the context vector of the pre-trained model.

[0244] For each span S, multi-classification prediction is required. The preset entity categories can be set according to actual needs. For example, the category SHA1 corresponds to the SHA1 hash value used to identify files or data; the category TOOL corresponds to the tools used in network security or attacks, etc. The softmax function is used to predict the probability distribution of each category:

[0245]

[0246] Among them, C is the set of all preset entity categories, W c and b c is the parameter of category c.

[0247] The final prediction of the model is the prediction for the class with the highest probability:

[0248]

[0249] A span-based key-value storage structure can be used to fuse the global features of these spans. Assume that the vector stored in the i-th slot ui in the storage matrix U is (H Si , p i ), where H Si is the context representation of span S, p i is the corresponding predicted probability.

[0250] When multiple spans belong to the same preset entity category c, the global features can be fused by weighted averaging. Let v Si S is the span i The feature vector of , its global feature can be calculated by weighted average as:

[0251]

[0252] The gate mechanism is used to balance local features and global features. The calculation process of the gate mechanism is as follows:

[0253]

[0254] Among them, g s is the gate weight, which is usually calculated through a neural network module. The specific calculation steps are as follows:

[0255] Input features: local features h s and the global feature v s ;

[0256] Calculate gate weights:

[0257]

[0258] Among them, W g and b g are the parameters of the network, Represents a feature concatenation operation.

[0259] Features after adjusting the door mechanism Input to MLP for classification or regression. The formula of MLP is as follows:

[0260]

[0261] Among them, W1, W2, W3 are the weight matrices in the MLP; b1, b2, b3 are the bias vectors, and σ is the activation function (such as ReLU or sigmoid). Finally, the output y of the MLP can be used to predict the category of each span.

[0262] The following is an example of text fragment entity extraction:

[0263] Text snippet: There are two main attack vectors that can exploit poorly protected session variables (such as cookies). One is to attack the local machine directly from a physical level, or indirectly through XSS and phishing. In addition, man-in-the-middle attacks (CAPEC-94) use network sniffers, proxies, or other intermediaries to intercept the victim's credentials and use these credentials to impersonate the victim to interact with the host. If there are no other protection mechanisms, once the credentials are intercepted, impersonation attacks become very easy for attackers. See also: CVE-2010-5148, CVE-2016-0353.

[0264] Entity extraction results:

[0265] entity category explain Cookies TOOL Tools related to session variables XSS VULN_TYPE Vulnerability type, cross-site scripting attack Phishing ATTACK_PATTERN Attack Mode Man-in-the-Middle Attack (CAPEC-94) ATTACK_PATTERN Man-in-the-middle attack patterns identified by CAPEC-94 Network Sniffer TOOL Tools used in the attack acting TOOL Tools used in the attack CVE-2010-5148 CVE_ID Vulnerability ID CVE-2016-0353 CVE_ID Vulnerability ID

[0266] The embodiment of the present application combines the local features (i.e., encoding) and the global features (the weighted average features of other spans in the same category) of each span, and intelligently balances the two through a gate mechanism, thereby generating a more comprehensive and accurate fusion feature, which effectively improves the richness and accuracy of information representation.

[0267] Figure 3It is a structural diagram of a virtual module provided by an embodiment of the present application, comprising: an entity extraction module 301, a vulnerability matching module 302, a task planning module 303, an API call planning module 304, a parameter matching module 305, a penetration execution module 306, a result data extraction module 307, and a data output module 308. The entity extraction module 301 extracts key entity information from the natural language description of the current vulnerability information input by the user, such as the IP, component type, vulnerability description, and attack path of the target system. The vulnerability matching module 302 uses LLM to calculate the similarity between the input current vulnerability information and the known vulnerability database, and finds the most similar vulnerability information and related EXP scripts. The task planning module 303 generates a specific penetration test task according to the LLM, and creates a task plan including scanning, vulnerability detection, and vulnerability exploitation by parsing the input test instructions. The API call planning module 304 uses LLM to generate a call dictionary for API calls to ensure that the call order meets the logical requirements of the penetration test. The parameter matching module 305 extracts key information through entity extraction technology, and matches according to the word vector similarity to ensure that the parameter information in the generated call dictionary strictly meets the customized API requirements. The penetration execution module 306 calls the API to perform penetration testing through the reflection mechanism according to the definition in the call dictionary, and automates the test operation. The result data extraction module 307 screens and extracts key information returned from the function call or service interface to provide data support for subsequent analysis. The data output module 308 outputs the data returned by the API to the outputBuffer, providing complete automated penetration test results to ensure the clarity and completeness of the results.

[0268] Figure 4 This is a module flow chart provided in the embodiment of the present application, see Figure 4 , local vulnerability information is the public vulnerability information on the Internet. The local vulnerability information and the corresponding EXP are stored in the vector database to obtain the known vulnerability database. The natural language description of the penetration target is the current vulnerability information entered by the user. The current vulnerability information is matched with the known vulnerability database. LLM outputs the matching result. 0-day vulnerability indicates an unknown vulnerability. If the current vulnerability information is not an unknown vulnerability, the task planning prompt (task prompt file) and the text of this penetration process (executable instance) are input into LLM to request task planning. LLM returns the penetration test task process (task planning). The obtained task planning and API scheduling prompt (API prompt file) are input into LLM. LLM returns the call dictionary. Based on the call dictionary, the reflection execution function is used to call the corresponding API to complete the test task and finally record the penetration test results.

[0269] Figure 5 This is a flow chart of the backend function execution provided by the embodiment of the present application, see Figure 5, initiate a post request to the interface, obtain the current vulnerability information entered by the user, call LLM to match the current vulnerability information with the local chromabd similar vulnerability information (known vulnerability database), retrieve similar vulnerability information and splice it with the current vulnerability information to obtain an executable instance, call LLM and load the task prompt file, LLM generates a task plan based on the executable instance and the task prompt file, that is, Figure 5 The task plan list in the API prompt file is called. LLM obtains the call dictionary based on the task plan and the API prompt file. The call dictionary matches the preset API specification vocabulary, so as to strictly match the parameters to the keyword table. Each subtask corresponds to a call dictionary. The call dictionary is read and the target function is called through reflection to execute the corresponding subtask. When there is no dependency between the steps in the subtask, it can be executed in parallel by the thread pool, extracting the penetration test results and returning to the previous section.

[0270] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the contents shown in the aforementioned penetration testing method embodiment based on LLM and entity extraction are implemented.

[0271] An electronic device is provided in an embodiment of the present application, such as Figure 6 As shown, Figure 6 The electronic device 600 shown includes: a processor 601 and a memory 603. The processor 601 and the memory 603 are connected, such as through a bus 602. Optionally, the electronic device 600 may also include a transceiver 604. It should be noted that in actual applications, the transceiver 604 is not limited to one, and the structure of the electronic device 600 does not constitute a limitation on the embodiments of the present application.

[0272] Processor 601 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 601 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0273] The bus 602 may include a path to transmit information between the above components. The bus 602 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 602 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but it does not mean that there is only one bus or only one type of bus.

[0274] The memory 603 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0275] The memory 603 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 601. The processor 601 is used to execute the application code stored in the memory 603 to implement the contents shown in the above-mentioned embodiment of the penetration testing method based on LLM and entity extraction.

[0276] Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0277] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer can execute the contents shown in the aforementioned penetration testing method embodiment based on LLM and entity extraction.

[0278] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0279] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A penetration testing method based on LLM and entity extraction, characterized in that: include: Obtain current vulnerability information input by the user, and use LLM to generate a task plan corresponding to the current vulnerability information based on the current vulnerability information and a known vulnerability database; Retrieving an API prompt file, inputting the task plan and the API prompt file into the LLM, and obtaining an initial call dictionary; Normalizing the initial call dictionary based on a preset API standard vocabulary to obtain a call dictionary; Calling the corresponding API based on the calling dictionary executes the task planning to obtain the penetration test result; The step of calling the corresponding API based on the calling dictionary to execute the task plan and obtain the penetration test result includes: Determine, based on the call dictionary, steps in the task plan that do not have dependencies under the same subtask, and submit the steps that do not have dependencies to a thread pool, wherein the task plan includes a scanning subtask, a vulnerability detection subtask, and a vulnerability exploitation subtask; Calling the corresponding API based on the calling dictionary to sequentially execute the scanning subtask, the vulnerability detection subtask, and the vulnerability exploitation subtask, wherein the thread pool processes the steps without dependency in parallel; The method further comprises: Obtain a threat intelligence document, and obtain a span in an encoded form based on the threat intelligence document; Constructing a span multi-classifier, using the span multi-classifier to determine the preset entity category to which each span belongs with the highest probability from multiple preset entity categories, and obtaining the categories and probabilities corresponding to each of the multiple spans; Combining the code and probability of each span into a feature vector, and constructing a storage matrix based on the feature vectors corresponding to each of the plurality of spans; Determine a fusion feature of each span based on the storage matrix; Input the fusion features of each span into the multi-layer perceptron to obtain the category of each span; The obtained categories of the multiple spans are stored as a preset API specification vocabulary.

2. The penetration testing method based on LLM and entity extraction according to claim 1, characterized in that: The using LLM to generate a task plan corresponding to the current vulnerability information based on the current vulnerability information and a known vulnerability database includes: Using the LLM to search the known vulnerability database for similar vulnerabilities matching the current vulnerability information and the EXP scripts of the similar vulnerabilities; The current vulnerability information and the EXP script are spliced ​​to obtain an executable instance; Retrieve a task prompt file, input the executable instance and the task prompt file into the LLM, and obtain a task plan corresponding to the current vulnerability information.

3. The penetration testing method based on LLM and entity extraction according to claim 1 is characterized in that: The step of normalizing the initial call dictionary based on the preset API standard vocabulary to obtain a call dictionary includes: Reading the initial call dictionary into a parameter format corresponding to the preset API specification vocabulary; Convert the initial call dictionary after adjusting the parameter format into a vector form; Matching the initial call dictionary in vector form with the preset API specification vocabulary, performing a replacement operation on each successfully matched keyword in the initial call dictionary, and obtaining a call dictionary after replacement; Wherein, for any successfully matched keyword in the initial call dictionary, the replacement operation is: determining a standard word that successfully matches the keyword in the preset API standard word list, and replacing the keyword with the standard word.

4. The penetration testing method based on LLM and entity extraction according to claim 1 is characterized in that: The task planning includes a plurality of subtasks with execution order, and each subtask corresponds to an API prompt file; The step of inputting the task plan and the API prompt file into the LLM to obtain an initial call dictionary includes: Using the LLM to process the multiple subtasks in order of execution, and using the model output containing the API call sequence obtained after processing the last subtask as the initial call dictionary; Wherein, any subtask among the multiple subtasks is used as a target subtask, and when the target subtask is the first subtask to be executed, the target subtask is processed by using the LLM, including: inputting the target subtask and the API prompt file corresponding to the target subtask into the LLM, and obtaining the model output corresponding to the target subtask; When the target subtask is not the first subtask to be executed, the target subtask is processed using the LLM, including: inputting the target subtask, the API prompt file corresponding to the target subtask, and the model output corresponding to the previous task of the target subtask into the LLM to obtain the model output corresponding to the target subtask.

5. The penetration testing method based on LLM and entity extraction according to claim 1 is characterized in that: The determining of the fusion feature of each span based on the storage matrix includes: Taking any span among the multiple spans as a target span, taking the encoding of the target span as a local feature of the target span, and taking the category corresponding to the target span as a target category; Determine a span set whose category is the target category from the multiple spans; Based on the weighted average of each span feature vector in the span set, a global feature of the target span is obtained; The local features and global features of the target span are balanced through a gate mechanism to obtain a fusion feature of the target span.

6. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the steps of the penetration testing method based on LLM and entity extraction described in any one of claims 1 to 5.

7. An electronic device, characterized in that: include: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the penetration testing method based on LLM and entity extraction as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the penetration testing method based on LLM and entity extraction as described in any one of claims 1 to 5.

Images