Python library fuzzy test drive generation method based on large language model

Abstract

The invention focuses on the field of fuzz testing, and particularly relates to intelligent auxiliary fuzz testing drive code generation by using a large language model. The invention provides a convenient, fast and automatic fuzz test driving code generation method for Python library code fuzz test by using a large language model and combining a test function and a sample code. The method comprises the steps that Python library source code analysis and drive code sample information extraction are conducted through a static analysis tool, and a data set of a test function and a sample code is obtained; constructing a prompt word engineering structure comprising a drive sample code, a generation task description, a variation strategy introduction, a test function information set, a drive sample code and a code standardization description, and generating zero-sample, less-sample and thinking chain prompts according to the number of samples; building a large language model agent end, transmitting cue words to open-source and closed-source large language models in an HTTP / HTTPS (Hyper Text Transfer Protocol / Hyper Text Transfer Protocol Secure Socket) message mode, and performing reasoning generation of fuzzy test driving codes; after the drive codes are obtained, the fuzz test drive codes which do not pass the pilot run test are generated and screened for multiple rounds, and finally a fuzz test drive code set to be subjected to vulnerability mining is obtained; and performing vulnerability mining on the test function by utilizing the obtained fuzzy test driving code set, and storing a result to a database for analysis.

Description

Technical Field

[0001] The present invention focuses on the field of fuzz testing, and specifically involves using a large language model to generate fuzzy test driver code for Python library code. It provides a non-artificial, high-efficiency, and highly adaptable intelligent method for generating fuzzy test driver code for Python library code. Background Art

[0002] Large Language Model (LLM) is based on the pre-trained language model. It increases the complexity and generalization ability of the model by increasing the model scale, adjusting the model structure, adding more tasks, etc., and finally trains it into a Transformer language model with billions or even tens of billions of parameters. Its outstanding leap ability is called "emergence ability". The development of large language models has gone through the development stages of language models, statistical language models, neural language models, and pre-trained language models. The main working principle is training and reasoning. The overall process of large language models includes: data collection, labeling, pre-training, fine-tuning, reasoning generation. The bottom layer of large language models is a set of neural networks. These neural networks can be regarded as a highly advanced and complex form of pattern recognition. They can process language efficiently and generate high-quality texts by using the relationship between words and phrases through the core components encoder and decoder. As the core part of the large language model, the encoder and decoder extract the meaning of sentences from a series of natural language texts, understand the relationship between words and phrases, and thus adapt to a variety of language tasks in specific fields. The encoder usually takes a sequence of variable length as input and processes the input sequence to generate a context vector; the decoder, by accepting the input of the encoder, predicts the subsequent tags in the target sequence. According to the degree of openness, large language models can be divided into open source and closed source. Representative open source models include LLaMA and representative closed source models include GPT.

[0003] Python, as a high-level interpreted programming language, is one of the fastest growing programming languages ​​with its concise and beautiful language features. In the field of artificial intelligence and machine learning, it provides a wide range of functions for the development of data science, big data applications, cloud computing, artificial intelligence, machine learning and other fields with its own concise coding and high readability, as well as a large number of libraries available to users. Faced with a large number of repeatedly reused Python libraries, once vulnerabilities appear, software developed based on libraries such as NumPy, Scipy, and TensorFlow can be easily exploited by attackers, resulting in the leakage of key information of enterprises and the decline of software quality.

[0004] Fuzz testing technology is a popular and effective software detection technology for detecting bugs and vulnerabilities. In many mainstream software companies and security companies, fuzz testing technology is often used as part of security detection to discover defects and vulnerabilities. The core idea is to use a large amount of random, unexpected, automatically or semi-automatically generated random data to "bomb" the data, and the initial input data can also be called a test sample. The fuzz test driver code is loaded into the fuzz test engine as input code, and data is continuously injected into the test target in the target program to trigger the vulnerability and find the vulnerability. Its main function is to act as a bridge between the fuzz test engine and the target program, so it plays a vital role in the entire fuzz test activity.

[0005] Recently, there have been related research schemes that assist fuzz testing based on large language models, and some results have been achieved, but these research schemes focus too much on C++ code and code vulnerabilities, and do not take into account that one of the core of fuzz testing is to generate fuzz test drivers, and do not focus on fuzz test driver code generation for Python code. In order to further improve the efficiency of fuzz test driver code generation, the present invention explores the use of large language models as a basis, makes full use of sample codes of open source fuzz test projects, combines static analysis tools, constructs fuzz test driver code generation prompts for Python libraries, interacts with large language model agents, generates fuzz test driver code, and mines Python library code vulnerabilities. Summary of the invention

[0006] In order to generate fuzzy test driver code more efficiently, the present invention proposes a Python library fuzzy test driver generation method based on a large language model. The method mainly includes the following steps:

[0007] 1. Perform static analysis on the code in the Python library through static analysis tools to obtain a set of test function information, and extract the Python library fuzz test driver from the open source fuzz testing project as sample code;

[0008] 2. Combine the fuzz test sample code and test function information to build zero-sample, few-sample and thought chain prompts including driver sample code, generation task description, mutation strategy introduction, test function information set and code standardization description, and instruct the large language model to generate fuzz test driver code according to the corresponding prompt words;

[0009] 3. Conduct a trial run test on the generated fuzz test driver code. After multiple rounds of large language model agent generation and trial run tests, the final driver code set to be exploited is obtained. The driver code is loaded into the fuzz test engine to perform fuzz testing on the Python library code.

[0010] 4. Store the vulnerability mining result information in the database and analyze the fuzzy test vulnerability mining results.

[0011] The specific details of this method are described in detail below:

[0012] 1. Source code analysis and sample extraction

[0013] After the Python library code and the open source fuzz testing project source code are downloaded, static analysis tools are used to extract the function signature, parameters, return value, document string and other information of the test function from the Python library code, and the fuzz test driver sample code of the Python library is extracted from the open source fuzz testing project, and ranked and numbered according to the cyclomatic complexity to generate a test data set <test ID number, library name, version, source code file name, test function name, function signature, parameters, return value, document string, cyclomatic complexity score, source code content>. For the sample code, its data set is set to <sample ID number, library name, source code file name, sample source code, sample code flag>.

[0014] 2. Build prompts and generate driver code

[0015] After source code analysis and sample extraction, the obtained data set information is used as an important part of the Python fuzz test driver code generation prompt. The fuzz test driver code generation prompt should include the description of the standardized sample and the test function information. The driver sample code comes from the official example of the fuzz test engine, the mutation strategy introduction comes from the data mutation strategy of the fuzz test engine, and the code standardization description indicates the format specification of the driver code generation:

[0016] (1) Constructing the initial prompt

[0017] The prompt content is constructed in a hierarchical manner according to the order of driver sample code, generation task description, mutation strategy introduction, test function information set, driver sample code, and code standardization description. The number of samples in the constructed prompt is divided into zero sample, few sample, and thinking chain prompts according to the number of samples. Different prompt words are constructed by setting different templates for the prompt words. The basic structure of the entire prompt word is as follows Figure 1 shown.

[0018] (2) Generate fuzz test driver code

[0019] After filtering and formatting the prompts, the prompts and information are sent to the remote large language model in the form of HTTP / HTTPS messages. The agent uses the large language model to generate and infer the driver code. The generated fuzz test driver code needs to be named according to fuzz_library name_test function.

[0020] 3. Multiple rounds of driver code generation and screening

[0021] After the build prompt and driver code generation phase, the generated files of the driver code are tested for a trial run. However, the fuzz test driver program may not completely pass the trial run test. Next, it will enter multi-agent generation and repeated screening generation, and generate multiple fuzz test driver codes through multiple large language models. Finally, the driver codes that pass the trial run test will be sorted according to the token size. The driver code multi-round generation and screening process is as follows:

[0022] (1) Interacting with the Large Language Model Agent

[0023] According to the previous step, based on the prompts of the test data set and the sample data set, more repair content is added, and multiple rounds of multiple large language model agents are interacted again. The large language model name, model service address, model service port, maximum token length, maximum number of attempts, and driver code generation prompts are sent to the large language model for a new round of reasoning generation. The main content sent is shown in Table 1:

[0024] Table 1 Agent content sending table

[0025] Content Name Contents Large language model name The name of the model generated by inference Model service address Local or remote service address Model service port Local or remote service port number Maximum token length The maximum size of the driver code content returned Maximum number of attempts Maximum number of attempts to generate fuzz test driver code Driver code generation tips Contains the test data set and sample data set content, complete prompt

[0026] After passing the necessary parameters to the large language model agent through HTTP / HTTPS messages, according to the prompt content, after the large language model is generated and inferred, the fuzzy test driver code under zero-sample, few-sample, and thinking chain prompts is obtained respectively. The obtained fuzzy test driver code is sent to the test environment for trial run test and further screening.

[0027] (2) Add the generated fuzz test driver code to the list of vulnerabilities to be mined

[0028] After the driver code screening and generation phase, the fuzz test driver code that has passed the trial run test will be further batch processed to fix some errors, remove unnecessary comments and imports, etc., and the final results will be stored in the list of vulnerabilities to be mined.

[0029] 4. Vulnerability mining and result collection

[0030] After multiple rounds of driver code generation and screening in the previous step, a whole list of vulnerabilities to be mined that can be fuzz tested has been obtained. For the whole list of vulnerabilities to be mined that can be fuzz tested, it is first sorted by content size, and finally sent to the fuzz testing engine in order for vulnerability mining, and the results are saved in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 :Prompt word engineering structure diagram DETAILED DESCRIPTION

[0032] The present invention is directed to a Python library fuzzy test driver generation method based on a large language model, comprising: performing static source code analysis on downloaded Python library codes and extracting sample codes from open source fuzzy test projects, generating test data sets and sample code sets from the extracted test function information and sample code information, combining them with driver sample codes, generation task instructions, mutation strategy introductions, and code normalization instructions, constructing zero sample, few sample, and thinking chain prompts according to the number of samples, inputting the prompts into a large language model agent end to perform fuzzy test driver code reasoning generation of test functions, forming a list of to-be-mined vulnerabilities through multiple rounds of generation, screening, and driver codes tested by trial operation, configuring a Python library code test environment, loading the driver code into a fuzzy test engine for vulnerability mining, and finally storing the vulnerability mining information in a database.

[0033] The specific embodiments of the present invention are further described as follows:

[0034] Step 1: Source code analysis and sample extraction

[0035] S101: Use Git tool to download Python library code, filter the Python library code, remove unnecessary Python code, use static analysis tool to extract the cyclomatic complexity, function signature, parameters, return value, document string and other information of the function to be tested; select fuzzy test driver sample code from open source fuzz testing project as sample code for fuzzy test driver code generation.

[0036] S102: Construct a test data set <test ID number, library name, version, source code file name, test function name, function signature, parameters, return value, document string, cyclomatic complexity score, source code content>. For the sample code, its data set structure is set to <sample ID number, library name, source code file name, sample source code, sample code flag>.

[0037] Step 2: Build prompt and generate driver code

[0038] Deploy the large language model agent and make it an important component for driving the interaction between code generation and the large language model. Combine the test data set and sample data set obtained in the previous step to build zero-sample, few-sample, and thinking chain prompts, and interact with the large language model agent to perform fuzzy test driving code reasoning generation.

[0039] S201: First, construct a basic prompt word engineering structure including driver sample code, generation task description, mutation strategy introduction, test function information set, driver sample code, and code standardization description. According to the number of samples, it is divided into zero sample, few sample, and thinking chain prompt templates to instruct the large language model to generate fuzzy test driver code according to the corresponding prompt words.

[0040] S202: Interact with the large language model via HTTP / HTTPS messages through the large language model proxy, and instruct the large language model to generate fuzzy test driver code through a driver code generation prompt.

[0041] Step 3: Multiple rounds of driver code generation and screening

[0042] S301: The obtained fuzzy test driver code is first tested through a trial run to determine whether the driver code can be tested.

[0043] S302: Send the failed fuzz test driver code, the large language model name, model service address, model service port, maximum token length, maximum number of attempts, and driver code generation prompt to the large language model agent, re-interact with the large language model, perform multiple rounds of generation, and obtain a reliable fuzz test driver code that passes the trial run test.

[0044] S303: The obtained fuzz test driver code is stored in the format of fuzz_library name_function name and added to the list of vulnerabilities to be mined.

[0045] Step 4: Vulnerability mining and result collection

[0046] S401: Using the obtained list of vulnerabilities to be mined as input to the fuzz testing engine, and automatically instructing the fuzz testing engine to perform fuzz testing on the test function.

[0047] S402: Extract vulnerability mining information, store it in the database according to library name_model name, test function, coverage, number of crashes, number of exceptions, and number of errors, and analyze the vulnerability mining results.

Claims

1. A Python library fuzz test driver generation method based on a large language model, characterized in that The implementation steps are: (1) Use static analysis tools to perform static analysis on the code in the Python library to obtain a set of test function information. Extract the Python library fuzz testing driver from the open source fuzz testing project as sample code; (2) Combining the fuzz test sample code and test function information, construct zero-sample, few-sample and thought chain prompts including driver sample code, generation task description, mutation strategy introduction, test function information set, driver sample code and code standardization description, and instruct the large language model to generate fuzz test driver code according to the corresponding prompt words; (3) The generated fuzz test driver code is tested on a trial run. After multiple rounds of large language model agent generation and trial run tests, the final set of driver codes to be mined for vulnerabilities is obtained, and the driver code is loaded into the fuzz test engine. Fuzz testing of Python library code; (4) Store the vulnerability mining result information in the database and analyze the fuzzy test vulnerability mining results.

2. According to a method for generating a Python library fuzzy test driver based on a large language model according to claim 1, it is characterized in that This method uses static analysis tools to perform static analysis on the code in the Python library to obtain a test function information set, and extracts the Python library fuzz test driver from the open source fuzz testing project as the sample code stage: (1) Select the Python library code with high usage rate, extract the function signature, parameters, return value, and document string information from its source code through static analysis tools, and extract the fuzz test driver sample code from the open source fuzz test project; (2) For the extracted test function information, generate a test data set <test ID number, library name, version, source code file name, test function name, function signature, parameters, return value, document string, cyclomatic complexity score, source code content>. For the sample code, its data set is set to <sample ID number, library name, source code file name, sample source code, sample code flag>.

3. According to a method for generating a Python library fuzzy test driver based on a large language model according to claim 2, it is characterized in that This method combines fuzz test sample code and test function information to construct zero-sample, few-sample and thought chain prompts including driver sample code, generation task description, mutation strategy introduction, test function information set, driver sample code, code standardization description, and instruct the large language model to generate fuzz test driver code according to the corresponding prompt words: (1) Build an agent based on a large language model, set up multiple open source and closed source large model agents, and provide zero-sample, few-sample, and thought chain prompt templates as the basis for the next step of building a complete prompt word; (2) Based on the test data set and sample data set, arrange and fill in the prompt content in the order of driver sample code, generation task description, mutation strategy introduction, test function information set, driver sample code, and code standardization description. Generate zero-sample, few-sample, and thought chain prompts. After filtering and formatting the prompts, input the prompts into the large language model agent and send them to the remote large language model in the form of HTTP / HTTPS messages to generate fuzzy test driver code.

4. According to claim 3, a Python library fuzzy test driver generation method based on a large language model is characterized in that This method conducts a trial run test on the generated fuzz test driver code. After multiple rounds of large language model agent generation and trial run tests, the final driver code set to be exploited is obtained. The driver code is loaded into the fuzz test engine, and the Python library code is subjected to the fuzz test phase: (1) Based on the fuzz test driver code obtained in the previous step, after trial run test, it is labeled and classified according to whether it passes the trial run test; (2) For the code that fails the trial run test, multiple rounds of interaction with multiple large language model agents are performed again, and the large language model name, model service address, model service port, maximum token length, maximum number of attempts, and driver code generation prompts are sent to the large language model for inference generation; (3) After multiple rounds of generation and screening, the fuzzy test driver code that finally passes the trial run test is obtained and saved as a set of driver codes to be exploited for vulnerabilities.

5. A Python library fuzzy test driver generation method based on a large language model according to claim 4, characterized in that This method stores the vulnerability mining result information in the database and analyzes the fuzzy test vulnerability mining results: (1) Combine the fuzz test driver code obtained in the previous step with the Python code fuzz test engine to perform fuzz testing on the selected Python library code to obtain vulnerability information including coverage, number of crashes, number of exceptions, and number of errors; (2) Based on the results of vulnerability mining of the test function by the fuzz testing engine, the test function information, prompt words, and vulnerability mining results are stored in the database for further analysis.

Images