Method for assisting database fuzzy testing based on large language model

Abstract

The invention discloses a method and a system for assisting an enhanced database fuzz test based on a large language model. The method comprises the following steps: performing a fuzzy test on a target database, continuously generating variant SQL statements, monitoring whether the database crashes or not, and collecting a code coverage rate. For the variable SQL, if the variable SQL triggers database crash and logic errors or improves code coverage rate, resetting variation times to be zero; otherwise, accumulating the variation times. And when the number of variation times reaches a preset variation bottleneck threshold value, introducing a large language model for variation, and inputting the information of the combined database, the recent SQL, the execution result and the prediction target into the model according to a designed prompt project to obtain a new variation SQL statement. According to the method, the coverage rate bottleneck in a traditional database fuzzy test can be effectively broken through, and the diversity and the code coverage rate of the SQL are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of database fuzz testing, and in particular to a method for assisting in enhancing the coverage of database fuzz testing by using a large language model as a guide. Background Art

[0002] With the rapid development of information technology, database systems have been widely used in various industries and fields, such as finance, medical care, e-commerce, government affairs, etc. A large amount of critical data is stored in the database system, and its security and reliability are directly related to the operation of the enterprise and the interests of users. Therefore, it is crucial to ensure the security and vulnerability of the database system. Fuzz testing is a vulnerability detection technology based on anomaly injection. It is widely used in database vulnerability detection due to its high degree of automation and strong versatility. It inputs random data into the target system and determines whether there is a vulnerability based on whether an anomaly occurs in the tested system. Its basic principle is to trigger possible vulnerabilities in the target system by generating a large amount of random or semi-random data as input.

[0003] However, existing fuzz testing technology has many shortcomings in database vulnerability detection. First, test case construction is extremely difficult. There are many types of database test cases, covering various data types, operation methods, and query statements. Manual construction alone is labor-intensive and inefficient, and it is difficult to meet comprehensive testing needs, especially in the face of complex database systems. Secondly, database input specifications are complex, with specific data acceptance conditions and database table field length restrictions. The randomness of fuzz testing causes a large amount of test data to be unacceptable, wasting resources and reducing test efficiency.

[0004] In order to overcome these difficulties and improve the coverage and effect of database fuzz testing, researchers have proposed various methods to generate syntactically and semantically correct mutation test cases. The first type is to use rule-based case generation and mutation guided by coverage feedback. For example, Squirrel, a classic open source database memory vulnerability fuzz testing detection tool, is based on the AFL framework, and adds a syntax-based mutation algorithm and a semantically guided instantiation algorithm, and introduces a code coverage feedback mechanism to guide seed selection. However, this method does not have a sufficient understanding of the internal structure of the database and potential vulnerability patterns, and it is difficult to cover the logic and code outside the rules, making it impossible for many test cases to effectively hit the vulnerability area, resulting in extended testing time and low efficiency. In actual testing, the mutation effect is often slowed down, making it difficult to improve test coverage. In the face of complex database structures and application scenarios, it is easy to fall into local optimality and cannot fully explore the code path.

[0005] The second category is to use machine learning and data mining technology to generate more realistic and diverse test cases by learning and simulating data distribution and query patterns in real workloads. However, test case synthesis faces the challenges of grammatical and semantic correctness requirements and rich grammatical features. The synthesis method of the query synthesizer needs to meet multiple conditions, and it is difficult to ensure test coverage. The database management system has a complex architecture and diverse functions. The query optimizer's rule combination is complex, and large-scale data and high-concurrency operations must also be considered.

[0006] Existing test case generation methods have their own advantages, but they also have obvious limitations. Rule-based mutation methods are difficult to fully cover all code paths due to insufficient understanding of complex database structures. On the other hand, generation methods based on machine learning and data mining techniques often have difficulty ensuring grammatical correctness and sufficient test coverage when faced with the syntax, semantics, and large-scale concurrent operations of complex database management systems.

[0007] With the rapid development of large language models, by describing query conditions and expected results through dialogue with them, it is possible to generate diverse test cases and trigger different query paths and boundary conditions. In order to solve the problem faced by the prior art that test cases are difficult to cover logic and code outside the rules, the present invention provides a database fuzz testing method based on a large language model to assist in enhancing SQL diversity and improving code coverage. Summary of the invention

[0008] The present invention makes full use of the powerful learning ability of the large language model and its understanding of SQL-related knowledge, and applies it to the database fuzz testing process. When the database fuzz testing encounters a bottleneck in generating variant SQL, a targeted prompting project is designed to introduce a large language model for interaction, generate more effective SQL, improve the diversity of SQL variations, increase code coverage, and thus improve the possibility of discovering database memory vulnerabilities and logic vulnerabilities. The present invention provides a method for assisting database fuzz testing based on a large language model. The system overview involved in the described method is as follows: Figure 1 ,include:

[0009] 1. Initialization phase

[0010] Before the fuzz test begins, you need to prepare the target database, fuzz test engine tools, seed SQL, large language model, etc., and initialize the relevant mutation cycle conditions: mutation times and mutation bottleneck threshold. Including:

[0011] (1) Select the test target, compile the instrumentation, and establish a connection

[0012] The initial state of the test target is an empty database that has been instrumented. Before each test, the target database must be identified, and then the appropriate instrumentation tool must be selected to instrument and compile it, so as to record the execution path of the test case in the target database, collect code coverage data, and provide guidance for subsequent SQL mutation. Then, according to the type of database, the connection interface is customized to establish a connection between the fuzz test engine and the target database.

[0013] (2) Prepare the initial seed SQL

[0014] When conducting the first fuzz test, you usually need to prepare initial seed SQL, which includes common combination statements such as creating tables, inserting data, and querying data in tables. The initial seed SQL is not only the starting point of the fuzz test, but also plays a key role in the subsequent process.

[0015] In subsequent tests, the initial seed SQL provides a basis for analysis and mutation for the fuzz testing engine and large language model. These seeds contain the structural information and operation logic of the database, and can guide the subsequent generated mutation SQL to maintain grammatical and semantic correctness. At the same time, the quality of the seed SQL directly affects the breadth and efficiency of the test. High-quality seeds can cover complex code paths faster and help discover potential database vulnerabilities.

[0016] (3) Choose a large language model

[0017] Choose a suitable large language model based on the characteristics and requirements of database fuzz testing. Consider factors such as model performance, understanding of SQL language and database concepts, and scalability. You can choose a large language model that has been pre-trained and fine-tuned on database-related tasks, such as Code Llama.

[0018] Develop an interface for interacting with the large language model, including functions such as establishing connections, sending, and receiving, to ensure that prompt statements can be accurately and quickly transmitted to the model, and that new variant SQL statements returned by the model can be received in a timely manner.

[0019] (4) Initialize mutation loop conditions

[0020] The fuzz testing engine will execute three steps in a loop: generating mutation SQL, sending SQL to the target database for execution, and monitoring whether the database crashes or whether the code coverage rate increases. The loop will be terminated by sending an interrupt or timeout. During the entire fuzz testing process, a global variable called the number of mutations is maintained and initialized to zero; and a mutation bottleneck threshold is preset, which can be adjusted according to the actual mutation bottleneck situation. When the number of mutations reaches the preset mutation bottleneck threshold, it indicates that the traditional mutation method can no longer effectively explore new code branches, and enters the mutation bottleneck state. A large language model is introduced for mutation, and the number of mutations is reset to zero.

[0021] 2. Large Language Model Mutation Stage

[0022] The goal of this phase is to call the large language model through the interactive module ChatLLM to perform mutation operations on SQL statements to break through the current coverage bottleneck. In response to the bottleneck situation, a prompt statement containing key information such as the current status of the database (such as the structure, table, field, etc. of the database), the executed operations (the SQL statements that have been entered and their execution results), and the expected goals (the types of vulnerabilities expected to be discovered or the specific code areas that need to be covered, etc.) is constructed. The specific steps are as follows:

[0023] (1) Analyze the current status of the database

[0024] Through the database management system interface and tools, obtain detailed structural information of the database, including table name, field type, constraints, etc. This information is organized and summarized, and presented in the prompt statement in a clear and accurate text form to facilitate subsequent analysis and mutation operations. The database information description part uses CREATE TABLE to display the table structure, clearly specify the data type of each field, and list the primary key and foreign key relationship format, for example:

[0025] Table schema:

[0026] CREATE TABLE Users(user_id INT PRIMARY KEY,username VARCHAR(100),password VARCHAR(100),email VARCHAR(100));

[0027] …

[0028] (2) Organize the executed SQL

[0029] Record the executed variant SQL statements in detail, including the statement content, execution time, and returned results (such as query result set, whether the execution is successful, error information, etc.). Arrange the information of the five most recent SQL executions in chronological order or other logical order so that the test history can be clearly presented in the prompt statement. The description part of the SQL information starts with a numerical sequence number and is in the format of SQL statement and execution result. For example:

[0030] Five recent executed SQLs:

[0031] 1.SELECT*FROM Users WHERE username='admin'; Return result:[queryresult set or error message]

[0032] …

[0033] (3) Clarify expected goals

[0034] In order to give full play to the role of the large language model in seed mutation, you first need to set it as a seed mutator that matches the target database version. During the interaction process, the large language model will appear as the SQL mutation role of the database, for example: "You are the seed mutator for the database Mysql 8.4.0.".

[0035] Secondly, according to the specific goals and requirements of the test, clearly set the expected results of the fuzz test. These goals include improving code coverage, triggering potential database vulnerabilities, or verifying certain complex logical anomalies. In order for the large language model to accurately target these goals when generating mutation SQL, the test expectations must be clearly expressed in the prompt statement. For example: "The intended goal of this SQL mutation of the database is to detect the presence of database vulnerabilities and to cover as many code branches as possible, please perform the semantic aspects of the mutation according to the SQL syntax.".

[0036] (4)Combined prompt statements

[0037] Combine the above collected and organized information such as the current status of the database, executed operations, and expected goals to construct a complete prompt statement. Ensure that the prompt statement has clear logic and accurate content, and can provide sufficient context information for the large language model to better understand the current test situation and requirements. For example:

[0038] "You are the seed mutator for the database MySQL 8.4.0.

[0039] Table schema:

[0040] CREATE TABLE Users(user_id INT PRIMARY KEY,username VARCHAR(100),password VARCHAR(100),email VARCHAR(100));

[0041] …

[0042] Here are the last five executed SQL statements and their results:

[0043] 1.SELECT*FROM Users WHERE username='admin'; Return result:[queryresultset]

[0044] …

[0045] The intended goal of this SQL mutation of the database is to detect the presence of database vulnerabilities and to cover as many code branches as possible,please perform the semantic aspects of the mutation according to the SQL syntax."

[0046] 3. Testing and result collection phase

[0047] The function of this stage is to send the mutated SQL to the target database for execution, output the fuzz test results, and record the database status. Based on the collected SQL execution results, database status, and code coverage, the mutation times are updated to provide guidance for subsequent mutation cycles and optimize the test process. Including:

[0048] (1) Collect SQL results and monitor database status

[0049] Record the results of SQL execution in the target database, and monitor whether the database crashes or triggers logic errors. These SQL execution results and database status data can be used as input for SQL mutation using a large language model to further optimize the fuzz testing strategy. The subsequent mutation process can more accurately cover deep-level code logic and potential abnormal scenarios, thereby improving the comprehensiveness and efficiency of the test.

[0050] (2) Collecting code coverage

[0051] The fuzz testing engine evaluates the effect of SQL mutation by collecting and analyzing code coverage feedback, and adjusts the subsequent mutation direction accordingly. Coverage data usually takes basic block coverage and branch coverage as core indicators, which can intuitively reflect the depth of exploration of the database code by the current test. Higher code coverage means greater potential for vulnerability mining.

[0052] (3) Update the number of mutations

[0053] The mutation count indicates the number of worthless seeds observed so far. If the seed SQL triggers a database crash, a logic error, or improves code coverage, the seed is considered valuable and the mutation count is reset to zero; otherwise, the seed is considered worthless and the mutation count is accumulated. The mutation count is used to determine whether the current mutation has reached the bottleneck threshold, thereby deciding whether to introduce a large language model to optimize the test effect.

[0054] Based on the above method, the fuzz testing engine inputs the mutated SQL into the database for execution, and monitors its execution results, database status, and code coverage. When the number of seed mutations reaches the preset bottleneck threshold, the large language model is introduced for interaction. The input of the interaction includes database information, recently executed SQL, and task requirements. The large language model mutates based on the original database status and seed text, thereby increasing the diversity of SQL and the possibility of exploring deep-level code logic, further improving code coverage and the efficiency of discovering database vulnerabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 :System Overview Diagram

[0056] Figure 2 :System flow chart

[0057] Figure 3 : Interaction example diagram DETAILED DESCRIPTION

[0058] In order to illustrate the purpose, technical solutions and advantages of the present invention, the following is a detailed description of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The operation is mainly divided into two parts. The first part is the database fuzz testing part. The specific process is as follows: Figure 2 As shown in the figure, the second part is the large language model interaction module part. The interaction example template is as follows Figure 3 shown.

[0060] 1. Database fuzz testing part

[0061] (1) Algorithm Description

[0062] Algorithm input: DB, SQL, T

[0063] Algorithm Output: Mutate-SQL

[0064] Note: DB is the database being fuzz tested, SQL is the corresponding SQL statement, T is the total fuzz testing time, and Mutate-SQL is the mutated SQL statement.

[0065] (2) Algorithm steps:

[0066] 1) Initialize the number of mutations, mutation bottleneck threshold, database DB, fuzz testing engine and large language model;

[0067] 2) Execute the fuzz test and stop when the specified fuzz test time T is reached or a termination signal is received;

[0068] 3) If the number of mutations reaches the mutation bottleneck threshold, the interactive module ChatLLM is called to generate new SQL for mutation using the large language model;

[0069] 4) Send the mutated SQL to the database and execute it, monitoring whether the SQL triggers database crashes, logical errors, or improves code coverage.

[0070] 5) Update the mutation count. If the SQL fails to improve the code coverage or triggers a database crash or logic error, the SQL is considered to be worthless and the mutation count is accumulated. Otherwise, the SQL is considered to be valuable and the mutation count is reset to zero.

[0071] The pseudo code is as follows:

[0072]

[0073] 2. Large language model interaction module

[0074] (1) Algorithm Description

[0075] Algorithm input: C, SQLs

[0076] Algorithm Output: Mutate-SQL

[0077] Note: C is the database connection information, SQLs is the SQL statements that have been executed recently for 5 times, and Mutate-SQL is the mutated SQL statement.

[0078] (2) Algorithm steps:

[0079] 1) Read the basic description information of the database according to the database connection;

[0080] 2) The read basic description information, historical SQL statement set and prompt suggestions are combined and input into the large language model;

[0081] 3) Parse the large language model output and read it into the mutated SQL statement.

[0082] The pseudo code is as follows:

[0083]

Claims

1. A method for assisting in enhancing database fuzz testing coverage based on a large language model, characterized in that The implementation steps are: (1) Perform fuzz testing on the database, continuously input SQL statements, monitor database status, collect code coverage, and update mutation counts; (2) When the number of mutations reaches the preset mutation bottleneck threshold, it is determined that the mutation has encountered a bottleneck, and a large language model is introduced to assist in mutation of SQL; (3) Construct prompt statements containing key information such as database status, executed operations, and expected goals for bottleneck situations; (4) Input the prompt statement into the large language model for interaction, obtain new mutated SQL statements, and change the mutation direction; (5) Repeat the above steps to continuously improve the code coverage of database fuzz testing and ensure the security and reliability of the database system software code.

2. The method according to claim 1, characterized in that The monitoring of database status, collection of code coverage, and updating of mutation times include: Monitor the fuzz testing process in real time and maintain a global variable called mutation count, which is initialized to zero and indicates the number of worthless seeds observed so far. If the seed SQL triggers a database crash, a logic error, or improves code coverage, the seed is considered valuable and the mutation count is reset to zero; otherwise, the seed is considered worthless and the mutation count is accumulated. The mutation count is used to determine whether the current mutation has reached the bottleneck threshold, thereby deciding whether to introduce a large language model to optimize the test effect.

3. The method according to claim 1, characterized in that The step of judging whether a bottleneck is encountered according to the number of mutations includes: Before the fuzz test starts, a mutation bottleneck threshold is preset, which can be adjusted according to the actual mutation bottleneck situation. When the number of mutations reaches the preset mutation bottleneck threshold, it indicates that the traditional mutation method can no longer effectively explore new code branches and enters the mutation bottleneck state. At this time, the large language model is introduced for mutation and the number of mutations is reset to zero.

4. The method according to claim 1, characterized in that: The prompt statement containing key information such as the database status, executed operations and expected goals is constructed for the bottleneck situation, specifically including: (1) Analyze the current status of the database table structure, field attributes, keys, etc. in detail, and describe the database information in the format of the database definition language CREATE TABLE; (2) Organize the list of executed SQL statements, including the specific content of each statement, execution time, key features of the returned results, etc., as an important part of the prompt statement; (3) Clearly set the large language model as a seed mutator that matches the target database version, and explain the expected goals of this fuzz test, such as the specific types of vulnerabilities expected to be discovered, the key code areas that need to be covered, etc., so that the prompt statement has a clear direction and purpose.

5. The method according to claim 1, characterized in that The prompt statement is input into the large language model for interaction to obtain a new variant SQL statement, specifically including: Select a large language model that has been pre-trained and fine-tuned on database-related tasks, such as Code Llama, and optimize the interface connection with the model to ensure that prompt statements can be accurately and quickly transmitted to the large language model; input the combined prompt engineering statements into the large language model to obtain new variant SQL statements.

Images