LLM intelligent contract fuzzy test seed generation method based on antagonism role

Abstract

The invention belongs to the field of software testing, particularly relates to an intelligent contract vulnerability detection problem, and provides an LLM intelligent contract fuzzy test seed generation method based on an antagonism role. According to the method, a method for establishing antagonism roles for LLM is provided, a seed set is generated for fuzzy testing, and two key roles, namely a generator and an evaluator, are set for LLM. The generator is responsible for generating a preliminary test case set. And the evaluator is responsible for optimizing the generated test case set, the quality and availability of the test case set are improved through an adversarial method, and finally the optimized test case set is converted into a seed set to be loaded into a fuzzy test. The invention provides the intelligent contract fuzzy test seed generation method with high instruction coverage rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and specifically relates to the problem of smart contract detection. A method for generating LLM smart contract fuzzy test seeds based on adversarial roles is provided. Background Art

[0002] Smart contracts are automated computing programs that run on blockchains. They execute transactions and operations based on predefined rules and conditions under contract terms without the need for third-party trust, thus achieving decentralized programmable functions. Smart contracts can automatically execute contract terms, manage digital assets, conduct financial transactions, etc., providing a secure, transparent and tamper-proof digital contract solution. Due to the immutability and decentralization of blockchains and smart contracts, smart contracts cannot be modified once deployed on the blockchain. Although immutability ensures trust and transparency, it also means that any vulnerabilities or errors in the code cannot be easily corrected without deploying a new version of the contract. However, the frequent security incidents in smart contracts not only cause huge economic losses, but also threaten the stability and trust of the blockchain ecosystem.

[0003] Smart contract security issues have attracted much attention. The complexity and particularity of smart contracts make it difficult for traditional software testing technologies to fully cover their potential vulnerabilities and security risks. Fuzz testing is a method of automatic test case generation that aims to discover vulnerabilities and defects in software by injecting a large amount of abnormal or randomly generated data into the tested system. Fuzz testing focuses on triggering abnormal behavior of the program under test, thereby revealing potential security vulnerabilities, runtime errors, and stability issues. Fuzz testing technology has become one of the most effective means in the field of software testing and vulnerability mining, and can discover defects in the program that cause system operation abnormalities. Therefore, smart contract fuzz testing, as an effective security testing method, is being widely used in security assessment and vulnerability mining of smart contracts. A common fuzz testing method is mutation-based fuzzing, in which a set of seed inputs are first created, which are usually valid input data, and then these seed data are repeatedly and slightly randomly modified (mutated) to generate a large number of new test cases.

[0004] However, traditional smart contract fuzz testing methods have some limitations, which may affect their effectiveness and reliability in smart contract security assessment and vulnerability mining. First, traditional fuzz testing often relies on randomly generated initial seed sets. This random generation method often cannot fully cover the complex logic and boundary conditions of smart contracts, resulting in insufficient seed coverage. Smart contracts usually contain complex conditional branches, loop structures, and data interactions, as well as complex logic that interacts with blockchains. It is difficult for traditional random generation methods to generate representative seed sets, which limits the effectiveness of fuzz testing. Second, traditional fuzz testing often ignores the grammatical and semantic constraints in smart contracts. The grammatical and semantic rules of smart contracts are crucial to the correctness and security of contracts, but traditional fuzz testing methods often ignore these rules, resulting in the generated seeds may not conform to the grammatical rules of smart contracts, and may even contain logic that does not conform to the semantics of the contract. In this case, even if the seeds are generated, it is difficult to effectively trigger potential vulnerabilities in smart contracts, thereby reducing the effectiveness of fuzz testing. Summary of the invention

[0005] The problem solved by the present invention is to overcome the problem of insufficient coverage caused by the random generation of initial seed sets in existing smart contract fuzz testing technology. An adversarial role is established for LLM to generate the initial seed set of smart contracts, thereby improving the coverage of smart contract fuzz testing.

[0006] When fuzz testing an Ethereum smart contract, a local test network is first established, and then the contract to be tested is deployed on the network. The execution process of the Ethereum Virtual Machine (EVM) involves multiple steps. First, when a transaction is broadcast to the Ethereum network, the node collects it and verifies its validity. Once verified, the transaction enters the pending transaction pool and waits to be packaged into the block. When a miner mines a new block, it selects some transactions in the transaction pool and includes them in the new block. Then, all nodes receive the new block and start executing the transactions therein. For each transaction, the EVM loads the bytecode of the contract and performs the corresponding operations, which may result in a change of state. Finally, when all transactions are processed, the new block will be added to the blockchain, the state update will be saved, and the execution results of the transaction will be recorded. The present invention is a LLM smart contract fuzz test seed generation method based on adversarial roles, which specifically includes the following steps:

[0007] Step A: Smart contract compilation and deployment and fuzz testing environment initialization;

[0008] Step B: LLM adversarial role loading;

[0009] Step C: LLM-based smart contract fuzz testing seed generation;

[0010] Step D: Load the seed into the fuzz test.

[0011] Preferably, in step A of the method, the specific steps of smart contract compilation and deployment and fuzz testing environment initialization are as follows:

[0012] Step a1: Collect the source code of the smart contract to be tested and upgrade the Solidity version in the smart contract to a version compatible with the solc compiler version.

[0013] Step a2: The paths of dependent files are specified in the source code of the smart contract to be tested, and these files are in the same directory or subdirectory as the main contract file;

[0014] Step a3: Use the Solidity compiler Solc to compile the smart contract source code to generate the application binary interface file and the Ethereum virtual machine bytecode file;

[0015] Step a4: Initialize the test case database to store the test cases generated by the generator and the test cases optimized by the judge;

[0016] Step a5: Initialize the log database to store the instruction coverage and branch coverage during the test process;

[0017] Step a6: Initialize the seed database to store the seed sets before and during the fuzz testing;

[0018] Preferably, in step B of the method, the specific steps of LLM adversarial role loading are as follows:

[0019] Step b1: configure the LLM temperature parameter value, T = 0.7 / MT, where T is the temperature value and MT is the maximum temperature value;

[0020] Step b2: Configure the number of LLM generator roles;

[0021] Step b3: Configure the generator prompt words, design basic prompt words, and request LLM to generate a test case set based on the smart contract source code;

[0022] Step b4: Configure the judge prompt words, design basic prompt words, and request LLM to optimize the test cases generated by the generator;

[0023] Step b5: Optimize the prompt words in steps b2 and b3, and ask LLM to improve and optimize the vocabulary and sentences in the basic prompt words;

[0024] Preferably, in step C of the method, the specific steps of generating the smart contract fuzz test seed based on LLM are as follows:

[0025] Step c1: Collect smart contract source code, binary interface and context information;

[0026] Step c2: The request generator generates test cases according to the smart contract and writes the generated test cases into the test case database;

[0027] Step c3: Request the judge to optimize the test case and write the optimized test case into the test case database;

[0028] Step c4: Convert the test cases in the test case database into JSON format and write them into a JSON file;

[0029] Preferably, in step D of the method, the specific steps of loading the seed into the fuzzy test are as follows:

[0030] Step d1: Obtain the seed format of the fuzz testing system;

[0031] Step d2: Convert the test case in JSON format in step a4 into a seed and write it into the seed database;

[0032] Step d3: Load the seeds generated in step d2 as the initial seed set into the fuzz testing system;

[0033] Step d4: perform fuzz testing and record the branch coverage and instruction coverage during the fuzz testing process;

[0034] Beneficial effects:

[0035] There are some limitations in traditional smart contract fuzz testing methods, which may affect their effectiveness and reliability in smart contract security assessment and vulnerability mining. First, traditional fuzz testing often relies on randomly generated initial seed sets. This random generation method often cannot fully cover the complex logic and boundary conditions of smart contracts, resulting in insufficient seed coverage. Smart contracts usually contain complex conditional branches, loop structures, and data interactions, as well as complex logic that interacts with blockchains. It is difficult for traditional random generation methods to generate representative seed sets, which limits the effectiveness of fuzz testing. Second, traditional fuzz testing often ignores the grammatical and semantic constraints in smart contracts. The grammatical and semantic rules of smart contracts are crucial to the correctness and security of contracts, but traditional fuzz testing methods often ignore these rules, resulting in the generated seeds may not conform to the grammatical rules of smart contracts, and may even contain logic that does not conform to the semantics of the contract. In this case, even if the seeds are generated, it is difficult to effectively trigger potential vulnerabilities in smart contracts, thereby reducing the effectiveness of fuzz testing. To this end, the present invention provides a method for generating seeds for fuzz testing of LLM smart contract vulnerabilities based on adversarial roles. Our method first uses LLM to conduct in-depth analysis of the source code of the smart contract to be tested before the fuzz test is executed to generate a high-coverage test case set. Specifically, first, in order to guide LLM to generate and optimize the test case set, we designed an effective prompt word system. Compared with the single-step request LLM, these prompt words can help LLM understand the function and structure of the smart contract, so as to generate test cases more targeted and improve the coverage and quality of the test case set. Secondly, we propose a method for establishing an adversarial proxy role for LLM to optimize the generated test cases. We set two key roles for LLM: generator and judge. The generator is responsible for generating a preliminary test case set based on the language understanding and generation capabilities of LLM. The judge is responsible for optimizing the generated test case set, and improving the quality and availability of the test case set through adversarial methods. This adversarial mechanism ensures that the generated test case set meets the requirements of the smart contract in terms of syntax, semantics and logic, thereby improving the effectiveness and accuracy of the test case set. Finally, we convert the test case set into the seed set of the target fuzz testing system and load it into the fuzz testing system as the initial seed set.

[0036] The experimental results of typical examples verify the effectiveness of this method: 130 smart contracts crawled from the Ethereum are selected as the data set, and the performance of instruction coverage is compared with the industry's advanced fuzz testing tool Ityfuzz. The experimental results of the comparison between this method and Ityfuzz are shown in Table 1. As can be seen from Table 1, the instruction coverage of this method reaches 44.1% after running for 10 seconds, and it is better than the Ityfuzz tool in the following 50 seconds. Therefore, this method can effectively improve the efficiency of fuzz testing and instruction coverage.

[0037] Table 1 Performance of this method and Ityfuzz in instruction coverage

[0038] BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are only used to more fully illustrate the process of the present invention and do not constitute a limitation on the scope of the present invention.

[0040] Figure 1 Flowchart of the seed generation method for adversarial role-based fuzz testing of LLM smart contracts. Specific implementation plan

[0041] The present invention will be further explained below with reference to the accompanying drawings. Figure 1 This is a flow chart of the LLM smart contract fuzzy test seed generation method based on adversarial roles in the present invention, which mainly includes the following steps:

[0042] Step A: Smart contract compilation and deployment and fuzz testing environment initialization;

[0043] Step B: LLM adversarial role loading;

[0044] Step C: LLM-based smart contract fuzz testing seed generation;

[0045] Step D: Load the seed into the fuzz test.

[0046] Preferably, in step A of the method, the specific steps of smart contract compilation and deployment and fuzz testing environment initialization are as follows:

[0047] Step a1: Collect the source code of the smart contract to be tested and upgrade the Solidity version in the smart contract to a version compatible with the solc compiler version.

[0048] Step a2: The paths of dependent files are specified in the source code of the smart contract to be tested, and these files are in the same directory or subdirectory as the main contract file;

[0049] Step a3: Use the Solidity compiler Solc to compile the smart contract source code to generate the application binary interface file and the Ethereum virtual machine bytecode file;

[0050] Step a4: Initialize the test case database to store the test cases generated by the generator and the test cases optimized by the judge;

[0051] Step a5: Initialize the log database to store the instruction coverage and branch coverage during the test process;

[0052] Step a6: Initialize the seed database to store the seed sets before and during the fuzz testing;

[0053] Preferably, in step B of the method, the specific steps of LLM adversarial role loading are as follows:

[0054] Step b1: configure the LLM temperature parameter value, T = 0.7 / MT, where T is the temperature value and MT is the maximum temperature value. Taking the forwarding interface ChatAnywhere as an example, such as the request in Table 2, the temperature value is set to 0.7;

[0055] Table 2 Interface request examples

[0056]

[0057] Step b2: Configure the number of LLM generator roles;

[0058] Step b3: Configure the generator prompt and design a basic prompt: "You are a generator of smart contract test cases. {Contract source code} is one of my smart contracts. Please generate enough test cases to cover all functions, branches and statements within functions." to request LLM to generate a test case set based on the smart contract source code;

[0059] Step b4: Configure the judge prompt words and design basic prompt words: "You are a judge of a smart contract test case. The above test cases do not reach 100% coverage, so I need you to generate more test cases based on the test cases and source codes I gave you to make the instruction coverage of the test case set reach 100%.", to request the LLM to optimize the test cases generated by the generator;

[0060] Step b5: Optimize the prompt words in steps b2 and b3: "{original prompt words}, the above is a prompt word I provided, please improve and optimize the vocabulary and sentences in it.", to request LLM to improve and optimize the vocabulary and sentences in the basic prompt words;

[0061] Preferably, in step C of the method, the specific steps of generating the smart contract fuzz test seed based on LLM are as follows:

[0062] Step c1: Collect smart contract source code, binary interface and context information. The binary interface is the abi file compiled and generated in step a3. The abi file defines how the contract methods are called, as well as the parameters and return value formats required to call them. Context information is all the explanatory information about the smart contract, such as the comment information at the beginning of the smart contract file, which generally summarizes the functions of the smart contract;

[0063] Step c2: The request generator generates test cases based on the smart contract and writes the generated test cases into the test case database. The test case database stores data in the form of key-value pairs, where the key is the smart contract name and the value is the generated test case set;

[0064] Step c3: Request the judge to optimize the test case and write the optimized test case into the test case database;

[0065] Step c4: Convert the test cases in the test case database into JSON format and write them into a JSON file;

[0066] Preferably, in step D of the method, the specific steps of loading the seed into the fuzzy test are as follows:

[0067] Step d1: Obtain the seed format of the fuzz testing system. Taking the experimental benchmark system Ityfuzz as an example, the seed format is shown in Table 3;

[0068] Table 3 Seed format example

[0069]

[0070] Step d2: Convert the test case in JSON format in step a4 into a seed and write it into the seed database. The seed format shown in Table 3 contains the following information: the caller of the contract, the contract address, the called function, and the function parameter value. Except for the function parameter value, the rest of the information is static information, so only convert the test case in JSON format into the data field in the table;

[0071] Step d3: Load the seeds generated in step d2 as the initial seed set into the fuzz test system. Taking the experimental benchmark system Ityfuzz as an example, initialCorpus is used to specify the initial seed set path when the fuzz test is started, as shown in Table 4;

[0072] Table 4 Example of loading the initial seed set

[0073]

[0074] Step d4: Execute fuzz testing and record the branch coverage and instruction coverage during the fuzz testing process.

Claims

1. A method for generating seeds for smart contract fuzz testing based on adversarial role LLM. Its characteristics include the following steps: Step A: Smart contract compilation and deployment and fuzz testing environment initialization; Step B: LLM adversarial role loading; Step C: LLM-based smart contract fuzz testing seed generation; Step D: Load the seed into the fuzz test.

2. The adversarial role-based LLM smart contract fuzz testing seed generation method according to claim 1 is characterized in that: In step A of the method, the specific steps of smart contract compilation and deployment and fuzz testing environment initialization are as follows: Step a1: Collect the source code of the smart contract to be tested, and upgrade the Solidity version in the smart contract to a version compatible with the solc compiler version; Step a2: The paths of dependent files are specified in the source code of the smart contract to be tested, and these files are in the same directory or subdirectory as the main contract file; Step a3: Use the Solidity compiler Solc to compile the smart contract source code to generate the application binary interface file and the Ethereum virtual machine bytecode file; Step a4: Initialize the test case database to store the test cases generated by the generator and the test cases optimized by the judge; Step a5: Initialize the log database to store the instruction coverage and branch coverage during the test process; Step a6: Initialize the seed database to store the seed sets before and during the fuzz testing.

3. The adversarial role-based LLM smart contract fuzz testing seed generation method according to claim 1, characterized in that: In step B of the method, the specific steps of LLM adversarial role loading are as follows: Step b1: configure the LLM temperature parameter value, T = 0.7 / MT, where T is the temperature value and MT is the maximum temperature value; Step b2: Configure the number of LLM generator roles; Step b3: Configure the generator prompt words, design basic prompt words, and request LLM to generate a test case set based on the smart contract source code; Step b4: Configure the judge prompt words, design basic prompt words, and request LLM to optimize the test cases generated by the generator; Step b5: Optimize the prompt words in steps b2 and b3, and ask LLM to improve and optimize the vocabulary and sentences in the basic prompt words.

4. The method for generating LLM smart contract fuzz testing seeds based on adversarial roles according to claim 1, characterized in that: In step C of the method, the specific steps of generating smart contract fuzz test seeds based on LLM are as follows: Step c1: Collect smart contract source code, binary interface and context information; Step c2: The request generator generates test cases according to the smart contract and writes the generated test cases into the test case database; Step c3: Request the judge to optimize the test case and write the optimized test case into the test case database; Step c4: Convert the test cases in the test case database into JSON format and write them into a JSON file.

5. The adversarial role-based LLM smart contract fuzz testing seed generation method according to claim 1, characterized in that: In step D of the method, the specific steps of loading the seed into the fuzzy test are as follows: Step d1: Obtain the seed format of the fuzz testing system; Step d2: Convert the test case in JSON format in step c4 into a seed and write it into the seed database; Step d3: Load the seeds generated in step d2 as the initial seed set into the fuzz testing system; Step d4: Execute fuzz testing and record the branch coverage and instruction coverage during the fuzz testing process.

Images