A method, system, and storage medium for evaluating blockchain smart programming models based on functional testing.

Abstract

This application discloses a method, system, and storage medium for evaluating blockchain smart programming models based on functional testing, comprising: generating initial requirements; generating multiple smart contract codes that meet the same initial requirements; registering and filtering functions, variables, and events, and supplementing them into the initial requirements to form detailed requirements; generating complete smart contract code based on the detailed requirements; testing whether the complete smart contract code is compileable; generating multiple test codes in batches for the main functions in the detailed requirements to form a test code set; combining the test code set with the detailed requirements to form a blockchain smart programming model test suite; generating multiple blockchain smart programming model test suites based on different smart contract types to form a blockchain smart programming model test dataset; and testing the blockchain smart programming model to be tested based on the blockchain smart programming model test dataset. This invention can accurately evaluate the ability of a smart programming model to generate smart contracts.

Description

Technical Field

[0001] This invention relates to the field of blockchain technology, and in particular to a method, system, and storage medium for evaluating blockchain smart programming models based on functional testing. Background Technology

[0002] In recent years, large language models for smart contract generation have emerged. These blockchain programming models provide new ideas and convenient approaches for smart contract development, enabling the rapid generation of corresponding smart contract code frameworks based on given requirements, saving a significant amount of manpower and time costs.

[0003] Meanwhile, these blockchain smart programming models have also exposed numerous problems in practical applications. Because smart contracts involve a large amount of asset transactions and critical business logic, they have extremely high requirements for accuracy, security, and performance. However, different types of large language models currently have varying generation capabilities and strengths, and their ability to generate smart contracts remains to be assessed. Currently, testing methods and systems for blockchain smart programming models primarily test smart contracts at the function level, often neglecting joint testing of multiple functions. However, since smart contracts often have specific functions, each requiring the collaboration of multiple different functions, testing only at the function level lacks comprehensiveness. This can not only lead to errors in the execution of smart contracts generated by the blockchain smart programming model, thus affecting normal business operations, but also potentially cause serious security vulnerabilities and risks such as asset losses.

[0004] Evaluating blockchain smart programming models is of great significance. It can not only identify problems in smart contracts, but also guide the blockchain smart programming model to modify the generated smart contract code and learn the reasons for the problems by feeding back the problems. Summary of the Invention

[0005] In view of the technical problems existing in the prior art, the present invention aims to provide a testing method based on smart contract functional testing to further evaluate the capabilities of blockchain smart programming models. The method aims to accurately evaluate the ability of smart programming models to generate smart contracts, ensure the quality of smart contracts, and promote the sound development of blockchain technology.

[0006] In a first aspect, the present invention provides a method for evaluating a blockchain smart programming model based on functional testing, comprising the following steps:

[0007] Step 1: Generate initial smart contract requirements;

[0008] Step 2: Using a large language model, generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in Step 1 in different sessions;

[0009] Step 3: Based on the multiple smart contract codes generated in Step 2, register all functions, variables and events that appear and filter them to supplement the initial smart contract requirements to form detailed smart contract requirements;

[0010] Step 4: Based on detailed smart contract requirements, generate complete smart contract code using a programming model;

[0011] Step 5: Test whether the complete smart contract code can be compiled. If the compilation fails, repeat step 4; if the compilation succeeds, proceed to step 6.

[0012] Step 6: Using a large language model, generate multiple test codes in batches to form a test code set for the main functions in the detailed smart contract requirements;

[0013] Step 7: Combine the detailed smart contract requirements and test code set to form a blockchain smart programming model test suite;

[0014] Step 8: Repeat steps 1 to 7 to generate multiple blockchain smart programming model test suites based on different smart contract types, forming a blockchain smart programming model test dataset;

[0015] Step 9: Test the blockchain smart programming model to be tested based on the blockchain smart programming model test dataset.

[0016] Furthermore, the preliminary smart contract requirements in step 1 include the overall contract type, contract name, main functions, main variables, functions and their events.

[0017] Furthermore, the programming model is a large language model or a template-based smart contract automatic generation model.

[0018] Furthermore, the filtering rule in step 3 is as follows: functions, variables, and events that occur at a preset frequency or higher are retained, while functions, variables, and events that do not occur at the required frequency are discarded.

[0019] Furthermore, the preset frequency is 2 / 3 of the number of generation times, rounded down.

[0020] Furthermore, the testing content in step 6 includes handling input / output boundary conditions, simulating complex interactions and user scenarios, constructing functions without parameters, determining function fallback in a specific way, setting the number of tokens, testing event triggering, and testing common smart contract issues.

[0021] Furthermore, the blockchain smart programming model test suite includes detailed smart contract requirements and a test code set consisting of one or more smart contract test codes.

[0022] Furthermore, in step 9, the Pass@k metric is used to measure the performance of the smart contract generated by the blockchain programming model in a single function test.

[0023] On the other hand, the present invention provides a blockchain smart programming model evaluation system based on functional testing, comprising:

[0024] Module 1: It is used to generate initial smart contract requirements;

[0025] Module 2: It is used to generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in different sessions, using a large language model;

[0026] Module 3: It is used to register and filter all functions, variables and events that appear based on the generated smart contract code, and supplement them into the initial smart contract requirements to form detailed smart contract requirements;

[0027] Module 4: It is used to generate complete smart contract code based on detailed smart contract requirements using a programming model;

[0028] Module 5: This is used to test whether the complete smart contract code can be compiled. If compilation fails, repeat the steps in Module 4; if compilation succeeds, execute the steps in Module 6.

[0029] Module Six: It is used to generate multiple test codes in batches to form a test code set based on the main functions in the detailed smart contract requirements using a large language model;

[0030] Module 7: It is used to combine detailed smart contract requirements and test code sets to form a blockchain smart programming model test suite;

[0031] Module 8: It is used to generate multiple blockchain smart programming model test suites based on different smart contract types, forming a blockchain smart programming model test dataset;

[0032] Module Nine: It is used to test the blockchain smart programming model to be tested based on the blockchain smart programming model test dataset.

[0033] Thirdly, the present invention provides a computer-readable medium, wherein the computer-readable medium is a server workstation;

[0034] The server workstation stores the computer program executed by the electronic device. When the computer program runs on the electronic device, it causes the electronic device to perform the steps in the blockchain smart programming model evaluation method based on functional testing.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This invention studies an evaluation method for performing single-function and multi-function tests on smart contracts and further assessing the capabilities of blockchain smart programming models in smart contract generation tasks. By guiding a large language model to generate a smart contract test suite and testing the smart contracts generated by the blockchain smart programming model, the ability of the smart programming model to generate smart contracts can be accurately evaluated. Furthermore, through single-function and multi-function testing, this invention overcomes the shortcomings of traditional smart contract testing methods that only perform function-level testing. It can not only detect whether the generated smart contracts have correct functionalities but also test whether the smart contracts meet functional requirements, ensuring the usability, accuracy, and security of smart contracts in complex application scenarios. By evaluating the blockchain smart programming model, its capabilities in smart contract generation tasks can be examined, helping technicians select or adjust the smart programming model used for blockchain smart contract generation based on the evaluation results. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart of an embodiment of the present invention.

[0039] Figure 2 This is an example diagram illustrating the user requirements proposed in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of the components of the blockchain smart programming model test kit according to an embodiment of the present invention. Detailed Implementation

[0041] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0042] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between units shown or discussed may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, which are not limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0043] Example 1

[0044] like Figure 1 As shown, this embodiment provides a method for evaluating blockchain smart programming models based on functional testing, including the following steps:

[0045] Step 1: Generate initial smart contract requirements;

[0046] Step 2: Using a large language model, generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in Step 1 in different sessions;

[0047] Step 3: Based on the multiple smart contract codes generated in Step 2, register all functions, variables and events that appear and filter them to supplement the initial smart contract requirements to form detailed smart contract requirements;

[0048] Step 4: Based on detailed smart contract requirements, generate complete smart contract code using a programming model;

[0049] Step 5: Test whether the complete smart contract code can be compiled. If the compilation fails, repeat step 4; if the compilation succeeds, proceed to step 6.

[0050] Step 6: Using a large language model, generate multiple test codes in batches to form a test code set for the main functions in the detailed smart contract requirements;

[0051] Step 7: Combine the detailed smart contract requirements and test code set to form a blockchain smart programming model test suite;

[0052] Step 8: Repeat steps 1 to 7 to generate multiple blockchain smart programming model test suites based on different smart contract types, forming a blockchain smart programming model test dataset;

[0053] Step 9: Test the blockchain smart programming model to be tested based on the blockchain smart programming model test dataset.

[0054] In this embodiment, the smart contract type is specified as

ERC-721NFT Contract

[0055] The specific steps are as follows:

[0056] Step 1: Use the large language model (ChatGPT-4o) to generate preliminary requirements based on the specified smart contract type. For example... Figure 2As shown, the generated smart contract requirements are as follows: This smart contract is an [ERC-721 NFT contract], please name it [UniqueArt]. Its main functions include [minting unique NFTs, transferring ownership, and managing authorization]. The main variables include [tokenID (unique token identifier), ownerAddress (owner address), approvedAddress (authorization address)]. The main functions are [mintNFT(address to, uint256 tokenId) (minting NFT), transferNFT(address from, address to, uint256 tokenId) (transferring NFT ownership), approve(address approved, uint256 tokenId) (authorization management)]. The main events include [NFT minting complete, ownership transfer, authorization complete].

[0057] Step 2: Generate the corresponding smart contract code based on the initial requirements. The generated smart contract code will include: a contract named UniqueArt, the declarations of variables in the initial requirements, the implementations of functions in the initial requirements, and the definitions of events in the initial requirements.

[0058] Step 3: Based on user requirements, generate 10 copies of smart contract code that meet the requirements using the large language model (ChatGPT-4o). The filtering operation is illustrated using statistical analysis of variables as an example. In the variable names, `ownerAddress` appears 10 times, `approveAddress` appears 10 times, and `tokenID` appears 2 times. According to the method requirements, functions with a frequency reaching 2 / 3 of the generation count are retained. Therefore, `tokenID` is discarded, and the remaining function content is retained. After statistical analysis and filtering of functions, variables, and events, the detailed requirements are obtained.

[0059] Step 4: Generate the smart contract again based on the detailed requirements. The generated smart contract code is named the complete smart contract, where the names, functions, variables, and events will correspond one-to-one with the detailed requirements.

[0060] Step 5: Use a smart contract compiler to check if the smart contract can be compiled. If the smart contract generated in Step 4 can be compiled, it can proceed to the next step. If the smart contract generated in Step 4 cannot be compiled, it needs to be regenerated.

[0061] Step 6: As Figure 3As shown, the functional descriptions in the detailed requirements are combined with the complete smart contract generated in step 5, and test code is generated using a large language model (ChatGPT-4o). In this embodiment, the smart contract's functions are described as follows: its main functions include [minting unique NFTs, transferring ownership, and managing authorization]. Therefore, smart contract test code needs to be generated in three stages, corresponding to three sets of smart contract test code. These three sets of smart contract test code constitute the test code set.

[0062] Step 7: Integrate the detailed requirements of this embodiment with the test code corresponding to all the functions described to form a test suite with one-to-many detailed requirements and test code.

[0063] Step 8: Using different smart contract types as inputs to Step 1, repeat Steps 1 to 7 to generate multiple blockchain smart programming model test suites under different contract type backgrounds, forming a blockchain smart programming model test dataset.

[0064] Step 9: For the blockchain smart programming model to be tested, use the blockchain smart programming model test dataset generated in Step 8 for testing. Regarding metrics, based on the Pass@k concept, the method uses single-function Pass@k to measure the performance of the smart contract generated by the blockchain smart programming model in a single-function test. Simultaneously, the method also applies a multi-function Pass@k metric to measure the performance of the smart contract generated by the blockchain smart programming model in multi-function joint testing scenarios.

[0065] The formula for calculating Pass@k is:

[0066]

[0067] The calculation method for Pass@10 is as follows: The model generates 20 smart contract codes, and then randomly selects 10 of them. A code passes the unit test if at least one of the randomly selected smart contract codes passes the unit test. In the formula, 'c' represents the number of randomly selected smart contract codes that pass the unit test.

[0068] In this embodiment, single-function Pass@10 means that the model generates 20 code samples, and then randomly selects 10 of them. A smart contract passes if at least one of the randomly selected samples passes the single-function test. Multi-function Pass@10 means that the model generates 20 code samples, and then randomly selects 10 of them. A smart contract passes if at least one of the randomly selected samples passes the multi-function test, which is a combination of multiple single-function test codes. The single-function Pass@k and multi-function Pass@k metrics can be used to evaluate the performance of the blockchain functional programming model in smart contract generation tasks. Based on the detailed requirements in this embodiment, the blockchain smart programming models (GPT-4o, GeMini1.5-Flash, and Qwen-plus) are required to perform smart contract generation tasks, and their single-function Pass@10 and multi-function Pass@10 scores for the generated smart contract code are tested. The test results show that GPT-4o's single-function Pass@10 score is 36.03%, and its multi-function Pass@10 score is 10.94%. FeMini1.5-Flash achieves a single-function Pass@10 of 18.37% and a multi-function Pass@10 of 6.21%. Qwen-plus achieves a single-function Pass@10 of 21.66% and a multi-function Pass@10 of 7.02%. The comparison shows that GPT-4o performs best in both single-function and multi-function Pass@10 metrics among the three blockchain smart programming models, indicating that GPT-4o's smart contract generation performance is superior to the other two blockchain smart programming models.

[0069] Example 2

[0070] This embodiment provides a blockchain smart programming model evaluation system based on functional testing, including:

[0071] Module 1: It is used to generate initial smart contract requirements based on the programming model;

[0072] Module 2: It is used to generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in different sessions, using a large language model;

[0073] Module 3: It is used to register and filter all functions, variables and events that appear based on the generated smart contract code, and supplement them into the initial smart contract requirements to form detailed smart contract requirements;

[0074] Module 4: It is used to generate complete smart contract code based on detailed smart contract requirements using a programming model;

[0075] Module 5: This is used to test whether the complete smart contract code can be compiled. If compilation fails, repeat the steps in Module 4; if compilation succeeds, execute the steps in Module 6.

[0076] Module Six: It is used to generate multiple test codes in batches to form a test code set based on the main functions in the detailed smart contract requirements using a large language model;

[0077] Module 7: It is used to combine detailed smart contract requirements and test code sets to form a blockchain smart programming model test suite;

[0078] Module 8: It is used to generate multiple blockchain smart programming model test suites based on different smart contract types, forming a blockchain smart programming model test dataset;

[0079] Module Jiu: It is used to test the blockchain smart programming model to be tested based on the blockchain smart programming model test dataset.

[0080] Example 3

[0081] A specific embodiment of the present invention also provides a computer-readable medium.

[0082] The computer-readable medium is a server workstation;

[0083] The server workstation stores computer programs executed by the electronic device. When the computer program runs on the electronic device, it causes the electronic device to perform the steps of an embodiment of the present invention related to a blockchain smart programming model evaluation method based on functional testing.

[0084] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0085] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0086] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for evaluating blockchain smart programming models based on functional testing, characterized in that, Includes the following steps: Step 1: Generate initial smart contract requirements; Step 2: Using a large language model, generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in Step 1 in different sessions; Step 3: Based on the multiple smart contract codes generated in Step 2, register all functions, variables and events that appear and filter them to supplement the initial smart contract requirements to form detailed smart contract requirements; Step 4: Based on detailed smart contract requirements, generate complete smart contract code using a programming model; Step 5: Test whether the complete smart contract code can be compiled. If the compilation fails, repeat step 4; if the compilation succeeds, proceed to step 6. Step 6: Using a large language model, generate multiple test codes in batches to form a test code set for the main functions in the detailed smart contract requirements; Step 7: Combine the detailed smart contract requirements and test code set to form a blockchain smart programming model test suite; Step 8: Repeat steps 1-7 to generate multiple blockchain smart programming model test suites based on different smart contract types, forming a blockchain smart programming model test dataset; Step 9: Test the blockchain smart programming model to be tested based on the blockchain smart programming model test dataset.

2. The method for evaluating a blockchain smart programming model based on functional testing according to claim 1, characterized in that, The initial smart contract requirements in step 1 include the overall contract type, contract name, main functions, main variables, functions and their events.

3. The method for evaluating a blockchain smart programming model based on functional testing according to claim 1, characterized in that, The programming model is either a large language model or a template-based smart contract automatic generation model.

4. The method for evaluating a blockchain smart programming model based on functional testing according to claim 1, characterized in that, The filtering rule in step 3 is as follows: functions, variables, and events that occur at or above a preset frequency are retained, while functions, variables, and events that do not occur at the required frequency are discarded.

5. The method for evaluating a blockchain smart programming model based on functional testing according to claim 4, characterized in that, The preset frequency is 2 / 3 of the number of generation times, rounded down.

6. The method for evaluating a blockchain smart programming model based on functional testing according to claim 1, characterized in that, The testing content in step 6 includes handling input / output boundary conditions, simulating complex interactions and user scenarios, constructing functions without parameters, determining function fallback in specific ways, setting the number of tokens, testing event triggering, and testing common smart contract issues.

7. The method for evaluating a blockchain smart programming model based on functional testing according to claim 1, characterized in that, The blockchain smart programming model test suite includes detailed smart contract requirements and a test code set consisting of one or more smart contract test codes.

8. The method for evaluating a blockchain smart programming model based on functional testing according to claim 1, characterized in that, In step 9, the Pass@k metric is used to measure the performance of the smart contract generated by the blockchain programming model in a single function test.

9. A blockchain intelligent programming model evaluation system based on functional testing, characterized in that, include: Module 1: Generating initial smart contract requirements; Module 2: It is used to generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in different sessions, using a large language model; Module 3: It is used to register and filter all functions, variables and events that appear based on the generated smart contract code, and supplement them into the initial smart contract requirements to form detailed smart contract requirements; Module 4: It is used to generate complete smart contract code based on detailed smart contract requirements using a programming model; Module 5: This is used to test whether the complete smart contract code can be compiled. If compilation fails, repeat the steps in Module 4; if compilation succeeds, execute the steps in Module 6. Module Six: It is used to generate multiple test codes in batches to form a test code set based on the main functions in the detailed smart contract requirements using a large language model; Module 7: It is used to combine detailed smart contract requirements and test code sets to form a blockchain smart programming model test suite; Module 8: It is used to generate multiple blockchain smart programming model test suites based on different smart contract types, forming a blockchain smart programming model test dataset; Module Nine: It is used to test the blockchain smart programming model to be tested based on the blockchain smart programming model test dataset; The blockchain smart programming model evaluation system based on functional testing is used to perform the steps in the blockchain smart programming model evaluation method based on functional testing according to any one of claims 1-8.

10. A computer-readable medium, characterized in that, The computer-readable medium is a server workstation; The server workstation stores a computer program executed by the electronic device. When the computer program is run on the electronic device, it causes the electronic device to perform the steps in the blockchain smart programming model evaluation method based on functional testing according to any one of claims 1-8.

Images