Block chain intelligent programming model evaluation method and system based on function test, and storage medium

Abstract

The invention discloses a function test-based block chain intelligent programming model evaluation method and system and a storage medium. The method comprises the following steps of: generating an initial demand; generating a plurality of smart contract codes meeting the demand based on the same initial demand; registering and screening functions, variables and events, and supplementing the functions, variables and events into the initial demand to form a detailed demand; generating a complete smart contract code based on the detailed demand; testing whether the complete intelligent contract code can be compiled or not; generating a plurality of test codes in batches according to main functions in the detailed requirements to form a test code set; combining with detailed demands to form a block chain intelligent programming model test suite; generating a plurality of block chain intelligent programming model test suites based on different intelligent contract types, and forming a block chain intelligent programming model test data set; and testing the to-be-tested block chain intelligent programming model based on the block chain intelligent programming model test data set. The method can accurately evaluate the capability of the intelligent programming model in the aspect of generating the intelligent contract.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to a blockchain intelligent programming model evaluation method, system and storage medium based on functional testing. Background Art

[0002] In recent years, large language models applied to smart contract generation tasks have emerged. This type of blockchain programming model provides new ideas and convenient ways for the development of smart contracts. It can quickly generate corresponding smart contract code frameworks based on given requirements, saving a lot of manpower and time costs.

[0003] At the same time, these blockchain intelligent programming models have also exposed many problems in practical applications. Since smart contracts involve a large number of asset transactions and key business logic, they have extremely high requirements for accuracy, security, and performance. However, the generation capabilities and areas of expertise of different types of large language models are different, and their capabilities in smart contract generation tasks need to be examined. At present, the testing methods and systems for blockchain intelligent programming models mainly test smart contracts at the function level, and often ignore the joint testing of multiple functions. However, since smart contracts often have specific functions, each function requires the collaboration of multiple different functions, so only testing smart contracts at the function level lacks comprehensiveness. This may not only cause errors in the execution of smart contracts generated by blockchain intelligent programming models, thereby affecting the normal development of business, but may also cause serious security vulnerabilities and cause risks such as asset losses.

[0004] It is of great significance to evaluate the blockchain intelligent programming model, which can not only discover the problems in the smart contract, but also guide it to modify the generated smart contract code and learn the reasons for the problems by feeding the problems back to the blockchain intelligent programming model. Summary of the invention

[0005] In response to the technical problems existing in the prior art, the present invention aims to provide a testing method based on smart contract function testing to further evaluate the capabilities of blockchain intelligent programming models, aiming to accurately evaluate the capabilities of intelligent programming models in generating smart contracts, ensure the quality of smart contracts, and promote the steady development of blockchain technology.

[0006] In a first aspect, the present invention provides a blockchain intelligent programming model evaluation method based on functional testing, comprising the following steps:

[0007] Step 1: Generate initial smart contract requirements;

[0008] Step 2: Using the large language model, generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in step 1 in different sessions;

[0009] Step 3: Based on the multiple smart contract codes generated in step 2, register and filter all the functions, variables, and events that appear, and add them to the initial smart contract requirements to form detailed smart contract requirements;

[0010] Step 4: Based on the detailed smart contract requirements, use the programming model to generate complete smart contract code;

[0011] Step 5: Test whether the complete smart contract code can be compiled. If the compilation fails, repeat step 4; if the compilation succeeds, go to step 6;

[0012] Step 6: Use the large language model to generate multiple test codes in batches for the main functions in the detailed smart contract requirements to form a test code set;

[0013] Step 7: Combine the detailed smart contract requirements and test code set to form a blockchain smart programming model test suite;

[0014] Step 8: Repeat steps 1 to 7 to generate multiple blockchain intelligent programming model test suites based on different smart contract types to form a blockchain intelligent programming model test data set;

[0015] Step 9: Test the blockchain intelligent programming model to be tested based on the blockchain intelligent programming model test data set.

[0016] Furthermore, the preliminary smart contract requirements in step 1 include the overall contract type, contract naming, main functions, main variables, functions and their events.

[0017] Furthermore, the programming model is a large language model or a template-based smart contract automatic generation model.

[0018] Furthermore, the screening rule in step 3 is: functions, variables and events whose occurrence frequency reaches a preset frequency or above are retained, and functions, variables and events whose occurrence frequency does not meet the requirement are discarded.

[0019] Furthermore, the preset frequency is 2 / 3 of the number of times generated and rounded down.

[0020] Furthermore, the test content in step 6 includes processing of input and output boundary conditions, simulation of complex interactions and user scenarios, constructor without parameter passing, specific method for determining function fallback, setting of token quantity, event triggering test, and test for common problems of smart contracts.

[0021] Furthermore, the blockchain intelligent programming model test suite includes detailed smart contract requirements and a test code set consisting of one or more smart contract test codes.

[0022] Furthermore, in step 9, the performance of the smart contract generated by the blockchain programming model in a single functional test is measured based on the Pass@k indicator.

[0023] On the other hand, the present invention provides a blockchain intelligent programming model evaluation system based on functional testing, comprising:

[0024] Module 1: It is used to generate initial smart contract requirements;

[0025] Module 2: It is used to generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in different sessions using a large language model;

[0026] Module 3: It is used to register and filter all functions, variables and events that appear based on the generated multiple smart contract codes, and supplement the initial smart contract requirements to form detailed smart contract requirements;

[0027] Module 4: It is used to generate complete smart contract code based on detailed smart contract requirements using programming models;

[0028] Module 5: It is used to test whether the complete smart contract code can be compiled. If the compilation fails, repeat the steps in module 4; if the compilation succeeds, execute the steps in module 6;

[0029] Module 6: It is used to use the large language model to generate multiple test codes in batches to form a test code set for the main functions in the detailed smart contract requirements;

[0030] Module 7: It is used to combine detailed smart contract requirements and test code sets to form a blockchain smart programming model test suite;

[0031] Module 8: It is used to generate multiple blockchain intelligent programming model test suites based on different smart contract types to form a blockchain intelligent programming model test data set;

[0032] Module 9: It is used to test the blockchain intelligent programming model to be tested based on the blockchain intelligent programming model test data set.

[0033] In a third aspect, the present invention provides a computer-readable medium, wherein the computer-readable medium is a server workstation;

[0034] The server workstation stores a computer program executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps in the blockchain intelligent programming model evaluation method based on functional testing.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention studies an evaluation method for performing single-function and multi-function tests on smart contracts and further evaluating the ability of blockchain intelligent programming models in smart contract generation tasks. By guiding a large language model to generate a smart contract test suite and testing the smart contracts generated by the blockchain intelligent programming model, the ability of the intelligent programming model in generating smart contracts can be accurately evaluated. In addition, through single-function and multi-function tests, the present invention makes up for the shortcomings of traditional smart contract testing methods that only use function-level tests on smart contracts. It can not only detect whether the generated smart contract has the correct function function, but also test whether the smart contract meets the functional requirements, and can ensure the availability, accuracy and security of smart contracts in complex application scenarios. By evaluating the blockchain intelligent programming model, the ability of the blockchain intelligent programming model in smart contract generation tasks can be examined, which helps technicians select or adjust the intelligent programming model used for blockchain smart contract generation according to the evaluation situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flow chart of an embodiment of the present invention.

[0039] Figure 2 This is an example diagram of the user requirements proposed in the embodiment of the present invention.

[0040] Figure 3 It is a schematic diagram of the composition of the blockchain intelligent programming model test suite according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0042] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in this application is a logical division. There may be other division methods in actual application. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection between units may be electrical or other similar forms, which are not limited in this application. Moreover, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0043] Example 1

[0044] like Figure 1 As shown, this embodiment provides a blockchain intelligent programming model evaluation method based on functional testing, comprising the following steps:

[0045] Step 1: Generate initial smart contract requirements;

[0046] Step 2: Using the large language model, generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in step 1 in different sessions;

[0047] Step 3: Based on the multiple smart contract codes generated in step 2, register and filter all the functions, variables, and events that appear, and add them to the initial smart contract requirements to form detailed smart contract requirements;

[0048] Step 4: Based on the detailed smart contract requirements, use the programming model to generate complete smart contract code;

[0049] Step 5: Test whether the complete smart contract code can be compiled. If the compilation fails, repeat step 4; if the compilation succeeds, go to step 6;

[0050] Step 6: Use the large language model to generate multiple test codes in batches for the main functions in the detailed smart contract requirements to form a test code set;

[0051] Step 7: Combine the detailed smart contract requirements and test code set to form a blockchain smart programming model test suite;

[0052] Step 8: Repeat steps 1 to 7 to generate multiple blockchain intelligent programming model test suites based on different smart contract types to form a blockchain intelligent programming model test data set;

[0053] Step 9: Test the blockchain intelligent programming model to be tested based on the blockchain intelligent programming model test data set.

[0054] In this embodiment, the specified smart contract type is [ERC-721NFT Contract], which is described as creating non-fungible tokens, which are usually used for collectibles, game props and unique assets.

[0055] The specific steps are as follows:

[0056] Step 1: Generate preliminary requirements based on the specified smart contract type using a large language model (ChatGPT-4o). Figure 2As shown, the generated smart contract requirements are as follows: This smart contract is an [ERC-721NFT contract], please name the smart contract [UniqueArt]. Its main functions include [minting unique NFT, transferring ownership, managing authorization]. The main variables include [tokenID (token unique identifier), ownerAddress (owner address), approvedAddress (authorized address)]. The main functions are [mintNFT (address to, uint256 tokenId) (minting NFT), transferNFT (address from, address to, uint256 tokenId) (transfer NFT ownership), approve (addressapproved, uint256tokenId) (authorization management)]. The main events include [NFT casting completed, ownership transfer, authorization completed].

[0057] Step 2: Generate the corresponding smart contract code based on the initial requirements. The generated smart contract code will include: a contract named UniqueArt, declarations of variables in the initial requirements, implementations of functions in the initial requirements, and definitions of events in the initial requirements.

[0058] Step 3: According to user needs, use the large language model (ChatGPT-4o) to repeatedly generate 10 copies of smart contract codes that meet the requirements. Take the statistical analysis of variables as an example to demonstrate the screening operation. In the variable name, ownerAddress appears 10 times, approveAddress appears 10 times, and tokenID appears 2 times. According to the method requirements, keep the functions with a frequency of 2 / 3 of the generated times. Therefore, tokenID is discarded and the rest of the function content is retained. After counting and filtering the functions, variables, and events respectively, the detailed requirements are obtained.

[0059] Step 4: Generate the smart contract again according to the detailed requirements. The generated smart contract code is called a complete smart contract, in which the names, functions, variables and events correspond to the detailed requirements.

[0060] Step 5: Use the smart contract compiler to check whether the smart contract can be compiled. For the smart contract generated in step 4, since it can be compiled, it can be processed in the next step. If the smart contract generated in step 4 cannot be compiled, it needs to be regenerated.

[0061] Step 6: Figure 3As shown, the functional description in the detailed requirements and the complete smart contract generated in step 5 are combined, and the test code is generated using the large language model (ChatGPT-4o). The description of the smart contract functions in the requirements of this embodiment is as follows: its main functions include [casting unique NFTs, transferring ownership, and managing authorization]. Therefore, it is necessary to generate smart contract test codes in 3 times, and the detailed requirements correspond to 3 smart contract test codes. The 3 smart contract test codes constitute the test code set.

[0062] Step 7: Integrate the detailed requirements of this embodiment with the test codes corresponding to all the functions described therein to form a one-to-many test suite of detailed requirements and test codes.

[0063] Step 8: Use different smart contract types as the input of step 1, repeat steps 1 to 7, generate multiple blockchain intelligent programming model test suites under different contract type backgrounds, and form a blockchain intelligent programming model test data set.

[0064] Step 9: For the blockchain intelligent programming model to be tested, use the blockchain intelligent programming model test data set generated in step 8 for testing. In terms of indicators, based on the idea of ​​Pass@k, the method uses single-function Pass@k to measure the performance of the smart contract generated by the blockchain intelligent programming model in a single function test. At the same time, the method also uses multi-function Pass@k indicators to measure the performance of the smart contract generated by the blockchain intelligent programming model in the case of multiple function joint tests.

[0065] The calculation formula of Pass@k is:

[0066]

[0067] Pass@10 is calculated as follows: the model generates 20 smart contract codes and randomly selects 10 of them. If at least one of the randomly selected smart contract codes passes the unit test, the test is passed. The c in the formula represents the number of randomly selected smart contract codes that can pass the unit test.

[0068] In this embodiment, single-function Pass@10 means that the model generates 20 codes, and then randomly selects 10 of them. If at least one of the randomly selected smart contract codes passes the single-function test, the test is passed. Multi-function Pass@10 means that the model generates 20 codes, and then randomly selects 10 of them. If at least one of the randomly selected smart contract codes passes the multi-function test formed by a combination of multiple single-function test codes, the test is passed. Through the single-function Pass@k and multi-function Pass@k indicators, the performance of the blockchain functional programming model in the smart contract generation task can be evaluated and evaluated. Based on the detailed requirements in the embodiment, the blockchain intelligent programming model (GPT-4o, GeMini1.5-Flash and Qwen-plus) is required to perform the smart contract generation task and test the single-function Pass@10 and multi-function Pass@10 of the generated smart contract code. After testing, the single-function Pass@10 of GPT-4o is 36.03%, and the multi-function Pass@10 is 10.94%. FeMini1.5-Flash has a single-function Pass@10 of 18.37% and a multi-function Pass@10 of 6.21%. Qwen-plus has a single-function Pass@10 of 21.66% and a multi-function Pass@10 of 7.02%. By comparison, it can be seen that among the three blockchain intelligent programming models, GPT-4o performs best in both single-function Pass@10 and multi-function Pass@10 indicators. It can be seen that GPT-4o's smart contract generation performance is better than the other two blockchain intelligent programming models.

[0069] Example 2

[0070] This embodiment provides a blockchain intelligent programming model evaluation system based on functional testing, including:

[0071] Module 1: It is used to generate initial smart contract requirements based on the programming model;

[0072] Module 2: It is used to generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in different sessions using a large language model;

[0073] Module 3: It is used to register and filter all functions, variables and events that appear based on the generated multiple smart contract codes, and supplement the initial smart contract requirements to form detailed smart contract requirements;

[0074] Module 4: It is used to generate complete smart contract code based on detailed smart contract requirements using programming models;

[0075] Module 5: It is used to test whether the complete smart contract code can be compiled. If the compilation fails, repeat the steps in module 4; if the compilation succeeds, execute the steps in module 6;

[0076] Module 6: It is used to use the large language model to generate multiple test codes in batches to form a test code set for the main functions in the detailed smart contract requirements;

[0077] Module 7: It is used to combine detailed smart contract requirements and test code sets to form a blockchain smart programming model test suite;

[0078] Module 8: It is used to generate multiple blockchain intelligent programming model test suites based on different smart contract types to form a blockchain intelligent programming model test data set;

[0079] Module 1: It is used to test the blockchain intelligent programming model to be tested based on the blockchain intelligent programming model test data set.

[0080] Example 3

[0081] A specific embodiment of the present invention also provides a computer-readable medium.

[0082] The computer readable medium is a server workstation;

[0083] The server workstation stores a computer program executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of an embodiment of the present invention involving a blockchain intelligent programming model evaluation method based on functional testing.

[0084] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

[0085] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0086] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A blockchain intelligent programming model evaluation method based on functional testing, characterized in that: The following steps are involved: Step 1: Generate initial smart contract requirements; Step 2: Using the large language model, generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in step 1 in different sessions; Step 3: Based on the multiple smart contract codes generated in step 2, register and filter all the functions, variables, and events that appear, and add them to the initial smart contract requirements to form detailed smart contract requirements; Step 4: Based on the detailed smart contract requirements, use the programming model to generate complete smart contract code; Step 5: Test whether the complete smart contract code can be compiled. If the compilation fails, repeat step 4; if the compilation succeeds, go to step 6; Step 6: Use the large language model to generate multiple test codes in batches for the main functions in the detailed smart contract requirements to form a test code set; Step 7: Combine the detailed smart contract requirements and test code set to form a blockchain smart programming model test suite; Step 8: Repeat steps 1 to 7 to generate multiple blockchain intelligent programming model test suites based on different smart contract types to form a blockchain intelligent programming model test data set; Step 9: Test the blockchain intelligent programming model to be tested based on the blockchain intelligent programming model test data set.

2. According to claim 1, a blockchain intelligent programming model evaluation method based on functional testing is characterized in that: The preliminary smart contract requirements in step 1 include the overall contract type, contract naming, main functions, main variables, functions and their events.

3. According to a functional testing-based blockchain intelligent programming model evaluation method according to claim 1, it is characterized in that: The programming model is a large language model or a template-based smart contract automatic generation model.

4. According to a functional testing-based blockchain intelligent programming model evaluation method according to claim 1, it is characterized in that: The screening rule in step 3 is: functions, variables and events whose occurrence frequency reaches a preset frequency or above are retained, and functions, variables and events whose occurrence frequency does not meet the requirement are discarded.

5. According to claim 4, a blockchain intelligent programming model evaluation method based on functional testing is characterized in that: The preset frequency is 2 / 3 of the number of times generated and is rounded down.

6. According to a functional testing-based blockchain intelligent programming model evaluation method according to claim 1, it is characterized in that: The test content in step 6 includes processing of input and output boundary conditions, simulation of complex interactions and user scenarios, constructor without parameter passing, specific method to determine function fallback, setting of token quantity, event triggering test and test for common problems of smart contracts.

7. According to claim 1, a blockchain intelligent programming model evaluation method based on functional testing is characterized in that: The blockchain intelligent programming model test suite includes detailed smart contract requirements and a test code set consisting of one or more smart contract test codes.

8. According to a functional testing-based blockchain intelligent programming model evaluation method according to claim 1, it is characterized in that: In step 9, the performance of the smart contract generated by the blockchain programming model in a single functional test is measured based on the Pass@k indicator.

9. According to a functional testing-based blockchain intelligent programming model evaluation system according to claim 1, it is characterized in that: include: Module 1: Generate initial smart contract requirements; Module 2: It is used to generate multiple smart contract codes that meet the initial smart contract requirements based on the same initial smart contract requirements generated in different sessions using a large language model; Module 3: It is used to register and filter all functions, variables and events that appear based on the generated multiple smart contract codes, and supplement the initial smart contract requirements to form detailed smart contract requirements; Module 4: It is used to generate complete smart contract code based on detailed smart contract requirements using programming models; Module 5: It is used to test whether the complete smart contract code can be compiled. If the compilation fails, repeat the steps in module 4; if the compilation succeeds, execute the steps in module 6; Module 6: It is used to use the large language model to generate multiple test codes in batches to form a test code set for the main functions in the detailed smart contract requirements; Module 7: It is used to combine detailed smart contract requirements and test code sets to form a blockchain smart programming model test suite; Module 8: It is used to generate multiple blockchain intelligent programming model test suites based on different smart contract types to form a blockchain intelligent programming model test data set; Module 9: It is used to test the blockchain intelligent programming model to be tested based on the blockchain intelligent programming model test data set; The functional testing-based blockchain intelligent programming model evaluation system is used to execute the steps in the functional testing-based blockchain intelligent programming model evaluation method of any one of claims 1-8.

10. A computer-readable medium, characterized in that The computer readable medium is a server workstation; The server workstation stores a computer program executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps in the functional testing-based blockchain intelligent programming model evaluation method of any one of claims 1 to 8.

Images