A method of generating a large language model for a blockchain smart contract code

Abstract

The present application relates to the technical field of machine learning and blockchain, and particularly relates to a method for generating a large language model of a blockchain smart contract code. A CodeBC model is generated through smart contract code, including the following steps: task definition; code filling, vulnerability detection; label-guided instruction fine-tuning. The present application proposes a safer blockchain smart contract code generation model CodeBC, which adopts a two-stage fine-tuning method to enhance the model's ability to program and understand human instructions for smart contract codes. Specifically, the first stage adopts a multi-task learning fine-tuning strategy including code filling and vulnerability detection tasks to enhance the model's understanding of smart contract codes and its ability to perceive security vulnerabilities; in the second stage, label-guided instruction fine-tuning is adopted to improve the model's understanding of language instructions and its ability to align the generated safe contract codes.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and blockchain technology, and in particular to a method for generating a large-scale language model from blockchain smart contract code. Background Art

[0002] To ensure transaction security and transparency, blockchains require developers to predefine transaction rules through smart contract code. These rules stipulate that when someone calls the code, it should automatically execute without an intermediary. While this facilitates on-chain transactions, it also creates security risks. If malicious actors call vulnerable code, these automatically executed codes could cause significant financial losses to users. Therefore, the security of smart contract code is paramount in the blockchain domain.

[0003] Currently, smart contracts are handcrafted by developers, facing challenges such as developer shortages, low development efficiency, and high security risks. (1) Smart contract code development requires specialized knowledge, resulting in a significant mismatch between the number of developers and users: blockchain users exceed 100 million, but there are only a little over 20,000 active developers. (2) The time-consuming nature of handwritten code significantly impacts development efficiency, often involving code with similar logical structures and requiring weeks to months of development time. (3) The lack of professionalism on the part of some developers leads to frequent security incidents and escalating economic losses. Summary of the Invention

[0004] This paper proposes a more secure blockchain smart contract code generation model CodeBC, which adopts a two-stage fine-tuning method to enhance the model's ability to program smart contract code and understand human instructions.

[0005] The technical solution adopted by the present invention is: a method for generating a large language model from blockchain smart contract code, generating a CodeBC model through smart contract code, including the following steps:

[0006] Step 1, task definition: D={(S1,T1,C1),...,(S N ,T N ,C N )} represents the dataset of smart contract code generation task, where S i is a natural language instruction, T i It is a safety label, C i Is the smart contract code, given a natural language instruction where s i It is instruction S i The word, label T i ∈{correct,bug}, the goal of the CodeBC model is to i and instruction Si The generated probability P(C i |S i ,T i ) to generate code where c i is the code token, P(C i |S i ,T i ) can be represented as:

[0007]

[0008] Step two, code filling: using the efficient training method of CodeLlama, the text is divided into three parts, prefix, infix and suffix, and by adjusting the position of each part, the model can make full use of the context information when predicting the infix code, where PRE, MID and SUF represent the start of each part, for the i-th example C i =(Pre i ,Mid i ,Suf i )∈D, we minimize the negative log-likelihood of the target Mid i :

[0009]

[0010] where m t represents the token of Mid i , θ represents the model parameters of CodeLlama;

[0011] Step three, vulnerability detection: strengthen the model's perception of security vulnerabilities in smart contracts, and provide alignment direction for subsequent label-guided instruction fine-tuning, given code C i , if C i has no vulnerabilities, the true label output is T gi ={[Tag] <correct>[ / Tag]}, otherwise T gi = {[Tag] <bug>[ / Tag]}, wherein [Tag] and [ / Tag] are the start and end marks of the code vulnerability detection result;

[0012] Step four, instruction fine-tuning guided by labels: use labels to help the model better distinguish between safe code and code with security vulnerabilities, given the human instruction S i and label information T i The goal of the code generation model is to generate real code C i , where T i ∈{correct,bug} indicates whether the code C i has a vulnerability, and the model optimization goal is to minimize the negative log-likelihood of the target code:

[0013]

[0014] Where c t is the t-th word element of C i , and θ D is the model parameter of the multi-task fine-tuned CodeLlama.

[0015] As a further improvement of the application, in step one, the overall fine-tuning structure of the CodeBC model consists of two stages. The first stage is a multi-task learning fine-tuning, in which the two tasks are code filling task optimization P(C i ) and vulnerability detection task optimization P(T i |C i ) to improve the generation probability P(C i |S i ,T i ); The second stage is instruction fine-tuning guided by labels, which directly optimizes P(C i |S i ,T i ) based on different T i , i.e. "correct" and "bug", and limits T i =correct during model inference to generate more secure contract code.

[0016] As a further improvement of the application, in step two, since some smart contracts are short in length, and the contract usually starts with the version number of solidity used, completely randomly dividing the contract code into three parts may make it difficult for the model to effectively learn the context information. Therefore, when dividing the code, the contract code is evenly divided into five parts according to the number of characters, and a random split point is selected in the second and fourth parts from left to right to ensure that the data participating in training has effective context information, and the combination method is randomly selected in PSM and SPM with a probability of 0.5.

[0017] As a further improvement of the application, in step three, the model performs the vulnerability detection task under the simple prompt of "whether the smart contract code is a correct solution", and the target sentence T gi The negative log likelihood of T

[0018]

[0019] Where t j is a token of T gi , and theta I is the model parameter of CodeLlama after code filling task fine-tuning.

[0020] As a further improvement of the application, in step four, given the artificial instruction S i and the restrictive label information T i = correct, the model will generate safer code C p with the help of T i = correct.

[0021] Advantages of the application: the application first introduces a large language model-based code generation model into the field of blockchain, promoting interdisciplinary research; the application proposes a customized two-stage fine-tuning strategy, which uses multi-task learning to assist the model to meet the specific needs of the blockchain field, and uses label-guided fine-tuning to enhance the understanding of artificial instructions on the basis of improving the model's alignment safety capability; the application constructs the first artificial evaluation dataset for a blockchain contract code generator, and the experimental results prove the effectiveness of the proposed model. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a schematic diagram of the CodeBC model of the method of the application for generating a large language model of a blockchain smart contract code. DETAILED DESCRIPTION

[0023] In order to make the technical problems, technical solutions and beneficial effects of the present application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the examples described herein are only used to explain the application and do not limit the application.

[0024] The application provides a method for generating a large language model of a blockchain smart contract code, which generates a CodeBC model through a smart contract code, including the following steps:

[0025] Step one, task definition: define D = {(S1, T1, C1),..., (S N , T N , C N )} represents the dataset of smart contract code generation task, where S i is a natural language instruction, T i It is a safety label, C i Is the smart contract code, given a natural language instruction where s i It is instruction S i The word, label T i ∈{correct,bug}, the goal of the CodeBC model is to i and instruction S i The resulting generation probability P(C i |S i ,T i ) to generate code where c i is a code word, P(C i |S i ,T i ) can be expressed as:

[0026]

[0027] like Figure 1 As shown in the figure, the overall fine-tuning structure of the CodeBC model consists of two stages. In the first stage, the two tasks in the multi-task learning fine-tuning are the code filling task to optimize P(C i ) and vulnerability detection task optimization P(T i |C i ), in order to increase the generation probability P(C i |S i ,T i ); The second stage of label-guided instruction fine-tuning is based on different T i , namely "correct" and "bug", directly on P(C i |S i ,T i ) for optimization and restrict T in the model inference phase i =correct to generate more secure contract code.

[0028] like Figure 1 In this paper, (1) multi-task learning is performed: (1-1) first fine-tuning is performed on the basic pre-trained language model using smart contract code; (1-2) then training is continued on the model with smart contract programming capabilities with the goal of vulnerability detection. (2) Finally, label-guided instruction fine-tuning is performed on the model trained in the first stage to align with the vulnerability detection labels in the first stage.

[0029] Step two, code filling: using the efficient training method of CodeLlama, the text is divided into three parts, prefix, infix and suffix, and by adjusting the position of each part, the model can make full use of the context information when predicting the infix code, where PRE, MID and SUF represent the start of each part, and for the ith example C i = (Pre i , Mid i , Suf i ) e D, we minimize the negative log-likelihood of the target Mid i :

[0030]

[0031] where m t represents the word of Mid i , and θ represents the model parameters of CodeLlama;

[0032] Since some smart contracts are short in length, and the contract usually starts with the version number of solidity used, completely randomly dividing the contract code into three parts may make the model unable to effectively learn the context information, therefore, when dividing the code, the contract code is evenly divided into five parts according to the number of characters, and the dividing point is randomly selected in the second and fourth parts from left to right, to ensure that the data participating in training has effective context information, and the combination method is randomly selected in PSM and SPM with a probability of 0.5.

[0033] Step three, vulnerability detection: strengthen the model's perception of security vulnerabilities in smart contracts, and provide alignment direction for subsequent label-guided instruction fine-tuning, given code C i , if C i has no vulnerabilities, the true label output is T gi = { [Tag] <correct>[ / Tag]}, otherwise T gi = {[Tag] <bug>[ / Tag]}, wherein [Tag], [ / Tag] are the marks of the beginning and the end of the code vulnerability detection result respectively;

[0034] The model performs the vulnerability detection task under the simple prompt "whether the smart contract code is a correct solution", and the target sentence T gi The negative log-likelihood is:

[0035]

[0036] Where t j is the word of T gi , θ I is the model parameter of the CodeLlama after code filling task fine-tuning.

[0037] Step four, instruction fine-tuning guided by labels: use labels to help the model better distinguish between safe code and code with security vulnerabilities, given artificial instructions S i and label information T i , the goal of the code generation model is to generate real code C i , where T i ∈{correct,bug} indicates whether the code C i has vulnerabilities, and the model optimization goal is to minimize the negative log-likelihood of the target code:

[0038]

[0039] Where c t is the t-th word of C i , θ D is the model parameter of CodeLlama after multi-task fine-tuning,

[0040] Given artificial instructions S i and restrictive label information T i = correct, the model will generate safer code C p with the help of T i = correct.

[0041] The application discloses a code generation model CodeBC for a blockchain smart contract, and designs a two-stage fine-tuning strategy for the security requirements of the smart contract. In order to verify the performance of CodeBC in the smart contract generation task, the Blockchain-HumanEval dataset is proposed. The experimental results show that the model improves the BLEU, CodeBLEU, compilation rate and vulnerability rate by nearly 17%, 13%, 42% and 36% respectively, and the proportion of safe contract codes increases from 40% to 78%. In the future, the control flow graph of the smart contract will be used to design a more stringent security strategy to ensure that the model can generate safe and usable codes.

[0042] The above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.< / bug> < / correct> < / bug> < / correct>

Claims

1. A method for generating a large language model from blockchain smart contract code, generating a CodeBC model through smart contract code, characterized in that: The following steps are involved: Step 1, task definition: D={(S1,T1,C1),…,(S N ,T N ,C N )} represents the dataset of smart contract code generation task, where S i is a natural language instruction, T i It is a safety label, C i Is the smart contract code, given a natural language instruction where s i It is instruction S i The word, label T i ∈{correct,bug}, the goal of the CodeBC model is to i and instruction S i The resulting generation probability P(C i |S i ,T i ) to generate code where c i is a code word, P(C i |S i ,T i ) can be expressed as: Step 2: Code filling: Using CodeLlama's efficient training method, the text is divided into three parts: prefix, infix, and suffix. By adjusting the position of each part, the model can make full use of context information when predicting the infix code. Among them, PRE, MID, and SUF are used to indicate the beginning of each part. For the i-th example C i =(Pre i ,Mid i ,Suf i )∈D, we minimize the target Mid i Negative log-likelihood of : where m t Indicates Mid i The word unit, θ represents the model parameters of CodeLlama; Step 3, vulnerability detection: Strengthen the model's ability to perceive security vulnerabilities in smart contracts and provide alignment direction for subsequent label-guided instruction fine-tuning. Given code C i , if C i There is no loophole, and the true label output is otherwise [Tag] and [ / Tag] are used as the start and end marks of the code vulnerability detection results respectively; Step 4: Label-guided instruction fine-tuning: Use labels to help the model better distinguish between secure code and code with security vulnerabilities. Given a manual instruction S i and label information T i , the goal of the code generation model is to generate real code C i , where T i ∈{correct,bug} represents code C i Is there a vulnerability? The model optimization goal is to minimize the negative log-likelihood of the target code: where c t C i The tth word, θ D Model parameters of CodeLlama after fine-tuning for multiple tasks.

2. A method for generating a large language model from blockchain smart contract code according to claim 1, characterized in that: In the step 1, the overall fine-tuning structure of the CodeBC model consists of two stages. The two tasks in the first stage of multi-task learning fine-tuning are the code filling task to optimize P(C i ) and vulnerability detection task optimization P(T i |C i ), in order to increase the generation probability P(C i |S i ,T i ); The second stage of label-guided instruction fine-tuning is based on different T i , namely "correct" and "bug", directly on P(C i |S i ,T i ) for optimization and restrict T in the model inference phase i =correct to generate more secure contract code.

3. The method for generating a large language model from blockchain smart contract code according to claim 1, characterized in that: In step 2, since some smart contracts are short and contracts usually start with the Solidity version number used, completely randomly dividing the contract code into three parts may prevent the model from effectively learning context information. Therefore, when splitting the code, the contract code is evenly divided into five parts based on the number of characters, and the dividing points are randomly selected from the second and fourth parts from left to right to ensure that the data involved in training has valid context information. The combination method is randomly selected in PSM and SPM with a probability of 0.

5.

4. The method for generating a large language model from blockchain smart contract code according to claim 1, characterized in that: In step 3, the model performs the vulnerability detection task under the simple prompt of "Is this smart contract code the correct solution?". The target sentence The negative log-likelihood of is: where t j for The word θ I Model parameters of CodeLlama after fine-tuning for the code completion task.

5. The method for generating a large language model from blockchain smart contract code according to claim 1, characterized in that: In step 4, a manual instruction S is given. i and restrictive label information T i =correct, the model will be i =correct to generate safer code C p .

Images