Method for generating large language model through block chain intelligent contract code

Abstract

The invention relates to the technical field of machine learning and block chains, in particular to a method for generating a large language model through a block chain intelligent contract code. A CodeBC model is generated through an intelligent contract code, and the method comprises the following steps: task definition; code filling and vulnerability detection; and tag-guided instruction fine tuning. The invention provides a safer block chain smart contract code generation model CodeBC. A two-stage fine tuning method is adopted to enhance the ability of the model to program smart contract codes and understand human instructions. Specifically, in the first stage, a multi-task learning fine tuning strategy including a code filling task and a vulnerability detection task is adopted to enhance the understanding of a model on an intelligent contract code and the perception ability of the model on security vulnerabilities; and in the second stage, tag-guided instruction fine tuning is adopted, so that the ability of the model to generate a security contract code is aligned while the language instruction understanding of the model is improved.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and blockchain technology, and in particular to a method for generating a large language model from blockchain smart contract code. Background Art

[0002] To ensure the security and transparency of transactions, blockchain requires developers to predefine transaction rules through smart contract code and stipulate that when someone calls the code, the code should be automatically executed without an intermediary. While facilitating on-chain transactions, it also brings security risks. If malicious actors call codes with vulnerabilities, these automatically executed codes may cause significant financial losses to users. Therefore, in the blockchain domain, the security of smart contract code is crucial.

[0003] Currently, smart contracts are handcrafted by developers, facing challenges such as developer shortage, low development efficiency, and high security risks. (1) Smart contract code development requires professional knowledge, resulting in a serious mismatch between the number of developers and users. There are more than 100 million blockchain users, but only more than 20,000 active developers. (2) The time-consuming nature of hand-written code greatly affects development efficiency. It usually involves codes with similar logical structures and takes weeks to months to develop. (3) Some developers lack professional qualities, resulting in frequent security incidents and escalating economic losses. Summary of the invention

[0004] This paper proposes a more secure blockchain smart contract code generation model CodeBC, which adopts a two-stage fine-tuning method to enhance the model's ability to program smart contract code and understand human instructions.

[0005] The technical solution adopted by the present invention is: a method for generating a large language model from a blockchain smart contract code, generating a CodeBC model through smart contract code, comprising the following steps:

[0006] Step 1: Task definition: D = {(S 1 ,T 1 ,C 1 ),...,(S N ,T N ,C N )} represents the dataset of smart contract code generation task, where S i is a natural language instruction, T i It is a safety label, C i is the smart contract code, given a natural language instruction where s i It is the instruction S i The word, label T i∈{correct,bug}, the goal of the CodeBC model is to i and instruction S i The resulting generation probability P(C i |S i ,T i ) to generate code where c i is a code word, P(C i |S i ,T i ) can be expressed as:

[0007]

[0008] Step 2: Code filling: CodeLlama's efficient training method is used to divide the text into three parts: prefix, infix, and suffix. By adjusting the position of each part, the model can make full use of context information when predicting the infix code. PRE, MID, and SUF are used to indicate the beginning of each part. For the i-th example C i =(Pre i ,Mid i ,Suf i )∈D, we minimize the objective Mid i The negative log-likelihood of :

[0009]

[0010] Where m t Indicates Mid i , θ represents the model parameters of CodeLlama;

[0011] Step 3: Vulnerability detection: Enhance the model’s ability to detect security vulnerabilities in smart contracts and provide alignment directions for subsequent label-guided instruction fine-tuning. Given code C i , if C i There is no loophole, the actual label output is T gi ={[Tag] <correct>[ / Tag]}, otherwise T gi ={[Tag] <bug>[ / Tag]}, where [Tag] and [ / Tag] are used as the start and end marks of the code vulnerability detection results respectively;

[0012] Step 4: Label-guided instruction fine-tuning: Use labels to help the model better distinguish between secure code and code with security vulnerabilities. Given a manual instruction S i and label information T i The goal of the code generation model is to generate real code C i , where T i ∈{correct,bug} represents code C i Is there a vulnerability? The model optimization goal is to minimize the negative log-likelihood of the target code:

[0013]

[0014] where c t C i The tth word, θ D Model parameters of CodeLlama after fine-tuning for multiple tasks.

[0015] As a further improvement of the present invention, in step 1, the overall fine-tuning structure of the CodeBC model consists of two stages. The two tasks in the first stage of multi-task learning fine-tuning are code filling task optimization P(C i ) and vulnerability detection task optimization P(T i |C i ) to increase the generation probability P(C i |S i ,T i ); The second stage of label-guided instruction fine-tuning is based on different T i , namely "correct" and "bug", directly affect P(C i |S i ,T i ) and restrict T in the model inference phase. i =correct to generate more secure contract code.

[0016] As a further improvement of the present invention, in the step 2, since some smart contracts are short in length and the contract usually starts with the solidity version number used, completely randomly dividing the contract code into three parts may make the model unable to effectively learn context information. Therefore, when splitting the code, the contract code is evenly divided into five parts according to the number of characters, and the dividing points are randomly selected from the second and fourth parts from left to right to ensure that the data involved in the training has valid context information, and the combination method is randomly selected in PSM and SPM with a probability of 0.5.

[0017] As a further improvement of the present invention, in step 3, the model performs the vulnerability detection task under the simple prompt of "Is the smart contract code the correct solution?", and the target sentence T gi The negative log-likelihood of is:

[0018]

[0019] where t j T gi The word unit, θ I Model parameters of CodeLlama after fine-tuning for the code completion task.

[0020] As a further improvement of the present invention, in step 4, given the manual instruction S i and restrictive label information T i =correct, the model will be i =correct to generate safer code C p .

[0021] Beneficial effects of the present invention: The present invention introduces the code generation model based on the large language model into the blockchain field for the first time, promoting interdisciplinary research; the present invention proposes a customized two-stage fine-tuning strategy, which assists the model to meet the specific needs of the blockchain field through multi-task learning, and uses a label-guided fine-tuning method to enhance the understanding of manual instructions on the basis of improving the model alignment security capability; the present invention constructs the first artificial evaluation dataset of the blockchain contract code generator, and proves the effectiveness of the proposed model through experimental results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the CodeBC model of a method for generating a large language model from blockchain smart contract code in the present invention. DETAILED DESCRIPTION

[0023] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The present invention provides a method for generating a large language model from a blockchain smart contract code, and generating a CodeBC model through the smart contract code, comprising the following steps:

[0025] Step 1: Task definition: D = {(S 1 ,T 1 ,C 1 ),...,(S N ,T N ,C N )} represents the dataset of smart contract code generation task, where S i is a natural language instruction, T i It is a safety label, C i is the smart contract code, given a natural language instruction where s i It is the instruction S i The word, label T i ∈{correct,bug}, the goal of the CodeBC model is to i and instruction S i The resulting generation probability P(C i |S i ,T i ) to generate code where c i is a code word, P(C i |S i ,T i ) can be expressed as:

[0026]

[0027] like Figure 1 As shown in the figure, the overall fine-tuning structure of the CodeBC model consists of two stages. In the first stage, the two tasks in the multi-task learning fine-tuning are the code filling task to optimize P(C i ) and vulnerability detection task optimization P(T i |C i ) to increase the generation probability P(C i |S i ,T i ); The second stage of label-guided instruction fine-tuning is based on different T i , namely "correct" and "bug", directly affect P(C i |S i ,T i ) and restrict T in the model inference phase. i =correct to generate more secure contract code.

[0028] like Figure 1 In this paper, (1) multi-task learning is performed: (1-1) first fine-tuning is performed on the basic pre-trained language model using smart contract code; (1-2) then training is continued on the model with smart contract programming capabilities with the goal of vulnerability detection. (2) Finally, label-guided instruction fine-tuning is performed on the model trained in the first stage to align with the vulnerability detection labels in the first stage.

[0029] Step 2: Code filling: CodeLlama's efficient training method is used to divide the text into three parts: prefix, infix, and suffix. By adjusting the position of each part, the model can make full use of context information when predicting the infix code. PRE, MID, and SUF are used to indicate the beginning of each part. For the i-th example C i =(Pre i ,Mid i ,Suf i )∈D, we minimize the objective Mid i The negative log-likelihood of :

[0030]

[0031] Where m t Indicates Mid i , θ represents the model parameters of CodeLlama;

[0032] Since some smart contracts are short in length and the contract usually starts with the solidity version number used, completely randomly dividing the contract code into three parts may make the model unable to effectively learn context information. Therefore, when splitting the code, the contract code is evenly divided into five parts according to the number of characters, and the dividing points are randomly selected from the second and fourth parts from left to right to ensure that the data involved in the training has valid context information, and the combination method is randomly selected in PSM and SPM with a probability of 0.5.

[0033] Step 3: Vulnerability detection: Enhance the model’s ability to detect security vulnerabilities in smart contracts and provide alignment directions for subsequent label-guided instruction fine-tuning. Given code C i , if C i There is no loophole, the actual label output is T gi ={[Tag] <correct>[ / Tag]}, otherwise T gi ={[Tag] <bug>[ / Tag]}, where [Tag] and [ / Tag] are used as the start and end marks of the code vulnerability detection results respectively;

[0034] The model performs the vulnerability detection task under the simple prompt "Is this smart contract code the correct solution?", and the target sentence T gi The negative log-likelihood of is:

[0035]

[0036] where t j T gi The word unit, θ I Model parameters of CodeLlama after fine-tuning for the code completion task.

[0037] Step 4: Label-guided instruction fine-tuning: Use labels to help the model better distinguish between secure code and code with security vulnerabilities. Given a manual instruction S i and label information T i The goal of the code generation model is to generate real code C i , where T i ∈{correct,bug} represents code C i Is there a vulnerability? The model optimization goal is to minimize the negative log-likelihood of the target code:

[0038]

[0039] where c t C i The tth word, θ D Model parameters of CodeLlama after multi-task fine-tuning,

[0040] Given a manual instruction S i and restrictive label information T i =correct, the model will be i =correct to generate safer code C p .

[0041] The present invention proposes a code generation model CodeBC for blockchain smart contracts, and designs a two-stage fine-tuning strategy for the security requirements of smart contracts. In order to verify the performance of CodeBC on the smart contract generation task, the Blockchain-HumanEval dataset is proposed. Experimental results show that the model improves BLEU, CodeBLEU, compilation rate, and vulnerability rate by nearly 17%, 13%, 42%, and 36%, respectively, and the proportion of contract code that can be safely used increases from 40% to 78%. In the future, this application will use the control flow graph of smart contracts to design stricter security policies to ensure that the model can generate safe and usable code.

[0042] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / bug> < / correct> < / bug> < / correct>

Claims

1. A method for generating a large language model from blockchain smart contract code, generating a CodeBC model through smart contract code, characterized in that: The following steps are involved: Step 1, task definition: D = {(S1, T1, C1), …, (S N ,T N ,C N )} represents the dataset of smart contract code generation task, where S i is a natural language instruction, T i It is a safety label, C i is the smart contract code, given a natural language instruction where s i It is the instruction S i The word, label T i ∈{correct,bug}, the goal of the CodeBC model is to i and instruction S i The resulting generation probability P(C i |S i ,T i ) to generate code where c i is a code word, P(C i |S i ,T i ) can be expressed as: Step 2: Code filling: CodeLlama's efficient training method is used to divide the text into three parts: prefix, infix, and suffix. By adjusting the position of each part, the model can make full use of context information when predicting the infix code. PRE, MID, and SUF are used to indicate the beginning of each part. For the i-th example C i =(Pre i ,Mid i ,Suf i )∈D, we minimize the objective Mid i The negative log-likelihood of : Where m t Indicates Mid i , θ represents the model parameters of CodeLlama; Step 3: Vulnerability detection: Enhance the model’s ability to detect security vulnerabilities in smart contracts and provide alignment directions for subsequent label-guided instruction fine-tuning. Given code C i , if C i There is no loophole, and the true label output is otherwise [Tag] and [ / Tag] are used as the start and end marks of the code vulnerability detection results respectively; Step 4: Label-guided instruction fine-tuning: Use labels to help the model better distinguish between secure code and code with security vulnerabilities. Given a manual instruction S i and label information T i The goal of the code generation model is to generate real code C i , where T i ∈{correct,bug} represents code C i Is there a vulnerability? The model optimization goal is to minimize the negative log-likelihood of the target code: where c t C i The tth word, θ D Model parameters of CodeLlama after fine-tuning for multiple tasks.

2. A method for generating a large language model from blockchain smart contract code according to claim 1, characterized in that: In step 1, the overall fine-tuning structure of the CodeBC model consists of two stages. The two tasks in the first stage of multi-task learning fine-tuning are code filling task optimization P(C i ) and vulnerability detection task optimization P(T i |C i ) to increase the generation probability P(C i |S i ,T i ); The second stage of label-guided instruction fine-tuning is based on different T i , namely "correct" and "bug", directly affect P(C i |S i ,T i ) and restrict T in the model inference phase. i =correct to generate more secure contract code.

3. According to claim 1, a method for generating a large language model from blockchain smart contract code is characterized in that: In the step 2, since some smart contracts are short and the contracts usually start with the solidity version number used, completely randomly dividing the contract code into three parts may make the model unable to effectively learn context information. Therefore, when splitting the code, the contract code is evenly divided into five parts according to the number of characters, and the dividing points are randomly selected from the second and fourth parts from left to right to ensure that the data involved in the training has valid context information, and the combination method is randomly selected in PSM and SPM with a probability of 0.

5.

4. According to claim 1, a method for generating a large language model from blockchain smart contract code is characterized in that: In step 3, the model performs the vulnerability detection task under the simple prompt of "Is this smart contract code the correct solution?". The target sentence The negative log-likelihood of is: where t j for The word unit, θ I Model parameters of CodeLlama after fine-tuning for the code completion task.

5. According to claim 1, a method for generating a large language model from blockchain smart contract code is characterized in that: In step 4, a manual instruction S is given. i and restrictive label information T i =correct, the model will be i =correct to generate safer code C p .

Images