Intelligent contract vulnerability detection method based on Bi-LSTM and improved transformer

Abstract

The invention discloses an intelligent contract vulnerability detection method based on Bi-LSTM and an improved transformer, and relates to the technical field of intelligent contract security, and the method comprises the following steps: problem definition: giving a specific definition for a problem to be solved; data processing: obtaining a contract vulnerability detection related code, converting the contract code into a code token sequence, and then converting the code token sequence into a vector; extracting composite semantic features: extracting horizontal features of each row of codes and global features between the rows based on Bi-LSTM (Bidirectional Long Short Term Memory); judging the security of the smart contract: processing semantics in the smart contract by using a Transformer model and obtaining a context association degree, introducing Position Embedding to supplement position information of each token, calculating an attention weight of each element in an input sequence, and outputting a judgment result; according to the method, through combination of Bi-LSTM and Transform, complex semantics and structural features of the smart contract are effectively captured, the accuracy of vulnerability detection is greatly improved, and particularly, the method has a remarkable speed advantage in processing large-scale smart contract data.

Description

Technical Field

[0001] The present invention relates to the field of smart contract security technology, and in particular to a smart contract vulnerability detection method based on Bi-LSTM and improved transformer. Background Art

[0002] Since the birth of Bitcoin in 2008, blockchain technology has developed rapidly. As a distributed ledger technology, its core features include decentralization, immutability and transparency. As an important application of blockchain technology, smart contracts are an automatically executed digital agreement that is widely used in financial services, the Internet of Things, supply chain management and other fields. However, the security issues of smart contracts are becoming increasingly prominent. Once a smart contract is deployed, it cannot be modified and potential vulnerabilities will exist permanently. At the same time, the transparency of smart contracts makes it a target for attackers. In recent years, serious security incidents have occurred many times, causing huge property losses. Therefore, ensuring the security of smart contracts has become a key issue in the current development of blockchain technology.

[0003] Existing smart contract vulnerability detection technologies mainly rely on static analysis methods such as symbolic execution, fuzz testing, and taint analysis. Although these methods have certain effects in detecting known vulnerabilities, they show problems such as low efficiency and high false positive rate in large-scale smart contracts. In addition, with the introduction of deep learning technology, the accuracy of vulnerability detection has improved, but deep learning models have a "black box" problem, which makes it difficult to explain their decision-making process, resulting in the inability to clearly locate vulnerabilities. Therefore, a smart contract vulnerability detection method based on Bi-LSTM and improved transformer is needed to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a smart contract vulnerability detection method based on Bi-LSTM and improved transformer to solve the problems existing in the prior art proposed in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The smart contract vulnerability detection method based on Bi-LSTM and improved transformer includes the following steps:

[0007] S1: Problem definition: Define the input smart contract as C = {C1, C2, ...C t ,...C n}, where C t Represents a smart contract. The source code of a smart contract consists of L = {l1,l2,...l k ,...l n}, where lk represents a line. In order to extract the grammatical features of each line of code, the code line is split to obtain the code tokens used for embedding the model. The model is represented by the objective function f st , the mathematical model is expressed as:

[0008]

[0009] S2: Data processing: Obtain contract vulnerability detection related codes, convert the contract codes into code token sequences, and then convert the code token sequences into vectors;

[0010] S3: Extracting composite semantic features: Extracting horizontal features of each line of code and global features between each line based on Bi-LSTM;

[0011] S4: Determine the security of smart contracts: Use the Transformer model to process the semantics in the smart contract and obtain the context relevance, introduce Positional Embedding to supplement the position information of each token, use the self-attention mechanism to calculate the attention weight of each element in the input sequence, and output the judgment result through the classifier.

[0012] Preferably, the specific steps of obtaining the contract vulnerability detection related code in S2 are:

[0013] (1) Delete irrelevant code such as comments, implementations, and blank lines;

[0014] (2) Process the code line by line: Replace the class name with CL line by line i ,i∈N+, replace global variable names and local variable names with VAR and LV, and replace function names with FUN i ,i∈N+FUN.

[0015] Preferably, the specific steps of converting the code token sequence into a vector in S2 are:

[0016] (1) Add tokens line by line to the code token sequence;

[0017] (2) When combining all tokens into a code token sequence, first add the contract address at the beginning of the code token sequence;

[0018] (3) Add an index to the end of each line before adding it to the code token sequence;

[0019] (4) Finally, the smart contract code is converted into a code token sequence.

[0020] Preferably, the specific steps of extracting the composite semantic features in S3 are:

[0021] Bi-LSTM is composed of LSTM. The input sequence is first processed by LSTM. LSTM processes the sequence as follows:

[0022] f t =σ g (W f x t +U f h t-1 +b f ) (1)

[0023] i t =σ g (W i x t +U i h t-1 +b i ) (2)

[0024] o t =σ g (W o x t +U o h t-1 +b o ) (3)

[0025]

[0026]

[0027] h t =o t ⊙σ h (c t ) (6)

[0028] In formula (1) to formula (4), x t Represents the input vector of the current time step t, which represents the input feature at the current moment; σ g is the activation function sigmoid, which is a vector between 0 and 1 and is used to control the opening and closing states of each gate; i, f, o and c are the input gate, forget gate, output gate and memory gate respectively; in training, w, u and b are used to automatically learn and update the weight matrix and bias parameters respectively; in formula (6), σ h is the activation function tanh, which maps the state of the memory unit to the range of [-1,1] to generate the hidden state; h t is the hidden state of the current time step t, indicating the output at that moment;

[0029] Bi-LSTM consists of two LSTMs in opposite directions, one for forward propagation and the other for backward propagation; each LSTM unit has its own hidden state, and the output is a composite representation; the two composite representations are usually spliced ​​together to form the final output of the current time step; in order to fully extract the contextual semantic information of the smart contract, a DBLSTM model is constructed, including two Bi-LSTMs. The first LSTM is used to extract the semantic features of each line of code, and the second LSTM is used to extract the logical features of the contract code. Finally, the composite semantic features of the contract code are obtained through logical fusion, which is recorded as CCF:

[0030] h i =f1(w1x i +w2h i +1) (7)

[0031] H i =f2(w3xi+w5H i +1) (8)

[0032] Y i =f3(w4hi+w6H i ) (9)

[0033]

[0034]

[0035] Among them, the vector data x of the input contract code = {x t1 ,x t2 ,…x tp},x i As input data, the DBLSTM model is used to obtain the contract code context information features, x i (i=1,2,...,t) represents the input data at the corresponding time, h i (i=1,2,...,t) represents the hidden state of the LSTM in the forward iteration at time t, H i (i=1,2,...,t) represents the LSTM hidden state of the backward iteration, Y i (i=1,2,...,t) represents the corresponding output data, w i (i=1,2,...,6) represents the weight corresponding to each layer, and the calculation formula of the parameter output process of Bi-LSTM is Eq.

[0036] Preferably, the specific steps of determining the security of the smart contract in S4 are:

[0037] After semantic feature extraction, the final feature vector of each node is obtained. At the input layer, the model converts the contract code sequence into a vector representation through Inputs input embedding, and introduces Positional Embedding to supplement the position information of each token;

[0038] The model uses the self-attention mechanism to calculate the attention weight of each element in the input sequence, thereby reflecting the importance of different code snippets in the context. Self-attention generates the attention output attn_output by correlating each token with other tokens, capturing the dependencies and logical connections between code snippets. The core of the self-attention mechanism is dot-product attention, and its calculation steps are as follows:

[0039] Q m =XW q (12)

[0040] K m =XW k (13)

[0041] V m =XW v (14)

[0042]

[0043] MultiHead=concat(head1,…head i ,…head n ) (16)

[0044] MAT=Norm(X+MultiHead×W o ) (17)

[0045] V=Maxpool(MAT) (18)

[0046] Where X is x t The word embedding matrix, W q , W k , W v is the weight matrix to be trained, W k ∈R d×k , Then connect n attention heads in parallel. For each attention head, there is When n head vectors are merged, residual addition, normalization and maximum pooling are performed to finally obtain the word vector representation V;

[0047] Finally, the output vector V of the self-attention layer is further processed by a feedforward neural network and used as the input of the classifier. The classifier determines whether there are vulnerabilities in the smart contract by processing the features extracted by the attention mechanism.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. The present invention effectively captures the complex semantics and structural features of smart contracts through the combination of Bi-LSTM and Transformer, greatly improving the accuracy of vulnerability detection, especially having a significant speed advantage when processing large-scale smart contract data.

[0050] 2. The present invention introduces an explanatory self-attention mechanism, which solves the "black box" problem that is difficult to explain in traditional deep learning models, making vulnerability detection results more transparent, allowing developers to accurately locate vulnerabilities and improving the credibility of detection results.

[0051] 3. The present invention is not only suitable for detecting common smart contract vulnerabilities (such as reentrancy attacks, timestamp dependence, etc.), but also has good scalability and adaptability, and can deal with unknown vulnerabilities that may appear in the future, thereby providing more comprehensive security protection for blockchain applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart of the smart contract vulnerability detection method of the present invention.

[0053] Figure 2 The figure is a schematic diagram of converting the contract code into a code token sequence according to the present invention.

[0054] Figure 3 This is a schematic diagram of converting a label sequence into a vector in the present invention.

[0055] Figure 4 A schematic diagram of extracting the semantics and global features of a contract according to the present invention.

[0056] Figure 5 This is the structural diagram of the LSTM model of the present invention.

[0057] Figure 6 This is the improved Transformer flow chart of the present invention. DETAILED DESCRIPTION

[0058] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0059] The BLTA-Learning model proposed in the present invention integrates Bi-LSTM, Transformer and self-attention network, which can effectively extract the semantic information of smart contracts and improve the interpretability of the model. Due to the determinism of the execution of smart contract vulnerability code and the horizontal and vertical correlation of contract code, the advantage of Bi-LSTM is that it can effectively extract text features, capture global context and long-term dependencies, so that it has a strong advantage in multi-directional semantic acquisition. Compared with other models, Bi-Transformer has parallel computing capabilities and flexible hierarchical structures, and can increase the self-attention layer to improve the interpretability of the model. Compared with other smart contract detections, the solution to achieve the best detection effect is the fusion of semantic structure and semantic features. Semantic structure features generally generate structure graphs for learning, which often causes the loss of semantic structure information. In order to solve this problem, the present invention obtains the semantic structure features of the contract through Transformer, avoiding the loss of semantic structure information.

[0060] See also Figure 1-6 , the present invention provides the following technical solutions:

[0061] The smart contract vulnerability detection method based on Bi-LSTM and improved transformer includes the following steps:

[0062] S1: Problem definition: The purpose of this invention is to extract the time and structure characteristics of smart contracts and detect their vulnerabilities. A specific definition of the problem to be solved by this invention is given. The data flow of the modeling of a specific vulnerability prediction problem is represented as follows: The input smart contract is defined as C = {C1, C2, ... C t ,...C n}, where C t Represents a smart contract. The source code of a smart contract consists of L = {l1,l2,...l k ,...l n}, where lk represents a line. In order to extract the grammatical features of each line of code, the code line is split to obtain the code tokens used for embedding the model. The model is represented by the objective function f st , the mathematical model is expressed as:

[0063]

[0064] S2: Data processing: Obtain the contract vulnerability detection related code, convert the contract code into a code token sequence, and then convert the code token sequence into a vector; the specific steps for obtaining the contract vulnerability detection related code are:

[0065] (1) Delete irrelevant code such as comments, implementations, and blank lines;

[0066] (2) Process the code line by line: Replace the class name with CL line by line i ,i∈N+, replace global variable names and local variable names with VAR and LV, and replace function names with FUN i ,i∈N+FUN.

[0067] like Figure 3 As shown in the figure, the specific steps to convert the code token sequence into a vector are:

[0068] (1) Add tokens line by line to the code token sequence;

[0069] (2) When combining all tokens into a code token sequence, first add the contract address at the beginning of the code token sequence;

[0070] (3) Add an index to the end of each line before adding it to the code token sequence;

[0071] (4) Finally, the smart contract code is converted into a code token sequence.

[0072] The input of the deep learning model requires a vector of fixed dimension, so the code token sequence needs to be converted into a vector through the embedding technology; work2dec is a commonly used embedding technology, which can capture the semantic relationship between words and infer the meaning of words through the distance between words in the vector space. It can also reduce the dimension of the feature space and improve the training efficiency of the model. The present invention constructs a data set by processing the code token sequence with work2dec.

[0073] S3: The logic of the contract code is complex and cannot be changed once deployed. It also involves financial security issues. When detecting vulnerabilities, more comprehensive contextual information is needed to improve detection efficiency. Compared with Bi-LSTM, which is suitable for short-range dependency feature extraction, Transformer introduces a self-attention mechanism, which is more suitable for long-range dependency feature extraction and can better obtain contextual information. In order to fully extract semantic and grammatical logic features, the model is used to extract horizontal features of each line of code and global features between each line based on Bi-LSTM. The steps of horizontal and vertical feature extraction are as follows: extract semantic horizontal features of each line of smart contract code through Bi-LSTM; extract semantic horizontal features of each line of code from the output of Bi-LSTM; the detailed workflow of the model is as follows: Figure 4 shown.

[0074] Semantic features are one of the main features of a contract. Currently, most researchers mainly extract semantic features from two different levels of representation: bytecode and contract token sequence. Contract token sequence is a high-level representation of contract code, which is closer to the structure and semantics of the code. Therefore, the mainstream research methods for smart contract vulnerability detection mostly start from the token sequence of the contract. For processing time series data of contract tokens, recurrent neural networks have inherent advantages. Compared with RNN and LSTM, Bi-LSTM can obtain forward and backward semantic information and has better performance. Therefore, the present invention uses Bi-LSTM to extract the semantic features of the contract, and uses its bidirectional context understanding characteristics to fully obtain the semantic information of the contract code.

[0075] The specific steps for extracting composite semantic features are:

[0076] Bi-LSTM is composed of LSTM. The input sequence is first processed by LSTM. LSTM processes the sequence as follows:

[0077] f t =σ g (W f x t +U f h t-1 +b f ) (1)

[0078] i t =σ g (W i x t +U i h t-1 +b i ) (2)

[0079] o t =σ g (W o x t +U o h t-1 +b o ) (3)

[0080]

[0081]

[0082] h t =o t ⊙σ h (c t ) (6)

[0083] In formula (1) to formula (4), x tRepresents the input vector of the current time step t, which represents the input feature at the current moment; σ g is the activation function sigmoid, which is a vector between 0 and 1 and is used to control the opening and closing states of each gate; i, f, o and c are the input gate, forget gate, output gate and memory gate respectively; in training, w, u and b are used to automatically learn and update the weight matrix and bias parameters respectively; in formula (6), σ h is the activation function tanh, which maps the state of the memory unit to the range of [-1,1] to generate the hidden state; h t is the hidden state of the current time step t, indicating the output at that moment;

[0084] Bi-LSTM consists of two LSTMs in opposite directions, one for forward propagation and the other for backward propagation; each LSTM unit has its own hidden state, and the output is a composite representation; the two composite representations are usually spliced ​​together to form the final output of the current time step; in order to fully extract the contextual semantic information of the smart contract, a DBLSTM model is constructed, including two Bi-LSTMs. The first LSTM is used to extract the semantic features of each line of code, and the second LSTM is used to extract the logical features of the contract code. Finally, the composite semantic features of the contract code are obtained through logical fusion, which is recorded as CCF:

[0085] h i =f1(w1x i +w2h i +1) (7)

[0086] H i =f2(w3xi+w5H i +1) (8)

[0087] Y i =f3(w4hi+w6H i ) (9)

[0088]

[0089]

[0090] Among them, the vector data x of the input contract code is t1 ,x t2 ,…x tp},x i As input data, the DBLSTM model is used to obtain the contract code context information features, x i (i=1,2,...,t) represents the input data at the corresponding time, h i (i=1,2,...,t) represents the hidden state of the LSTM in the forward iteration at time t, Hi (i=1,2,...,t) represents the LSTM hidden state of the backward iteration, Y i (i=1,2,...,t) represents the corresponding output data, w i (i=1,2,...,6) represents the weight corresponding to each layer, and the calculation formula of the parameter output process of Bi-LSTM is Eq.

[0091] S4: Determine the security of smart contracts: Use the Transformer model to process the semantics in the smart contract and obtain the context relevance, introduce Positional Embedding to supplement the position information of each token, use the self-attention mechanism to calculate the attention weight of each element in the input sequence, and output the judgment result through the classifier.

[0092] The specific steps for determining the security of the smart contract in S4 are:

[0093] After semantic feature extraction, the final feature vector of each node is obtained. The Transformer model is a deep learning architecture based on the self-attention mechanism, which is particularly suitable for processing complex semantics and logic in smart contracts. At the input layer, the model converts the contract code sequence into a vector representation through Inputs input embedding, and introduces Positional Embedding to supplement the position information of each token; this position information helps the model capture the dependencies between code snippets and better understand the overall logical structure of the contract.

[0094] The model uses the self-attention mechanism to calculate the attention weight of each element in the input sequence, thereby reflecting the importance of different code snippets in the context. Self-attention generates an attention output attn_output by correlating each token with other tokens, capturing the dependencies and logical connections between code snippets. Since the query vector (Q), key value (K), and value (V) in self-attention all come from the same input sequence, it is called "internal attention". This is different from the traditional attention mechanism, in which the query vector usually comes from external input. Through this adaptive weight adjustment, the self-attention mechanism can learn long-distance dependencies more effectively. The core of the self-attention mechanism is dot-product attention, and its calculation steps are as follows:

[0095] Q m =XW q (12)

[0096] K m =XW k (13)

[0097] Vm =XW v (14)

[0098]

[0099] MultiHead=concat(head1,…head i ,…head n ) (16)

[0100] MAT=Norm(X+MultiHead×W o ) (17)

[0101] V=Maxpool(MAT) (18)

[0102] Where X is x t The word embedding matrix, W q , W k , W v is the weight matrix to be trained, W k ∈R d×k , Then connect n attention heads in parallel. For each attention head, there is When n head vectors are merged, residual addition, normalization and maximum pooling are performed to finally obtain the word vector representation V;

[0103] The output vector V of the self-attention layer is further processed by the feedforward neural network and used as the input of the classifier. The classifier processes the features extracted by the attention mechanism to determine whether there are vulnerabilities in the smart contract. 1 indicates a vulnerability and 0 indicates no vulnerability. The stability and effectiveness of the model during the calculation process are ensured through the Add & Layer Norm steps. Ultimately, the results output by the classifier can effectively identify the security of the contract and provide a high degree of interpretability for vulnerability detection in smart contracts.

[0104] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A smart contract vulnerability detection method based on Bi-LSTM and improved transformer, characterized in that: The following steps are involved: S1: Problem definition: Define the input smart contract as C = {C1, C2, ...C t ,...C n }, where C t Represents a smart contract. The source code of a smart contract consists of L = {l1,l2,...l k ,...l n }, where lk represents a line. In order to extract the grammatical features of each line of code, the code line is split to obtain the code tokens used for embedding the model. The model is represented by the objective function f st , the mathematical model is expressed as: S2: Data processing: Obtain contract vulnerability detection related codes, convert the contract codes into code token sequences, and then convert the code token sequences into vectors; S3: Extracting composite semantic features: Extracting horizontal features of each line of code and global features between each line based on Bi-LSTM; S4: Determine the security of smart contracts: Use the Transformer model to process the semantics in the smart contract and obtain the context relevance, introduce Positional Embedding to supplement the position information of each token, use the self-attention mechanism to calculate the attention weight of each element in the input sequence, and output the judgment result through the classifier.

2. The smart contract vulnerability detection method based on Bi-LSTM and improved transformer according to claim 1 is characterized in that: The specific steps for obtaining the contract vulnerability detection related code in S2 are: (1) Delete irrelevant code such as comments, implementations, and blank lines; (2) Process the code line by line: Replace the class name with CL line by line i ,i∈N+, replace global variable names and local variable names with VAR and LV, and replace function names with FUN i ,i∈N+FUN.

3. The smart contract vulnerability detection method based on Bi-LSTM and improved transformer according to claim 1 is characterized in that: The specific steps of converting the code token sequence into a vector in S2 are: (1) Add tokens line by line to the code token sequence; (2) When combining all tokens into a code token sequence, first add the contract address at the beginning of the code token sequence; (3) Add an index to the end of each line before adding it to the code token sequence; (4) Finally, the smart contract code is converted into a code token sequence.

4. The smart contract vulnerability detection method based on Bi-LSTM and improved transformer according to claim 1, characterized in that: The specific steps of extracting composite semantic features in S3 are: Bi-LSTM is composed of LSTM. The input sequence is first processed by LSTM. LSTM processes the sequence as follows: f t =σ g (W f x t +U f h t-1 +b f ) (1) i t =σ g (W i x t +U i h t-1 +b i ) (2) the t =s g (W o x t +U o h t-1 +b o ) (3) h t =o t ⊙σ h (c t ) (6) In formula (1) to formula (4), x t Represents the input vector of the current time step t, which represents the input feature at the current moment; σ g is the activation function sigmoid, which is a vector between 0 and 1 and is used to control the opening and closing states of each gate; i, f, o and c are the input gate, forget gate, output gate and memory gate respectively; in training, w, u and b are used to automatically learn and update the weight matrix and bias parameters respectively; in formula (6), σ h is the activation function tanh, which maps the state of the memory unit to the range of [-1,1] to generate the hidden state; h t is the hidden state of the current time step t, indicating the output at that moment; Bi-LSTM consists of two LSTMs in opposite directions, one for forward propagation and the other for backward propagation; each LSTM unit has its own hidden state, and the output is a composite representation; the two composite representations are usually spliced ​​together to form the final output of the current time step; in order to fully extract the contextual semantic information of the smart contract, a DBLSTM model is constructed, including two Bi-LSTMs. The first LSTM is used to extract the semantic features of each line of code, and the second LSTM is used to extract the logical features of the contract code. Finally, the composite semantic features of the contract code are obtained through logical fusion, which is recorded as CCF: h i =f1(w1x i +w2h i +1) (7) <h2 style=";text-align:left;direction:ltr">H<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> =f2(w3xi+w5H<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> +1) (8) <h2 style=";text-align:left;direction:ltr">Y<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> =f3(w4hi+w6H<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> ) (9) Among them, the vector data x of the input contract code = {x t1 ,x t2 ,…x tp },x i As input data, the DBLSTM model is used to obtain the contract code context information features, x i (i=1,2,...,t) represents the input data at the corresponding time, h i (i=1,2,...,t) represents the hidden state of the LSTM in the forward iteration at time t, H i (i=1,2,...,t) represents the LSTM hidden state of the backward iteration, Y i (i=1,2,...,t) represents the corresponding output data, w i (i=1,2,...,6) represents the weight corresponding to each layer, and the calculation formula of the parameter output process of Bi-LSTM is Eq.

5. The smart contract vulnerability detection method based on Bi-LSTM and improved transformer according to claim 1 is characterized in that: The specific steps for determining the security of the smart contract in S4 are: After semantic feature extraction, the final feature vector of each node is obtained. At the input layer, the model converts the contract code sequence into a vector representation through Inputs input embedding, and introduces Positional Embedding to supplement the position information of each token; The model uses the self-attention mechanism to calculate the attention weight of each element in the input sequence, thereby reflecting the importance of different code snippets in the context. Self-attention generates the attention output attn_output by correlating each token with other tokens, capturing the dependencies and logical connections between code snippets. The core of the self-attention mechanism is dot-product attention, and its calculation steps are as follows: Q m =XW q (12) K m =XW k (13) V m =XW v (14) MultiHead=concat(head1,…head i ,…head n ) (16) MAT=Norm(X+MultiHead×W o ) (17) V=Maxpool(MAT) (18) Where X is x t The word embedding matrix, W q , W k , W v is the weight matrix to be trained, W k ∈R d ×k , Then connect n attention heads in parallel. For each attention head, we have When n head vectors are merged, residual addition, normalization and maximum pooling are performed to finally obtain the word vector representation V; Finally, the output vector V of the self-attention layer is further processed by a feedforward neural network and used as the input of the classifier. The classifier determines whether there are vulnerabilities in the smart contract by processing the features extracted by the attention mechanism.

Images