A code semantic clone detection method based on RvNN+Transformer neural network model

Abstract

The present invention discloses a code semantic clone detection method based on RvNN+Transformer neural network model. Given two code snippets, their abstract syntax trees are first split to the statement level, and then the tree structure information of each statement level is obtained through RvNN, and then the order information of each statement tree is obtained through Transformer with directional information, and then the final code representation vector is generated through the maximum pooling layer, and the absolute value of the difference between the two code representation vectors is compared to determine whether the two codes are semantic clones. The main idea of ​​the present invention is to use Transformer-encoder with directional information to improve the current model. Most of the existing research still uses LSTM or GRU, but it is not suitable for parallel computing and cannot pay attention to global information well. Experiments show that compared with the prior art, the present invention has a higher F1 value and consumes less time.

Description

Technical Field

[0001] The present invention relates to the field of computer code cloning, and in particular to a code semantic cloning detection method based on an RvNN+Transformer neural network model. Background Art

[0002] Code cloning refers to two or more identical or similar source code fragments in a code base. Issues related to code cloning are important research topics in the field of software engineering. Code cloning is a common phenomenon in software development. Although it can improve development efficiency, code cloning can also have a negative impact on the system, including reducing software stability, causing code base redundancy, and propagating software defects.

[0003] Code clones are divided into four types. Type 1 is a pair of code where two code snippets are exactly the same except for spaces and comments; Type 2 is a pair of code where everything is the same except for variable names, type names, and function names; Type 3 is a pair of code that is similar but has some statements added or deleted, or uses different identifiers, text, types, spaces, layouts, and comments; Type 4 is heterogeneous code with the same function, which is not similar in text or syntax, but has semantic similarities. The detection difficulty of types 1-4 increases step by step. There are still few tools for detecting code semantic clones of type 4, so the present invention focuses on detecting code semantic clones.

[0004] In recent years, many code clone detection tools based on deep learning have emerged. However, most of them train code snippets through LSTM or GRU. Most studies have shown that these models are suitable for training sequence input. However, due to the sequence input characteristics of the model, it cannot make good use of the parallel computing capabilities of the GPU, and the problem of gradient disappearance will occur for sequences that are too long. The Transformer model can input the entire sequence at once and combine the position encoding of the sequence, so that the parallel computing capabilities of the GPU can be well utilized and the problem of gradient disappearance can be effectively reduced. The Transformer model includes Encoder and Decoder modules. The Encoder module is generally used to generate an intermediate vector representation of the original data, and the Decoder module is generally used to convert the intermediate vector into data related to the original data, such as Chinese-English translation. Since the present invention is aimed at the vector representation of the training code, the Transformer-encoder module is used to train the model. The Transformer-encoder module can increase the training speed of the model, and the fusion of RvNN and Transformer-encoder can also enable the model to have better results when detecting code semantic clones. Summary of the invention

[0005] The present invention aims to improve the accuracy and performance of code semantic clone detection, and proposes a code semantic clone detection method based on RvNN+Transformer neural network model. The method first converts the code into an abstract syntax tree representation, then splits the abstract syntax tree at the statement level, inputs the split statement tree from bottom to top into the RvNN model to train the corresponding statement tree vector, and then simultaneously inputs multiple statement tree vectors into the Transformer-encoder module with direction information to generate a new statement tree vector, and finally outputs the vector representation of the code through the maximum pooling layer. Whether code semantic clones are constituted is detected by comparing the distance between the vector representations of two codes.

[0006] The present invention provides a code semantic clone detection method based on the RvNN+Transformer neural network model, which specifically includes the following process:

[0007] Step 1: Preprocess the code. If a complete abstract syntax tree is directly input into the model, the gradient may disappear. Therefore, it needs to be split first. The abstract syntax tree can be split into a set of statement trees. For nested statements, the code in the curly braces can be split out, such as if, for, while. Then each statement tree is stored locally in order. The statement tree is stored in the form of a multi-level array. The value of the first level is the parent node of the second level, and so on.

[0008] Step 2: Pre-training of the code. Since the deep learning model must pass in vectors, each token of the code needs to be converted into a vector first. Use the Skip-Gram model of Word2vec to train all the tokens of the code in the dataset and generate the word vector corresponding to each token. The dimension d of the word vector is 128.

[0009] Step 3: Build a neural network model. The model consists of the RvNN sentence tree encoding layer, the Transformer-encoder layer, and the maximum pooling layer. The following is a detailed introduction to each module:

[0010] In order to obtain the structural information on the sentence tree through training, the sentence tree needs to be input into the RvNN model from bottom to top. The specific conversion process is as follows:

[0011] Perform a depth-first traversal of a code statement tree and use the trained Skip-Gram model to convert each node value into a d-dimensional vector V i Then, the vector formula (1) corresponding to each node in the statement tree is updated so that it contains the child node information.

[0012]

[0013] Among them, v n represents the vector corresponding to node n, is the weight matrix, k is the encoding dimension, C is the number of child nodes corresponding to node n, and b n is the deviation, h i is the updated vector of child node i, σ is the activation function, and this paper uses the identity function.

[0014] In order to obtain the most important features in the node vector, after completing the vector update of each node, use formula (2) to update all the node vectors h of the statement tree. i Perform maximum pooling to obtain the final vector representation of the sentence tree.

[0015] e t =[max(h i1 ),…,max(h ik )],i=1,…,N (2)

[0016] Where N is the number of nodes in the statement tree.

[0017] Finally, a complete abstract syntax tree of the code is represented as [e1,…,e i ,…,e t ], i∈[1,t]. Where t represents the number of statement trees.

[0018] Since the order information of the code is also important, the Transformer-encoder model is needed to train the order information between the various statement trees of the code. The Transformer-encoder model needs to use the attention calculation score formula with relative position information, the formula is as follows:

[0019]

[0020]

[0021]

[0022] Att(Q,K,V)=softmax(A rel )V (6)

[0023] Where H is the sum of the input sentence tree vector e and the positional encoding, with a dimension of d k , Q, V are obtained by mapping the vector H through the corresponding matrix W. K is used to enhance the information of the input vector H, so no matrix mapping is used. The input vector t is the target token index, j is the context token index, represents the similarity between the i-th token and the j-th token, u and v are learnable parameters, R t-jIt is the relative position encoding. The A matrix is ​​normalized by softmax and then multiplied by V to get the final representation.

[0024] The sentence vector transformed by the Transformer-encoder layer is then subjected to maximum pooling to obtain the final code representation r. The formula is as follows:

[0025] r=[max(V1),…,max(V i )],i=1,…,N (7)

[0026] Where N represents the total number of nodes corresponding to a statement tree.

[0027] Step 4: Design a Siamese architecture for code semantic clone detection, which includes: converting the given two codes into two sets of statement trees as input, and then passing each statement tree into the RvNN model from bottom to top. Then, the model designed in step 3 outputs the final vector representation, and calculates the vector distance between the two codes to identify whether the two codes are cloned.

[0028] Compared with the prior art, the present invention has the following characteristics:

[0029] 1. The Transformer-encoder module can make the model training faster.

[0030] 2. The fusion of RvNN and Transformer-encoder can make the model have higher detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is an overall flow chart of a code semantic clone detection method based on the RvNN+Transformer neural network model of the present invention;

[0032] Figure 2 This is an example diagram of a multi-level array storage representation for a code generation statement tree;

[0033] Figure 3 It is the overall architecture diagram of the model of the present invention. DETAILED DESCRIPTION

[0034] The present invention proposes a code semantic clone detection method based on the RvNN+Transformer neural network model, which can well identify whether two codes are semantic clones.

[0035] The code semantic clone detection method based on the RvNN+Transformer neural network model proposed by the present invention is described in detail below in conjunction with specific implementation. Figure 1As shown in FIG. 1 , it is an overall flow chart of a code semantic clone detection method based on the RvNN+Transformer neural network model of the present invention. The flow specifically includes the following processing:

[0036] Step 1: Preprocess the code. The code is parsed into an abstract syntax tree using the pyparser tool, and then split into a set of statement trees. For nested statements, the code in the curly braces can be split out, such as if, for, while. Each statement tree is then stored locally in order.

[0037] like Figure 2 As shown in the figure, a code is split into multiple statement trees through statement levels. Each statement tree stores node values ​​by level. The sample code generates the following array:

[0038] [['FuncDef',['Decl',['FuncDecl',['main',['int']]]]],['Compound'],['Decl',['n',['int']]],['Decl',['a',['int' ]],['0']],['Decl',['b',['int']],['0']],['Decl',['c',['int']],['0']],['Decl',['t',['int']]],['FuncCall',['sca nf'],['ExprList',['"%d"'],['&',['n']]]],['For',['>',['n'],['0']]],['Compound'],['=',['t'],['%',['n'],['10'] ]],['FuncCall',['printf'],['ExprList',['"%d"'],['t']]],[' / =',['n'],['10']],['End'],['Return',['0']],['End']]

[0039] This array is composed of arrays formed by multiple statement trees.

[0040] For example, the first array ['FuncDef',['Decl',['FuncDecl',['main',['int']]]]] represents the first statement tree. Its root node is FuncDef (method definition), its child node is Decl (declaration), its child node is FuncDecl (method declaration), and its child nodes are main (method name) and int (type).

[0041] Step 2, pre-training of code. The data set used in the present invention is OJClone, which is a large C / C++ program data set consisting of 104 programming tasks, each with 500 source codes. Because the purpose of the present invention is to test code semantic cloning, different source codes for the same task are regarded as code semantic clones. The original data set is too large, so only the first 15 programming tasks are selected from OJ, each with 500 source codes. Even so, it will generate more than 28 million clone pairs, which is very time-consuming, so 50,000 samples were randomly selected. For the data set, it is randomly split into three parts, including a training set, a validation set, and a test set, with a ratio of 3:1:1. The Skip-Gram model of Word2vec is used to train all tokens of the code in the data set to generate a word vector corresponding to each token. The dimension d of the word vector is 128.

[0042] With the index corresponding to each word vector, we can convert the example code into the index corresponding to the word, as follows:

[0043] [[30,[1,[29,[40,[2]]]]],[6],[1,[13,[2]]],[1,[12,[2]],[5]],[1,[24,[2]],[5]],[1,[33,[2]],[5]],[1,[43,[2]]],[9,

[28] ,[10,

[32] ,[26,

[13] ]]],[15,[56,

[13] ,[5]]],[6],[4,

[43] ,[80,

[13] ,

[107] ]],[9,

[27] ,[10,

[32] ,

[43] ]],[1656,

[13] ,

[107] ],[7],[35,[5]],[7]]

[0044] For example, the name of the word vector corresponding to 30 is FuncDef.

[0045] Step 3: Establish a neural network model and conduct training. Figure 3 As shown in the figure, this model consists of an RvNN sentence tree encoding layer, a Transformer-encoder layer, and a maximum pooling layer. Each module is introduced in detail below.

[0046] In order to obtain the structural information on the sentence tree through training, the sentence tree needs to be input into the RvNN model from bottom to top. Taking the first sentence tree array [30,[1,[29,[40,[2]]]]] in the sample code as an example, each index will be converted into a vector according to the word vector table generated by Word2vec and then input into the model. First, the vector corresponding to index 2 is input to generate the hidden state h1, and then the hidden state generated by the vector corresponding to 40 is input and added to h1 to get h2, so that the parent node 40 can contain the information of node 2.

[0047] In order to obtain the most important features in the node vector, after completing the vector update of each node, use formula (8) to update all the node vectors h of the statement tree. i Perform maximum pooling to obtain the final vector representation of the sentence tree.

[0048] e t =[max(h i1 ),…,max(h ik )],i=1,…,N (8)

[0049] Where N is the number of nodes in the statement tree.

[0050] Finally, a complete abstract syntax tree of the code is represented as [e1,…,e i ,…,e t ], i∈[1,t]. Where t represents the number of statement trees.

[0051] Since the order information of the code is also important, the Transformer-encoder model is needed to train the order information between the various statement trees of the code. The Transformer-encoder model needs to use the attention calculation score formula with relative position information, the formula is as follows:

[0052]

[0053]

[0054]

[0055] Att(Q,K,V)=sof tmax(A rel )V (12)

[0056] Where H is the sum of the input sentence tree vector e and the positional encoding, with a dimension of d k , Q, V are obtained by mapping the vector H through the corresponding matrix W. K is used to enhance the information of the input vector H, so no matrix mapping is used. The input vector t is the target token index, j is the context token index, and A i,jrepresents the similarity between the i-th token and the j-th token, u and v are learnable parameters, R t-j It is the relative position encoding. The A matrix is ​​normalized by softmax and then multiplied by V to get the final representation.

[0057] The sentence vector transformed by the Transformer-encoder layer is then subjected to maximum pooling to obtain the final code representation r. The formula is as follows:

[0058] r=[max(V1),…,max(V i )],i=1,…,N (13)

[0059] Where N represents the total number of nodes corresponding to a statement tree.

[0060] The training parameters are as follows:

[0061] The Transformer head is set to 4,d k The number of layers is set to 128, the dimension of the feedforward neural network is 256, and the dropout is set to 0.5. The batch size of the training is set to 32, and the epoch is set to 10. The present invention uses the label smoothing formula:

[0062]

[0063] Where ε=0.05.

[0064] The distance between two codes is calculated by r = |r1-r2|. The model outputs The loss function used is binary cross entropy, and the formula is as follows:

[0065]

[0066] Where θ represents the parameters of all weight matrices in the model and y is the true label.

[0067] Finally, the present invention uses AdaMax optimizer to train the model.

[0068] Step 4: Model training results

[0069] Since code clone detection can be regarded as a binary classification problem, the commonly used precision (P), recall (R) and F1 metric (F1) are selected as evaluation indicators. The experimental results of the present invention are compared with the current better ASTNN model. The experimental results are shown in Table 1:

[0070] Table 1 Results on OJClone

[0071]

[0072] Compared with the ASTNN method, the p-value of the invention is lower, but the R-value and F1-value are higher, so the invention has a better effect in detecting code semantic clones. And because the Transformer-encoder utilizes the parallel computing capability of the GPU, the speed of training the model is also faster than that of ASTNN.

Claims

1. A code semantic clone detection method based on RvNN+Transformer neural network model, characterized in that: The method specifically includes the following processes: Step 1: Code preprocessing; If a complete abstract syntax tree is directly input into the model, the gradient may disappear. Therefore, it is necessary to split the abstract syntax tree into a set of statement trees. For statements with nested structures, the code in the curly braces is split out. Then each statement tree is stored locally in order. The statement tree is stored in the form of a multi-level array, and the value of the first level is the parent node of the second level. Step 2: Pre-training of the code. Since the deep learning model must pass in vectors, each token of the code needs to be converted into a vector first. Use the Skip-Gram model of Word2vec to train all the tokens of the code in the dataset and generate the word vector corresponding to each token. The dimension d of the word vector is 128. Step 3: Establish a neural network model, which consists of an RvNN sentence tree encoding layer, a Transformer-encoder layer, and a maximum pooling layer; In order to obtain the structural information on the sentence tree through training, the sentence tree needs to be input into the RvNN model from bottom to top. The specific transformation process is as follows: Perform a depth-first traversal of a code statement tree and use the trained Skip-Gram model to convert each node value into a d-dimensional vector V i ; Then update the vector formula (1) corresponding to each node in the statement tree so that it contains the child node information; Among them, v n represents the vector corresponding to node n, is the weight matrix, k is the encoding dimension, C is the number of child nodes corresponding to node n, and b n is the deviation, h i is the updated vector of child node i, σ is the activation function; In order to obtain the most important features in the node vector, after completing the vector update of each node, use formula (2) to update the vector h of all child nodes i of the statement tree i Perform maximum pooling to obtain the final vector representation of the sentence tree; e t =[max(h i1 ),…,max(h ik )],i=1,…,N (2) Where N is the number of nodes in the statement tree; Finally, a complete abstract syntax tree of the code is represented as [e1,…,e i ,…,e t ], i∈[1,t]; where t represents the number of statement trees; Since the order information of the code is also important, the Transformer-encoder model is needed to train the order information between the various statement trees of the code; the Transformer-encoder model needs to use the attention calculation score formula with relative position information, the formula is as follows: That(Q,K,V)=softmax(A rel )V (6) Where H is the sum of the input sentence tree vector e and the positional encoding, with a dimension of d k , Q, V are obtained by mapping the vector H through the corresponding matrix W; K is used to enhance the information of the input vector H, so no matrix mapping is used; the input vector t is the target token index, j is the context token index, represents the similarity between the i-th token and the j-th token, u and v are learnable parameters, R t-j It is the relative position encoding; the A matrix is ​​normalized by softmax and then multiplied by V to get the final representation; The sentence vector transformed by the Transformer-encoder layer is then max-pooled to obtain the final code representation r; the formula is as follows: r=[max(V1),…,max(V i )],i=1,…,N (7) Among them, N represents the total number of nodes corresponding to a statement tree; Step 4: Design a Siamese architecture for code semantic clone detection, which specifically includes: converting the given two codes into two sets of statement trees as input, and then passing each statement tree into the RvNN model from bottom to top; then outputting the final vector representation through the model designed in step 3, and calculating the distance between the two code vectors to determine whether the two codes are cloned.

Images