Code clone detection method and device based on multistage prediction and medium

Abstract

The invention discloses a code clone detection method and device based on multistage prediction and a medium, and the method comprises the steps: firstly, carrying out the AST analysis of a code snippet, generating an AST node coding sequence through traversal, converting a code text into a Token sequence, and carrying out the recognition of the Token sequence; according to the method, feature vectors are extracted from an AST node sequence and a Token sequence through a deep learning model of a similar structure, a node type statistical vector is obtained based on an AST coding sequence, and finally, multi-level prediction is carried out: similarity calculation is carried out on codes in sequence according to obtained abstract syntax tree features, text features and node type features of the codes, and the codes are obtained. The code pair is finally determined to be a clone pair only when all levels of predictions confirm that the clone exists. According to the method, the code characteristics are analyzed from multiple dimensions through a multi-level prediction mechanism, so that the accuracy of cloning detection is improved, and an efficient and reliable method is provided for code cloning detection.

Description

Technical Field

[0001] The present invention relates to the technical field of code clone detection, and in particular to a code clone detection method, device and medium based on multi-level prediction. Background Art

[0002] In the modern software development process, code clone detection is a crucial task. With the rapid development of the software industry and the sharp increase in private code libraries and open-source software projects, a large number of code segments have been accumulated. These code segments contain rich function implementations, and the effective reuse of existing code segments has become the key to accelerating the software development process. However, how to find code segments that meet the needs of developers from a huge code library and accelerate software development through reuse and modification is a problem that the code recommendation system needs to solve. At the same time, how to identify the clone relationships in the code, improve the maintenance efficiency of the reused code, and reduce the maintenance cost are the main application scenarios of code clone detection.

[0003] The key to code clone detection lies in how to accurately represent and compare code segments. At present, code representation methods mainly include several ways such as symbol-based, abstract syntax tree (AST)-based, and graph structure-based. The symbol-based method represents the code by decomposing it into a symbol sequence. This method is simple and intuitive, but often ignores the semantic information of the code. The graph structure-based methods, such as control flow graphs or data dependence graphs, can better represent the execution flow and data relationships of the code, but have deficiencies in expressing the structural order and text information of the code. In contrast, the AST-based method has received extensive attention because it can capture both the structure and semantic information of the code. The abstract syntax tree is a tree-like representation of the source code, where each node represents a construct in the code, such as an expression, a statement, etc. By analyzing the structure and node types of the AST, the logic and organization of the code can be deeply understood. However, there are also some challenges in directly using the AST for code clone detection. First, the structure of the AST is complex, and direct processing requires a large amount of computing resources. Second, the original AST contains a large amount of detailed information, which may interfere with clone detection.

[0004] To overcome these challenges, researchers have proposed a variety of AST-based code representation and processing methods. For example, by refining the AST to simplify its structure and extract key information; or converting the AST into an encoded sequence for easy similarity calculation. These methods have improved the efficiency and accuracy of code clone detection to a certain extent, but there is still room for improvement. In addition, with the development of deep learning technology, researchers have begun to explore using deep learning models to automatically learn feature representations from code. By training deep learning models, useful features can be extracted from the text or structure of the code for code clone detection, and this method is expected to further improve the accuracy and efficiency of detection. Summary of the Invention

[0005] Object of the Invention: The present invention provides a method, device and medium for code clone detection assisted by multi-level deep learning, which can effectively extract code features from different levels and improve the accuracy and efficiency of code clone detection.

[0006] Technical Solution: A method for code clone detection assisted by multi-level deep learning according to the present invention specifically includes the following steps:

[0007] Step 1: Convert the code into an Abstract Syntax Tree (AST) and encode it.

[0008] Step 2: Convert the source code text into a Token sequence, and use the word embedding method to convert the tokens into numerical word vectors, representing the text information of the code with a digital vector sequence.

[0009] Step 3: Based on deep learning, perform semantic feature extraction on the AST encoding sequence and the Token encoding sequence respectively, extract the functional semantic information in the code abstract syntax tree and the text, and generate functional semantic prediction vectors that can reflect the information of the code abstract syntax tree and the text respectively.

[0010] Step 4: Extract node type information from the AST encoding sequence to generate a node type statistical vector.

[0011] Step 5: Multi-stage filtering for clone detection: Based on the AST functional filtering stage and the source text functional filtering stage, determine candidate pairs; then determine clone pairs through the node type metric filtering stage.

[0012] Further, the implementation process of the above Step 1 is as follows:

[0013] First, convert the source code into an Abstract Syntax Tree (AST) to capture the structural information of the code; then refine the original AST to remove redundant information, traverse the refined AST, extract and encode the node path information, and generate an AST encoding sequence that can represent the structural features of the code.

[0014] Further, the implementation process of the above Step 3 is as follows:

[0015] Input the AST encoding sequence obtained in Step 1 into a bidirectional long short-term memory network, and use the code classification task as the downstream target; learn the information of the code abstract syntax tree contained in the AST encoding sequence to generate a functional prediction vector AstFP; this functional prediction vector represents the probability that the code snippet is classified into different functions, reflecting the structural features of the code abstract syntax tree.

[0016] Input the Token encoding sequence obtained in step 2 into a bidirectional long short-term memory network, and use the code classification task as the downstream target; learn the source code text information contained in the Token encoding sequence to generate a functional prediction vector SrcFP; this functional prediction vector represents the probability that a code snippet is classified into different functions and reflects the source text features of the code.

[0017] Further, the node type statistical vector in step 4 is obtained by counting the occurrence frequency of each node type in the code snippet from the AST encoding sequence.

[0018] Further, the AST functional filtering stage in step 5 is specifically as follows:

[0019] Based on the AST functional prediction vector, perform the first-level clone detection filtering. Use the cosine similarity to calculate the similarity between two AST functional prediction vectors. If the similarity is greater than the given threshold, the two snippets are considered a candidate pair; the threshold is 0.9.

[0020] Further, the source text functional filtering stage in step 5 is specifically as follows:

[0021] For the code pairs detected through the AST functional filtering stage, use the cosine similarity to calculate the similarity between two source text functional prediction vectors. If the similarity is greater than the given threshold, the two snippets are considered a candidate pair.

[0022] Further, the node type metric filtering stage in step 5 is specifically as follows:

[0023] For the code pairs that pass through the AST functional filtering stage and the source text functional filtering stage, based on the node type statistical vector, perform node type metric filtering; use the cosine similarity to calculate the similarity between two node type statistical vectors. If the similarity is greater than the given threshold, then finally the two snippets are considered a clone pair.

[0024] A storage medium according to the present invention, on which a computer program is stored, and when the computer program is executed by at least one processor, the steps of the code clone detection method based on multi-level prediction as described above are implemented.

[0025] An electronic device according to the present invention, including a memory and a processor, wherein: the memory is used to store a computer program that can run on the processor; the processor is used to execute the steps of the code clone detection method based on multi-level prediction as described above when running the computer program.

[0026] Advantages: Compared with the prior art, the advantages of the present invention are as follows: The present invention respectively generates an AST-based coding sequence and a source text-based coding sequence, and then uses a deep learning model to extract their features to generate vector representations. In addition, a node type statistical vector is generated to provide additional structural information for the code. These three vectors can respectively reflect information from different angles of the code, which is beneficial to improving the accuracy of code clone detection; by implementing multi-level prediction of AST functional filtering, source text functional filtering, and node type metric filtering, the present invention can evaluate the similarity of code pairs at different levels, thereby significantly improving the efficiency, accuracy, and reliability of clone detection. Brief Description of the Drawings

[0027] Figure 1 It is a flowchart of a code clone detection method based on multi-level prediction;

[0028] Figure 2 It is a flowchart of the deep learning feature extraction proposed by the present invention;

[0029] Figure 3 It is a flowchart of the multi-level filtering method proposed by the present invention. Detailed Embodiment

[0030] The present invention will be further described in detail below with reference to the accompanying drawings.

[0031] As Figure 1 shown, the present invention proposes a code clone detection method based on multi-level prediction. First, the source code is converted into an abstract syntax tree (AST) and refined to extract key structural information; then, the source code text is converted into a vector sequence using a word embedding method; then, two semantic feature vectors are extracted from the AST coding sequence and the Token vector sequence respectively using deep learning technology to provide rich semantic and structural information for code clone detection; multi-level prediction of AST functional filtering, source text functional filtering, and node type metric filtering is implemented. Specifically, it includes the following steps:

[0032] Step 1: Convert the code to an abstract syntax tree (AST) and encode it.

[0033] First, the source code is converted into an abstract syntax tree (AST) to capture the structural information of the code, and then the original AST is refined to remove redundant information. The refined AST is traversed to extract and encode the node path information, generating an AST coding sequence that can represent the structural features of the code.

[0034] Generate an AST encoding sequence that can represent the characteristics of the code structure. This sequence is composed of the nodes contained in the abstract syntax tree of the code in the pre-order traversal order. The encoding of each node is (structural encoding, node type encoding). The node type encoding is three digits, such as 001, and the structural encoding is four digits, such as +002. The structural encoding represents the layer difference between this node and the previous node in the pre-order traversal sequence. The first digit is "+", which means that the layer of the syntax tree where this node is located is larger than that of the previous node, and "-" is the opposite.

[0035] Step 2: Code text representation.

[0036] Apply the word embedding model Word2Vec to convert the code text into a sequence of Token vectors. This model will learn the semantic information of the code, convert the characters in the code text into a series of word tokens (Tokens) with semantic information. Finally, the code snippet will be converted into a sequence of Token vectors, representing the text information of the code.

[0037] Step 3: Deep learning feature extraction.

[0038] Use a deep learning model to extract features from the AST encoding sequence and the Token encoding sequence respectively, specifically as Figure 2 shown. Input the AST encoding sequence and the Token encoding sequence obtained in Step 1 and Step 2 into a bidirectional long short-term memory network (BiLSTM), and use the code classification task as the downstream target. This network will extract the abstract syntax tree information and text information of the code, and then generate a functional prediction vector AstFP that can reflect the abstract syntax tree information of the code and a functional prediction vector SrcFP that can reflect the text information of the code through a code classifier.

[0039] Input the AST encoding sequence obtained in Step 1 into a bidirectional long short-term memory network, and use the code classification task as the downstream target. This network will learn the abstract syntax tree information contained in the AST encoding sequence and generate a functional prediction vector AstFP. This functional prediction vector can represent the probability that the code snippet is classified into different functions and can reflect the abstract syntax tree structure characteristics of the code.

[0040] Input the Token encoding sequence obtained in Step 2 into a bidirectional long short-term memory network, and use the code classification task as the downstream target. This network will learn the source code text information contained in the Token encoding sequence and generate a functional prediction vector SrcFP. This functional prediction vector can represent the probability that the code snippet is classified into different functions and can reflect the source text characteristics of the code.

[0041] Step 4: Obtain the node type statistical vector based on the AST node encoding sequence.

[0042] Further obtain node type information from the AST encoded sequence, and count the occurrence frequency of each node type in the code snippet, so as to obtain a vector containing the frequency information of each type of node, that is, the node type statistical vector of this code snippet. This step provides additional structural information for the code by analyzing the types of AST nodes.

[0043] Based on the obtained AST encoded sequence, only consider the type encodings in the sequence and count the frequencies of its various types of nodes. For example, the code snippet x can be represented as a bag-of-tokens Bagx = {(TCi, freqi)}. Each token in the bag is represented as a binary tuple (TCi, freqi), where TCi represents the node type in the AST and freqi represents the number of times TCi appears in the AST node sequence.

[0044] Step 5: Use the information obtained in the above steps for multi-level filtering, specifically as Figure 3 shown.

[0045] First, perform AST functional filtering. If it is determined to be a possible clone pair, enter the source text functional filtering stage. For the code that is also determined to be a possible clone pair in this stage, finally perform node type metric filtering. By screening the code pairs layer by layer through three stages, if the existence of cloning is confirmed in each layer of filtering, the code pair is determined to be a clone pair.

[0046] The AST functional filtering in the multi-level filtering step specifically includes: Based on the AST functional prediction vector, perform the first-level clone detection filtering. This step uses cosine similarity to calculate the similarity between two AST functional prediction vectors. If the similarity is greater than the given threshold of 0.9, the two fragments can be considered candidate pairs. This step mainly focuses on the structural similarity of the code and screens out possible clone pairs by comparing the similarity of the AST functional prediction vectors.

[0047] The text functional filtering in the multi-level filtering step specifically includes: Based on the source text functional prediction vector, perform the second-level clone detection filtering. This step uses cosine similarity to calculate the similarity between two source text functional prediction vectors. If the similarity is greater than the given threshold of 0.9, the two fragments are considered candidate pairs and enter the next-level filtering. This step mainly focuses on the text similarity of the code and further screens clone pairs by comparing the similarity of the source text functional prediction vectors.

[0048] The node type metric filtering in the multi-level filtering step specifically includes: For the code pairs that pass the first two rounds of detection, the node type statistical vector is used for final filtering. Based on the node type statistical vector, node type metric filtering is performed. This step uses cosine similarity to calculate the similarity between two node type statistical vectors. If the similarity is greater than the given threshold of 0.9, then finally the two segments are considered a clone pair.

[0049] The present invention also provides a storage medium, on which a computer program is stored. When the computer program is executed by at least one processor, the steps of the above-mentioned code clone detection method based on multi-level prediction are implemented.

[0050] The present invention also provides an electronic device, including a memory and a processor, wherein: the memory is used to store a computer program that can run on the processor; the processor is used to execute the steps of the above-mentioned code clone detection method based on multi-level prediction when running the computer program.

[0051] The present invention can evaluate the similarity of code pairs at different levels, thereby significantly improving the accuracy and reliability of clone detection. In addition, it also ensures high performance in the clone matching process, providing an efficient and accurate technical means for code clone detection.

[0052] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A code clone detection method based on multi-level prediction, characterized in that: The steps include: Step 1: Convert the code to abstract syntax tree AST and encode it; Step 2: Convert the source code text into a token sequence, and use the word embedding method to convert the token into a numerical word vector, using a digital vector sequence to represent the text information of the code; Step 3: Based on deep learning, semantic features are extracted from AST coding sequences and Token coding sequences respectively, the functional semantic information in the code abstract syntax tree and text is extracted, and functional semantic prediction vectors that can reflect the code abstract syntax tree information and text information are generated respectively; Step 4: Extract node type information from the AST encoding sequence and generate a node type statistical vector; Step 5: Multi-stage filtering clone detection: Determine candidate pairs based on the AST functional filtering stage and the source text functional filtering stage; then determine the clone pairs through the node type metric filtering stage.

2. The code clone detection method based on multi-level prediction according to claim 1, characterized in that: The implementation process of step 1 is as follows: First, the source code is converted into an abstract syntax tree AST to capture the structural information of the code; then the original AST is refined to remove redundant information, the refined AST is traversed, the node path information is extracted and encoded, and an AST encoding sequence that can represent the structural characteristics of the code is generated.

3. The code clone detection method based on multi-level prediction according to claim 1, characterized in that: The implementation process of step 3 is as follows: The AST encoding sequence obtained in step 1 is input into the bidirectional long short-term memory network, and the code classification task is used as the downstream target; the code abstract syntax tree information contained in the AST encoding sequence is learned to generate a functional prediction vector AstFP; the functional prediction vector represents the probability of the code fragment being classified into different functions, reflecting the abstract syntax tree structure characteristics of the code; The token encoding sequence obtained in step 2 is input into the bidirectional long short-term memory network, and the code classification task is used as the downstream target; the source code text information contained in the token encoding sequence is learned to generate a functional prediction vector SrcFP; the functional prediction vector represents the probability of the code fragment being classified into different functions, reflecting the source text characteristics of the code.

4. The code clone detection method based on multi-level prediction according to claim 1, characterized in that: The node type statistical vector described in step 4 is obtained by counting the occurrence frequency of each node type in the code snippet from the AST encoding sequence.

5. The code clone detection method based on multi-level prediction according to claim 1, characterized in that: The AST functional filtering stage described in step 5 is specifically as follows: Based on the AST functional prediction vector, the first level of clone detection filtering is performed, and the similarity between two AST functional prediction vectors is calculated using cosine similarity. If the similarity is greater than a given threshold, the two fragments are considered as candidate pairs.

6. The code clone detection method based on multi-level prediction according to claim 1, characterized in that: The source text functional filtering stage described in step 5 is specifically as follows: For the code pairs detected by the AST functional filtering stage, cosine similarity is used to calculate the similarity between the two source text functional prediction vectors. If the similarity is greater than a given threshold, the two segments are considered as candidate pairs.

7. The code clone detection method based on multi-level prediction according to claim 1, characterized in that: The node type metric filtering phase described in step 5 is specifically as follows: For the code pairs that have passed the AST functional filtering stage and the source text functional filtering stage, node type metric filtering is performed based on the node type statistical vector; the similarity between the two node type statistical vectors is calculated using cosine similarity, and if the similarity is greater than a given threshold, the two fragments are ultimately considered to be clone pairs.

8. The code clone detection method based on multi-level prediction according to claim 5, characterized in that: The threshold is 0.

9.

9. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by at least one processor, implements the steps of the code clone detection method based on multi-level prediction according to any one of claims 1 to 8.

10. An electronic device, characterized in that: comprising a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor, configured to execute the steps of the code clone detection method based on multi-level prediction according to any one of claims 1 to 8 when running the computer program.

Images