Code clone detection method and system based on abstract syntax tree optimization and multi-representation

Abstract

The application discloses a code clone detection method and system based on abstract syntax tree optimization and multi-representation, which comprises the following steps: compiling code text to obtain a corresponding abstract syntax tree; optimizing the abstract syntax tree, including removing the nodes generated by the compiler and the recovery nodes of the compilation errors, removing the declaration nodes and constant nodes, refining the expression nodes, and respectively converting the selection structure and the loop structure into corresponding unified sub-tree structures; traversing the optimized abstract syntax tree to obtain two sequences; inputting the two sequences into a Transformer network to output the feature fingerprints corresponding to the code text; obtaining a plurality of corresponding feature fingerprints according to a plurality of code texts; and if the cosine similarity of any two feature fingerprints is greater than a first set threshold, the two text codes corresponding to the two feature fingerprints are similar. The application optimizes the abstract syntax tree to improve the prediction result while reducing the prediction cost.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of code clone detection, and particularly relates to a code clone detection method and system based on abstract syntax tree optimization and multi-representation, a terminal device and a computer readable storage medium. BACKGROUND

[0002] Code clone is one of the important problems of software engineering tasks, and is widely used in identifying software plagiarism, code refactoring or code search. Meanwhile, code clone is one of the important software defect types, and most software refactoring needs to detect and identify code clone and refactor it. The current main methods have the problems of high training cost and low accuracy.

[0003] In the code clone detection industry, there are mainly two categories of traditional methods and model-based methods, and the specific combination with code can be divided into five methods based on text, based on lexicon, based on syntax, based on semantics and based on metric value.

[0004] The methods based on text and based on lexicon mostly use traditional methods for comparison, such as using KMP algorithm for direct matching of code text to find code duplication, or using filtered Token sequence for matching. This method can only detect part of the highly similar codes in code clone, and has low accuracy. The methods based on syntax or semantics mainly obtain the AST of the code and compare them. The traditional methods have methods such as similar subgraph identification and high-fuzzy net. The traditional methods perform poorly in accuracy. The model methods have methods such as word embedding and DNN network (ASTNN), LSTM network (such as At-BiLSTM), Transformer network (such as Code Token Learner, CTL) and graph neural network. These methods currently have the problems of high training cost and poor robustness. The method based on metric value only detects the code at a fixed granularity. This method has low accuracy in fine-grained code clone, and also cannot find some specific types of code clone. SUMMARY

[0005] In order to solve the above problems of the prior art, the application provides a code clone detection method and system based on abstract syntax tree optimization and multi-representation, a terminal device and a computer readable storage medium.

[0006] A first object of the application is to provide a code clone detection method based on abstract syntax tree optimization and multi-representation.

[0007] A second object of the application is to provide a code clone detection system based on abstract syntax tree optimization and multi-representation.

[0008] The third object of the present application is to provide a terminal device.

[0009] The fourth object of the present application is to provide a computer-readable storage medium.

[0010] The first object of the present application can be achieved by adopting the following technical solution:

[0011] A code clone detection method based on abstract syntax tree optimization and multi-representation, the method comprises:

[0012] Compiling the code text to obtain a corresponding abstract syntax tree AST;

[0013] Optimizing the abstract syntax tree AST, including removing the nodes generated by the compiler and the recovery nodes of the compilation errors, removing the declaration nodes and the constant nodes, refining the expression nodes, and respectively converting the selection structure and the loop structure into corresponding unified sub-tree structures;

[0014] Traversing the optimized abstract syntax tree AST to obtain a pre-order sequence and a post-order sequence;

[0015] Inputting the pre-order sequence and the post-order sequence into a multi-layer Transformer network to output a feature fingerprint corresponding to the code text;

[0016] According to a plurality of code texts, a plurality of corresponding feature fingerprints are obtained; if the cosine similarity of any two feature fingerprints is greater than a first set threshold, then the two code texts corresponding to the two feature fingerprints are similar.

[0017] Further, the inputting the pre-order sequence and the post-order sequence into the multi-layer Transformer network to output a plurality of feature fingerprints comprises:

[0018] Mapping each word in the pre-order sequence to an n-dimensional vector using an embedding layer to obtain a word vector sequence of the pre-order sequence; if the length of the pre-order sequence is less than n Token, pad is filled, and a mask is generated to shield the pad part; otherwise, the pre-order sequence is truncated to the first n Token;

[0019] Mapping each word in the post-order sequence to an n-dimensional vector using an embedding layer to obtain a word vector sequence of the post-order sequence; if the length of the post-order sequence is less than n Token, pad is filled, and a mask is generated to shield the pad part; otherwise, the post-order sequence is truncated to the first n Token;

[0020] The word vector sequence of the pre-order sequence and the word vector sequence of the post-order sequence pass through the multi-layer Transformer network respectively, and output a learned pre-order feature matrix and a post-order feature matrix respectively;

[0021] The pre-sequence feature matrix and the post-sequence feature matrix are regarded as two channels of a picture, and are combined through a convolution layer to obtain a combined feature matrix;

[0022] The combined feature matrix is output through a multi-layer Transformer network to obtain a learned feature matrix;

[0023] The learned feature matrix is regarded as a single-channel picture, and an output vector is obtained through a convolution layer and a RELU activation function, the output vector being a feature fingerprint.

[0024] Further, the removal of the nodes generated by the compiler and the recovery nodes of the compilation errors comprises:

[0025] The analysis of the code text by the compiler retains the implicit type conversion as an internal type conversion expression, deletes the node corresponding to the internal type conversion expression, and reconnects the upper and lower nodes;

[0026] The compiler places the node with a syntax error in a recovery expression, and deletes the node with a syntax error and its subtree.

[0027] Further, the expression node refinement comprises:

[0028] The binary operator expression and the unary operator expression are refined into the following structure: Binary / UnaryOperatorExpr_<expression symbol>_<return value type>.

[0029] Further, the conversion of the selection structure and the loop structure into corresponding unified subtree structures respectively comprises:

[0030] Each branch in the branch statement is disassembled into two parts: a condition and a statement, and the condition and the statement are attributed to the same selection unit node;

[0031] The first expression of the for statement is moved to the end of the for previous statement, and the third expression of the for statement is moved to the end of the loop body; in this way, the loop structure can be uniformly represented as a loop statement containing a condition sub-node and a compound statement sub-node.

[0032] Further, the compilation of the code text to obtain the corresponding abstract syntax tree AST comprises:

[0033] The suffix name of the file in which the code text is located is judged, and if it is not one of.cpp,.c,.h, the file suffix name is converted into.cpp;

[0034] The recoverable text sequence is obtained according to the code text in the file;

[0035] According to the text sequence, a corresponding abstract syntax tree AST is obtained.

[0036] Further, the method further comprises, for the case of a large number of code blocks, dividing the large number of code blocks into different code clone sets, comprising:

[0037] numbering all the code blocks, and the fingerprint number corresponding to each code block being the same as the number of the code block;

[0038] selecting the code block with the smallest number from the code blocks that have not been grouped as a reference code block, and marking the reference code block as having been grouped;

[0039] adding the reference code block to the current group, and for each code block that has not been grouped, if the cosine similarity of the feature fingerprint of the code block to the feature fingerprints of all the code blocks in the current group is greater than a second set threshold, then adding the code block to the current group and marking the code block as having been grouped; when all the code blocks that have not been grouped have been accessed, if the number of code blocks in the current group is more than 1, then regarding the current group as a code clone set;

[0040] if all the code blocks are marked as having been grouped, then the division ends, otherwise returning to select the code block with the smallest number from the code blocks that have not been grouped as a reference code block, and continuing to perform the subsequent operations.

[0041] The second object of the application can be achieved by adopting the following technical scheme:

[0042] A code clone detection system based on abstract syntax tree optimization and multi-representation, the system comprising:

[0043] a compiling module configured to compile a code text to obtain a corresponding abstract syntax tree AST;

[0044] an optimization module configured to optimize the abstract syntax tree AST, comprising removing compiler-generated nodes and recovery nodes of compilation errors, removing declaration nodes and constant nodes, refining expression nodes, and respectively converting selection structures and loop structures into corresponding uniform subtree structures;

[0045] a traversal module configured to traverse the optimized abstract syntax tree AST to obtain a pre-order sequence and a post-order sequence;

[0046] an output module configured to input the pre-order sequence and the post-order sequence into a multi-layer Transformer network to output a feature fingerprint corresponding to the code text;

[0047] a detection module configured to obtain a plurality of corresponding feature fingerprints according to a plurality of code texts; if the cosine similarity of any two feature fingerprints is greater than a first set threshold, then the two code texts corresponding to the two feature fingerprints are similar.

[0048] The third object of the present application can be achieved by adopting the technical scheme as follows:

[0049] A terminal device comprises a processor and a memory for storing a program executable by the processor, and the processor implements the code clone detection method described above when executing the program stored in the memory.

[0050] The fourth object of the present application can be achieved by adopting the technical scheme as follows:

[0051] A computer-readable storage medium stores a program, and the program is executed by a processor to implement the code clone detection method described above.

[0052] The present application has the following beneficial effects relative to the prior art:

[0053] 1. The present application can more effectively shorten the length of the AST sequence and increase the category of the AST node by optimizing the abstract syntax tree (AST), thereby improving the model prediction result while reducing the prediction cost of the model.

[0054] 2. The present application can obtain better prediction results than the ASTNN, At-BiLSTM, Code Token Learner, etc. model by combining the pre-order traversal and post-order traversal two kinds of representation methods, and using two kinds of representation can improve the robustness of the model, so that it has better use effect in the actual application environment.

[0055] 3. The present application uses a sequence-based model instead of a graph-based model to speed up the training speed of the model and reduce the deployment cost of the code clone method.

[0056] 4. The present application uses the pre-order and post-order two sequences to construct a twin neural network, and then completely uses the twin network to generate a feature fingerprint and a cosine similarity comparison process of the feature fingerprint, which is more convenient to deploy and faster in actual use compared with the Code Token Learner and other similar models using cross-attention. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the drawings shown.

[0058] Figure 1This is a flowchart of the code clone detection method based on abstract syntax tree optimization and multiple representations according to Embodiment 1 of the present invention;

[0059] Figure 2 This is a schematic diagram of AST optimization in Embodiment 1 of the present invention;

[0060] Figure 3 This is a schematic diagram of the output feature fingerprint based on the Transformer network in Embodiment 1 of the present invention;

[0061] Figure 4 This is a flowchart of the set partitioning in Embodiment 1 of the present invention;

[0062] Figure 5 This is a comparison of the performance of the method of Embodiment 1 of the present invention with several models on the OJClone dataset;

[0063] Figure 6 This is a structural block diagram of the code clone detection system based on abstract syntax tree optimization and multiple representations according to Embodiment 2 of the present invention;

[0064] Figure 7 This is a structural block diagram of the terminal device according to Embodiment 3 of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the specific embodiments described are merely used to explain this application and are not intended to limit this application.

[0066] Example 1:

[0067] like Figure 1 As shown, the code clone detection method based on abstract syntax tree optimization and multi-representation provided in this embodiment includes the following steps:

[0068] Step (1): Compile to obtain the Abstract Syntax Tree (AST) corresponding to the code text.

[0069] This step can be performed using the compiler clang or other tools to obtain the corresponding Abstract Syntax Tree (AST).

[0070] Furthermore, step (1) specifically includes:

[0071] (1.1) suffix name transformation: judge whether the suffix name of the file is one of.cpp / .c / .h, if not, transform the suffix name of the file to.cpp;

[0072] (1.2) obtain the recoverable AST sequence according to the code in the file: use the tool developed based on clang's libtooling to obtain a text sequence containing brackets and node names, the specific algorithm process is as follows:

[0073] (1.2.1) set the current node N as the root node of the AST tree, and the text sequence as S;

[0074] (1.2.2) obtain the name n of the current node N, S=S+n;

[0075] (1.2.3) obtain all child nodes N1, N2...N n, of the current node N;

[0076] (1.2.4) take the child nodes N1, N2...N n, as the current node N respectively, repeat steps (1.2.2) and (1.2.3) to obtain S1, S2...S n , S=S+[S1+S2+...+S n ];

[0077] Finally, the obtained S is the AST text sequence, and S is output.

[0078] (1.3) obtain the corresponding abstract syntax tree AST according to the recoverable AST sequence: restore the structure of the AST tree according to the obtained AST text sequence, the specific algorithm process is as follows:

[0079] (1.3.1) obtain the first word of S as T0, as the root node of the AST tree, T p =T0, T p2 =T0;

[0080] (1.3.2) obtain the next word of T0 as T, if T is "[", then T p2 =T p , T p =T, if T is "]", then T p =T p2 , if T is other words, then the parent node T p2 =T p of T;

[0081] (1.3.3) take T as T0, repeat step (1.3.2) until the end of sequence S is reached;

[0082] (1.3.4) output the generated AST tree.

[0083] Step (2): Optimization of the Abstract Syntax Tree AST.

[0084] Further, as shown in Figure 2 , step (2) specifically includes:

[0085] (2.1) Removing compiler-generated nodes and recovery nodes for compilation errors: The compiler analysis of the source code retains implicit type conversion as ImplictCastExpr (internal type conversion expression), this optimization step removes such nodes and reconnects the upper and lower nodes, the compiler places nodes with syntax errors in RecoryExpr (recovery expression), this optimization step removes the node and the subtree under the node;

[0086] (2.2) Removing declaration and constant nodes: There are DeclStmt (declaration statement) and other nodes related to variable declaration in the AST tree, which do not participate in the expression of the logical flow, in order to simplify the AST, these nodes and the corresponding subtree under the node will be removed;

[0087] (2.3) Refining expression nodes: There are BinaryOperatorExpr (binary operator expression) and UnaryOperatorExpr (unary operator expression) in the AST tree, this step refines the two expressions into the following structure Binary / UnaryOperatorExpr_<expression symbol>_<return value type>, for example, when a and b are both int, a BinaryOperatorExpr of a+b will be converted to BinaryOperatorExpr_+_Integer;

[0088] (2.4) Unified loop selection logic structure: The AST sub-tree structure of IfStmt, SwitchStmt of code AST is uniformly converted into the sub-tree structure of FixedSelectionStmt. The AST sub-tree structure of ForStmt, WhileStmt, DoStmt of code AST is uniformly converted into the sub-tree structure of FixedLoopStmt. The AST sub-tree structure of CompoundStmt of code AST is uniformly converted into the sub-tree structure of FixedCompoundStmt or deleted. The branch structure in C / C++ includes if statement and switch statement. Each branch in the branch statement can be decomposed into two parts: condition and statement. This step extracts the condition and statement in each branch into a select unit node, thereby realizing the unification of the branch statement. For the loop structure such as for statement and while statement, it is unified into a loop statement. This step moves the first expression (for-init part) of the for statement to the end of the for previous statement, without being reflected in the loop statement. At the same time, the third expression of the for statement is moved to the end of the loop body. In this way, for and while can be represented as a loop statement containing a condition sub-node and a compound statement sub-node. The unified structure diagram is shown in Figure x. As such a structure is unified, the algorithm provided by the embodiment can detect the code clone generated by the mutual rewriting of if statement and switch statement, and the mutual rewriting of for statement and while statement.

[0089] Step (3): traverse the AST to obtain pre-order and post-order sequences.

[0090] Step (4): input the pre-order and post-order sequences into the Transformer network to obtain the feature fingerprint.

[0091] Further, as shown in Figure 3 Step (4) specifically includes:

[0092] (4.1) Word vector embedding: Each word of the pre-order sequence and the post-order sequence is mapped to an n-dimensional vector by an embedding layer. At the same time, the maximum input length is defined as n Token, and the sequence less than n Token is filled with pad, and mask is generated to shield the pad part. The sequence exceeding n is truncated to the first 128 Token.

[0093] In the embodiment, n is 128.

[0094] (4.2) Transformer feature extraction: the pre-sequence and post-sequence word vector sequences pass through a multi-layer Transformer network. When the input is two [128, 128] pre-sequence and post-sequence matrices, the output is also two learned [128, 128] pre-sequence and post-sequence matrices;

[0095] (4.3) Feature fusion: the two feature matrices are regarded as two channels of a picture, and a convolution layer is used to combine the features, and finally a combined [128, 128] matrix is obtained;

[0096] (4.4) Transformer fusion learning: the combined [128, 128] matrix is further passed through a multi-layer Transformer network, and finally a learned [128, 128] output feature matrix is output;

[0097] (4.5) Feature extraction: the output feature matrix is regarded as a single-channel picture, and a convolution layer and a RELU activation function are used to obtain an output vector. According to the number of convolution output channels, output vectors of different sizes can be obtained. The default output channel is 4, and the output vector obtained is 512-dimensional.

[0098] Step (5): compare the feature fingerprints to determine whether they are similar.

[0099] According to steps (1)-(4), a plurality of feature fingerprints corresponding to a plurality of code texts can be obtained. The cosine similarity is used to compare the distance between two fingerprints. When the fingerprint similarity is greater than 0.7, it is considered to be similar.

[0100] Step (6): clone code set division.

[0101] This step is for the case of a large number of code blocks. The code clone set is divided by set.

[0102] Further, as shown in Figure 4 , step (6) specifically includes:

[0103] (6.1) Code block numbering: all code blocks are numbered. At the same time, the fingerprint number corresponding to each code block is the same as the code block number;

[0104] (6.2) Select reference code block: select the smallest numbered code block as the reference code block from the code blocks that have not been grouped, and mark the reference code block as grouped;

[0105] (6.3) Code block grouping: the benchmark code block is the first code block to join the current group. For each code block that has not been grouped, check whether the feature fingerprint of the code block meets the distance threshold limit with all the feature fingerprints corresponding to the code blocks in the current group. If it does, add the code block to the current group and mark the code block as grouped. When all ungrouped code blocks have been accessed, if the current group contains more than one code block, the current group is considered as a code clone set;

[0106] Specifically, if the cosine similarity of the feature fingerprint of the ungrouped code block with all the feature fingerprints corresponding to the code blocks in the current group is greater than 0.6, the code block is added to the current group and marked as grouped; otherwise, no processing is performed on the code block;

[0107] (6.4) Repeat steps (6.2) and (6.3) until all code blocks are marked as grouped, and all code clone sets are obtained.

[0108] As shown in Figure 5 , the accuracy, recall rate and F1 score of the method of the present application are higher than those of several existing model detection methods.

[0109] Those skilled in the art can understand that all or part of the steps in the method of the above embodiments can be instructed by a program to relevant hardware, and the corresponding program can be stored in a computer readable storage medium.

[0110] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. On the contrary, the depicted steps can change the order of execution. Additionally or alternatively, some steps can be omitted, combined into one step, and / or divided into multiple steps.

[0111] Embodiment 2:

[0112] As shown in Figure 6 , the embodiment provides a code clone detection system based on abstract syntax tree optimization and multi-representation, which includes a compiling module 601, an optimization module 602, a traversal module 603, an output module 604 and a detection module 605, wherein:

[0113] The compiling module 601 is configured to compile the code text to obtain a corresponding abstract syntax tree AST;

[0114] The optimization module 602 is used to optimize the abstract syntax tree (AST), including removing compiler-generated nodes and compilation error recovery nodes, removing declaration nodes and constant nodes, refining expression nodes, and converting selection structures and loop structures into corresponding unified subtree structures respectively.

[0115] Traversal module 603 is used to traverse the optimized Abstract Syntax Tree (AST) to obtain the preorder and postorder sequences.

[0116] The output module 604 is used to input the preorder and postorder sequences into a multi-layer Transformer network and output the feature fingerprint corresponding to the code text.

[0117] The detection module 605 is used to obtain multiple corresponding feature fingerprints based on multiple code texts; if the cosine similarity of any two feature fingerprints is greater than a first set threshold, then the two text codes corresponding to the two feature fingerprints are similar.

[0118] The specific implementation of each module in this embodiment can be found in Embodiment 1 above, and will not be repeated here. It should be noted that the system provided in this embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0119] Example 3:

[0120] This embodiment provides a terminal device, which can be a computer, such as... Figure 7 As shown, the processor 702, memory, input device 703, display 704, and network interface 705 are connected via system bus 701. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 706 and internal memory 707. The non-volatile storage medium 706 stores the operating system, computer programs, and database. The internal memory 707 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 702 executes the computer programs stored in the memory, it implements the code clone detection method of Embodiment 1 described above, as follows:

[0121] The code text is compiled to obtain the corresponding Abstract Syntax Tree (AST);

[0122] The abstract syntax tree (AST) is optimized by removing compiler-generated nodes and nodes that restore compilation errors, removing declaration nodes and constant nodes, refining expression nodes, and converting selection structures and loop structures into corresponding unified subtree structures.

[0123] Traverse the optimized abstract syntax tree AST to obtain a pre-order sequence and a post-order sequence;

[0124] Input the pre-order sequence and the post-order sequence into a multi-layer Transformer network, and output a feature fingerprint corresponding to the code text;

[0125] According to a plurality of code texts, a plurality of corresponding feature fingerprints are obtained; if the cosine similarity of any two feature fingerprints is greater than a first set threshold, then the two code texts corresponding to the two feature fingerprints are similar.

[0126] Embodiment 4:

[0127] The embodiment provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the code clone detection method of the above-mentioned embodiment 1, as follows:

[0128] Compiling the code text to obtain a corresponding abstract syntax tree AST;

[0129] Optimizing the abstract syntax tree AST, including removing the nodes generated by the compiler and the recovery nodes of the compilation errors, removing the declaration nodes and the constant nodes, refining the expression nodes, and respectively converting the selection structure and the loop structure into corresponding uniform sub-tree structures;

[0130] Traverse the optimized abstract syntax tree AST to obtain a pre-order sequence and a post-order sequence;

[0131] Input the pre-order sequence and the post-order sequence into a multi-layer Transformer network, and output a feature fingerprint corresponding to the code text;

[0132] According to a plurality of code texts, a plurality of corresponding feature fingerprints are obtained; if the cosine similarity of any two feature fingerprints is greater than a first set threshold, then the two code texts corresponding to the two feature fingerprints are similar.

[0133] It should be noted that the computer readable storage medium of the embodiment can be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] The above merely describes preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, within the scope disclosed by the present application, can make equivalent replacements or changes according to the technical scheme and inventive concept of the present application, and all of them shall fall within the protection scope of the present application.

Claims

1. A code clone detection method based on abstract syntax tree optimization and multi-representation, characterized in that, The method comprises: compiling the code text to obtain a corresponding abstract syntax tree AST; optimizing the abstract syntax tree AST, including removing compiler-generated nodes and recovery nodes of compilation errors, removing declaration nodes and constant nodes, refining expression nodes, and respectively converting selection structures and loop structures into corresponding unified sub-tree structures; traversing the optimized abstract syntax tree AST to obtain a pre-order sequence and a post-order sequence; inputting the pre-order sequence and the post-order sequence into a multi-layer Transformer network to output feature fingerprints corresponding to the code text; obtaining a plurality of corresponding feature fingerprints according to a plurality of code texts; if the cosine similarity of any two feature fingerprints is greater than a first set threshold, the two code texts corresponding to the two feature fingerprints are similar; wherein the removing of the compiler-generated nodes and the recovery nodes of the compilation errors comprises: deleting and reconnecting nodes corresponding to internal type conversion expressions; the internal type conversion expressions are obtained by the compiler analyzing the code text and retaining implicit type conversion; deleting nodes with syntax errors and their sub-trees; the nodes with syntax errors are placed in a recovery expression by the compiler; the refining of the expression nodes specifically comprises: refining binary operator expressions and unary operator expressions into the following structure: Binary / UnaryOperatorExpr_<expression symbol>_<return value type>; the converting of the selection structures and the loop structures into corresponding unified sub-tree structures respectively comprises: decomposing each branch in a branch statement into two parts: a condition and a statement, and grouping the condition and the statement into the same selection unit node; moving a first expression of a for statement to the end of a for preceding statement, and moving a third expression of the for statement to the end of a loop body; in this way, the loop structure can be uniformly represented as a loop statement containing a condition sub-node and a compound statement sub-node.

2. The code clone detection method of claim 1, wherein, the inputting of the pre-order sequence and the post-order sequence into the multi-layer Transformer network to output a plurality of feature fingerprints comprises: mapping each word in the pre-order sequence to an n-dimensional vector using an embedding layer to obtain a word vector sequence of the pre-order sequence; if the length of the pre-order sequence is less than n tokens, pad is used for padding, and a mask is generated to shield the pad part; otherwise, the pre-order sequence is truncated to the first n tokens; mapping each word in the post-order sequence to an n-dimensional vector using an embedding layer to obtain a word vector sequence of the post-order sequence; if the length of the post-order sequence is less than n tokens, pad is used for padding, and a mask is generated to shield the pad part; otherwise, the post-order sequence is truncated to the first n tokens; the word vector sequence of the pre-order sequence and the word vector sequence of the post-order sequence respectively pass through the multi-layer Transformer network to respectively output a pre-order feature matrix and a post-order feature matrix after learning; the pre-order feature matrix and the post-order feature matrix are regarded as two channels of a picture, and are combined through a convolution layer to obtain a combined feature matrix; The merged feature matrix is output through a multi-layer Transformer network to obtain a learned feature matrix; The learned feature matrix is regarded as a single-channel picture, and a convolution layer and a RELU activation function are used to obtain an output vector, which is a feature fingerprint.

3. The code clone detection method of claim 1, wherein, The code text is compiled to obtain a corresponding abstract syntax tree AST, including: If the file suffix is not one of.cpp,.c, and.h, the file suffix is converted to.cpp; A recoverable text sequence is obtained according to the code text in the file; A corresponding abstract syntax tree AST is obtained according to the text sequence.

4. The code clone detection method according to any one of claims 1 to 3, characterized in that, The method further includes, for the case of a large number of code blocks, dividing the large number of code blocks into different code clone sets, including: All code blocks are numbered, and the fingerprint number corresponding to each code block is the same as the number of the code block; The smallest numbered code block that has not been grouped is selected as a reference code block, and the reference code block is marked as having been grouped; The reference code block is added to the current group, and for each code block that has not been grouped, if the cosine similarity of the feature fingerprint of the code block and the feature fingerprints of all code blocks in the current group is greater than a second set threshold, the code block is added to the current group and marked as having been grouped; when all ungrouped code blocks have been accessed, if the number of code blocks in the current group is greater than 1, the current group is regarded as a code clone set; If all code blocks are marked as having been grouped, the division ends, otherwise, the smallest numbered code block that has not been grouped is selected as a reference code block, and the subsequent operations are continued.

5. A code clone detection system based on abstract syntax tree optimization and multi-representation, characterized in that, The system includes: A compilation module for compiling the code text to obtain a corresponding abstract syntax tree AST; An optimization module for optimizing the abstract syntax tree AST, including removing compiler-generated nodes and recovery nodes of compilation errors, removing declaration nodes and constant nodes, refining expression nodes, and respectively converting selection structures and loop structures into corresponding uniform subtree structures; A traversal module for traversing the optimized abstract syntax tree AST to obtain a pre-order sequence and a post-order sequence; An output module for inputting the pre-order sequence and the post-order sequence into a multi-layer Transformer network to output a feature fingerprint corresponding to the code text; A detection module for obtaining a plurality of corresponding feature fingerprints according to a plurality of code texts; if the cosine similarity of any two feature fingerprints is greater than a first set threshold, the two code texts corresponding to the two feature fingerprints are similar; The removal of the compiler-generated nodes and the recovery nodes of the compilation errors includes: The node corresponding to the internal type conversion expression is deleted and reconnected to the upper and lower nodes; the internal type conversion expression is obtained by the compiler analyzing the code text and retaining the implicit type conversion; The node with a syntax error and its subtree are deleted; the node with a syntax error is placed in the recovery expression by the compiler; The expression node refinement specifically includes: The binary operator expression and the unary operator expression are refined into the following structure: Binary / UnaryOperatorExpr_<expression symbol>_<return value type>; The converting the selection structure and the loop structure into the corresponding unified sub-tree structure respectively comprises: Each branch in the branch statement is split into two parts: a condition and a statement, and the condition and the statement are classified into the same selection unit node; The first expression of the for statement is moved to the end of the for preceding statement, and the third expression of the for statement is moved to the end of the loop body; thus the loop structure can be uniformly represented as a loop statement containing a condition sub-node and a compound statement sub-node.

6. A terminal device comprising a processor and a memory for storing a processor-executable program, characterized in that The processor executes the program stored in the memory, and the code clone detection method in any one of claims 1-4 is implemented.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor, and the code clone detection method in any one of claims 1-4 is implemented.

Images