Code clone detection method, device, equipment and medium based on hierarchical analysis
Through hierarchical analysis and decoupling attention mechanism processing AST, the problems of low code cloning detection efficiency and syntax structure damage in the existing technology are solved, and efficient and accurate code cloning detection is achieved, which is suitable for large-scale code bases.
Patent Information
- Application Number
- CN202411396600.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2044-10-09
AI Technical Summary
Existing code cloning detection technologies have problems such as inefficiency, gradient vanishing, syntax structure corruption and high computational overhead, especially when dealing with large code bases, it is difficult to effectively capture the complex relationships between code snippets.
Using a hierarchical analysis method, the abstract syntax tree (AST) is decomposed into manageable layers, and AST nodes are processed through relative position embedding matrix and decoupling attention mechanism, and the content and position information of the nodes are calculated layer by layer to form attention. Matrix for code cloning detection.
It improves the efficiency and accuracy of code cloning detection, reduces computational overhead, maintains the integrity of code syntax structure, solves the problem of gradient vanishing, and provides an efficient and scalable solution.
Smart Images
Figure CN118916077B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence and language processing technology, and in particular to a code clone detection method, device, equipment and medium based on hierarchical analysis. Background Art
[0002] Code cloning aims to identify code snippets with similar syntax or functions within a software system or between software systems to improve the maintainability and convenience of the software. Code clone detection mainly finds similar or repeated code snippets in the software system to perform code maintenance, refactoring and performance optimization, thereby improving software quality.
[0003] Existing code clone detection technologies include text matching, semantic analysis technology, and Abstract Syntax Tree (AST) matching. Text matching is the simplest code clone detection method. It finds similar fragments by directly comparing code text, but it is easily affected by changes in code format (such as comments and spaces), resulting in false positives or negatives. Semantic analysis technology performs clone detection by understanding the semantic information of the code. It can identify code fragments with different syntax but the same semantics, but its computational complexity is high, which has a certain impact on the performance of processing large-scale code bases. AST matching technology detects cloned code fragments by comparing the AST structure of the code. It can better capture the structure and logical relationship of the code and reduce false positives caused by format changes. However, due to the large and deep AST, the semantic information is not fully captured.
[0004] Among existing code clone detection technologies, recurrent neural network (RNN) and long short-term memory network (LSTM) models are widely used to process code sequence data. These models process code sequences by encoding them into fixed-length representations, and use recurrent units to process the timing information of the code during encoding and decoding, attempting to capture the similarities between code snippets. Although they show certain effects in code clone detection, they have the following disadvantages:
[0005] (1) Low efficiency. Due to the limitations of the loop structure, RNN and LSTM models cannot implement parallel processing well, resulting in high training and inference costs. The low efficiency problem is particularly obvious when dealing with large code bases.
[0006] (2) Gradient vanishing. In long sequence data, RNN and LSTM are prone to the gradient vanishing problem, making it difficult to capture long-distance dependencies, making it difficult for the model to learn the complex relationships between code snippets.
[0007] (3) Syntax structure destruction. When existing AST-based neural models process large and deeply nested ASTs, most existing processing methods will destroy the grammatical structure of the source code, weakening the model's ability to capture true semantics.
[0008] (4) High computational overhead. Existing code generation techniques use program dependency graphs (PDGs) and control flow graphs (CFGs) to represent functions, which is time-consuming and complex, resulting in high runtime overhead. This approach is not suitable for uncompilable or incomplete code snippets.
[0009] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention
[0010] The present invention aims to provide a code clone detection method and device based on hierarchical analysis to solve at least one of the above problems.
[0011] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:
[0012] A code clone detection method based on hierarchical analysis, comprising:
[0013] S1, obtain two source code fragments to be cloned and tested;
[0014] S2, respectively parse the two source code fragments and convert them into corresponding abstract syntax trees AST; wherein each node of the abstract syntax tree AST represents a syntax element of the source code;
[0015] S3, performing hierarchical extraction on the abstract syntax trees AST corresponding to the two source code fragments, respectively, decomposing them into manageable layers, and obtaining corresponding hierarchical ASTs;
[0016] S4, traverse each node of the corresponding hierarchical AST, calculate the relative distance between each node and another node in turn, obtain a relative position embedding matrix, and integrate the relative position embedding matrix into the node representation corresponding to the hierarchical AST to obtain a relative position hierarchical AST; wherein the relative position embedding matrix records the inter-layer node position information and intra-layer node position information of each node in the tree structure; each node of the relative position hierarchical AST contains the content information of the node itself and the relative position embedding information of the node;
[0017] S5, according to the relative position hierarchical AST, use the decoupled attention mechanism to process each node of the relative position hierarchical AST, obtain the content vector and position embedding vector of each node, and calculate the attention weights of the content vector to the content vector, the content vector to the position embedding vector, and the position embedding vector to the content vector, so as to obtain the attention matrix of the relative position hierarchical AST;
[0018] S6, inputting the relative position hierarchical AST and the corresponding attention matrix of the two source code fragments into the Siamese network for clone detection, calculating the similarity, and obtaining the clone detection results of the two source code fragments according to the similarity.
[0019] Preferably, when performing hierarchical extraction on the abstract syntax tree AST, the syntax elements of the source code file are extracted in sequence according to the structure of the source code; the syntax elements include function method names, local variable declarations, control flow structures and return statements.
[0020] Preferably, when calculating the relative distance between each node and another node in sequence, the expression of the relative distance is:
[0021] = ;
[0022] in, Represents a node in a hierarchical AST To Node The distance between nodes is , i and j represent the index of the node in the path.
[0023] Preferably, at each node of the relative position hierarchical AST, the content information of the node itself is fused with the relative position embedding information of the node to integrate the relative position embedding information into the node representation corresponding to the hierarchical AST; wherein the fusion includes addition or splicing.
[0024] Preferably, the content vector is used to record the content information of the node in the hierarchical AST; the position embedding vector is used to record the relative position information of the node in the hierarchical AST; the attention matrix of the hierarchical AST is obtained by respectively calculating the dot product of the content vectors of two nodes, the dot product of the content vector and the position embedding vector, and the dot product of the position embedding vector and the content vector, and the calculation formula is:
[0025] = + ;
[0026] in, Represents the attention weight of the i-th node and the j-th node, that is, the value of the i-th row and j-th column in the attention matrix; and Represent the content vectors of the i-th node and the j-th node respectively; The embedding vector representing the relative position of the i-th node and the j-th node; represents the embedding vector of the relative position of the j-th node to the i-th node; T represents the transposed matrix.
[0027] Preferably, when judging the clone detection result, when the similarity is greater than a set threshold, the two source code fragments are determined to be a pair of clone codes.
[0028] Preferably, the inter-layer node position information refers to the relative position relationship between different levels in the entire tree structure, reflecting the global hierarchy of the node in the entire tree structure; the intra-layer node position information refers to the relative position relationship between each node in the same level in the tree structure, reflecting the interdependence of the local structure.
[0029] The present invention also provides a code clone detection device based on hierarchical analysis, comprising:
[0030] An acquisition unit, used for acquiring two source code fragments to be cloned and tested;
[0031] An AST parsing unit, used to parse the two source code fragments and convert them into corresponding abstract syntax trees AST; wherein each node of the abstract syntax tree AST represents a syntax element of the source code;
[0032] An AST hierarchical unit, used to perform hierarchical extraction on the abstract syntax trees AST corresponding to the two source code fragments, decompose them into manageable layers, and obtain corresponding hierarchical ASTs;
[0033] A relative position embedding unit is used to traverse each node of the corresponding hierarchical AST, calculate the relative distance between each node and another node in turn, obtain a relative position embedding matrix, and integrate the relative position embedding matrix into the node representation corresponding to the hierarchical AST to obtain a relative position hierarchical AST; wherein the relative position embedding matrix records the inter-layer node position information and intra-layer node position information of each node in the tree structure; each node of the relative position hierarchical AST contains the content information of the node itself and the relative position embedding information of the node;
[0034] The attention calculation unit is used to process each node of the relative position hierarchical AST using the decoupled attention mechanism according to the relative position hierarchical AST, obtain the content vector and position embedding vector of each node, and respectively calculate the attention weights of the content vector to the content vector, the content vector to the position embedding vector, and the position embedding vector to the content vector, so as to obtain the attention matrix of the relative position hierarchical AST;
[0035] The clone detection unit is used to input the relative position hierarchical AST and the corresponding attention matrix of the two source code fragments into the Siamese network for clone detection, calculate the similarity, and obtain the clone detection results of the two source code fragments according to the similarity.
[0036] The present invention also provides a code clone detection device based on hierarchical parsing, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program can be executed by the processor to implement a code clone detection method based on hierarchical parsing as described above.
[0037] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the code clone detection method based on hierarchical parsing as described above is implemented.
[0038] In summary, compared with the prior art, the present invention has the following beneficial effects:
[0039] The code clone detection method based on hierarchical parsing proposed by the present invention decomposes the abstract syntax trees AST of two source code fragments into manageable layers respectively, so that each layer can be processed sequentially and more accurately. Through hierarchical and sequential extraction of AST, the code is parsed into hierarchical ASTs, and the model can more effectively capture the structure and logic information of the function in the source code. Each layer in the hierarchical AST only contains nodes of a specific type, thereby reducing the complexity of a single layer. By processing layer by layer, the number of nodes processed by the model each time can be effectively reduced, thereby improving the efficiency and accuracy of processing large ASTs.
[0040] In addition, the present invention records the relative position relationship of AST nodes through the relative position embedding matrix to fully understand the structural position information and content information of AST. At the same time, the present invention combines the decoupled attention mechanism to process the content information and position information of the node separately through independent attention heads, so that the model can learn the complex relationship between content and position in a more fine-grained manner, thereby improving the efficiency and accuracy of code clone detection, solving the gradient vanishing problem, improving the interpretability of the model, maintaining the integrity of the code syntax structure, and reducing the computational overhead, thereby providing an efficient and scalable solution for large-scale code clone detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0042] Figure 1 A schematic diagram of a code clone detection method based on hierarchical analysis provided in Example 1 of the present invention.
[0043] Figure 2 This is an overall structural diagram of a code clone detection method based on hierarchical analysis provided in Example 1 of the present invention.
[0044] Figure 3 This is a diagram of the AST hierarchical structure provided in Example 1 of the present invention.
[0045] Figure 4 This is a diagram of the relative position embedding matrix structure of the tree provided in the first embodiment of the present invention.
[0046] Figure 5 A schematic diagram of a code clone detection device based on hierarchical analysis provided in Embodiment 2 of the present invention.
[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0049] Embodiment 1
[0050] Embodiment 1 of the present invention provides a code clone detection method based on hierarchical analysis, which can be implemented by a code clone detection device based on hierarchical analysis (hereinafter referred to as clone detection), and in particular, is executed by one or more processors in the clone detection.
[0051] In this embodiment, the clone detection may be an electronic device equipped with a processor, the processor having a computer program of the code clone detection method based on hierarchical analysis and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.
[0052] In this embodiment, the Abstract Syntax Tree (AST) is a tree representation of the source code, where each node of the tree represents a structure in the source code. AST does not represent every detail in the source code (such as spaces, comments, etc.), but only represents the structure and syntax elements of the program. This representation makes it easier for tools such as compilers and interpreters to analyze and transform the source code. When parsing the source code, the compiler or interpreter first converts the source code into a series of tokens, and then builds AST based on these tokens.
[0053] Existing recurrent neural networks (RNN) and long short-term memory networks (LSTM) cannot effectively process data in parallel due to the limitations of the loop structure, resulting in high training and reasoning costs. This inefficiency problem is particularly prominent when processing large code bases. The present invention introduces a hierarchical parsing method to decompose the abstract syntax tree (AST) into manageable layers, so that the processing of each layer can be carried out in parallel, thereby significantly improving the overall processing efficiency.
[0054] Many methods ignore the order and structure information inherent in the code writing process. The order of the code usually reflects the execution flow of the program. Usually, a piece of code will first contain a function declaration, followed by the statements in the function body. This order information is crucial to the understanding of the code. This observation provides important inspiration for our Hierarchical-AST method.
[0055] When processing large and deeply nested ASTs, AST-based neural models often destroy the original grammatical structure of the source code. The hierarchical parsing method proposed in this invention maintains the original structure of the AST, and each layer only contains nodes of a specific type, thereby more realistically reflecting the grammatical structure of the source code and enhancing the model's ability to capture complex semantics.
[0056] For sentences or images with clear structural rules, it is easy to give positional embeddings based on the relative positions of the rules. However, AST syntax trees have different structural characteristics, and positional embeddings commonly used for image, video, and natural language processing cannot be directly applied to AST syntax trees. In this case, the tree-relative positional embedding we proposed can more accurately capture the hierarchical and sequential relationships between nodes, and capture the relationships between nodes by introducing a node matrix.
[0057] like Figure 1-Figure 2 As shown, a code clone detection method based on hierarchical analysis includes steps S1 to S6.
[0058] S1, obtain two source code fragments to be cloned and tested;
[0059] S2, respectively parse the two source code fragments and convert them into corresponding abstract syntax trees AST; wherein each node of the abstract syntax tree AST represents a syntax element of the source code.
[0060] In this embodiment, the source code fragment is input into the parser tree_sitter to generate the corresponding AST. Each node represents a grammatical element in the code (such as function, variable, control flow, etc.). Of course, other parsers such as ANTLR, oslyn, Babel, etc. can also be used, which is not limited here.
[0061] S3, performing hierarchical extraction on the abstract syntax trees AST corresponding to the two source code fragments, respectively, decomposing them into manageable layers, and obtaining corresponding hierarchical ASTs.
[0062] In the process of hierarchical parsing of AST (abstract syntax tree) in another preferred embodiment, we first start with the function method name. This step helps the model understand the overall function of the code. Then, the declaration of each local variable and the main control flow structure, such as loop statements and conditional statements, are gradually extracted. These extraction steps ensure that the control flow and logical structure of the code are parsed layer by layer. By identifying and decomposing these grammatical elements, the model can accurately understand the operation sequence and data processing process of the code. Finally, extracting key nodes such as return statements can help the model grasp the final output and purpose of the code. This hierarchical parsing method allows the model to obtain structural and semantic information in a gradually refined manner when processing complex code, thereby improving the accuracy of understanding.
[0063] The “model” mentioned in this embodiment refers to the model corresponding to the method of the present invention.
[0064] In another preferred embodiment, Figure 3The example of layered AST extraction of one of the code snippets shown in the figure. We first start with the function definition node and extract the function definition (Method Declaration, line 1), because the method name plays a vital role in understanding the function's function, which can help the model understand that this is a bubble sort function at the first time. Next, we extracted the first local variable declaration (Local VariableDeclaration, line 2) in the function, which is the key variable for controlling the loop. Then we extracted the entire while loop structure (whileStatement, lines 3-13), which is the core of the function. Then, we understand the internal structure of the function layer by layer. In the loop structure, we separately extracted the assignment operation of the variable f (line 4) to help the model understand the control flow of the loop. Next, we extracted the for loop (For Statement, lines 5-12), which is the main part of the sorting logic, including the comparison and exchange of array elements. Subsequently, we extracted the if statement (If Statement, lines 6-11) from the for statement module and extracted the local variable declarations (local variable declaration) in the if statement module one by one. Finally, the function’s return statement (Return Statement, line 14) is extracted to indicate the function’s output, thereby helping the model better understand the function’s overall logic and purpose.
[0065] S4, traverse each node of the corresponding hierarchical AST, calculate the relative distance between each node and another node in turn, obtain the relative position embedding matrix, and integrate the relative position embedding matrix into the node representation corresponding to the hierarchical AST to obtain the relative position hierarchical AST; wherein, the relative position embedding matrix records the inter-layer node position information and intra-layer node position information of each node in the tree structure; each node of the relative position hierarchical AST contains the content information of the node itself and the relative position embedding information of the node.
[0066] In another preferred embodiment, further, for each path in the tree structure (a path from a root node to a leaf node or from one node to another node), it is necessary to traverse all pairs of nodes in the path and calculate the relative distances between these nodes. This relative distance is calculated based on the order in which the nodes appear in the path. The relative distance expression is:
[0067] = ;
[0068] in, Represents a node in a hierarchical AST To Node The distance between nodes is , i and j represent the index of the node in the path.
[0069] After determining the relative position measurement method, you need to initialize a relative position embedding matrix. Assuming the maximum possible relative distance is max_distance, the size of the relative position embedding matrix should be (2 * max_distance +1) * embedding_dim, where embedding_dim is the dimension of the embedding matrix.
[0070] Then, for each node pair in the hierarchical AST, their relative distance is found, and each vector of the relative position embedding matrix represents the relative position between the node pairs, that is, the relative position embedding vector. The relative position embedding vector is fused with the content information of the node itself by addition or concatenation to integrate the relative position embedding information into the node representation corresponding to the hierarchical AST.
[0071] This integration can help the model understand the structural relationship between nodes in the AST, especially when capturing complex dependencies between nodes such as parent-child and sibling nodes, relative position embedding is particularly important.
[0072] like Figure 4 The relative position embedding matrix structure of the if_statement module shown in the figure. The relative position embedding vector records the relative position relationship between AST nodes, and provides the position information of the nodes between layers and the position information of the nodes within the layer of the AST. The relative position embedding vector is stored in the relative position embedding matrix P, which contains the position relationship between each node, helping the model to better understand the structural information of the abstract syntax tree.
[0073] Specifically, for the layered AST syntax tree, the relative position embedding matrix of the tree actually provides two key position information: on the one hand, it captures the relative position relationship between different levels, thereby reflecting the global level of the node in the entire tree structure; on the other hand, it records the relative position relationship between nodes in the same level, thereby describing the interdependence of the local structure in detail. These two position relationships record the level and relative position of the node in the tree structure, enabling the model to more comprehensively process the complex structure of the AST.
[0074] S5, according to the relative position hierarchical AST, use the decoupled attention mechanism to process each node of the relative position hierarchical AST, obtain the content vector and position embedding vector of each node, and calculate the attention weights of the content vector to the content vector, the content vector to the position embedding vector, and the position embedding vector to the content vector, so as to obtain the attention matrix of the relative position hierarchical AST.
[0075] When processing AST, a decoupled attention mechanism is applied to process the content information and position information of the node separately through independent attention heads, enabling the model to learn the complex relationship between content and position in a more fine-grained manner.
[0076] At this time, each node of AST is represented as two independent vectors, a content vector and a position embedding vector. The content vector is used to record the content information of the node in the hierarchical AST; the position embedding vector is used to record the relative position information of the node in the hierarchical AST.
[0077] In another preferred embodiment, the attention matrix of the hierarchical AST is obtained by respectively calculating the dot product of the content vectors of two nodes, the dot product of the content vector and the position embedding vector, and the dot product of the position embedding vector and the content vector. The calculation formula is:
[0078] = + ;
[0079] in, Represents the attention weight of the i-th node and the j-th node, that is, the value of the i-th row and j-th column in the attention matrix; and Represent the content vectors of the i-th node and the j-th node respectively; The embedding vector representing the relative position of the i-th node and the j-th node; represents the embedding vector of the relative position of the j-th node to the i-th node; T represents the transposed matrix.
[0080] Other attention weight calculation methods can also be used to separate the content and relative position of the node, so that the model can more accurately capture the dependencies between nodes in the syntax tree. The grammatical relationship between nodes may depend not only on their content, but also on their relative positions in the syntax tree.
[0081] After calculating the attention weights of all nodes, the attention matrix of the entire syntax tree is obtained.
[0082] S6, inputting the relative position hierarchical AST and the corresponding attention matrix of the two source code fragments into the Siamese network for clone detection, calculating the similarity, and obtaining the clone detection results of the two source code fragments according to the similarity.
[0083] In this embodiment, the Siamese network is a special neural network structure, which consists of two identical sub-networks that share the same weights and parameters. After the input data is processed by the two sub-networks respectively, the output features are used to calculate the similarity or distance between the two. Among them, the sub-network can use two identical CNNs, RNNs or more complex networks (such as Transformer-based models) as sub-networks.
[0084] Clone detection outputs the input data as two vectors, and the similarity is obtained by calculating the distance between the two vectors to measure the semantic relevance of the input data.
[0085] When judging the clone detection result, when the similarity is greater than a set threshold, the two source code fragments are determined to be a pair of clone codes, otherwise the two source code fragments do not belong to a pair of clone codes. The threshold here can be set according to actual conditions, such as the threshold is set to 0.5, when the calculated similarity value is greater than 0.5, it is determined that the two input source codes are a pair of clones.
[0086] In summary, compared with the prior art, the present invention has the following beneficial effects:
[0087] (1) The present invention parses AST layer by layer, starting from the top-level node (such as function definition), and gradually goes deeper into local variable declarations, control structures (such as loops, conditional statements), and finally return statements. This process enables the model to systematically decompose complex code logic and understand the overall goal of the code and the functions of each component. The specific steps are:
[0088] Starting from function definition → extracting local variable declarations → parsing control flow structures (such as loops, conditional statements) → identifying key nodes (such as return statements), this series of steps ensures that the model will not be overwhelmed by the huge structure when processing complex code, and can understand the role of each part layer by layer.
[0089] (2) By decomposing complex syntax trees, each level of parsing helps the model focus on understanding the syntax and logic of the current level, rather than trying to Figure 1 This layered approach reduces computational complexity, avoids misunderstanding or improper processing of large or complex code, and captures more fine-grained dependencies.
[0090] The specific steps are: gradually parse each layer in the syntax tree → focus on processing local structures and variables → process content and control structures separately to ensure that each part is accurately understood.
[0091] (3) In the traditional attention mechanism, content and position information are mixed in a single vector for processing, which may cause confusion in the model when capturing the grammatical relationship between words in a sentence, especially when semantically similar words appear in different positions. The present invention decouples the attention mechanism, represents content information and position information separately, and calculates their attention weights separately. This enables the model to process content and position independently, avoiding the problem of information confusion.
[0092] (4) The combination of the hierarchical parsing method, relative position embedding matrix and decoupled attention mechanism proposed in the present invention can provide an effective solution to the shortcomings of the existing technology in code clone detection. This not only improves the processing efficiency and solves the gradient vanishing problem, but also enhances the interpretability of the model, maintains the integrity of the grammatical structure, and reduces the computational overhead, making the application of the present invention in large-scale code clone detection more efficient and accurate, providing strong support for software maintenance and optimization.
[0093] Embodiment 2
[0094] like Figure 5 As shown, the second embodiment of the present invention further provides a code clone detection device based on hierarchical analysis, comprising:
[0095] An acquisition unit, used for acquiring two source code fragments to be cloned and tested;
[0096] An AST parsing unit, used to parse the two source code fragments and convert them into corresponding abstract syntax trees AST; wherein each node of the abstract syntax tree AST represents a syntax element of the source code;
[0097] An AST hierarchical unit, used to perform hierarchical extraction on the abstract syntax trees AST corresponding to the two source code fragments, decompose them into manageable layers, and obtain corresponding hierarchical ASTs;
[0098] A relative position embedding unit is used to traverse each node of the corresponding hierarchical AST, calculate the relative distance between each node and another node in turn, obtain a relative position embedding matrix, and integrate the relative position embedding matrix into the node representation corresponding to the hierarchical AST to obtain a relative position hierarchical AST; wherein the relative position embedding matrix records the inter-layer node position information and intra-layer node position information of each node in the tree structure; each node of the relative position hierarchical AST contains the content information of the node itself and the relative position embedding information of the node;
[0099] The attention calculation unit is used to process each node of the relative position hierarchical AST using the decoupled attention mechanism according to the relative position hierarchical AST, obtain the content vector and position embedding vector of each node, and respectively calculate the attention weights of the content vector to the content vector, the content vector to the position embedding vector, and the position embedding vector to the content vector, so as to obtain the attention matrix of the relative position hierarchical AST;
[0100] The clone detection unit is used to input the relative position hierarchical AST and the corresponding attention matrix of the two source code fragments into the Siamese network for clone detection, calculate the similarity, and obtain the clone detection results of the two source code fragments according to the similarity.
[0101] Embodiment 3
[0102] The third embodiment of the present invention further provides a code clone detection device based on hierarchical parsing, which includes a memory and a processor. The memory stores a computer program, and the computer program can be executed by the processor to implement the code clone detection method based on hierarchical parsing as described above.
[0103] Embodiment 4
[0104] The fourth embodiment of the present invention further provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the code clone detection method based on hierarchical parsing as described above is implemented.
[0105] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0106] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0107] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.
[0108] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0109] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0110] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0111] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.
[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A code clone detection method based on hierarchical analysis, characterized in that: include: S1, obtain two source code fragments to be cloned and tested; S2, respectively parse the two source code fragments and convert them into corresponding abstract syntax trees AST; wherein each node of the abstract syntax tree AST represents a syntax element of the source code; S3, performing hierarchical extraction on the abstract syntax trees AST corresponding to the two source code fragments, respectively, decomposing them into manageable layers, and obtaining corresponding hierarchical ASTs; S4, traverse each node of the corresponding hierarchical AST, calculate the relative distance between each node and another node in turn, obtain a relative position embedding matrix, and integrate the relative position embedding matrix into the node representation corresponding to the hierarchical AST to obtain a relative position hierarchical AST; wherein the relative position embedding matrix records the inter-layer node position information and intra-layer node position information of each node in the tree structure; each node of the relative position hierarchical AST contains the content information of the node itself and the relative position embedding information of the node; S5, according to the relative position hierarchical AST, use the decoupled attention mechanism to process each node of the relative position hierarchical AST, obtain the content vector and position embedding vector of each node, and calculate the attention weights of the content vector to the content vector, the content vector to the position embedding vector, and the position embedding vector to the content vector, so as to obtain the attention matrix of the relative position hierarchical AST; S6, inputting the relative position hierarchical AST and the corresponding attention matrix of the two source code fragments into the Siamese network for clone detection, calculating the similarity, and obtaining the clone detection results of the two source code fragments according to the similarity.
2. A code clone detection method based on hierarchical analysis according to claim 1, characterized in that When performing hierarchical extraction on the abstract syntax tree AST, the syntax elements of the source code file are extracted in sequence according to the structure of the source code; the syntax elements include function method names, local variable declarations, control flow structures and return statements.
3. The code clone detection method based on hierarchical analysis according to claim 1 is characterized in that ,When calculating the relative distance between each node and another node in turn, the expression of the relative distance is: = ； in, Represents a node in a hierarchical AST To Node The distance between nodes is , i and j represent the index of the node in the path.
4. The code clone detection method based on hierarchical analysis according to claim 1 is characterized in that ,At each node of the relative position hierarchical AST, the content information of the node itself is fused with the relative position embedding information of the node to integrate the relative position embedding information into the node representation corresponding to the hierarchical AST; wherein the fusion includes addition or splicing.
5. The code clone detection method based on hierarchical analysis according to claim 3 is characterized in that , the content vector is used to record the content information of the node in the hierarchical AST; the position embedding vector is used to record the relative position information of the node in the hierarchical AST; the attention matrix of the hierarchical AST is obtained by respectively calculating the dot product of the content vectors of two nodes, the dot product of the content vector and the position embedding vector, and the dot product of the position embedding vector and the content vector. The calculation formula is: = + ； in, Represents the attention weight of the i-th node and the j-th node, that is, the value of the i-th row and j-th column in the attention matrix; and Represent the content vectors of the i-th node and the j-th node respectively; The embedding vector representing the relative position of the i-th node and the j-th node; represents the embedding vector of the relative position of the j-th node to the i-th node; T represents the transposed matrix.
6. The code clone detection method based on hierarchical analysis according to claim 1 is characterized in that ,When judging the clone detection result, when the similarity is greater than the set threshold, the two ,source code fragments are determined to be a pair of clone codes.
7. The code clone detection method based on hierarchical analysis according to claim 1 is characterized in that ,The inter-layer node position information refers to the relative position relationship between different levels in the entire tree structure, reflecting the global hierarchy of the node in the entire tree structure; ,the intra-layer node position information refers to the relative position relationship between the nodes in the same level in the tree structure, reflecting the interdependence of the local structure.
8. A code clone detection device based on hierarchical analysis, characterized in that: include: An acquisition unit, used for acquiring two source code fragments to be cloned and tested; An AST parsing unit, used to parse the two source code fragments and convert them into corresponding abstract syntax trees AST; wherein each node of the abstract syntax tree AST represents a syntax element of the source code; An AST hierarchical unit, used to perform hierarchical extraction on the abstract syntax trees AST corresponding to the two source code fragments, decompose them into manageable layers, and obtain corresponding hierarchical ASTs; A relative position embedding unit is used to traverse each node of the corresponding hierarchical AST, calculate the relative distance between each node and another node in turn, obtain a relative position embedding matrix, and integrate the relative position embedding matrix into the node representation corresponding to the hierarchical AST to obtain a relative position hierarchical AST; wherein the relative position embedding matrix records the inter-layer node position information and intra-layer node position information of each node in the tree structure; each node of the relative position hierarchical AST contains the content information of the node itself and the relative position embedding information of the node; The attention calculation unit is used to process each node of the relative position hierarchical AST using the decoupled attention mechanism according to the relative position hierarchical AST, obtain the content vector and position embedding vector of each node, and respectively calculate the attention weights of the content vector to the content vector, the content vector to the position embedding vector, and the position embedding vector to the content vector, so as to obtain the attention matrix of the relative position hierarchical AST; The clone detection unit is used to input the relative position hierarchical AST and the corresponding attention matrix of the two source code fragments into the Siamese network for clone detection, calculate the similarity, and obtain the clone detection results of the two source code fragments according to the similarity.
9. A code clone detection device based on hierarchical analysis, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program can be executed by the processor to implement a code clone detection method based on hierarchical parsing as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the code clone detection method based on hierarchical parsing as described in any one of claims 1 to 7 is implemented.