Code clone detection methods, devices, equipment, storage media, and program products

Abstract

This application discloses a code clone detection method, apparatus, device, storage medium, and program product, belonging to the field of computer technology. The method includes: acquiring the syntactic and semantic information of code source files; fusing the syntactic and semantic information to obtain code fusion information; extracting features from the code fusion information to obtain feature vectors of the code source files; and comparing the feature vectors of each code source file to obtain clone detection results. Clone detection based on the syntactic and semantic analysis results of the source code can effectively identify the impact of minor syntactic changes on similarity and effectively identify homogeneous but heterogeneous code fragments, thus helping to improve the detection accuracy of various types of cloned code.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a code clone detection method, apparatus, device, storage medium, and program product. Background Technology

[0002] Code clone detection is one of the most fundamental software engineering techniques, primarily applied to code reuse, security, and software engineering performance management. Code clones can be broadly categorized into four types: Type 1: Except for spaces, line breaks, and comments, the code content is completely identical; Type 2: Except for renaming variables, methods, and class names, the code content is completely identical; Type 3: Code snippets have partial modifications, or the order of statements or modules has been adjusted; Type 4: Code snippets are inconsistent, but their semantics and functionalities are consistent.

[0003] Code clone detection in related technologies mainly involves simple preprocessing of the code text and overall hashing to calculate similarity. However, the above methods only support the first two types of clone detection, have low detection accuracy, weak generalization ability, and are prone to false positives or false negatives. Summary of the Invention

[0004] This application provides a code clone detection method, apparatus, device, storage medium, and program product, which helps improve the detection accuracy of various types of cloned code. The technical solution is as follows:

[0005] On the one hand, this application provides a code clone detection method, the method comprising:

[0006] Obtain the syntax and semantic information of the source code file, wherein the syntax information is used to indicate the syntax structure of the code and the semantic information is used to indicate the meaning of the code;

[0007] The syntactic information and the semantic information are fused to obtain code fusion information;

[0008] Feature extraction is performed on the code fusion information to obtain the feature vector of the code source file;

[0009] Based on the feature vectors of each source code file, feature comparison is performed to obtain clone detection results, which are used to indicate the similarity between the source code files.

[0010] On the other hand, this application provides a code clone detection device, the device comprising:

[0011] The acquisition module is used to acquire the syntax information and semantic information of the code source file. The syntax information is used to indicate the syntax structure of the code, and the semantic information is used to indicate the meaning of the code.

[0012] An information fusion module is used to fuse the syntactic information and the semantic information to obtain code fusion information;

[0013] The feature extraction module is used to extract features from the code fusion information to obtain the feature vector of the code source file;

[0014] The similarity evaluation module is used to perform feature comparison based on the feature vectors of each code source file to obtain clone detection results, which are used to indicate the similarity between the code source files.

[0015] On the other hand, this application provides a computer device including a processor and a memory; the memory stores at least one program, which is loaded and executed by the processor to implement the code clone detection method as described above.

[0016] On the other hand, this application provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the code clone detection method as described above.

[0017] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the code clone detection method provided in various alternative implementations of the above aspect.

[0018] The technical solutions provided in this application have at least the following beneficial effects:

[0019] In this embodiment, the syntactic and semantic information of the source code file is fused to obtain the fused features of the source code file. Then, feature comparison is performed based on the fused features of the source code file to be detected to obtain the similarity score. By fully considering both syntactic and semantic features in clone detection, the impact of minor syntactic changes on similarity can be effectively identified, and homogeneous yet heterogeneous code fragments can be effectively recognized, thus improving the detection accuracy of various types of cloned code. Attached Figure Description

[0020] Figure 1 This is a flowchart of a code clone detection method provided in an exemplary embodiment of this application;

[0021] Figure 2This is a flowchart of a code clone detection method provided in another exemplary embodiment of this application;

[0022] Figure 3 This is a schematic diagram of code fusion information provided in an exemplary embodiment of this application;

[0023] Figure 4 This is a framework diagram of a code clone detection method provided in an exemplary embodiment of this application;

[0024] Figure 5 This is a flowchart of a code clone detection method provided in another exemplary embodiment of this application;

[0025] Figure 6 This is a schematic diagram of a similarity calculation process provided in an exemplary embodiment of this application;

[0026] Figure 7 This is a structural block diagram of a code clone detection device provided in an exemplary embodiment of this application;

[0027] Figure 8 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0029] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0030] First, a brief introduction to the terms used in the embodiments of this application:

[0031] Abstract Syntax Tree (AST): A structured tree that is abstracted from source code based on programming language keywords, syntax rules, etc.

[0032] Control Flow Graph (CFG): A structured graph formed in code functions and methods based on jump statements (IF, ELSE, SWITCH, GOTO, etc.).

[0033] Call Graph (CG): A graph showing the call relationships between functions and methods.

[0034] Dataflow (DL): A graph showing the activity of global variables, local variables, and static variables in the code during the program's lifetime.

[0035] Program Dependency Graph (PDG): Composed of a control flow graph and a data flow graph, it is divided into data dependencies and control dependencies.

[0036] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0037] Clone detection technologies based on code analysis can be mainly divided into the following three categories:

[0038] 1. Text-based methods treat the source code as a sequence of strings and detect errors by matching the longest repeating string sequence, without transforming the source code. This method is relatively effective in handling Type 1 problems and can also handle Type 2 and Type 3 problems to a large extent.

[0039] 2. Token-based methods utilize lexical and syntactic analysis to transform source code into a token sequence. The source code is then analyzed by matching the longest repeating subsequence. This method effectively solves Type 1 and Type 2 problems, but not Type 3 problems.

[0040] 3. Metric-based methods calculate metrics and contrast values ​​for statements within functions or classes to reflect differences between code segments. These methods can also resolve Type 1, Type 2, and Type 4 problems, but not Type 3 problems.

[0041] To address the problems existing in the aforementioned related technologies, this application provides a code clone detection method capable of detecting the four cloning scenarios described above. The application scenarios of the code clone detection method provided in this application are illustrated below.

[0042] 1. Assist engineers in monitoring code duplication rate:

[0043] The code clone detection method provided in this application can be integrated into a fast, efficient, and high-precision code duplication rate detection tool. For example, by embedding it into the continuous integration pipeline of Development and Operations, it can remind engineers whether code changes follow the reuse principle when code is merged; during software architecture review and upgrade, it can provide engineers with a duplication rate report of the entire system to assist architects in making architectural design decisions; in addition, it can also assist researchers in checking for code duplication submitted by others.

[0044] 2. Assist developers in managing large warehouse operations:

[0045] In the centralized hosting model, various business lines are centrally managed. For a large number of similar or repetitive components such as gateways, configuration files, databases, and middleware within these business lines, designing and developing code clone detection tools has significant engineering value in assisting in the convergence and unified maintenance of common libraries, components, and utility classes. By using the code clone detection method provided in this application to detect code clones in the basic code modules of various businesses within the centralized hosting system, the common basic code modules of various businesses migrated to the centralized hosting system can be extracted for unified governance and maintenance.

[0046] Please refer to Figure 1 This document illustrates a flowchart of a code clone detection method provided in an exemplary embodiment of this application. This embodiment describes the method as being executed by a computer device capable of compiling code. The method includes the following steps:

[0047] Step 101: Obtain the syntax information and semantic information of the source code file. The syntax information is used to indicate the syntax structure of the code, and the semantic information is used to indicate the meaning of the code.

[0048] Syntactic information in code represents specific code statements, such as keywords, variable names, and symbols. Semantic information, on the other hand, represents the meaning of the code, that is, the function it performs.

[0049] In one possible implementation, the computer device acquires two or more source code files to be tested, and for each source code file, acquires its syntax information and semantic information.

[0050] To illustrate, the computer device identifies source code file a and source code file b to be detected based on the clone detection operation in the clone detection interface. The computer device acquires the syntax and semantic information of source code file a, as well as the syntax and semantic information of source code file b.

[0051] Step 102: Perform information fusion on syntactic and semantic information to obtain code fusion information.

[0052] The computer equipment performs information fusion on each source code file to be tested, obtaining fused code information for each source code file. This fused code information incorporates both syntactic and semantic features of the code.

[0053] In a schematic representation, a computer device fuses the syntactic and semantic information of a source code file to obtain a tree structure that reflects both syntax and semantics. Nodes in this tree structure contain information such as keywords, variable names, and statement meanings, and also include edges that reflect the flow of data or code control.

[0054] Step 103: Extract features from the code fusion information to obtain the feature vector of the code source file.

[0055] In one possible implementation, the computer device performs feature extraction based on code fusion information to obtain the feature vector corresponding to the code source file. Illustratively, the feature vector of the code source file is the set of feature vectors of each node in the aforementioned tree structure, and the feature vector of the code source file contains the semantic relationships between the nodes.

[0056] Step 104: Based on the feature vectors of each source code file, feature comparison is performed to obtain clone detection results. The clone detection results are used to indicate the similarity between source code files.

[0057] Computer devices calculate similarity based on feature vectors of source code files, and this similarity represents the similarity between the source code files. In one possible implementation, the clone detection results output by the computer device include the similarity between source code files, as well as the similar code snippets involved, for user reference.

[0058] As an illustration, the computer device hashes the feature vectors of the source code files to obtain the hash value corresponding to the feature vector of each source code file. Then, similarity is calculated based on the hash value to obtain the clone detection result.

[0059] In summary, in this embodiment, the syntactic and semantic information of the source code file is fused to obtain the fused features of the source code file. Then, feature comparison is performed based on the fused features of the source code file to be detected to obtain the similarity score. By fully considering both syntactic and semantic features in clone detection, the impact of minor syntactic changes on similarity scores can be effectively identified, as well as homogeneous but heterogeneous code fragments can be effectively recognized, thus improving the detection accuracy of various types of cloned code.

[0060] In one possible implementation, the computer device is based on, but is not limited to, abstract syntax trees, function call graphs, control flow graphs, and data flows. Figure 4 Information is fused from various sources. The abstract syntax tree reflects the code syntax, while the function call graph, control flow graph, and data flow graph reflect the code semantics.

[0061] Please refer to Figure 2 This embodiment illustrates a flowchart of a code clone detection method provided in another exemplary embodiment of this application. The method is described using an example where the method is executed by a computer device capable of compiling code. The method includes the following steps:

[0062] Step 201: Obtain the syntax information and semantic information of the source code file. The syntax information is used to indicate the syntax structure of the code, and the semantic information is used to indicate the meaning of the code.

[0063] In one possible implementation, the syntactic information includes a method-level abstract syntax tree (ABAT), and the semantic information includes a function call graph, a control flow graph, and a data flow graph. The method-level ABAT is a structured tree abstracted from the source code based on programming language keywords and syntax rules. The function call graph is used to represent the call relationships between functions. Typically, one function corresponds to one (or more) method-level ABATs.

[0064] Optionally, the computer device can use a code compiler or open-source tools to generate a method-level abstract syntax tree for the code source file, as well as construct the function call graph, control flow graph, and data flow graph corresponding to the code source file.

[0065] Step 202: Integrate the information from the method-level abstract syntax tree and the function call graph to obtain the function-level abstract syntax tree.

[0066] The computer device directly compiles only the method-level abstract syntax tree. After fusing information with the function call graph, a function-level abstract syntax tree is obtained. This function-level abstract syntax tree is still a tree that reflects the syntactic structure of the source code, but it can also reflect the semantic information of the calling relationships between functions.

[0067] In one possible implementation, step 202 specifically includes the following steps 202a to 202b:

[0068] Step 202a: Normalize the functions in the source code file based on the function call graph to obtain the target function. The target function is an independent function that has no call relationship with other functions, or a set of at least two functions that have a call relationship.

[0069] In one possible implementation, the computer device extracts function calls from the source code file and connects functions with call relationships to form a new function, i.e., the target function. For functions without call relationships, the computer device identifies the function itself as a target function. Therefore, functions with call relationships constitute a new function node.

[0070] For example, a source code file contains functions a, b, c, and d, where functions a, c, and d have call relationships. A computer device normalizes this function call graph to obtain target function A and target function B, where target function A includes functions a, c, and d, and target function B is function b.

[0071] Step 202b: Connect the method-level abstract syntax trees corresponding to each code segment based on the objective function to obtain the function-level abstract syntax tree.

[0072] After normalizing the initial function, the computer device connects the method-level abstract syntax trees into a function-level abstract syntax tree based on the target function. For example, in step 202a above, the computer device connects the method-level abstract syntax trees of functions a, c, and d based on the call relationships between them. This integrates syntactic information and function call relationships into the abstract syntax tree.

[0073] The nodes of an abstract syntax tree contain a large number of variables, keywords, and symbols. In one possible implementation, to eliminate the influence of variable names on the clone detection process, after step 202b above, the code clone detection method provided in this application embodiment may further include the following steps 202c to 202d:

[0074] Step 202c: Obtain the initial variable names contained in the nodes of the function-level abstract syntax tree.

[0075] Step 202d: Normalize the initial variable names and replace them with the target variable names.

[0076] The target variable name consists of a variable type name and a counter number i. The variable type name is used to represent the variable type to which the variable belongs, and the counter number is used to represent the i-th variable belonging to the variable type, where i is a positive integer.

[0077] Changing variable names does not affect the actual content of the code; what truly impacts the meaning are the variable types and their assigned values. However, different variable names can affect the final similarity calculation results. Therefore, computer systems normalize variable names in source code files, replacing custom names with variable types, thus preventing inaccurate clone detection results caused by changes in variable names.

[0078] To illustrate, computer devices categorize variable types into five classes, including but not limited to: Integer (INT), Float (FLOAT), Character (CHAR), String (STRING), and Boolean (BOOL). Variable names are then composed of the variable type and a counter digit. For example, for the first integer variable 'a' in the code source file, the computer device replaces its name with "INT1" and performs the same replacement on all instances of 'a' throughout the file.

[0079] In another possible implementation, the computer device may also normalize variable names before performing information fusion on the method-level abstract syntax tree.

[0080] Optionally, before performing information fusion on the method-level abstract syntax tree, the computer device also removes spaces, comments, and symbols (such as semicolons) from the code to reduce the impact of these non-substantive changes on the clone detection results.

[0081] Step 203: Integrate the information from the function-level abstract syntax tree, control flow graph, and data flow graph to obtain code fusion information.

[0082] Function-level abstract syntax trees (ASTs) contain syntactic information and function call information from the source code, representing another form of code. However, ASTs are often quite large, making direct feature extraction and analysis difficult. Therefore, computer devices utilize semantic information, including but not limited to data flow and control flow, to segment and partition function-level ASTs. The control flow graph contains branch and jump information from the code, while the data flow graph contains information about the data flow direction within the code.

[0083] In one possible implementation, step 203 specifically includes the following steps 203a to 203b:

[0084] Step 203a: Insert the corresponding function-level abstract syntax tree under the statement node of the control flow graph.

[0085] The computer device determines the correspondence between statement nodes and abstract syntax trees (APIs), and connects the corresponding APIs under each statement node, thereby achieving the fusion of function-level APIs and control flow graphs. Illustratively, the computer device connects statement nodes in the control flow graph to their corresponding parent nodes in the APIs.

[0086] Step 203b: Based on the data flow graph, add edges between the statement nodes of the control flow graph to indicate the direction of data flow, thereby obtaining code fusion information.

[0087] Computer devices, based on a constructed data flow graph, add edges pointing to data flows between statement nodes in the control flow graph. This achieves the fusion of the method-level abstract syntax tree, function call graph, control flow graph, and data flow graph to obtain code fusion information.

[0088] There is no strict order of execution between steps 203a and 203b.

[0089] As an example, the following code snippet is given:

[0090]

[0091] The computer device performs information fusion on the above code snippet, resulting in code fusion information, namely a tree diagram that integrates syntax nodes, function call relationships, control flow nodes, data flow direction, and control flow direction, as shown below. Figure 3 As shown in the diagram. The nodes "Entry," "Declaration," and "Judgment" are statement nodes in the control flow graph, corresponding to the statements in the code above. For example, "Entry" corresponds to the statement "public static void main(String[]args)". The tree diagram connected below the "Declaration," "Judgment," and "Assignment" control flow nodes is a function-level abstract syntax graph, where nodes represent keywords, variables, assignments, symbols, etc. The statement nodes in the control flow graph are connected by edges indicating the direction of control flow (solid lines with arrows between statement nodes in the diagram). For example, after the "Judgment" node, there are two branches: one pointing to the first assignment node, and the other pointing to the second assignment node. Furthermore, the statement nodes in the control flow graph are also connected by edges indicating the direction of data flow (dashed lines with arrows between statement nodes in the diagram). For example, the statement node used to declare the assignment of variable 'a' points to the judgment node and the first assignment node. Figure 3 The fusion diagram shown reflects both syntactic and semantic information.

[0092] Step 204: Extract features from the code fusion information to obtain the feature vector of the code source file.

[0093] Step 205: Based on the feature vectors of each source code file, feature comparison is performed to obtain clone detection results. The clone detection results are used to indicate the similarity between source code files.

[0094] The specific implementation of steps 204 to 205 can be referred to steps 103 to 104 above, and will not be repeated here in the embodiments of this application.

[0095] Some related technologies utilize tree structures for code clone detection, such as abstract syntax trees. However, these methods detect clones by searching similar subtrees within the complete tree structure, which can only identify clones with identical code snippets. They may produce false positives for clones with minor modifications that result in inconsistent statement or module order.

[0096] In this embodiment, the method-level abstract syntax tree is first connected based on the function call graph to obtain the function-level abstract syntax tree. Then, the function-level abstract syntax tree is partitioned and supplemented with information based on the control flow graph and data flow graph. Feature extraction and similarity calculation are performed based on the fused information. This approach can effectively identify the impact of minor syntactic changes on similarity (i.e., low-quality code duplication) and also effectively identify homogeneous but heterogeneous code fragments (i.e., semantically similar code modules). Secondly, variable names are normalized by using variable types and count numbers to replace custom variable names, thereby reducing the impact of variable name changes on the detection results.

[0097] In conjunction with the above embodiments, Figure 4 The diagram illustrates the framework of the code clone detection method. The computer device performs the following operations on source code files m1 and m2: generating a method-level abstract syntax tree, a function call graph, a control flow graph, and a data flow graph; concatenating the method-level abstract syntax tree based on the function call graph to obtain a function-level abstract syntax tree; pruning the function-level abstract syntax tree based on the control flow graph and data flow graph to obtain a simplified abstract syntax tree; and extracting features from the simplified abstract syntax tree to obtain function features (i.e., feature vectors of the source code files). Next, the computer device calculates the feature similarity between source code files m1 and m2, obtaining a similarity score and similar lines in the code.

[0098] In one possible implementation, feature extraction of the source code file is actually feature extraction of each node in the abstract syntax tree, that is, the feature vector of the source code file is a set of node feature vectors.

[0099] Please refer to Figure 5 This embodiment illustrates a flowchart of a code clone detection method provided in another exemplary embodiment of this application. The method is described using an example where the method is executed by a computer device capable of compiling code. The method includes the following steps:

[0100] Step 501: Obtain the syntax information and semantic information of the source code file. The syntax information is used to indicate the syntax structure of the code, and the semantic information is used to indicate the meaning of the code.

[0101] Step 502: Information fusion is performed on the syntactic information and semantic information to obtain code fusion information.

[0102] The specific implementation of steps 501 to 502 can be referred to steps 101 to 102 above, and will not be repeated here in the embodiments of this application.

[0103] Step 503: Extract features from the syntax nodes in the fused abstract syntax tree to obtain node feature vectors.

[0104] In one possible implementation, the computer device extracts features from the non-keyword nodes (usually leaf nodes) in the final fused abstract syntax tree, obtaining node feature vectors corresponding to each non-keyword node. By extracting features of the keyword nodes themselves, the relationship features between non-keyword nodes and their parent nodes, the relationship features between the parent nodes of non-keyword nodes and the parent nodes of other non-keyword nodes, and the relationship features between adjacent non-keyword nodes, feature extraction of the entire code source file is achieved.

[0105] Specifically, step 503 includes the following step 503a:

[0106] Step 503a: Extract features from the syntax nodes based on at least one of node features, source features, sibling features, variable features, and data flow features to obtain node feature vectors.

[0107] Among them, node features represent grammatical information, source features represent the parent node to which the grammatical node belongs and the corresponding branch order, sibling features represent adjacent nodes, variable features represent the calling position of variables in non-keyword nodes in functions, and data flow features represent the data flow pointing relationship and the data flow being pointed to relationship corresponding to the control flow node (i.e., statement node) where the grammatical node is located.

[0108] Step 504: Determine the union of the feature vectors of the nodes in the fused abstract syntax tree to obtain the feature vectors of the source code file.

[0109] If a computer device represents a non-key child node based on the above five features, then the features of an AST tree can be represented as the pathogenesis of the features of all non-key nodes of the AST tree, which is F(m), where m represents the AST tree.

[0110] Step 505: Input the feature vectors of each source code file into the similarity algorithm to obtain the similarity score and the set of similar nodes output by the similarity algorithm.

[0111] In one possible implementation, the computer device calculates a similarity score between feature vectors using a similarity algorithm, which serves as the similarity score for the source code files. This set of similar nodes contains similar syntactic nodes. Illustratively, assuming two AST trees m1 and m2 exist, the computer device calculates the similarity between the two feature vectors as follows: Figure 6 As shown.

[0112] Step 506: Based on the mapping relationship between syntax nodes and code snippets, and the set of similar nodes, determine the similar code snippets between code source files.

[0113] like Figure 6 As shown, set R represents a set of similar non-keyword nodes. Computer devices determine similar code segments based on the correspondence between nodes and code segments, using similar nodes as a reverse framework.

[0114] Step 507: Highlight similar code snippets in the source code file through the results display interface.

[0115] In one possible implementation, the computer device displays the clone detection results through a result display interface of the code clone detection tool. This interface displays a similarity score and the source code that participated in the detection. Based on the identified similar code snippets, the computer device highlights these snippets in the source code within the result display interface. For example, similar lines of code in two source files might be highlighted with the same background color to indicate the location of the cloned code snippets to the user.

[0116] In this embodiment, after obtaining the fused abstract syntax tree, features are extracted from each non-keyword node from five aspects: the node itself, the node's source, adjacent nodes, variable call locations, and data flow directions. The union of these features yields the feature vector of the entire source code file, improving feature extraction efficiency and comprehensiveness, thereby enhancing the accuracy of clone detection results. Furthermore, similar code snippets are highlighted to facilitate quick code clone confirmation by the user.

[0117] Figure 7 This is a structural block diagram of a code clone detection device provided in an exemplary embodiment of this application. The device includes the following structure:

[0118] The acquisition module 701 is used to acquire the syntax information and semantic information of the code source file. The syntax information is used to indicate the syntax structure of the code, and the semantic information is used to indicate the meaning of the code.

[0119] Information fusion module 702 is used to fuse the syntactic information and the semantic information to obtain code fusion information;

[0120] Feature extraction module 703 is used to extract features from the code fusion information to obtain the feature vector of the code source file;

[0121] The similarity evaluation module 704 is used to perform feature comparison based on the feature vectors of each code source file to obtain clone detection results, which are used to indicate the similarity between the code source files.

[0122] Optionally, the syntactic information includes a method-level abstract syntax tree, and the semantic information includes a function call graph, a control flow graph, and a data flow graph;

[0123] The information fusion module 702 is also used for:

[0124] Information fusion is performed on the method-level abstract syntax tree and the function call graph to obtain a function-level abstract syntax tree;

[0125] Information is fused from the function-level abstract syntax tree, the control flow graph, and the data flow graph to obtain the code fusion information.

[0126] Optionally, the information fusion module 702 is further configured to:

[0127] Based on the function call graph, the functions in the source code file are normalized to obtain the target function. The target function is an independent function that has no call relationship with other functions, or a set of at least two functions that have a call relationship.

[0128] Based on the objective function, the method-level abstract syntax trees corresponding to each code segment are connected to obtain the function-level abstract syntax tree.

[0129] Optionally, the acquisition module 701 is further configured to:

[0130] Obtain the initial variable names contained in the nodes of the function-level abstract syntax tree;

[0131] The initial variable name is normalized and replaced with the target variable name. The target variable name consists of a variable type name and a count number i. The variable type name is used to represent the variable type to which the variable belongs, and the count number is used to represent the i-th variable belonging to the variable type, where i is a positive integer.

[0132] Optionally, the information fusion module 702 is further configured to:

[0133] Insert the corresponding function-level abstract syntax tree under the statement node of the control flow graph;

[0134] Based on the data flow graph, edges indicating the direction of data flow are added between the statement nodes in the control flow graph to obtain the code fusion information.

[0135] Optionally, the code fusion information is a fusion abstract syntax tree, the nodes in the fusion abstract syntax tree include syntax nodes and control flow nodes, and the syntax nodes are clustered based on function call relationships, and the edges in the fusion abstract syntax tree are used to indicate the direction of control flow or data flow.

[0136] The feature extraction module 703 is also used for:

[0137] Feature extraction is performed on the syntax nodes in the fused abstract syntax tree to obtain node feature vectors;

[0138] The feature vector of the source code file is obtained by determining the union of the node feature vectors in the fused abstract syntax tree.

[0139] Optionally, the feature extraction module 703 is further configured to:

[0140] The syntax node is feature extracted based on at least one of node features, source features, sibling features, variable features, and data flow features to obtain the node feature vector.

[0141] The node features represent grammatical information, the source features represent the parent node to which the grammatical node belongs and the corresponding branch order, the sibling features represent adjacent nodes, the variable features represent the calling position of variables in non-keyword nodes in functions, and the data flow features represent the data flow pointing relationship and the data flow being pointed to relationship corresponding to the control flow node to which the grammatical node is located.

[0142] Optionally, the similarity evaluation module 704 is further configured to:

[0143] The feature vectors of each source code file are input into a similarity algorithm to obtain a similarity score and a set of similar nodes, wherein the set of similar nodes contains similar syntax nodes.

[0144] Optionally, the device further includes a result display module, used for:

[0145] Based on the mapping relationship between the syntax nodes and code snippets, and the set of similar nodes, similar code snippets between the source code files are determined;

[0146] The results display interface highlights similar code snippets in the source code file.

[0147] In summary, in this embodiment, the syntactic and semantic information of the source code file is fused to obtain the fused features of the source code file. Then, feature comparison is performed based on the fused features of the source code file to be detected to obtain the similarity score. By fully considering both syntactic and semantic features in clone detection, the impact of minor syntactic changes on similarity scores can be effectively identified, as well as homogeneous but heterogeneous code fragments can be effectively recognized, thus improving the detection accuracy of various types of cloned code.

[0148] Please refer to Figure 8 The diagram illustrates the structure of a computer device provided in one embodiment of this application.

[0149] The computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including random access memory (RAM) 802 and read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. The computer device 800 also includes a basic input / output (I / O) controller 806 that facilitates information transfer between various components within the computer, and a mass storage device 807 for storing the operating system 813, application programs 814, and other program modules 815.

[0150] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 for user input, such as a mouse or keyboard. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include the input / output controller 810 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0151] The mass storage device 807 is connected to the central processing unit 801 via a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable media provide non-volatile storage for the computer device 800. That is, the mass storage device 807 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0152] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), flash memory or other solid-state storage technologies, CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 804 and mass storage device 807 described above can be collectively referred to as memory.

[0153] According to various embodiments of this application, the computer device 800 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 800 can be connected to a network 812 via a network interface unit 811 connected to the system bus 805, or the network interface unit 811 can be used to connect to other types of networks or remote computer systems (not shown).

[0154] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the code clone detection method described in the above embodiments.

[0155] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the code clone detection method provided in various alternative implementations of the above aspect.

[0156] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0157] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the source code files involved in this application were obtained under full authorization.

[0158] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A code clone detection method, characterized in that, The method includes: Obtain the syntax and semantic information of the code source file. The syntax information is used to indicate the syntax structure of the code, and the semantic information is used to indicate the meaning of the code. The syntax information includes a method-level abstract syntax tree, and the semantic information includes a function call graph, a control flow graph, and a data flow graph. The function call graph is a graph used to represent the calling relationship between functions. The control flow graph contains branch jump information in the code, and the data flow graph contains data flow information in the code. Information fusion is performed on the method-level abstract syntax tree and the function call graph to obtain a function-level abstract syntax tree, which is used to represent the source code syntax structure and the call relationship between functions. Insert the corresponding function-level abstract syntax tree under the statement node of the control flow graph; based on the data flow graph, add edges between the statement nodes of the control flow graph to indicate the data flow direction, thereby obtaining code fusion information, which is a fused abstract syntax tree. The nodes in the fused abstract syntax tree include syntax nodes and control flow nodes, and the syntax nodes are clustered based on function call relationships. The edges in the fused abstract syntax tree are used to indicate the control flow direction or the data flow direction. Based on at least one of node features, source features, sibling features, variable features, and data flow features, feature extraction is performed on the syntax nodes in the fused abstract syntax tree to obtain node feature vectors; the union of the node feature vectors in the fused abstract syntax tree is determined to obtain the feature vector of the code source file. Based on the feature vectors of each source code file, feature comparison is performed to obtain clone detection results, which are used to indicate the similarity between the source code files.

2. The method according to claim 1, characterized in that, The process of fusing information from the method-level abstract syntax tree and the function call graph to obtain a function-level abstract syntax tree includes: Based on the function call graph, the functions in the source code file are normalized to obtain the target function. The target function is an independent function that has no call relationship with other functions, or a set of at least two functions that have a call relationship. Based on the objective function, the method-level abstract syntax trees corresponding to each code segment are connected to obtain the function-level abstract syntax tree.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the initial variable names contained in the nodes of the function-level abstract syntax tree; The initial variable name is normalized and replaced with the target variable name. The target variable name consists of a variable type name and a count number i. The variable type name is used to represent the variable type to which the variable belongs, and the count number is used to represent the i-th variable belonging to the variable type, where i is a positive integer.

4. The method according to claim 1, characterized in that, in, The node feature represents the syntax information, the source feature represents the parent node to which the syntax node belongs and the corresponding branch order, the sibling feature represents adjacent nodes, the variable feature represents the calling position of the variable in the non-keyword node in the function, and the data flow feature represents the data flow pointing relationship and the data flow being pointed to relationship corresponding to the control flow node to which the syntax node is located.

5. The method according to claim 1, characterized in that, The feature comparison based on the feature vectors of each source code file to obtain clone detection results includes: The feature vectors of each source code file are input into a similarity algorithm to obtain a similarity score and a set of similar nodes, wherein the set of similar nodes contains similar syntax nodes.

6. The method according to claim 5, characterized in that, The method further includes: Based on the mapping relationship between the syntax nodes and code snippets, and the set of similar nodes, similar code snippets between the source code files are determined; The results display interface highlights similar code snippets in the source code file.

7. A code clone detection device, characterized in that, The device includes: The acquisition module is used to acquire the syntax information and semantic information of the code source file. The syntax information is used to indicate the syntax structure of the code, and the semantic information is used to indicate the meaning of the code. The syntax information includes a method-level abstract syntax tree, and the semantic information includes a function call graph, a control flow graph, and a data flow graph. The function call graph is a graph used to represent the calling relationship between functions, the control flow graph contains branch jump information in the code, and the data flow graph contains data flow information in the code. An information fusion module is used to fuse the method-level abstract syntax tree and the function call graph to obtain a function-level abstract syntax tree, which represents the source code syntax structure and the call relationship between functions; insert the corresponding function-level abstract syntax tree under the statement node of the control flow graph; based on the data flow graph, add edges between the statement nodes of the control flow graph to indicate the data flow direction to obtain code fusion information, which is a fused abstract syntax tree. The nodes in the fused abstract syntax tree include syntax nodes and control flow nodes, and the syntax nodes are clustered based on function call relationships. The edges in the fused abstract syntax tree are used to indicate the control flow direction or the data flow direction. The feature extraction module is used to extract features from the syntax nodes in the fused abstract syntax tree based on at least one of node features, source features, sibling features, variable features, and data flow features to obtain node feature vectors; and to determine the union of the node feature vectors in the fused abstract syntax tree to obtain the feature vector of the code source file. The similarity evaluation module is used to perform feature comparison based on the feature vectors of each code source file to obtain clone detection results, which are used to indicate the similarity between the code source files.

8. A computer device, characterized in that, The computer device includes a processor and a memory; the memory stores at least one program, which is loaded and executed by the processor to implement the code clone detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the code clone detection method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the code clone detection method as described in any one of claims 1 to 6.

Images