Code cloning detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a code clone detection method and device, electronic equipment and a storage medium, and belongs to the technical field of software. The method comprises the steps that for each to-be-detected function in a to-be-detected code file, a syntax element sequence of the to-be-detected function is generated based on an abstract syntax tree corresponding to the to-be-detected function; generating a feature vector and a first index of the to-be-detected function based on the syntax element sequence of the to-be-detected function; querying each code file in a code file library by using the first index to obtain at least one objective function; a first index exists in the at least one objective function; performing similarity comparison on the feature vector of the at least one target function and the feature vector of the to-be-detected function to obtain a similarity comparison result; and based on the similarity comparison result of each to-be-detected function in the to-be-detected code file, determining a code clone detection result of the to-be-detected code file. The detection efficiency of code clone detection is improved, and the accuracy of code clone detection is improved at the same time.

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to a code clone detection method, device, electronic device and storage medium. Background Art

[0002] Code cloning refers to the repeated use of the same or similar code snippets in different code locations. This phenomenon may be caused by a variety of reasons, such as code reuse, copy-and-paste programming, etc. Although code cloning can improve development efficiency to a certain extent, it also brings many problems, such as code redundancy, difficulty in maintenance, and vulnerability propagation. Therefore, code clone detection technology plays an important role in software engineering, which can help development teams identify and manage duplicate code in the code base, and improve code quality and maintenance efficiency.

[0003] In the prior art, many code clone detection methods have introduced indexes, which can significantly improve detection efficiency. However, some detection methods build indexes based on code lines or blocks. Although this will increase detection speed and reduce memory requirements, too large a detection granularity is likely to miss a large number of results. Some code clone detection methods mainly rely on the index of surface lexical tokens, that is, first build a vocabulary mapping from all tokens to unique integer IDs, and then convert each token into its corresponding integer ID according to the vocabulary to generate a corresponding index sequence. Although this method is simple and efficient, and also achieves a finer granularity, it is insufficient in accuracy. Surface lexical tokens cannot fully express the syntax and semantic structure of the code, which leads to false positives and false negatives in the detection process.

[0004] Therefore, the code clone detection method in the prior art has the problems of low detection accuracy and low detection efficiency. Summary of the invention

[0005] The embodiments of the present application provide a code clone detection method, device, electronic device and storage medium to improve the detection accuracy and detection efficiency of code clone detection.

[0006] In a first aspect, an embodiment of the present application provides a code clone detection method, the method comprising:

[0007] For each function to be tested in the code file to be tested, perform the following operations:

[0008] Generate a syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected;

[0009] Generate a feature vector and a first index of the function to be detected based on the syntax element sequence of the function to be detected;

[0010] Using the first index to query each code file in the code file library to determine the query result;

[0011] If at least one target function is found, a feature vector of the at least one target function is compared with a feature vector of the function to be detected for similarity to obtain a similarity comparison result; wherein the first index exists in the at least one target function;

[0012] Based on the similarity comparison result of each function to be detected in the code file to be detected, a code clone detection result of the code file to be detected is determined.

[0013] In some embodiments, generating a first index of the function to be detected based on the syntax element sequence of the function to be detected includes:

[0014] Determining the number of syntax elements constituting the first index based on the total number of syntax elements in the syntax element sequence of the function to be detected;

[0015] Combining the syntax elements in the syntax element sequence of the function to be detected according to the number of syntax elements of the first index to obtain at least one syntax element combination;

[0016] The at least one syntax element is combined to determine a first index of the function to be detected.

[0017] In some embodiments, combining the syntax elements in the syntax element sequence of the function to be detected according to the number of syntax elements of the first index to obtain at least one syntax element combination includes:

[0018] According to the arrangement order of each syntax element in the syntax element sequence of the function to be detected, each syntax element is taken as the first syntax element in turn;

[0019] Based on the number of syntax elements of the first index, starting from the first syntax element, a corresponding number of syntax elements are selected in order from the syntax element sequence of the function to be detected to combine and obtain the syntax element combination.

[0020] In some embodiments, the at least one objective function is obtained by querying in the following manner:

[0021] Using the first index, querying the second index corresponding to each source function in each code file pre-generated in the code file library;

[0022] Determine at least one source function corresponding to the second index which is the same as the first index as a candidate function;

[0023] For any candidate function, determine the number of indexes in the candidate function that are the same as the function to be detected;

[0024] The candidate function whose corresponding index quantity is higher than a preset quantity threshold is determined as the target function.

[0025] In some embodiments, the performing similarity comparison between the feature vector of the at least one target function and the feature vector of the function to be detected to obtain a similarity comparison result includes:

[0026] For any target function, similarity calculation is performed based on the feature vector corresponding to the target function pre-generated in the code file library and the feature vector corresponding to the function to be detected;

[0027] If the similarity exceeds a preset similarity threshold, the function to be detected is determined as a clone function of the target function.

[0028] In some embodiments, generating a feature vector of the function to be detected based on the syntax element sequence of the function to be detected includes:

[0029] Input the grammatical element sequence into the Word2Vec model to obtain a word vector for each grammatical element in the grammatical element sequence;

[0030] A feature vector of the function to be detected is generated based on the word vector of each grammatical element in the grammatical element sequence.

[0031] In some embodiments, the generating the syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected includes:

[0032] Parsing the function to be detected to generate an abstract syntax tree of the function to be detected; wherein each node of the abstract syntax tree is used to represent a grammatical structure in the function to be detected;

[0033] Depth-first traversal of the abstract syntax tree, and determining each traversed node of the abstract syntax tree as a syntax element corresponding to the node;

[0034] The syntax elements corresponding to the nodes are merged in a traversal order to obtain a syntax element sequence of the function to be detected.

[0035] In some embodiments, the feature vector and the second index corresponding to each source function in each code file in the code file library are pre-generated in the following manner:

[0036] Based on the code files, determining each source function in the code files;

[0037] For any source function, generating a syntax element sequence of the source function based on an abstract syntax tree corresponding to the source function;

[0038] Based on the syntax element sequence of the source function, a feature vector and a second index of the source function are generated.

[0039] In a second aspect, an embodiment of the present application provides a code clone detection device, including:

[0040] The execution module is used to perform the following operations for each function to be detected in the code file to be detected:

[0041] Generate a syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected;

[0042] Generate a feature vector and a first index of the function to be detected based on the syntax element sequence of the function to be detected;

[0043] Using the first index to query each code file in the code file library to determine the query result;

[0044] If at least one target function is found, a feature vector of the at least one target function is compared with a feature vector of the function to be detected for similarity to obtain a similarity comparison result; wherein the first index exists in the at least one target function;

[0045] The determination module is used to determine the code clone detection result of the code file to be detected based on the similarity comparison result of each function to be detected in the code file to be detected.

[0046] In some embodiments, when executing the step of generating a first index of the function to be detected based on the syntax element sequence of the function to be detected, the execution module is specifically configured to:

[0047] Determining the number of syntax elements constituting the first index based on the total number of syntax elements in the syntax element sequence of the function to be detected;

[0048] Combining the syntax elements in the syntax element sequence of the function to be detected according to the number of syntax elements of the first index to obtain at least one syntax element combination;

[0049] The at least one syntax element is combined to determine a first index of the function to be detected.

[0050] In some embodiments, when combining the syntax elements in the syntax element sequence of the function to be detected according to the number of syntax elements of the first index to obtain at least one syntax element combination, the execution module is specifically used to:

[0051] According to the arrangement order of each syntax element in the syntax element sequence of the function to be detected, each syntax element is taken as the first syntax element in turn;

[0052] Based on the number of syntax elements of the first index, starting from the first syntax element, a corresponding number of syntax elements are selected in order from the syntax element sequence of the function to be detected to combine and obtain the syntax element combination.

[0053] In some embodiments, the execution module is specifically configured to obtain the at least one objective function by querying in the following manner:

[0054] Using the first index, querying the second index corresponding to each source function in each code file pre-generated in the code file library;

[0055] Determine at least one source function corresponding to the second index which is the same as the first index as a candidate function;

[0056] For any candidate function, determine the number of indexes in the candidate function that are the same as the function to be detected;

[0057] The candidate function whose corresponding index quantity is higher than a preset quantity threshold is determined as the target function.

[0058] In some embodiments, when performing a similarity comparison between the feature vector of the at least one target function and the feature vector of the function to be detected to obtain a similarity comparison result, the execution module is specifically used to:

[0059] For any target function, similarity calculation is performed based on the feature vector corresponding to the target function pre-generated in the code file library and the feature vector corresponding to the function to be detected;

[0060] If the similarity exceeds a preset similarity threshold, the function to be detected is determined as a clone function of the target function.

[0061] In some embodiments, when the step of generating a feature vector of the function to be detected based on the syntax element sequence of the function to be detected is executed, the execution module is specifically used to:

[0062] Input the grammatical element sequence into the Word2Vec model to obtain a word vector for each grammatical element in the grammatical element sequence;

[0063] A feature vector of the function to be detected is generated based on the word vector of each grammatical element in the grammatical element sequence.

[0064] In some embodiments, when executing the step of generating a sequence of syntax elements of the function to be detected based on the abstract syntax tree corresponding to the function to be detected, the execution module is specifically used to:

[0065] Parsing the function to be detected to generate an abstract syntax tree of the function to be detected; wherein each node of the abstract syntax tree is used to represent a grammatical structure in the function to be detected;

[0066] Depth-first traversal of the abstract syntax tree, and determining each traversed node of the abstract syntax tree as a syntax element corresponding to the node;

[0067] The syntax elements corresponding to the nodes are merged in a traversal order to obtain a syntax element sequence of the function to be detected.

[0068] In some embodiments, the feature vector and the second index corresponding to each source function in each code file in the code file library are pre-generated in the following manner:

[0069] Based on the code files, determining each source function in the code files;

[0070] For any source function, generating a syntax element sequence of the source function based on an abstract syntax tree corresponding to the source function;

[0071] Based on the syntax element sequence of the source function, a feature vector and a second index of the source function are generated.

[0072] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0073] The memory stores a computer program executable by at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned code clone detection method.

[0074] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the above-mentioned code clone detection method.

[0075] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when called and executed by an electronic device, enables the electronic device to execute the above-mentioned code cloning detection method.

[0076] In an embodiment of the present application, for each function to be detected in the code file to be detected, a syntax element sequence of the function to be detected is generated based on the abstract syntax tree corresponding to the function to be detected; based on the syntax element sequence of the function to be detected, a feature vector and a first index of the function to be detected are generated; the first index is used to query each code file in the code file library to obtain at least one target function; the first index exists in at least one target function; the feature vector of at least one target function is compared with the feature vector of the function to be detected for similarity to obtain a similarity comparison result; based on the similarity comparison result of each function to be detected in the code file to be detected, the code clone detection result of the code file to be detected is determined. In this way, the first index is generated according to the syntax element sequence of the function to be detected, at least one target function is first determined by the first search, and then the feature vector of the target function is compared with the feature vector of the function to be detected, thereby improving the detection efficiency of code clone detection and the accuracy of code clone detection.

[0077] Other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0079] Figure 1 A flowchart of a code clone detection method provided in an embodiment of the present application;

[0080] Figure 2 A flow chart of a method for generating a first index provided in an embodiment of the present application;

[0081] Figure 3 A schematic diagram of a Token combination of a function to be detected provided in an embodiment of the present application;

[0082] Figure 4 A flow chart of a method for generating a characteristic vector and a second index corresponding to a source function provided in an embodiment of the present application;

[0083] Figure 5 A flow chart of a method for determining an objective function provided in an embodiment of the present application;

[0084] Figure 6 A schematic diagram of the structure of a code cloning detection device provided in an embodiment of the present application;

[0085] Figure 7 A schematic diagram of the hardware structure of an electronic device for implementing a code cloning detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0087] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0088] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0089] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0090] Before introducing the code clone detection method provided by the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0091] Code cloning refers to the repeated use of the same or similar code snippets in different code locations. This phenomenon may be caused by a variety of reasons, such as code reuse, copy-and-paste programming, etc. Although code cloning can improve development efficiency to a certain extent, it also brings many problems, such as code redundancy, difficulty in maintenance, and vulnerability propagation. Therefore, code clone detection technology plays an important role in software engineering, which can help development teams identify and manage duplicate code in the code base, and improve code quality and maintenance efficiency.

[0092] In the prior art, many code clone detection methods have introduced indexes, which can significantly improve detection efficiency. However, some detection methods build indexes based on code lines or blocks. Although this will increase detection speed and reduce memory requirements, excessive detection granularity is likely to miss a large number of results. Some code clone detection methods mainly rely on the index of surface lexical tokens, that is, first build a vocabulary mapping from all tokens to unique integer IDs, and then convert each token into its corresponding integer ID according to the vocabulary to generate a corresponding index sequence. Although this method is simple and efficient, and also has a finer granularity, it is insufficient in accuracy. Surface lexical tokens cannot fully express the syntax and semantic structure of the code, which makes it easy for false positives and false negatives to occur during the detection process. Therefore, the code clone detection methods in the prior art have the problems of low detection accuracy and low detection efficiency.

[0093] In view of this, in order to solve the problems of low detection accuracy and low detection efficiency in the prior art code clone detection method, the present application embodiment provides a code clone detection method, device, equipment and medium. The following describes some preferred embodiments of the present application in conjunction with the accompanying drawings.

[0094] The code clone detection method provided in this application can be applied to code clone detection scenarios of various large-scale software projects, such as code plagiarism detection in academic institutions, open source communities or enterprises, open source code vulnerability detection in enterprises, and project self-research degree analysis in internal enterprise project development.

[0095] It is worth noting that the application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0096] Please refer to Figure 1 , is a flowchart of a code clone detection method provided in an embodiment of the present application, the method comprising the following steps.

[0097] In step 101, for each function to be detected in the code file to be detected, the following operations are performed:

[0098] Since the operation for each function to be detected in the code file to be detected is the same, referring to steps 1011 to 1014 below, the operation performed on one function to be detected is taken as an example for a detailed description.

[0099] In step 1011, a syntax element sequence of the function to be detected is generated based on an abstract syntax tree corresponding to the function to be detected.

[0100] In specific implementation, the function to be detected can be parsed to generate an abstract syntax tree of the function to be detected; wherein each node of the abstract syntax tree is used to represent a syntax structure in the function to be detected; the abstract syntax tree is traversed depth-first, and each node of the traversed abstract syntax tree is determined as a syntax element corresponding to the node; the syntax elements corresponding to the nodes are merged in the traversal order to obtain a syntax element sequence of the function to be detected.

[0101] For example, use some open source tools (Antlr4, JDT) to generate the Abstract Syntax Tree (AST) of the function to be tested, where AST is a tree structure and each node represents a grammatical structure in the function to be tested. Then, select a traversal method, such as depth-first traversal (DFS), starting from the root node, and traverse each node layer by layer. The traversed nodes are determined as the grammatical elements corresponding to the nodes, and the grammatical elements corresponding to all nodes are merged into a complete grammatical element sequence in the traversal order to ensure that the order of the grammatical element sequence is consistent with the order in the function to be tested.

[0102] In step 1012, a feature vector and a first index of the function to be detected are generated based on the syntax element sequence of the function to be detected.

[0103] In specific implementation, after obtaining the syntax element sequence of the function to be detected, the feature vector of the function to be detected can be generated through the following steps.

[0104] Step 1: Input the grammatical element sequence into the Word2Vec model to obtain the word vector of each grammatical element in the grammatical element sequence.

[0105] In the specific implementation, taking the grammatical element as Token as an example, the Token sequence can be input into the trained Word2Vec model to generate the word vector of each Token. Assuming that the Token sequence of the function to be detected is: ['def','add','(','a',',','b',')',':','return','a','+','b'], each word vector is 3-dimensional, then the word vector of each Token is obtained from the Word2Vec model, such as: def->[0.1,0.2,0.3], add->[0.4,0.5,0.6], (->[0.7,0.8,0.9], etc.

[0106] Step 2: Generate a feature vector of the function to be detected based on the word vector of each grammatical element in the grammatical element sequence.

[0107] In specific implementation, methods such as average pooling, maximum pooling, minimum pooling, and concatenation can be used to combine the word vectors of each grammatical element to generate a feature vector of the function to be detected. The specific method is not limited in this application.

[0108] In specific implementation, after obtaining the syntax element sequence of the function to be detected, it can be done by Figure 2 The flowchart of the method for generating the first index shown generates the first index of the function to be detected.

[0109] Please refer to Figure 2 , Figure 2 A flow chart of a method for generating a first index provided in an embodiment of the present application, the method comprising the following steps.

[0110] In step 201, the number of syntax elements constituting the first index is determined based on the total number of syntax elements in the syntax element sequence of the function to be detected.

[0111] In the specific implementation, the syntax element is still taken as an example. For example, if the total number of tokens in the token sequence is 10, then the number of tokens constituting the first index can be set to 3 or 5. For another example, if the total number of tokens in the token sequence is 5, then the number of tokens constituting the first index can be set to 3. The specific setting can be made by the technician according to the needs.

[0112] In step 202, the syntax elements in the syntax element sequence of the function to be detected are combined according to the number of syntax elements of the first index to obtain at least one syntax element combination.

[0113] In a specific implementation, each syntax element can be used as the first syntax element in turn according to the arrangement order of each syntax element in the syntax element sequence of the function to be detected; based on the number of syntax elements of the first index, starting from the first syntax element, a corresponding number of syntax elements are selected in sequence from the syntax element sequence of the function to be detected for combination to obtain a syntax element combination.

[0114] For example, still taking the syntax element as Token, set the number of Tokens of the first index to 3, and the Token sequence of function A is (assign, name_a, binop_plus, name_b, name_c), where assign: indicates that this is an assignment statement, name_a: indicates the target variable of the assignment, binop_plus: indicates an addition operation, name_b and name_c: these two are the operands of the addition operation, and function A indicates that the result of adding name_b and name_c is assigned to name_a.

[0115] Then, we can take each Token as the first Token in turn according to the arrangement order of each Token in the Token sequence of function A. For example, take assign as the first Token, and then select 2 more Tokens from the Token sequence in order, that is, the selected two Tokens are: name_a and binop_plus. Finally, assign, name_a and binop_plus are combined to obtain Token combination 1 (assign, name_a, binop_plus). Next time, take name_a as the first Token, and select 2 more Tokens from the Token sequence in order, that is, the selected two Tokens are: binop_plus and name_b. Finally, name_a, binop_plus and name_b are combined to obtain Token combination 2 (name_a, binop_plus, name_b). By analogy, we can obtain Token combination 3 (binop_plus, name_b, name_c), which is as follows. Figure 3 The token combination corresponding to the function A shown in the figure is: Figure 3 A schematic diagram of a Token combination of a function to be detected provided in an embodiment of the present application.

[0116] The above process of establishing grammatical element combinations can also be assisted by the search server Elasticsearch, which is a distributed, RESTful-style search and analysis engine built on the Lucene library. It is designed to handle large-scale data sets and provide real-time search, analysis, and visualization capabilities. By setting appropriate analyzers and tokenizers, Elasticsearch can index each n-gram (n consecutive tokens), thereby supporting efficient similarity searches.

[0117] For example, input the Token sequence of function A into the search server Elasticsearch, set the number of Tokens of the first index to 3, that is, set the value of n in n-gram to 3, and set the analyzer and tokenizer. Elasticsearch can analyze and tokenize the Token sequence of function A according to the set n-gram, and automatically generate 3 n-gram sequences, where n-gram sequence 1 is: (assign, name_a, binop_plus), n-gram sequence 2 is: (name_a, binop_plus, name_b), and n-gram sequence 2 is: (binop_plus, name_b, name_c), and each n-gram sequence corresponds to a Token combination.

[0118] In this way, by traversing the AST (abstract syntax tree) to generate a Token sequence, and using it to establish an n-gram index, unlike the traditional Token sequence based on surface lexical analysis, the Token sequence generated by AST has more semantic depth because it captures the grammatical hierarchy and logical structure of the code, rather than just the text features on the surface of the code. The index constructed by this method has a more detailed query granularity and more grammatical and semantic depth, which can detect more types of code clone relationships and solve more complex code clone detection problems.

[0119] In step 203, at least one syntax element is combined and determined as a first index of the function to be detected.

[0120] In specific implementation, each syntax element combination corresponds to a first index, that is, if the function to be detected is function A, there are 3 first indexes, namely: (assign, name_a, binop_plus), (name_a, binop_plus, name_b), and (binop_plus, name_b, name_c).

[0121] In step 1013, the first index is used to query each code file in the code file library to determine the query result.

[0122] During specific implementation, the code file library pre-generates the feature vector and the second index corresponding to each source function in each code file.

[0123] Figure 4 A flow chart of a method for generating a characteristic vector and a second index corresponding to a source function provided in an embodiment of the present application, the method comprising the following steps.

[0124] In step 401, based on each code file, each source function in each code file is determined.

[0125] During specific implementation, each source function may also be numbered to determine the source function identifier.

[0126] In step 402, for any source function, a syntax element sequence of the source function is generated based on the abstract syntax tree corresponding to the source function.

[0127] In specific implementation, the process of generating the syntax element sequence of the source function is consistent with the process of generating the syntax element sequence of the function to be detected, and can refer to step 1011, which will not be repeated here.

[0128] In step 403, a feature vector and a second index of the source function are generated based on the syntax element sequence of the source function.

[0129] In specific implementation, the process of generating the feature vector and the second index of the source function is consistent with the process of generating the feature vector and the first index of the function to be detected, which can be referred to step 1012 and will not be repeated here.

[0130] In specific implementation, if there are functions similar to the function to be retrieved in the code files pre-generated in the code file library, the query result includes at least one target function. If there are no functions similar to the function to be retrieved in the code files pre-generated in the code file library, there is no target function in the query result.

[0131] During specific implementation, the objective function can be determined according to the following steps. Figure 5 A flow chart of a method for determining an objective function provided in an embodiment of the present application, the method comprising the following steps.

[0132] In step 501, the first index is used to query the second index corresponding to each source function in each code file pre-generated in the code file library.

[0133] In the specific implementation, for example, the first index is (assign, name_a, binop_plus), then the second index that is the same as the first index is searched in the second index corresponding to each source function in each pre-generated code file. Assuming that there are 3 second indexes, namely: second index 1 (assign, name_a, binop_plus), second index 2 (assign, name_c, binop_plus), second index 3 (assign, name_d, binop_plus), then, the corresponding second index can be queried through the first index as second index 1.

[0134] In step 502, at least one source function corresponding to a second index that is the same as the first index is determined as a candidate function.

[0135] For example, the source functions corresponding to the second index 1 are: source function 1, source function 2 and source function 3, that is, the second index 1 exists in source function 1, source function 2 and source function 3, then source function 1, source function 2 and source function 3 can be determined as candidate functions.

[0136] In a specific implementation, if there are multiple first indexes, if each first index is used to find the same second index, a group of candidate functions can be determined for each first index.

[0137] Assume that there are 3 first indexes of function B, among which the first index 1 corresponds to the second index 1, the first index 2 corresponds to the second index 5, and the first index 3 corresponds to the second index 6. The source functions corresponding to the second index 1 are source function 1, source function 2 and source function 3, the source functions corresponding to the second index 5 are source function 1, source function 4 and source function 3, and the source functions corresponding to the second index 6 are source function 1, source function 5 and source function 6. Then the candidate function corresponding to the first index 1 is (source function 1, source function 2, source function 3), the candidate function corresponding to the first index 2 is (source function 1, source function 4, source function 3), and the candidate function corresponding to the first index 3 is (source function 1, source function 5, source function 6).

[0138] In step 503, for any candidate function, the number of indexes in the candidate function that are the same as the function to be detected is determined.

[0139] Still taking the candidate functions of the above function B as an example, it can be seen that the number of indexes that source function 1 has in common with function B is 3, the number of indexes that source function 2, source function 4, source function 5 and source function 6 have in common with function B is 1, and the number of indexes that source function 3 has in common with function B is 2.

[0140] In step 504, the candidate function whose corresponding index quantity is higher than a preset quantity threshold is determined as the target function.

[0141] In specific implementation, the preset quantity threshold can be determined according to the number of first indexes of the function to be detected. For example, when the number of first indexes of the function to be detected is 5, the preset quantity threshold is set to 3. For another example, when the number of first indexes of the function to be detected is 3, the preset quantity threshold is set to 2. Taking the candidate function of the above-mentioned function B as an example, assuming that the preset quantity threshold is set to 2, then only the number of indexes that are the same as function B by source function 1 is higher than 2, then source function 1 can be determined as the target function of function B. If the preset quantity threshold is set to 1, then the number of indexes that are the same as function B by source function 1 and source function 3 is higher than 1, then source function 1 and source function 3 can be determined as the target functions of function B.

[0142] In this way, the amount of data that needs to be processed by the comparison neural network is greatly reduced through the screening of indexes and target functions. At the same time, the pre-generated feature vectors of the source function can save a lot of time in feature extraction, making it possible to perform efficient and accurate clone detection in large-scale code file libraries, thus solving the limitations of traditional methods in dealing with complex code structures and large-scale data.

[0143] In step 1014, if at least one target function is found, a feature vector of the at least one target function is compared with a feature vector of the function to be detected for similarity to obtain a similarity comparison result; wherein the first index exists in at least one target function.

[0144] In the specific implementation, for any target function, a similarity calculation is performed based on the feature vector corresponding to the target function pre-generated in the code file library and the feature vector corresponding to the function to be detected; if the similarity exceeds the preset similarity threshold, the function to be detected is determined as a clone function of the target function.

[0145] In specific implementation, the feature vector of the target function can be compared with the feature vector of the function to be detected through a comparison neural network, wherein the comparison neural network has three layers, the size of each layer decreases continuously, and finally outputs a similarity.

[0146] For example, still taking the candidate function of the above-mentioned function B as an example, the feature vector of source function 1 is compared with the feature vector of function B for similarity to obtain similarity 1, and the feature vector of source function 3 is compared with the feature vector of function B for similarity to obtain similarity 2. If similarity 1 exceeds the preset similarity threshold, source function 1 is determined as a clone function of function B. If both similarity 1 and similarity 2 exceed the preset similarity threshold, source function 1 and source function 3 are determined as clone functions of function B.

[0147] In step 102, based on the similarity comparison result of each function to be detected in the code file to be detected, a code clone detection result of the code file to be detected is determined.

[0148] During specific implementation, the overall detection report of the code file to be detected can be generated by combining the similarity comparison results of each function to be detected in the code file to be detected.

[0149] In this way, a first index is generated according to a sequence of syntax elements of the function to be detected, at least one target function is first determined using a first search, and then a feature vector of the target function is compared with a feature vector of the function to be detected, thereby improving the detection efficiency of code clone detection and the accuracy of code clone detection.

[0150] Based on the same technical concept, the embodiment of the present application also provides a code cloning detection device. The principle of solving the problem by the code cloning detection device is similar to that of the above-mentioned code cloning detection method. Therefore, the implementation of the code cloning detection device can refer to the implementation of the code cloning detection method, and the repeated parts will not be repeated.

[0151] Figure 6 A schematic diagram of the structure of a code clone detection device provided in an embodiment of the present application includes an execution module 601 and a determination module 602.

[0152] The execution module 601 is used to perform the following operations for each function to be detected in the code file to be detected:

[0153] Generate a syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected;

[0154] Generate a feature vector and a first index of the function to be detected based on the syntax element sequence of the function to be detected;

[0155] Using the first index to query each code file in the code file library to determine the query result;

[0156] If at least one target function is found, a feature vector of the at least one target function is compared with a feature vector of the function to be detected for similarity to obtain a similarity comparison result; wherein the first index exists in the at least one target function;

[0157] The determination module 602 is used to determine the code clone detection result of the code file to be detected based on the similarity comparison result of each function to be detected in the code file to be detected.

[0158] In some embodiments, when executing the step of generating the first index of the function to be detected based on the syntax element sequence of the function to be detected, the execution module 601 is specifically used to:

[0159] Determining the number of syntax elements constituting the first index based on the total number of syntax elements in the syntax element sequence of the function to be detected;

[0160] Combining the syntax elements in the syntax element sequence of the function to be detected according to the number of syntax elements of the first index to obtain at least one syntax element combination;

[0161] The at least one syntax element is combined to determine a first index of the function to be detected.

[0162] In some embodiments, when combining the syntax elements in the syntax element sequence of the function to be detected according to the number of syntax elements of the first index to obtain at least one syntax element combination, the execution module 601 is specifically used to:

[0163] According to the arrangement order of each syntax element in the syntax element sequence of the function to be detected, each syntax element is taken as the first syntax element in turn;

[0164] Based on the number of syntax elements of the first index, starting from the first syntax element, a corresponding number of syntax elements are selected in order from the syntax element sequence of the function to be detected to combine and obtain the syntax element combination.

[0165] In some embodiments, the execution module 601 is specifically configured to obtain the at least one objective function by querying in the following manner:

[0166] Using the first index, querying the second index corresponding to each source function in each code file pre-generated in the code file library;

[0167] Determine at least one source function corresponding to the second index which is the same as the first index as a candidate function;

[0168] For any candidate function, determine the number of indexes in the candidate function that are the same as the function to be detected;

[0169] The candidate function whose corresponding index quantity is higher than a preset quantity threshold is determined as the target function.

[0170] In some embodiments, when performing a similarity comparison between the feature vector of the at least one target function and the feature vector of the function to be detected to obtain a similarity comparison result, the execution module 601 is specifically used to:

[0171] For any target function, similarity calculation is performed based on the feature vector corresponding to the target function pre-generated in the code file library and the feature vector corresponding to the function to be detected;

[0172] If the similarity exceeds a preset similarity threshold, the function to be detected is determined as a clone function of the target function.

[0173] In some embodiments, when the step of generating a feature vector of the function to be detected based on the syntax element sequence of the function to be detected is executed, the execution module 601 is specifically used to:

[0174] Input the grammatical element sequence into the Word2Vec model to obtain a word vector for each grammatical element in the grammatical element sequence;

[0175] A feature vector of the function to be detected is generated based on the word vector of each grammatical element in the grammatical element sequence.

[0176] In some embodiments, when executing the step of generating a syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected, the execution module 601 is specifically used to:

[0177] Parsing the function to be detected to generate an abstract syntax tree of the function to be detected; wherein each node of the abstract syntax tree is used to represent a grammatical structure in the function to be detected;

[0178] Depth-first traversal of the abstract syntax tree, and determining each traversed node of the abstract syntax tree as a syntax element corresponding to the node;

[0179] The syntax elements corresponding to the nodes are merged in a traversal order to obtain a syntax element sequence of the function to be detected.

[0180] In some embodiments, the feature vector and the second index corresponding to each source function in each code file in the code file library are pre-generated in the following manner:

[0181] Based on the code files, determining each source function in the code files;

[0182] For any source function, generating a syntax element sequence of the source function based on an abstract syntax tree corresponding to the source function;

[0183] Based on the syntax element sequence of the source function, a feature vector and a second index of the source function are generated.

[0184] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated in a processor, or may exist physically separately, or two or more modules may be integrated in one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0185] After introducing the code clone detection method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0186] Refer to the following Figure 7 The electronic device 130 implemented according to this embodiment of the present application is described. Figure 7 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0187] like Figure 7 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0188] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0189] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0190] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0191] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0192] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium, and the computer program product includes: computer program code, when the computer program code is run on a computer, the computer executes any of the code clone detection methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the page switching method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0193] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the code clone detection methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the page switching method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0194] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0195] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0196] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0198] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A code clone detection method, characterized in that: The method comprises: For each function to be tested in the code file to be tested, perform the following operations: Generate a syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected; Generate a feature vector and a first index of the function to be detected based on the syntax element sequence of the function to be detected; Using the first index to query each code file in the code file library to determine the query result; If at least one target function is found, a feature vector of the at least one target function is compared with a feature vector of the function to be detected for similarity to obtain a similarity comparison result; wherein the first index exists in the at least one target function; Based on the similarity comparison result of each function to be detected in the code file to be detected, a code clone detection result of the code file to be detected is determined.

2. The method according to claim 1, characterized in that Generating a first index of the function to be detected based on the syntax element sequence of the function to be detected includes: Determining the number of syntax elements constituting the first index based on the total number of syntax elements in the syntax element sequence of the function to be detected; Combining the syntax elements in the syntax element sequence of the function to be detected according to the number of syntax elements of the first index to obtain at least one syntax element combination; The at least one syntax element is combined to determine a first index of the function to be detected.

3. The method according to claim 2, characterized in that The combining, according to the number of syntax elements of the first index, the syntax elements in the syntax element sequence of the function to be detected to obtain at least one syntax element combination comprises: According to the arrangement order of each syntax element in the syntax element sequence of the function to be detected, each syntax element is taken as the first syntax element in turn; Based on the number of syntax elements of the first index, starting from the first syntax element, a corresponding number of syntax elements are selected in order from the syntax element sequence of the function to be detected to combine and obtain the syntax element combination.

4. The method according to claim 1, characterized in that The at least one objective function is obtained by querying in the following manner: Using the first index, querying the second index corresponding to each source function in each code file pre-generated in the code file library; Determine at least one source function corresponding to the second index which is the same as the first index as a candidate function; For any candidate function, determine the number of indexes in the candidate function that are the same as the function to be detected; The candidate function whose corresponding index quantity is higher than a preset quantity threshold is determined as the target function.

5. The method according to claim 1, characterized in that The performing similarity comparison between the feature vector of the at least one target function and the feature vector of the function to be detected to obtain a similarity comparison result includes: For any target function, similarity calculation is performed based on the feature vector corresponding to the target function pre-generated in the code file library and the feature vector corresponding to the function to be detected; If the similarity exceeds a preset similarity threshold, the function to be detected is determined as a clone function of the target function.

6. The method according to claim 1, characterized in that Generating a feature vector of the function to be detected based on the syntax element sequence of the function to be detected includes: Input the grammatical element sequence into the Word2Vec model to obtain a word vector for each grammatical element in the grammatical element sequence; A feature vector of the function to be detected is generated based on the word vector of each grammatical element in the grammatical element sequence.

7. The method according to any one of claims 1 to 6, characterized in that: The step of generating a syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected includes: Parsing the function to be detected to generate an abstract syntax tree of the function to be detected; wherein each node of the abstract syntax tree is used to represent a grammatical structure in the function to be detected; Depth-first traversal of the abstract syntax tree, and determining each traversed node of the abstract syntax tree as a syntax element corresponding to the node; The syntax elements corresponding to the nodes are merged in a traversal order to obtain a syntax element sequence of the function to be detected.

8. The method according to any one of claims 1 to 6, characterized in that: The feature vector and the second index corresponding to each source function in each code file in the code file library are pre-generated in the following manner: Based on the code files, determining each source function in the code files; For any source function, generating a syntax element sequence of the source function based on an abstract syntax tree corresponding to the source function; Based on the syntax element sequence of the source function, a feature vector and a second index of the source function are generated.

9. A code clone detection device, characterized in that: include: The execution module is used to perform the following operations for each function to be detected in the code file to be detected: Generate a syntax element sequence of the function to be detected based on the abstract syntax tree corresponding to the function to be detected; Generate a feature vector and a first index of the function to be detected based on the syntax element sequence of the function to be detected; Using the first index to query each code file in the code file library to determine the query result; If at least one target function is found, a feature vector of the at least one target function is compared with a feature vector of the function to be detected for similarity to obtain a similarity comparison result; wherein the first index exists in the at least one target function; The determination module is used to determine the code clone detection result of the code file to be detected based on the similarity comparison result of each function to be detected in the code file to be detected.

10. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method according to any one of claims 1 to 8.

Images