Code similarity detection method and device, storage medium and electronic equipment

Abstract

The application discloses a code similarity detection method and device, a storage medium and an electronic device, and belongs to the computer field. The method comprises the following steps: analyzing the syntax structure of a target code file, and generating an abstract syntax tree of the target code file, wherein the abstract syntax tree comprises a plurality of nodes, each node corresponds to a temporary array and a count array; and the code similarity of a code block in the target code file is calculated by using the temporary array and the count array. The application solves the technical problem that the code similarity error is large in the related art by character comparison, and improves the maintenance efficiency of repeated codes.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a method and apparatus for detecting code similarity, a storage medium, and an electronic device. Background Technology

[0002] In related technologies, maintaining duplicate code requires very high human resources (modifications in one place cannot be synchronized to all), so the extraction and abstraction of common code is of paramount importance in development.

[0003] Related technologies use a line-by-line scanning method for code comparison, comparing the consistency of text characters in each line and column. This character-based comparison is easily affected by interference, such as code obfuscation, special characters, or code blocks with identical logic but different variable names. Furthermore, character-based comparison in these technologies does not consider the degree of repetition in code logic; it only compares characters in each line and column. After code has been compiled or obfuscated, variable names, variable definitions, operators, and even operation methods may have changed. In such cases, character-based comparison alone will produce significant errors and cannot reflect the true state of the code.

[0004] There are currently no effective solutions to the aforementioned problems in the relevant technologies. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides a method and apparatus, storage medium and electronic device for detecting code similarity.

[0006] According to one aspect of the embodiments of this application, a method for detecting code similarity is provided, comprising: parsing the syntactic structure of a target code file, generating an abstract syntax tree of the target code file, wherein the abstract syntax tree includes a plurality of nodes, each node corresponding to a temporary array and a counting array; and calculating the code similarity of code blocks in the target code file using the temporary array and the counting array.

[0007] According to another aspect of the embodiments of this application, a code similarity detection device is also provided, comprising: a generation module, configured to parse the syntactic structure of a target code file and generate an abstract syntax tree of the target code file, wherein the abstract syntax tree includes a plurality of nodes, each node corresponding to a temporary array and a counting array; and a calculation module, configured to calculate the code similarity of code blocks in the target code file using the temporary array and the counting array.

[0008] Furthermore, the calculation module includes: a generation unit, configured to generate multiple counting arrays based on multiple temporary arrays in the AST, wherein the counting arrays include the following information of the variable definition expression: operation type, variable type, and number of variables; and a comparison unit, configured to compare a first counting array and a second counting array among the multiple counting arrays, and determine the similarity between the first counting array and the second counting array as the similarity between a first code block and a second code block in the target code file, wherein the first counting array corresponds to the first code block, and the second counting array corresponds to the second code block.

[0009] Furthermore, the generation unit includes: a first reading subunit, used to read a first key character from each temporary array in the AST, wherein the first key character is used to characterize the operation type of the variable definition expression; a second reading subunit, used to read a second key character from the temporary array and count the number of variables for the second key character, wherein the second key character is used to characterize the variable type under the current operation type, and each second key character corresponds to a variable type; and a generation subunit, used to generate a counting array using the first key character, the second key character, and the number of variables.

[0010] Furthermore, the comparison unit includes: a comparison subunit, used to compare the first operation type of the first counting array with the second operation type of the second counting array; a first calculation subunit, used to calculate a first quotient between the first number of variables in the first counting array and the second number of variables in the second counting array if the first operation type and the second operation type are the same, wherein the first quotient is used to characterize the first similarity between the first counting array and the second counting array; and a first determination subunit, used to determine the first quotient as the similarity between the first code block and the second code block in the target code file.

[0011] Further, the comparison unit includes: a comparison subunit, used to compare the first operation type of the first counting array with the second operation type of the second counting array; a second calculation subunit, used to calculate a second quotient between the number of first variable types in the first counting array and the number of second variable types in the second counting array if the first operation type and the second operation type are the same, wherein the number of first variable types and the number of second variable types are obtained statistically based on the types of variable types included in the first operation type and the second operation type, wherein the second quotient is used to characterize the second similarity between the first counting array and the second counting array; and a second determination subunit, used to determine the second quotient as the similarity between the first code block and the second code block in the target code file.

[0012] Further, the comparison unit includes: a comparison subunit, used to compare the first operation type of the first counting array with the second operation type of the second counting array; a third calculation subunit, used to determine the number of specific variables of the same variable type in the first counting array and the second counting array if the first operation type and the second operation type are the same, and to determine the total number of variables in the first counting array or the second counting array, and to calculate a third quotient value between the number of specific variables and the total number of variables, wherein the total number of variables in the first counting array and the second counting array are the same, and the third quotient value is used to characterize the third similarity between the first counting array and the second counting array; and a third determination subunit, used to determine the third quotient value as the similarity between the first code block and the second code block in the target code file.

[0013] Furthermore, the generation module includes: a segmentation unit, used to segment the code string in the target code file into lexical units, wherein each lexical unit corresponds to a code block; a generation unit, used to generate a list of lexical units based on the segmented multiple lexical units; and a transformation unit, used to convert the list of lexical units into an AST in units of variable definition expressions.

[0014] Furthermore, the transformation unit includes: a splitting subunit, used to split the lexical unit list into multiple temporary arrays based on variable definition expressions, wherein each temporary array corresponds to a variable definition expression, and each variable definition expression is identified by a key character; a mapping subunit, used to map the corresponding temporary array to a sub-AST for each variable definition expression in the lexical unit list; and a combining subunit, used to determine the multiple sub-ASTs as child nodes under a specified tree root node, and combine them to obtain a total AST.

[0015] Furthermore, the segmentation unit includes: a scanning subunit, configured to scan the code string from left to right, starting from the first character of the code string in the target code file, until the next empty string, and extract the continuous string before the empty string; a determining subunit, configured to determine the continuous string as a key character, continue scanning the code string from left to right until the nearest identifier after the key character, and determine the character set between the key character and the identifier as a lexical unit; and a processing subunit, configured to continue scanning the remaining code string from left to right on a newline until the last character of the code string is scanned.

[0016] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.

[0018] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any embodiment of the code similarity detection method described above.

[0019] The technical solutions provided in this application have the following advantages compared with the prior art:

[0020] In the embodiments of this application, by analyzing the similarity at the syntactic and logical levels, the influence of code compilation or some obfuscation and encryption methods can be avoided, the error can be reduced, the technical problem of large error in calculating code similarity by character comparison in related technologies can be solved, and the maintenance efficiency of duplicate code can be improved. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a code similarity detection method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the scanning code string in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the AST in the embodiments of this application;

[0026] Figure 4 These are two schematic diagrams illustrating the generation of the counting array in an embodiment of this application;

[0027] Figure 5 This is a comparative example diagram of two code blocks in an embodiment of this application;

[0028] Figure 6 This is a comparative schematic diagram of the embodiments in this application;

[0029] Figure 7 These are comparative example diagrams of three code blocks from embodiments of this application;

[0030] Figure 8 This is a structural block diagram of a code similarity detection device according to an embodiment of this application;

[0031] Figure 9 This is a structural diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another similar entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0034] This embodiment provides a method for detecting code similarity. Figure 1 This is a flowchart of a code similarity detection method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0035] Step S102: Parse the syntax structure of the target code file and generate an abstract syntax tree of the target code file, wherein the abstract syntax tree includes several nodes, and each node corresponds to a temporary array and a counting array;

[0036] In this embodiment, the abstract syntax tree (AST) is a tree-like representation of the abstract syntax structure of the source code in the target code file.

[0037] Step S104: Calculate the code similarity of code blocks in the target code file using the temporary array and the counting array;

[0038] This embodiment can calculate the logical similarity of code at the semantic level by using the temporary array and counting array corresponding to the AST, without relying on the compilation method of the code or the text itself.

[0039] Through the above steps, the syntactic structure of the target code file is parsed, and an abstract syntax tree (AST) is generated for the target code file. Each node of the AST corresponds to a temporary array. Then, the code similarity of the target code file is calculated using the AST. By analyzing the degree of similarity at the syntactic and logical levels, the influence of code compilation or some obfuscation and encryption methods can be avoided, reducing errors. This solves the technical problem of large errors in calculating code similarity through character comparison in related technologies, and improves the maintenance efficiency of duplicate code.

[0040] In this embodiment, parsing the syntax structure of the target code file and generating an abstract syntax tree for the target code file includes:

[0041] S11, the code string in the target code file is divided into lexical units, wherein each lexical unit corresponds to a code block;

[0042] S12, Generate a list of lexical units based on the segmented multiple lexical units;

[0043] S13, convert the list of lexical units into an AST, using variable definition expressions as units.

[0044] In one embodiment of this example, converting the lexical unit list into an AST based on variable definition expressions includes: splitting the lexical unit list into multiple temporary arrays based on variable definition expressions, wherein each temporary array corresponds to a variable definition expression, and each variable definition expression is identified by a key character; mapping the corresponding temporary array to a sub-AST for each variable definition expression in the lexical unit list; and determining the multiple sub-ASTs as child nodes under a specified root node, and combining them to obtain a total AST.

[0045] In one example, the key character in the variable definition expression is VariableDeclaration, and the temporary arrays are: VariableDeclaration, VariableDeclarator, Identifier, literal.

[0046] In this embodiment, segmenting the code string in the target code file into lexical units includes: starting from the first character of the code string in the target code file, scanning the code string from left to right until the next empty string, extracting the continuous string before the empty string; identifying the continuous string as a key character, continuing to scan the code string from left to right until the nearest identifier after the key character, and identifying the character set between the key character and the identifier as a lexical unit.

[0047] After obtaining the first lexical unit, the remaining code string is scanned from left to right (if there is still unscanned code string) until the last character of the code string is reached.

[0048] Optionally, the identifier can be a semicolon or a newline character. The first character of the code string is the first non-whitespace character starting from the first column (e.g., column 0) of the code string, beginning from the first line (e.g., line 0).

[0049] In the JavaScript language of this embodiment, the smallest logical execution unit is an expression (i.e., an AST node), such as var a = 1. During the scanning process, "v" is the first character, "var" is the key character, and "var a = 1" is an expression. Each expression has a key character to identify it, so the AST syntax tree can be constructed by identifying the key characters. For example, the string var is a key character in the code of the target code file (used to identify the variable definition expression VariableDeclaration).

[0050] Figure 2 This is a schematic diagram illustrating the scanning of a code string according to an embodiment of this application. By scanning characters row by row and column by column, a coordinate x (row number x) and y (column number y) are generated for each character, as shown below. Figure 2As shown, scanning starts from the first character x0y0. If it is an empty character, it is ignored. If x1y0 is the character 'v' (not an empty character), it is recorded first. Scanning continues from the next character 'a' (x2y0) until the next empty character (x4y0) appears. This gives us a continuous string 'var' from x1y0 to x3y0. In JavaScript, 'var' is a key character (VariableDeclaration). Based on the syntax of VariableDeclaration in JavaScript, we can determine that the nine characters (x1y0 to x9y0) between 'var' and the nearest semicolon (x10y0) or newline character (x11y0) (semicolons and newlines are special characters that separate expressions in JavaScript) constitute an expression.

[0051] After obtaining a complete expression through scanning, the variable definition expression (or simply expression) is then broken down. This complete variable definition expression can always be broken down into the following components (i.e., child nodes in the AST), each separated by an empty string. In an example of var a = 0, the following elements can be obtained by splitting it according to the code position, thus obtaining the corresponding temporary array:

[0052] The three characters of x1y0-x3y0 are var(VariableDeclarator);

[0053] x5y0 character a (Identifier) ​​(This could be multiple consecutive characters, but in this example there is only one character);

[0054] x7y0 always has the operator =;

[0055] x9y0 character 1 (literal) (This could be multiple consecutive characters; in this example, there is only one character.)

[0056] Additionally, in this example, "literal" is a simple literal definition. It could also be another expression. If it is an expression, it will be a child node of the variable definition expression "VariableDeclaration" mentioned above and will continue to be split and parsed according to the splitting process described above.

[0057] The other expression in the example above, `var b = 2;` and `var a = 1;`, are separated by a semicolon or a newline character, so they are two independent expressions. Both belong to the root node of the AST tree and are temporarily stored in an array, i.e., a temporary array, as shown below:

[0058]

[0059] Starting with identifying the variable definition expression `VariableDeclaration`, then breaking it down into four components: `VariableDeclaration`, `VariableDeclarator`, `Identifier`, and `literal`, and finally iterating through this temporary array, we can obtain the AST (Abstract Syntax Tree) structure of this expression. By repeating these steps until the last character of the code string is reached, we can construct the AST syntax tree for all expressions in the code string within the specified code file. Figure 3 This is a schematic diagram of the AST in an embodiment of this application, illustrating two nodes (lexical unit node 1 and lexical unit node 2), each corresponding to one of the two expressions. The elements in the AST are explained below:

[0060] Lexical unit: Each complete child node is a lexical unit;

[0061] Program: Root node of the tree;

[0062] VariableDeclaration: The expression used to define a variable.

[0063] VariableDeclarator: Lexical unit / child node / code block type (indicating variable definition);

[0064] kind: "var": Declares the variable type;

[0065] Identifier: Lexical unit / sub-node / code block type (indicates variable identifier definition);

[0066] name: "a": Declare the variable name;

[0067] Literal: The operation type of a lexical unit / child node / code block (indicating a literal definition here);

[0068] raw: "1": Declares a variable value (literal);

[0069] It should be noted that the values ​​(VariableDeclarator / Identifier / literal) in the above lexical unit / sub-node / code block types are enumeration types, and other types such as FunctionDeclaration, which represents method definitions, are similar.

[0070] In this embodiment, calculating the code similarity of the target code file using the AST includes:

[0071] S21, generate multiple counting arrays based on the multiple temporary arrays in the AST, wherein the counting arrays include the following information of the variable definition expression: operation type, variable type, and number of variables;

[0072] Operation type, variable type, and number of variables are the three dimensions for calculating the degree of repetition, because similarity only cares about the definition and form of variables and operations in the code, and does not need to care about whether the variable names and operator declarations are consistent.

[0073] Optionally, generating multiple counting arrays based on multiple temporary arrays in the AST includes: for each temporary array in the abstract syntax tree, reading a first key character in the temporary array, wherein the first key character is used to characterize the operation type of the variable definition expression; reading a second key character in the temporary array and counting the number of variables for the second key character, wherein the second key character is used to characterize the variable type under the current operation type, and each second key character corresponds to a variable type; and generating a counting array using the first key character, the second key character, and the number of variables.

[0074] In one example, any two code blocks are traversed sequentially from left to right according to the tree structure of the AST, or according to code blocks in the AST that meet preset conditions (such as only traversing code blocks whose operation type is a specified type). For example, the first and second code blocks contain the expressions "var a = 1" and "var b = 2" respectively, and temporary arrays are shown below:

[0075]

[0076] In the first code block, the variable definition expression is `VariableDeclaration`, which is further broken down into four components: `VariableDeclaration`, `VariableDeclarator`, `Identifier`, and `literal`. In the second code block, the variable definition expression is also `VariableDeclaration`, which is further broken down into four components: `VariableDeclaration`, `VariableDeclarator`, `Identifier`, and `literal`. Reading the temporary array from the first code block, the first key character is "1", indicating a literal definition operation. The second key character is "VariableDeclarator", indicating a variable declaration operation. The number of "var" characters is 1, and the number of variables is also 1. Similarly, reading the temporary array from the second code block, the first key character is "2", indicating a literal definition operation. The second key character is also "VariableDeclarator", indicating a variable declaration operation. The number of "var" characters is also 1, and the number of variables is also 1.

[0077] Figure 4 These are two schematic diagrams illustrating the generation of a counting array in an embodiment of this application. A second counting array (omitting the operation type) can be generated simultaneously with the temporary array. The structure is similar to that of the temporary array. Unlike the temporary array, the counting array does not need to describe the meaning of each node in the complete tree structure (ignoring parts that do not affect the comparison, since the comparison is of expression type rather than specific value). It only records the number of identical expressions at the same array position (i.e., the AST tree node position) for comparison and statistical purposes.

[0078] S22, compare the first counting array and the second counting array among the plurality of counting arrays, and determine the similarity between the first counting array and the second counting array as the similarity between the first code block and the second code block in the target code file, wherein the first counting array corresponds to the first code block, and the second counting array corresponds to the second code block. The first code block and the second code block are any two code blocks in the same target code file to be compared for similarity. The first counting array corresponds to the first temporary array generated for the first code block and is generated based on the first temporary array; the second counting array corresponds to the second temporary array generated for the second code block and is generated based on the second temporary array.

[0079] This explanation uses the expression "var=1" as an example. Figure 5 This is a comparison example diagram of two code blocks from an embodiment of this invention. The dashed boxes indicate the repeated parts. Figure 5In section 1 and 2, there are different implementations of a method definition expression named aFunc, but their functionality is duplicated. The following example illustrates how to use the counting array corresponding to the AST tree to find the duplicate parts (the dashed parts in the AST).

[0080] In one example, comparing a first counting array and a second counting array among the plurality of counting arrays, and determining the similarity between the first counting array and the second counting array as the similarity between a first code block and a second code block in the target code file, includes: comparing a first operation type of the first counting array with a second operation type of the second counting array; if the first operation type and the second operation type are the same, comparing a first variable type of the first counting array with a second variable type of the second counting array; if the first variable type of the first counting array and the second variable type of the second counting array are the same, comparing the number of first variables in the first counting array with the number of second variables in the second counting array; if the number of first variables in the first counting array and the number of second variables in the second counting array are the same, then the similarity between the first code block and the second code block in the target code file is confirmed to be 100%.

[0081] In another aspect of this example, if the first operation type and the second operation type are different, the similarity between the first code block and the second code block is directly determined to be 0.

[0082] Figure 6 This is a comparison diagram in an embodiment of this application. The expressions at each level of the counting array 1 are compared cyclically with those at each level of the counting array 2 until the fourth comparison finds the same type: FunctionDeclaration. Since the expressions are of the same type and are both method definition expressions, the comparison continues with the expression type: VariableDeclaration in the child nodes of that method definition expression. Both method definition expressions have exactly one type: VariableDeclaration in their child nodes, and num is 1. Therefore, the above can be considered... Figure 5 The aFunc methods in code blocks 1 and 2 have completely identical functionality, with a similarity of 100%. The above example represents a scenario with a 100% similarity.

[0083] In one example, comparing a first counting array and a second counting array among the plurality of counting arrays, and determining the similarity between the first counting array and the second counting array as the similarity between a first code block and a second code block in the target code file, includes: comparing a first operation type of the first counting array with a second operation type of the second counting array; if the first operation type and the second operation type are the same, calculating a first quotient between the number of first variables in the first counting array and the number of second variables in the second counting array, wherein the first quotient is used to characterize the first similarity between the first counting array and the second counting array; and determining the first quotient as the similarity between the first code block and the second code block in the target code file.

[0084] In one example, comparing a first counting array and a second counting array among the plurality of counting arrays, and determining the similarity between the first counting array and the second counting array as the similarity between the first code block and the second code block in the target code file, includes: comparing a first operation type of the first counting array with a second operation type of the second counting array; if the first operation type and the second operation type are the same, calculating a second quotient between the number of first variable types in the first counting array and the number of second variable types in the second counting array, wherein the number of first variable types and the number of second variable types are respectively obtained statistically based on the types of variable types included in the first operation type and the second operation type, and the second quotient is used to characterize the second similarity between the first counting array and the second counting array; and determining the second quotient as the similarity between the first code block and the second code block in the target code file.

[0085] In another aspect of this example, if the first operation type and the second operation type are different, the similarity between the first code block and the second code block is directly determined to be 0.

[0086] In one example, comparing a first counting array and a second counting array among the plurality of counting arrays, and determining the similarity between the first counting array and the second counting array as the similarity between a first code block and a second code block in the target code file, includes: comparing a first operation type of the first counting array with a second operation type of the second counting array; if the first operation type and the second operation type are the same, determining the number of specific variables of the same variable type in the first counting array and the second counting array, and determining the total number of variables in the first counting array or the second counting array, calculating a third quotient value between the number of specific variables and the total number of variables, wherein the total number of variables in the first counting array and the second counting array are the same, and the third quotient value is used to characterize the third similarity between the first counting array and the second counting array; and determining the third quotient value as the similarity between the first code block and the second code block in the target code file.

[0087] In another aspect of this example, if the first operation type and the second operation type are different, the similarity between the first code block and the second code block is directly determined to be 0.

[0088] Here's an example of three code blocks with a similarity of less than 100%. Code block 1 contains the expression "var a = 1", code block 2 contains two expressions "var a = 1" and "var b = 1", and code block 3 contains two expressions "var a = 1" and "function b()". The incompletely matching aFunc method definition expressions can be compared using a counting array to determine the similarity between the two code blocks. Figure 7 This is a comparison example diagram of three code blocks in an embodiment of this application, including code blocks 1, 2, and 3, which correspond to counting arrays 1, 2, and 3 respectively. Pairwise comparisons are performed. It should be noted that the counting arrays are generated from temporary arrays. Therefore, the pairwise comparisons are based on the operation type, variable type, number of variables, and number of variable types at the same array position (i.e., the tree node position in the abstract syntax tree). An example is provided below:

[0089] The method definition expression `type:FunctionDeclaration` in counting arrays 1 and 2 matches the `type:VariableDeclaration` (which is the only type) of the child nodes `nodes`, but the values ​​of the counts `num` are different. That is, the first operation type is the same as the second operation type, but the number of the first variable is different from the number of the second variable. Calculate the first quotient of the number of the first variable `num:1` in the first counting array and the number of the second variable `num:2` in the second counting array. Therefore, we can know that the functionality of the `aFunc` method in 1 and 2 is: `num:1 / num:2 = 50%`. Take 50% as the similarity between the first code block and the second code block in the target code file.

[0090] In counting arrays 1 and 3, although the number of `type:VariableDeclaration` nodes (`num`) is the same, counting array 3 has one more child node type (`FunctionDeclaration`). That is, the first operation type is the same as the second operation type, but the number of the first variable type is different from the number of the second variable type. Calculating the second quotient between the number of nodes for the first variable type in the first counting array (1) and the number of nodes for the second variable type in the second counting array (2), we can calculate: Count of nodes in counting array 1: 1 / Count of nodes in counting array 3: 2 = 50%. This 50% is taken as the similarity between the first and second code blocks in the target code file. In other examples, if the original number of nodes in counting array 1 is 3 and the original number of nodes in counting array 3 is 2, then the corresponding calculation is: Count of nodes in counting array 3: 2 / Count of nodes in counting array 1: 3 = 66.67%.

[0091] In counting arrays 2 and 3, the number of child nodes (nodes) is greater than 2 (the value of num equals the number of expressions of the same type in nodes). We need to remove expressions of the same type first. That is, expressions where the first and second operation types are the same, but the number of the first and second variable types exceeds the threshold, and there are identical variable types, we need to determine the number of specific variables of the same type in the first and second counting arrays. Counting arrays 2 and 3 have one variable of the same type: VariableDeclaration. Therefore, we need to remove one variable of type: VariableDeclaration from both counting arrays 2 and 3, resulting in a total of 1 specific variable of the same type. This leaves only one expression type in the child nodes of counting arrays 2 and 3, which can be interpreted as a total of 2 variables. Therefore, we can divide the number of removed expressions by the original number of nodes: Count of nodes removed from counting array 1 (counting array 3): 1 / Original number of nodes in counting array 1 (counting array 3): 2 = 50%. We will use 50% as the similarity between the first and second code blocks in the target code file.

[0092] This embodiment's solution is not based on character comparison, but rather on analyzing the similarity at the syntactic and logical levels. Therefore, it is not affected by code compilation or obfuscation / encryption methods. (Code compilation or obfuscation / encryption refers to using technical means to shuffle the code order, change variable names, encrypt variable values, etc., without changing the semantic and logical structure of the code.) Since maintaining duplicate code requires very high manpower costs (modifications in one place cannot be synchronized to all), the extraction and abstraction of common code is of paramount importance in development. This solution provides the conditions for extraction and abstraction, helping developers identify which code is worth making public, greatly reducing code maintenance costs.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0094] This embodiment also provides a code similarity detection device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0095] Figure 8 This is a structural block diagram of a code similarity detection device according to an embodiment of this application, such as... Figure 8 As shown, the device includes: a generation module 80 and a calculation module 82, wherein,

[0096] The generation module 80 is used to parse the syntax structure of the target code file and generate an abstract syntax tree of the target code file, wherein the abstract syntax tree includes several nodes, and each node corresponds to a temporary array and a counting array;

[0097] The calculation module 82 is used to calculate the code similarity of code blocks in the target code file using the temporary array and the counting array.

[0098] Optionally, the calculation module includes: a generation unit, configured to generate multiple counting arrays based on multiple temporary arrays in the AST, wherein the counting arrays include the following information of the variable definition expression: operation type, variable type, and number of variables; and a comparison unit, configured to compare a first counting array and a second counting array among the multiple counting arrays, and determine the similarity between the first counting array and the second counting array as the similarity between a first code block and a second code block in the target code file, wherein the first counting array corresponds to the first code block, and the second counting array corresponds to the second code block.

[0099] Optionally, the generation unit includes: a first reading subunit, configured to read a first key character from each temporary array in the AST, wherein the first key character is used to characterize the operation type of the variable definition expression; a second reading subunit, configured to read a second key character from the temporary array and count the number of variables for the second key character, wherein the second key character is used to characterize the variable type under the current operation type, and each second key character corresponds to a variable type; and a generation subunit, configured to generate a counting array using the first key character, the second key character, and the number of variables.

[0100] Optionally, the comparison unit includes: a comparison subunit, configured to compare a first operation type of the first counting array with a second operation type of the second counting array; a first calculation subunit, configured to calculate a first quotient between the number of first variables in the first counting array and the number of second variables in the second counting array if the first operation type and the second operation type are the same, wherein the first quotient is used to characterize a first similarity between the first counting array and the second counting array; and a first determination subunit, configured to determine the first quotient as the similarity between a first code block and a second code block in the target code file.

[0101] Optionally, the comparison unit includes: a comparison subunit, used to compare the first operation type of the first counting array with the second operation type of the second counting array; a second calculation subunit, used to calculate a second quotient between the number of first variable types in the first counting array and the number of second variable types in the second counting array if the first operation type and the second operation type are the same, wherein the number of first variable types and the number of second variable types are obtained statistically based on the types of variable types included in the first operation type and the second operation type, wherein the second quotient is used to characterize the second similarity between the first counting array and the second counting array; and a second determination subunit, used to determine the second quotient as the similarity between the first code block and the second code block in the target code file.

[0102] Optionally, the comparison unit includes: a comparison subunit, used to compare the first operation type of the first counting array with the second operation type of the second counting array; a third calculation subunit, used to determine the number of specific variables of the same variable type in the first counting array and the second counting array if the first operation type and the second operation type are the same, and to determine the total number of variables in the first counting array or the second counting array, and to calculate a third quotient value between the number of specific variables and the total number of variables, wherein the total number of variables in the first counting array and the second counting array are the same, and the third quotient value is used to characterize the third similarity between the first counting array and the second counting array; and a third determination subunit, used to determine the third quotient value as the similarity between the first code block and the second code block in the target code file.

[0103] Optionally, the generation module includes: a segmentation unit, used to segment the code string in the target code file into lexical units, wherein each lexical unit corresponds to a code block; a generation unit, used to generate a list of lexical units based on the segmented multiple lexical units; and a transformation unit, used to convert the list of lexical units into an AST in units of variable definition expressions.

[0104] Optionally, the transformation unit includes: a splitting subunit, used to split the lexical unit list into multiple temporary arrays based on variable definition expressions, wherein each temporary array corresponds to a variable definition expression, and each variable definition expression is identified by a key character; a mapping subunit, used to map the corresponding temporary array to a sub-AST for each variable definition expression in the lexical unit list; and a combining subunit, used to determine the multiple sub-ASTs as child nodes under a specified tree root node, and combine them to obtain a total AST.

[0105] Optionally, the segmentation unit includes: a scanning subunit, configured to scan the code string from left to right, starting from the first character of the code string in the target code file, until the next empty string, and extract the continuous string before the empty string; a determining subunit, configured to determine the continuous string as a key character, continue scanning the code string from left to right until the nearest identifier after the key character, and determine the character set between the key character and the identifier as a lexical unit; and a processing subunit, configured to continue scanning the remaining code string from left to right on a newline until the last character of the code string is scanned.

[0106] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0107] This application also provides an electronic device. Figure 9 This is a structural diagram of an electronic device according to an embodiment of this application, such as... Figure 9 As shown, it includes a processor 91, a communication interface 92, a memory 93, and a communication bus 94. The processor 91, the communication interface 92, and the memory 93 communicate with each other through the communication bus 94. The memory 93 is used to store computer programs. When the processor 91 executes the program stored in the memory 93, it implements the code similarity detection method described in any of the embodiments.

[0108] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0109] The communication interface is used for communication between the aforementioned terminal and other devices.

[0110] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0111] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0112] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the code similarity detection methods described in the above embodiments.

[0113] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any embodiment of the code similarity detection method described above.

[0114] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0115] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

[0116] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for detecting code similarity, characterized in that, include: The syntax structure of the target code file is parsed to generate an abstract syntax tree of the target code file, wherein the abstract syntax tree includes several nodes, and each node corresponds to a temporary array and a counting array; The temporary array and the counting array are used to calculate the code similarity of code blocks in the target code file; The calculation of code similarity of code blocks in the target code file using the temporary array and the counting array includes: generating multiple counting arrays based on multiple temporary arrays in the abstract syntax tree, wherein the counting array includes the following information of the variable definition expression: operation type, variable type, and number of variables; comparing the first counting array and the second counting array in the multiple counting arrays, and determining the similarity between the first counting array and the second counting array as the similarity between the first code block and the second code block in the target code file, wherein the first counting array corresponds to the first code block, and the second counting array corresponds to the second code block; wherein generating multiple counting arrays based on multiple temporary arrays in the abstract syntax tree includes: for each temporary array in the abstract syntax tree, reading the first key character in the temporary array, wherein the first key character is used to characterize the operation type of the variable definition expression; reading the second key character in the temporary array and counting the number of variables for the second key character, wherein the second key character is used to characterize the variable type under the current operation type, and each second key character corresponds to one variable type; generating a counting array using the first key character, the second key character, and the number of variables.

2. The method according to claim 1, characterized in that, Comparing the first and second count arrays among the plurality of count arrays, and determining the similarity between the first and second count arrays as the similarity between the first and second code blocks in the target code file, includes: Compare the first operation type of the first counting array with the second operation type of the second counting array; If the first operation type and the second operation type are the same, calculate the first quotient between the number of first variables in the first counting array and the number of second variables in the second counting array, wherein the first quotient is used to characterize the first similarity between the first counting array and the second counting array; The first quotient is determined as the similarity between the first code block and the second code block in the target code file.

3. The method according to claim 1, characterized in that, Comparing the first and second count arrays among the plurality of count arrays, and determining the similarity between the first and second count arrays as the similarity between the first and second code blocks in the target code file, includes: Compare the first operation type of the first counting array with the second operation type of the second counting array; If the first operation type and the second operation type are the same, calculate the second quotient between the number of first variable types in the first counting array and the number of second variable types in the second counting array; the number of first variable types and the number of second variable types are respectively obtained by statistical analysis based on the types of variable types included in the first operation type and the second operation type, wherein the second quotient is used to characterize the second similarity between the first counting array and the second counting array; The second quotient is determined as the similarity between the first code block and the second code block in the target code file.

4. The method according to claim 1, characterized in that, Comparing the first and second count arrays among the plurality of count arrays, and determining the similarity between the first and second count arrays as the similarity between the first and second code blocks in the target code file, includes: Compare the first operation type of the first counting array with the second operation type of the second counting array; If the first operation type and the second operation type are the same, determine the number of specific variables of the same variable type in the first counting array and the second counting array, and determine the total number of variables in the first counting array or the second counting array. Calculate the third quotient between the number of specific variables and the total number of variables, wherein the total number of variables in the first counting array and the second counting array are the same, and the third quotient is used to characterize the third similarity between the first counting array and the second counting array. The third quotient is determined as the similarity between the first code block and the second code block in the target code file.

5. The method according to claim 1, characterized in that, Parse the syntax structure of the target code file and generate an abstract syntax tree for the target code file, including: The code string in the target code file is divided into lexical units, where each lexical unit corresponds to a code block; Generate a list of lexical units based on the segmented multiple lexical units; The list of lexical units is converted into an abstract syntax tree, with each variable definition expression as a unit.

6. The method according to claim 5, characterized in that, Converting the list of lexical units into an abstract syntax tree, using variable definition expressions as units, includes: The list of lexical units is split into multiple temporary arrays based on variable definition expressions. Each temporary array corresponds to a variable definition expression, and each variable definition expression is identified by a key character. For each variable in the lexical unit list, define an expression and map the corresponding temporary array to a sub-abstract syntax tree; Multiple sub-abstract syntax trees are each identified as child nodes under a specified root node, and then combined to obtain a total abstract syntax tree.

7. The method according to claim 5, characterized in that, Segmenting the code string in the target code file into lexical units includes: Starting from the first character of the code string in the target code file, scan the code string from left to right until the next empty string, and extract the continuous string before the empty string; The continuous string is identified as a key character, and the code string is scanned from left to right until the nearest identifier after the key character is reached. The set of characters between the key character and the identifier is identified as a lexical unit.

8. A code similarity detection device, characterized in that, include: A generation module is used to parse the syntax structure of the target code file and generate an abstract syntax tree of the target code file, wherein the abstract syntax tree includes several nodes, and each node corresponds to a temporary array and a counting array; The calculation module is used to calculate the code similarity of code blocks in the target code file using the temporary array and the counting array; The calculation module includes: a generation unit, used to generate multiple counting arrays based on multiple temporary arrays in the abstract syntax tree, wherein the counting arrays include the following information of the variable definition expression: operation type, variable type, and number of variables; and a comparison unit, used to compare a first counting array and a second counting array among the multiple counting arrays, and determine the similarity between the first counting array and the second counting array as the similarity between a first code block and a second code block in the target code file, wherein the first counting array corresponds to the first code block, and the second counting array corresponds to the second code block. The generation unit includes: a first reading subunit, used to read a first key character from each temporary array in the abstract syntax tree, wherein the first key character is used to characterize the operation type of the variable definition expression; a second reading subunit, used to read a second key character from the temporary array and count the number of variables for the second key character, wherein the second key character is used to characterize the variable type under the current operation type, and each second key character corresponds to a variable type; and a generation subunit, used to generate a counting array using the first key character, the second key character, and the number of variables.

9. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein the computer program executes the method steps of any one of claims 1 to 7 when it is run.

10. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, communication interface, and memory communicate with each other via a communication bus; among which: Memory, used to store computer programs; A processor for performing the method steps of any one of claims 1 to 7 by running a computer program stored in memory.

11. A computer program product comprising computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the method steps of any one of claims 1 to 7.

Images