AST-based duplicated code detection and analysis method, apparatus and device, and storage medium

Abstract

The invention relates to the field of code detection, and discloses an AST-based duplicated code detection and analysis method, which comprises the following steps of: obtaining incremental codes after text cutting, generating an AST syntax tree by utilizing the incremental codes, extracting a tree characteristic value character string of the AST syntax tree and a class characteristic value character string of each class corresponding to the AST syntax tree, calculating the similarity between the tree feature value character strings and the class feature value character strings one by one, screening out classes corresponding to the class feature value character strings with the similarity smaller than a preset similarity threshold value, and performing text merging on the incremental codes and the classes corresponding to the screened class feature value character strings to obtain updated code classes, and generating an updated AST syntax tree by using the updated code class, and storing the updated AST syntax tree to a distributed file database. According to the method, the calculation efficiency of the similarity between codes can be improved.

Description

Technical Field

[0001] The present invention relates to the field of code detection, and in particular to an AST-based duplicate code detection and analysis method, device, electronic device and computer-readable storage medium. Background Art

[0002] AST stands for "Abstract Syntax Tree". An abstract syntax tree is a tree-like data structure used to represent the structure of source code. It is often used in compilers and interpreters to analyze and transform program code. Each node represents a grammatical structure in the source code, making the syntax and semantic analysis of the program more systematic and efficient.

[0003] With the advent of the information society, the product demand iteration of each software company is faster, and the amount of software code is increasing exponentially. Therefore, the larger the amount of stock code, the heavier the maintenance burden of software engineers. The similarity of traditional software codes can be determined through text comparison, which can quickly find the code generated by "copy-and-paste". However, some people may avoid text similarity comparison by adding redundant comments, changing variable names, changing line breaks, and other modifications.

[0004] The existing technology has the problem of low efficiency in calculating the similarity between codes. Summary of the invention

[0005] The present invention provides a duplicate code detection and analysis method, device and computer-readable storage medium based on AST, the main purpose of which is to solve the problem of low efficiency in calculating similarity between codes.

[0006] To achieve the above object, the present invention provides a duplicate code detection and analysis method based on AST, comprising:

[0007] Get the incremental code after text cutting;

[0008] Generate an AST syntax tree using the incremental code;

[0009] Extracting a tree feature value string of the AST syntax tree and a class feature value string corresponding to each class of the AST syntax tree;

[0010] Calculating the similarities between the tree feature value string and the class feature value string one by one, and filtering out the classes corresponding to the class feature value strings whose similarities are less than a preset similarity threshold;

[0011] Merge the incremental code with the class corresponding to the filtered class feature value string to obtain an update code class;

[0012] An updated AST syntax tree is generated using the updated code class, and the updated AST syntax tree is saved in a distributed file database.

[0013] In some embodiments, the step of obtaining the incremental code after text segmentation includes:

[0014] Get the number of lines of code to be split submitted by the user;

[0015] Determine whether the number of lines of code to be segmented is greater than a preset line number threshold;

[0016] If the number of lines of code to be segmented is greater than the line number threshold, determining the segmentation granularity to be a granularity greater than a preset granularity threshold;

[0017] If the number of lines of code to be segmented is less than or equal to the line number threshold, determining the segmentation granularity to be a granularity less than a preset granularity threshold;

[0018] The code to be split submitted by the user is divided according to the determined cutting granularity to obtain the incremental code.

[0019] In some embodiments, the generating an AST syntax tree using the incremental code includes:

[0020] Decomposing the incremental code into lexical units;

[0021] combining the lexical units into a grammatical structure;

[0022] The grammatical structure is converted to generate an AST grammar tree.

[0023] In some embodiments, the extracting of the tree feature value string of the AST syntax tree and the class feature value string of each class corresponding to the AST syntax tree includes:

[0024] Filter out the key nodes of the AST syntax tree;

[0025] Recursively construct a key syntax tree using the key nodes;

[0026] Traversing the key syntax tree in the order of left subtree, root node, and right subtree to obtain the characteristics of each node;

[0027] Using the features of each node to form a tree feature value string;

[0028] Extracting a class feature value from the features of each node;

[0029] The class feature values ​​are concatenated into a class feature value string.

[0030] In some embodiments, the step of calculating the similarities between the tree feature value strings and the class feature value strings one by one and selecting the classes corresponding to the class feature value strings whose similarities are less than a preset similarity threshold comprises:

[0031] Splitting the tree feature value string into a plurality of basic tree words, and splitting the class feature value string into a plurality of basic class words;

[0032] Calculating the occurrence frequencies of the several basic tree words and the several basic class words in the incremental code to obtain the word frequencies of the several basic tree words and the word frequencies of the several basic class words;

[0033] Converting the word frequencies of the several basic tree words and the word frequencies of the several basic class words into tree word frequency vectors and class word frequency vectors;

[0034] The similarity between the tree word frequency vector and the class word frequency vector is calculated using the cosine similarity formula.

[0035] In some embodiments, filtering out the class corresponding to the class feature value string whose similarity is less than a preset similarity threshold includes:

[0036] Determining whether the similarity is less than a preset similarity threshold;

[0037] If the similarity is greater than or equal to a preset similarity threshold, the incremental code is set as a warning, and the incremental code is sent to the reviewer to confirm whether to save it in the database;

[0038] If the similarity is less than a preset similarity threshold, the class corresponding to the class feature value string in the similarity will be retained.

[0039] In some embodiments, saving the updated AST syntax tree to a distributed file database includes:

[0040] Use the ToString method to obtain the string data of the updated AST syntax tree;

[0041] Construct a storage file for storing the string data of the updated AST syntax tree;

[0042] Writing the string data of the updated AST syntax tree into the storage file;

[0043] The storage file to which the string data has been written is saved to the distributed file database.

[0044] In order to solve the above problem, the present invention also provides a duplicate code detection and analysis device based on AST, the device comprising:

[0045] A data acquisition module is used to obtain the incremental code after text segmentation;

[0046] A syntax tree generation module, used to generate an AST syntax tree using the incremental code;

[0047] A feature extraction module, used to extract a tree feature value string of the AST syntax tree and a class feature value string corresponding to each class of the AST syntax tree;

[0048] A similarity comparison module, used to calculate the similarity between the tree feature value string and the class feature value string one by one, and filter out the classes corresponding to the class feature value strings whose similarity is less than a preset similarity threshold;

[0049] A text merging module, used for text merging the incremental code with the class corresponding to the filtered class feature value string to obtain an updated code class;

[0050] The data storage module is used to generate an updated AST syntax tree using the updated code class and save the updated AST syntax tree to a distributed file database.

[0051] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0052] at least one processor; and,

[0053] a memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned AST-based duplicate code detection and analysis method.

[0055] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned AST-based duplicate code detection and analysis method.

[0056] An embodiment of the present invention obtains incremental code after text segmentation, generates an AST syntax tree using the incremental code, extracts a tree feature value string of the AST syntax tree and a class feature value string corresponding to each class of the AST syntax tree, calculates the similarity between the tree feature value string and the class feature value string one by one, filters out the classes corresponding to the class feature value strings whose similarity is less than a preset similarity threshold, performs text merging on the incremental code and the classes corresponding to the filtered class feature value strings to obtain an updated code class, generates an updated AST syntax tree using the updated code class, and saves the updated AST syntax tree to a distributed file database.

[0057] Selecting a suitable text cutting strategy can reduce the number of code blocks that need to be processed, extract the tree feature value string of the AST syntax tree and the class feature value string corresponding to each class of the AST syntax tree, select and retain important nodes, reduce the complexity of the tree, and reduce the amount of calculation by screening out the classes corresponding to the class feature value strings whose similarity is less than a preset similarity threshold. Selecting a high-performance distributed database system supports fast reading, writing and retrieval, which speeds up subsequent similarity search and comparison. Therefore, the present invention can solve the problem of low efficiency in similarity calculation between code blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a duplicate code detection and analysis method based on AST provided in one embodiment of the present invention;

[0059] Figure 2 A functional module diagram of a duplicate code detection and analysis device based on AST provided in one embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of an electronic device for implementing the AST-based duplicate code detection and analysis method provided by an embodiment of the present invention.

[0061] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0063] The embodiment of the present invention provides a duplicate code detection and analysis method based on AST. The execution subject of the duplicate code detection and analysis method based on AST includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present invention. In other words, the duplicate code detection and analysis method based on AST can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0064] Reference Figure 1 FIG. 1 is a flow chart of a duplicate code detection and analysis method based on AST provided by an embodiment of the present invention. In this embodiment, the duplicate code detection and analysis method based on AST includes:

[0065] S1. Get the incremental code after text segmentation.

[0066] In an embodiment of the present invention, when a user submits a relatively large number of code blocks at one time, the code blocks are segmented according to the "code class" granularity. The granularity of the code segmentation will affect the working cost of the entire detection device. According to the actual number of code lines in the project, it is determined whether to use a granularity greater than the preset granularity threshold or a granularity less than the preset granularity threshold for text segmentation.

[0067] In detail, the step of obtaining the incremental code after text segmentation includes:

[0068] Get the number of lines of code to be split submitted by the user;

[0069] Determine whether the number of lines of code to be segmented is greater than a preset line number threshold;

[0070] If the number of lines of code to be segmented is greater than the line number threshold, determining the segmentation granularity to be a granularity greater than a preset granularity threshold;

[0071] If the number of lines of code to be segmented is less than or equal to the line number threshold, determining the segmentation granularity to be a granularity less than a preset granularity threshold;

[0072] The code to be split submitted by the user is divided according to the determined cutting granularity to obtain the incremental code.

[0073] In detail, the code to be segmented submitted by the user is obtained, and according to the number of lines of the code to be segmented, it is determined whether to use a larger granularity or a smaller granularity for text segmentation; for code whose number of lines of code to be segmented is greater than a preset line number threshold, cutting with a granularity greater than the preset granularity threshold will reduce the number of processing times; for code whose number of lines of code to be segmented is less than or equal to the preset line number threshold, cutting with a granularity less than the preset granularity threshold can improve the detection accuracy; finally, the code to be segmented is divided according to the determined granularity to obtain the incremental code after text segmentation.

[0074] By dividing the code to be split into classes, each class can be regarded as an independent functional unit. This approach not only optimizes the organizational structure of the code, but also improves the accuracy of duplicate code detection. Through this modular approach, the responsibilities and functions of each class become clearer, thereby simplifying code reuse and maintenance, while supporting more sophisticated automated analysis and team collaboration, ultimately improving code quality and development efficiency, and providing more accurate data for subsequent duplicate code detection and analysis.

[0075] Assuming that the number of lines of code to be split submitted by the user is 5000 lines, and the preset line number threshold is 1000 lines, the number of lines of code to be split 5000 lines is greater than the line number threshold 1000 lines, then a cutting granularity greater than the preset granularity threshold will be selected.

[0076] Assuming that the preset granularity threshold is 50 lines, the system may select 100 lines, which is larger than the preset granularity threshold of 50 lines, as the segmentation granularity. The 5,000 lines of code to be segmented will be segmented into 50 code blocks, each containing 100 lines of code. This segmentation method can ensure processing efficiency while retaining sufficient code context information.

[0077] Assuming that the number of lines of code to be split submitted by the user is 200 lines, and the preset line number threshold is 1000 lines, the number of lines of code to be split 200 lines is less than the line number threshold 1000 lines, then a cutting granularity smaller than the preset granularity threshold will be selected.

[0078] Assuming the preset granularity threshold is 50 lines, the system may select 20 lines less than the preset granularity threshold of 50 lines as the cutting granularity, and the 200 lines of code to be split will be split into 10 code blocks, each containing 20 lines of code. Each block contains enough detailed information to facilitate accurate duplicate code detection.

[0079] This adaptive code cutting method provides a good foundation for subsequent AST generation and analysis. Through reasonable code cutting, the processing efficiency can be improved while ensuring the analysis accuracy. This method is not only applicable to static code analysis, but also to incremental code analysis, effectively identifying the repeated parts in the newly added code, and effectively solving the efficiency and accuracy problems of the traditional fixed cutting granularity method when processing codes of different sizes.

[0080] S2. Generate an AST syntax tree using the incremental code.

[0081] In an embodiment of the present invention, the step of generating an AST syntax tree using the incremental code includes:

[0082] Decomposing the incremental code into lexical units;

[0083] combining the lexical units into a grammatical structure;

[0084] The grammatical structure is converted to generate an AST grammar tree.

[0085] In detail, the incremental code is decomposed into several meaningful code blocks, which are collectively referred to as lexical units. For example, int a=2, this code is decomposed into lexical units including: int (keyword), a (identifier), = (operator), 2 (literal), and the lexical units are combined into the main grammatical structure of the AST syntax tree composed of nested elements. Finally, the grammatical structure is converted to generate the AST syntax tree.

[0086] Decomposing the incremental code into lexical units can decompose the complex code structure into basic elements, making subsequent processing more accurate and efficient, helping to identify the basic components in the increment, and simplifying the complexity of grammatical analysis; combining lexical units into grammatical structures can capture the grammatical relationship of the code at the structural level, laying the foundation for building an accurate AST syntax tree. The clarification of the grammatical structure makes the subsequent AST syntax tree generation process more direct and reliable.

[0087] In the embodiment of the present invention, the process of combining lexical units into a grammatical structure is completed by a grammatical analyzer. The grammatical analyzer combines lexical units into a meaningful grammatical structure according to the grammatical rules of the programming language. For example, the above code snippet will be combined into a grammatical structure of variable declaration and initialization.

[0088] The AST syntax tree structures the source code into a tree structure, which can clearly express the syntax and semantic hierarchy of the code. This structured representation enables the compiler and parser to effectively analyze and transform the code, thereby realizing functions such as syntax checking, optimization, and code generation. In addition, the AST syntax tree also supports tools such as refactoring, static analysis, and code intelligence prompts, which improves development efficiency and code quality. By abstracting complex code structures into easy-to-operate tree data, the AST syntax tree not only helps developers understand and manage the code, but also enhances the maintainability and scalability of the code, and promotes the automation process of code analysis and optimization. In this process, the hierarchical relationship and semantic information of the code are clearly expressed. For example, a variable declaration will become a node in the AST syntax tree, and the child nodes of the AST syntax tree include the variable type, variable name, and initial value.

[0089] By gradually decomposing and reconstructing, not only the accuracy of code processing is improved, but also the tool's ability to understand and analyze the code is enhanced. The AST syntax tree provides a structured representation of the incremental code, making static analysis, optimization and reconstruction of the incremental code more efficient and accurate. The AST syntax tree makes the understanding and operation of the incremental code more intuitive, further improving development efficiency and code quality.

[0090] In detail, the present invention can be optimized and expanded in many ways. For example, a cache mechanism can be used to store common grammatical structures to accelerate the generation process of AST syntax trees. In addition, parallel processing technology can be used to simultaneously process the generation of AST syntax trees for multiple code fragments to further improve efficiency. At the same time, it can flexibly adapt to different programming languages ​​and code styles. For example, for object-oriented languages, special attention can be paid to the structure of classes and methods; for functional programming languages, the patterns of function definition and call can be analyzed in detail.

[0091] Compared with the traditional text-based code analysis method, the present invention has significant advantages, which can more deeply understand the structure and semantics of the code, thereby improving the accuracy of duplicate code detection. At the same time, through the optimized generation process, the present invention also greatly improves the processing efficiency, especially when facing a large code base, which not only saves computing resources, but also shortens the time of code analysis.

[0092] S3. Extract the tree feature value string of the AST syntax tree and the class feature value string of each class corresponding to the AST syntax tree.

[0093] In an embodiment of the present invention, the extracting of the tree feature value string of the AST syntax tree and the class feature value string of each class corresponding to the AST syntax tree includes:

[0094] Filter out the key nodes of the AST syntax tree;

[0095] Recursively construct a key syntax tree using the key nodes;

[0096] Traversing the key syntax tree in the order of left subtree, root node, and right subtree to obtain the characteristics of each node;

[0097] Using the features of each node to form a tree feature value string;

[0098] Extracting a class feature value from the features of each node;

[0099] The class feature values ​​are concatenated into a class feature value string.

[0100] In detail, the AST syntax tree is a data structure in code parsing. Each node usually represents a structure or element in the syntax, such as an expression, an operator, a control flow, etc. In program analysis, key nodes may be function definitions, class declarations, conditional statements, etc. These key nodes are identified by traversing the AST syntax tree. Traversal can use methods such as depth-first search (DFS) or breadth-first search (BFS) to mark nodes that meet the criteria as key nodes and extract them from the AST syntax tree.

[0101] Create a new tree node for each key node, and build the tree structure between key nodes according to the original parent-child relationship in the AST syntax tree. For example, if a key node has child nodes, corresponding child nodes should also be created in the key syntax tree. Starting from each key node, recursively build the subtree to ensure that the child nodes of each key node are also correctly added to the tree.

[0102] The key syntax tree is traversed to obtain the features of each node. The traversal is performed in the order of "left subtree, root node, right subtree". During the traversal, features are extracted for each node. These features may include the node type, value, number of child nodes, depth, etc.

[0103] Format the features of each node into a unified string representation, concatenate the formatted feature strings into a complete tree feature value string in the order of traversal, and concatenate all class feature values ​​into a class feature value string in order, which can be used as the input features of the model.

[0104] In the embodiment of the present invention, the selection of key nodes can be based on multiple strategies. Important grammatical structures such as function definition, class definition, loop structure, conditional statement, etc. can be selected as key nodes. Another method is to use a heuristic algorithm to determine the importance of a node based on factors such as the type, depth, and subtree size of the node.

[0105] The present invention can not only effectively extract the structural features of the code, but also reflect the semantic information of the code to a certain extent. By retaining the method name and parameter type, the basic function of the function can be captured; by retaining the if-else structure, the logical flow of the code can be reflected. This information is very valuable for subsequent similarity calculation and duplicate code detection.

[0106] In detail, extracting the tree feature value string of the AST syntax tree and the class feature value string corresponding to each class of the AST syntax tree can significantly improve the efficiency and accuracy of data processing. By using the key syntax tree structure and extracting core features, the feature value string finally generated is more representative, and subsequent duplicate code detection can be analyzed and compared more quickly.

[0107] S4. Calculate the similarities between the tree feature value character string and the class feature value character string one by one, and filter out the classes corresponding to the class feature value character strings whose similarities are less than a preset similarity threshold.

[0108] In the embodiment of the present invention, the step of calculating the similarity between the tree feature value string and the class feature value string one by one includes:

[0109] Splitting the tree feature value string into a plurality of basic tree words, and splitting the class feature value string into a plurality of basic class words;

[0110] Calculating the occurrence frequencies of the several basic tree words and the several basic class words in the incremental code to obtain the word frequencies of the several basic tree words and the word frequencies of the several basic class words;

[0111] Converting the word frequencies of the several basic tree words and the word frequencies of the several basic class words into tree word frequency vectors and class word frequency vectors;

[0112] The similarity between the tree word frequency vector and the class word frequency vector is calculated using the cosine similarity formula.

[0113] In detail, cosine similarity effectively measures the angular relationship between vectors, which is of great significance in code similarity analysis. Compared to other metrics such as Euclidean distance, cosine similarity is insensitive to the length of the vector, which makes it more suitable for comparing code snippets of different lengths.

[0114] The cosine similarity formula is as follows:

[0115]

[0116] Among them, A represents the tree word frequency vector, B represents the class word frequency vector, ||A|| represents the modulus of the tree word frequency vector, and ||B|| represents the modulus of the class word frequency vector.

[0117] Cosine similarity can measure the similarity between two vectors in high-dimensional space, and is not affected by the size of the vectors. Cosine similarity provides an effective way to measure the similarity of text, documents, or data features by calculating the cosine value of the angle between vectors. It can accurately capture the correlation and similarity between data, regardless of the absolute size of the data.

[0118] Cosine similarity can effectively measure the similarity between the tree word frequency vector and the class word frequency vector. Cosine similarity focuses on the relative frequency distribution of words rather than the overall length. By calculating cosine similarity, the structure and content similarity of incremental code snippets can be accurately compared to discover potential code duplication or similar patterns.

[0119] In detail, for the segmentation of basic vocabulary, segmentation can be performed based on the grammatical structure of the code, with each grammatical unit as a basic vocabulary, or several consecutive tokens can be combined into a basic vocabulary through the sliding window technology. When calculating the word frequency, different weights can be assigned according to the position of the word in the code structure, or the importance of the word frequency can be adjusted using the inverse document frequency (IDF). When converting the word frequency into a vector, a sparse vector representation can be used to save storage space and improve computational efficiency.

[0120] In an embodiment of the present invention, for Java code, key nodes such as method declaration, variable declaration, loop structure, etc. in the abstract syntax tree are first extracted as basic vocabulary. Then, the number of occurrences of these words in each class is counted to generate a word frequency vector. Finally, the cosine similarity formula is used to calculate the similarity between different classes. If it is found that the similarity exceeds a preset threshold, it is marked as potential duplicate code.

[0121] For Python code, since Python's syntax is more flexible, the indentation level can also be used as part of the basic vocabulary to better capture the structural features of Python code. At the same time, using the TF-IDF (term frequency-inverse document frequency) method to calculate the term frequency can reduce the weight of some common grammatical structures and highlight the truly distinguishing code features.

[0122] In the embodiment of the present invention, the step of filtering out the class corresponding to the class feature value string whose similarity is less than a preset similarity threshold comprises:

[0123] Determining whether the similarity is less than a preset similarity threshold;

[0124] If the similarity is greater than or equal to a preset similarity threshold, the incremental code is set as a warning, and the incremental code is sent to the reviewer to confirm whether to save it in the database;

[0125] If the similarity is less than a preset similarity threshold, the class corresponding to the class feature value string in the similarity will be retained.

[0126] In detail, different processing strategies are adopted according to the comparison results between the calculated similarity and the preset similarity threshold, which can not only automatically process obviously dissimilar codes, but also manually review the codes that may be duplicated, thereby improving the accuracy and reliability of duplicate code detection.

[0127] By calculating the similarity between the tree feature value string and the class feature value string, and filtering out the class feature value strings whose similarity is lower than a preset threshold, the duplicate code in the class can be effectively identified and reduced. Similarity calculation methods, such as edit distance, cosine similarity, or Jaccard similarity, can help quantify the similarity between two strings. After calculating the similarity, a similarity threshold, such as 0.5, is set as a screening criterion. If the similarity between the class feature value string and the target tree feature value string is less than the similarity threshold of 0.5, it is considered that the class of the class feature value string may contain code segments that are significantly different from the target tree structure, thereby identifying those code segments that may be redundant or repeated. If the similarity between the class feature value string and the target tree feature value string is greater than the similarity threshold of 0.5, the code is marked as a warning and sent to a designated reviewer. The reviewer can view the code through a dedicated interface, compare its differences with similar code, and make a decision to keep or modify it.

[0128] By reducing duplicate code, the maintainability and readability of the code can be improved, while reducing potential errors and performance issues caused by redundant code. Therefore, it not only helps to identify the actual duplicate code in the code, but also provides data support for code optimization, ensuring the overall quality and efficiency of the software system.

[0129] S5. Perform text merging on the incremental code and the class corresponding to the filtered class feature value string to obtain an updated code class.

[0130] In the embodiment of the present invention, the text merging of the incremental code with the class corresponding to the filtered class feature value string to obtain the updated code class includes:

[0131] Formatting the incremental code to obtain a formatted code;

[0132] The formatting code is merged with the class corresponding to the filtered class characteristic value string using a merge command to obtain an update code class.

[0133] Formatting the incremental code makes the code structure clearer and more standardized, thereby reducing potential syntax errors and format inconsistencies. The formatted code is easier to understand and maintain, which helps improve code readability and team collaboration efficiency. Using the merge command to merge the formatted code with the class corresponding to the filtered class feature value string can ensure that the new code is consistent with the structure and characteristics of the existing class, reduce the risk of code conflicts, and improve the accuracy of code merging and the convenience of maintenance.

[0134] In detail, in version control devices, such as Git, the merge command is typically used to merge changes from two or more branches into a single branch, integrating the incremental code into the class branch corresponding to the filtered class feature value string. The merge command allows multiple developers to work on different branches, and finally merge their changes, while maintaining the stability of the main branch, thereby improving the efficiency of text merging of incremental code with filtered classes.

[0135] S6. Generate an updated AST syntax tree using the updated code class, and save the updated AST syntax tree to a distributed file database.

[0136] In the embodiment of the present invention, JavaParser is used to parse the update code class, and the syntax tree generated by the parsing is traversed. The update nodes obtained by the traversal are used to generate an update AST syntax tree, which can accurately analyze and modify the code structure, so that developers can understand the components of the code, perform fine-grained code optimization or reconstruction, and ensure the consistency and correctness of the code. The updated AST syntax tree finally generated can provide a clear code view, support further automated processing and analysis, and improve the efficiency and accuracy of code management.

[0137] In an embodiment of the present invention, the step of saving the updated AST syntax tree to a distributed file database includes:

[0138] Use the ToString method to obtain the string data of the updated AST syntax tree;

[0139] Construct a storage file for storing the string data of the updated AST syntax tree;

[0140] Writing the string data of the updated AST syntax tree into the storage file;

[0141] The storage file to which the string data has been written is saved to the distributed file database.

[0142] In detail, use the ToString method in the programming language to convert the updated AST syntax tree into a string representation, and output the structure and content of the AST syntax tree in the form of a string to facilitate subsequent storage and processing. Create a file for storing AST string data, select an appropriate file format, and determine the storage location of the file. Write the generated AST string data to the file just created, use the storage system's API or tool to upload the file, and save the stored file to the distributed file database.

[0143] Furthermore, the ToString method provides a standard way to convert the update AST syntax tree into a string representation, so that the content of the update AST syntax tree can be easily output, recorded, or displayed. For example, during debugging, developers can use the ToString method to quickly view the internal state and data of the update AST syntax tree to help locate problems. The ToString method can customize the string representation of the update AST syntax tree to provide a more readable output, which is especially important for logging and user interface display. In addition, it helps to serialize the update AST syntax tree and convert the update AST syntax tree into a string form for easy storage or transmission, such as writing data to a file or sending it to the network. The ToString method simplifies the data presentation and processing of the update AST syntax tree, improving the maintainability and readability of the code.

[0144] By storing the string data of the updated AST syntax tree in a distributed file database, efficient data storage and management can be achieved, making full use of the scalability and fault tolerance of the distributed file system, which not only ensures the persistence and reliability of the data, but also supports the rapid access and analysis of large-scale data. At the same time, the use of a distributed file database improves the scalability of the device, allowing subsequent data processing and analysis to be performed in a distributed environment, thereby improving the overall processing efficiency.

[0145] like Figure 2 , which is a functional module diagram of an AST-based duplicate code detection and analysis device provided by an embodiment of the present invention.

[0146] The AST-based duplicate code detection and analysis device 100 of the present invention can be installed in an electronic device. According to the functions implemented, the AST-based duplicate code detection and analysis device 100 may include a data acquisition module 101, a syntax tree generation module 102, a feature extraction module 103, a similarity comparison module 104, a text merging module 105 and a data storage module 106. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0147] In this embodiment, the functions of each module / unit are as follows:

[0148] The data acquisition module 101 is used to obtain the incremental code after the text is cut;

[0149] A syntax tree generation module 102, used to generate an AST syntax tree using the incremental code;

[0150] A feature extraction module 103 is used to extract a tree feature value string of the AST syntax tree and a class feature value string corresponding to each class of the AST syntax tree;

[0151] A similarity comparison module 104 is used to calculate the similarity between the tree feature value string and the class feature value string one by one, and filter out the classes corresponding to the class feature value strings whose similarity is less than a preset similarity threshold;

[0152] A text merging module 105 is used to perform text merging of the incremental code with the class corresponding to the filtered class feature value string to obtain an updated code class;

[0153] The data storage module 106 is used to generate an updated AST syntax tree using the updated code class, and save the updated AST syntax tree to a distributed file database.

[0154] In detail, each module described in the AST-based duplicate code detection and analysis device 100 in the embodiment of the present invention is used in the same manner as described above. Figures 1 to 3 The same technical means as the AST-based duplicate code detection and analysis method described in , and can produce the same technical effects.

[0155] like Figure 3 , which is a schematic diagram of the structure of an electronic device for implementing a duplicate code detection and analysis method based on AST provided by an embodiment of the present invention.

[0156] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an AST-based duplicate code detection and analysis program.

[0157] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a duplicate code detection and analysis program based on AST, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data.

[0158] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 11 may be an internal storage unit of an electronic device in some embodiments, such as a mobile hard disk of the electronic device. The memory 11 may also be an external storage device of an electronic device in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device, such as the code of a duplicate code detection and analysis program based on AST, but may also be used to temporarily store data that has been output or is to be output.

[0159] The communication bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0160] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0161] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0162] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0163] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0164] The AST-based repeated code detection and analysis program stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve: obtaining incremental code after text segmentation; generating an AST syntax tree using the incremental code; extracting the tree feature value string of the AST syntax tree and the class feature value string corresponding to each class of the AST syntax tree; calculating the similarity between the tree feature value string and the class feature value string one by one, and filtering out the classes corresponding to the class feature value strings whose similarity is less than a preset similarity threshold; text merging the incremental code with the filtered class corresponding to the class feature value string to obtain an updated code class; generating an updated AST syntax tree using the updated code class, and saving the updated AST syntax tree to a distributed file database.

[0165] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0166] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0167] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor of an electronic device, it can achieve: obtaining incremental code after text segmentation; generating an AST syntax tree using the incremental code; extracting the tree feature value string of the AST syntax tree and the class feature value string corresponding to each class of the AST syntax tree; calculating the similarity between the tree feature value string and the class feature value string one by one, and filtering out the class corresponding to the class feature value string whose similarity is less than a preset similarity threshold; text merging the incremental code with the class corresponding to the filtered class feature value string to obtain an updated code class; generating an updated AST syntax tree using the updated code class, and saving the updated AST syntax tree to a distributed file database.

[0168] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0169] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0171] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0172] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is not limited only according to the above description, and it is intended that all changes within the meaning and scope of equivalent elements within the scope of protection are included in the present invention.

[0173] The embodiment of the present invention can acquire and process relevant data based on code detection, wherein code detection refers to checking errors, loopholes or problems that do not meet coding standards in the code through tools, mainly used to improve code quality and ensure software reliability and security.

[0174] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention. If software tools or components other than those of the company appear in the embodiments of the present invention, they are merely used for illustrative purposes and do not represent actual use.

Claims

1. A duplicate code detection and analysis method based on AST, characterized in that: The method comprises: Get the incremental code after text cutting; Generate an AST syntax tree using the incremental code; Extracting a tree feature value string of the AST syntax tree and a class feature value string corresponding to each class of the AST syntax tree; Calculating the similarities between the tree feature value string and the class feature value string one by one, and filtering out the classes corresponding to the class feature value strings whose similarities are less than a preset similarity threshold; Merge the incremental code with the class corresponding to the filtered class feature value string to obtain an update code class; An updated AST syntax tree is generated using the updated code class, and the updated AST syntax tree is saved in a distributed file database.

2. The AST-based duplicate code detection and analysis method according to claim 1, characterized in that: The step of obtaining the incremental code after text segmentation includes: Get the number of lines of code to be split submitted by the user; Determine whether the number of lines of code to be segmented is greater than a preset line number threshold; If the number of lines of code to be segmented is greater than the line number threshold, determining the segmentation granularity to be a granularity greater than a preset granularity threshold; If the number of lines of code to be segmented is less than or equal to the line number threshold, determining the segmentation granularity to be a granularity less than a preset granularity threshold; The code to be split submitted by the user is divided according to the determined cutting granularity to obtain the incremental code.

3. The AST-based duplicate code detection and analysis method according to claim 1, characterized in that: The step of generating an AST syntax tree using the incremental code includes: Decomposing the incremental code into lexical units; combining the lexical units into a grammatical structure; The grammatical structure is converted to generate an AST grammar tree.

4. The AST-based duplicate code detection and analysis method according to claim 1, characterized in that: The tree feature value string of the AST syntax tree and the class feature value string of each class corresponding to the AST syntax tree are extracted, including Filter out the key nodes of the AST syntax tree; Recursively construct a key syntax tree using the key nodes; Traversing the key syntax tree in the order of left subtree, root node, and right subtree to obtain the characteristics of each node; Using the features of each node to form a tree feature value string; Extracting a class feature value from the features of each node; The class feature values ​​are concatenated into a class feature value string.

5. The AST-based duplicate code detection and analysis method according to claim 1, characterized in that: The step of calculating the similarity between the tree feature value string and the class feature value string one by one includes: Splitting the tree feature value string into a plurality of basic tree words, and splitting the class feature value string into a plurality of basic class words; Calculating the occurrence frequencies of the several basic tree words and the several basic class words in the incremental code to obtain the word frequencies of the several basic tree words and the word frequencies of the several basic class words; Converting the word frequencies of the several basic tree words and the word frequencies of the several basic class words into tree word frequency vectors and class word frequency vectors; The similarity between the tree word frequency vector and the class word frequency vector is calculated using the cosine similarity formula.

6. The AST-based duplicate code detection and analysis method according to claim 1, characterized in that: The filtering out the class corresponding to the class feature value string whose similarity is less than a preset similarity threshold comprises: Determining whether the similarity is less than a preset similarity threshold; If the similarity is greater than or equal to a preset similarity threshold, the incremental code is set as a warning, and the incremental code is sent to the reviewer to confirm whether to save it in the database; If the similarity is less than a preset similarity threshold, the class corresponding to the class feature value string in the similarity will be retained.

7. The AST-based duplicate code detection and analysis method according to claim 1, characterized in that: Saving the updated AST syntax tree to a distributed file database includes: Use the ToString method to obtain the string data of the updated AST syntax tree; Construct a storage file for storing the string data of the updated AST syntax tree; Writing the string data of the updated AST syntax tree into the storage file; The storage file to which the string data has been written is saved to the distributed file database.

8. A duplicate code detection and analysis device based on AST, characterized in that: The device comprises: A data acquisition module is used to obtain the incremental code after text segmentation; A syntax tree generation module, used to generate an AST syntax tree using the incremental code; A feature extraction module, used to extract a tree feature value string of the AST syntax tree and a class feature value string corresponding to each class of the AST syntax tree; A similarity comparison module, used to calculate the similarity between the tree feature value string and the class feature value string one by one, and filter out the classes corresponding to the class feature value strings whose similarity is less than a preset similarity threshold; A text merging module, used for text merging the incremental code with the class corresponding to the filtered class feature value string to obtain an updated code class; The data storage module is used to generate an updated AST syntax tree using the updated code class and save the updated AST syntax tree to a distributed file database.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the AST-based duplicate code detection and analysis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the AST-based duplicate code detection and analysis method as described in any one of claims 1 to 7 is implemented.

Images