Multi-stage multi-level code feature efficient matching method

Abstract

The invention discloses a multi-stage multi-level code feature efficient matching method. The method comprises the steps that a to-be-matched standard code file of file feature information and file code fragment feature information is input; when the file suffix name and the feature value of the to-be-matched code file are the same as those in the code feature information knowledge base, taking the matched file in the knowledge base as a file-level matching result, otherwise, performing fragment-level code feature matching on the to-be-matched code file; sequentially matching the feature values of the feature information of the fragments in the to-be-matched file with the feature values of all code fragments in a knowledge base to obtain a fragment-level matching result; counting the number of successfully matched code snippets under different files in the knowledge base, and selecting the file with the maximum number as a file-level matching result; and counting the file number of the file-level matching result under each item in the knowledge base, selecting the item with the first sequence as an item similar to the to-be-matched code file as an item-level matching result, and evaluating the similarity between the to-be-matched file and the item-level matching result.

Description

Technical Field

[0001] The invention relates to a multi-stage and multi-level code feature efficient matching method, belonging to the technical field of intelligent software engineering. Background Art

[0002] With the rapid development of information technology, software is increasingly used in all walks of life, and the scale and complexity of software systems are also increasing. The code base involved in modern software development usually contains tens of thousands of lines of code. When tracing the source of these codes, feature matching becomes an indispensable part of the process.

[0003] Existing code feature matching methods usually use string matching and syntax and semantic analysis matching to identify and match code snippets for the fragments of two codes. Their limitations are firstly that they only support matching based on code snippets; secondly, they usually have the following defects in the matching process: string matching is completely based on character sequences and cannot capture the structure and semantic information of the code. For slightly modified code or code snippets with different syntax, the matching accuracy is poor. Syntax and semantic analysis matching is based on abstract syntax trees for matching. For large code bases and complex code structures, syntax analysis may become expensive and complicated, and it is necessary to consider the syntax and semantic rules of different programming languages, which increases the complexity of implementation, requires more computing resources and time, and has poor matching efficiency. In addition, for complex syntax and semantic relationships, false positives and false negatives may occur. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide a multi-stage and multi-level code feature efficient matching method, which can quickly and accurately locate the matching file-level and code fragment-level results by analyzing and matching code features of different granularities, and fit the project-level matching results, thereby significantly improving the efficiency and accuracy of code tracing.

[0005] The technical solution of the present invention is: a multi-stage and multi-level code feature efficient matching method, comprising:

[0006] S1 inputs the code file to be matched, including file feature information and file code fragment feature information;

[0007] Identify the programming language type of the code file according to the file suffix value in the input file feature information; assign a thread to each programming language to process subsequent matching tasks of the corresponding language;

[0008] In each thread, when the file suffix and feature value in the file feature information are the same as those in the code feature information knowledge base, the matching file in the knowledge base is taken as the file-level matching result, and the process directly proceeds to step S3; otherwise, the process proceeds to step S2 to perform segment-level code feature matching on the matching code file;

[0009] S2 sequentially matches the feature values ​​of the feature information of each code snippet in the code file to be matched with the feature values ​​of all code snippets in the knowledge base. If the same feature values ​​are matched, the code snippet successfully matches the current code snippet in the knowledge base, and a snippet-level matching result is obtained;

[0010] Count the number of successfully matched code snippets in different files in the knowledge base, sort the files in the knowledge base from high to low according to the number of matched code snippets in the files, and select the file with the largest number of matched code snippets as the file-level matching result;

[0011] S3 counts the number of files in the file-level matching results under each project in the knowledge base, and sorts each project from high to low according to the number of files that can be matched; the project that ranks first is selected as the project that is similar to the code file to be matched as the project-level matching result.

[0012] Preferably, the code feature information knowledge base stores file feature information and file code snippet feature information of all open source projects, and this information is stored in an open source distributed search and analysis engine index.

[0013] Preferably, the feature information of each code snippet in the code file to be matched includes a feature value of the code snippet, a starting line number of the code snippet, and an ending line number of the code snippet.

[0014] Preferably, the feature value during matching is a hash value.

[0015] Preferably, when matching the feature value of the feature information of each file code fragment in the code file to be matched with the feature values ​​of all file code fragments in the knowledge base: if there are two or more consecutive code fragment feature values ​​in the code file to be matched that are the same as two or more adjacent consecutive code fragment feature values ​​in the knowledge base matching file, then the adjacent consecutive fragments in the knowledge base matching file are aggregated.

[0016] Preferably, when adjacent continuous fragments are aggregated, the aggregation method is sequential superposition, that is, the starting line number of the aggregated code fragment is the starting line number of the first aggregated code fragment, and the ending line number is the ending line number of the last aggregated code fragment, and the aggregated matching code fragment composed of multiple code fragments is used as the fragment-level matching result.

[0017] Preferably, the method for calculating the degree of similarity between the code file to be matched and the project-level matching result is: obtaining the total number of code matching lines in the project, and calculating the ratio of the total number of code matching lines in the project to the total number of project code lines.

[0018] Preferably, the file feature information and the file code snippet feature information are both represented by standard JSON files.

[0019] Preferably, a file list of the corresponding language is read in each thread to perform file-level code feature matching and fragment-level code feature matching.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] (1) The present invention introduces a multi-stage and multi-level code feature matching method, which effectively improves the efficiency and accuracy of code feature matching through matching at different levels (code fragment level, file level, project level);

[0022] (2) The present invention adopts a multi-threaded processing strategy, where each language file is processed by one thread, and through thread pool management, the concurrent processing capability and system resource utilization are improved. This method is particularly important in the case of large-scale code bases and can significantly accelerate the matching process;

[0023] (3) The calculation of the project-level matching results of the present invention adopts the quantitative accumulation of the number of matching lines at the code snippet level of the file, so as to more accurately reflect the code similarity and duplication of the entire project;

[0024] (4) The multi-level matching method of the present invention is not only applicable to code similarity analysis, but also can be applied to code tracing, duplicate code detection and software quality assessment. Specifically:

[0025] Code traceability: The matching results can accurately track and analyze the source and change history of the code, helping developers understand the evolution process and maintenance requirements of the code;

[0026] Duplicate code detection: effectively detect and manage duplicate code in projects, improve code reuse rate and maintenance efficiency;

[0027] Software quality assessment: By analyzing code similarity and structure, the structural rationality of the software and the standardization of code design are evaluated to help improve software quality and maintainability;

[0028] (5) The present invention can be further optimized and customized according to the needs and characteristics of a specific project. For example, the parameter settings of the matching algorithm can be adjusted according to the programming language characteristics and code base size of the project to achieve a more optimized and efficient matching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a multi-stage and multi-level code feature matching flow chart of the present invention;

[0030] Figure 2 This is the multi-threaded processing flow of the matching task of the present invention. DETAILED DESCRIPTION

[0031] In order to solve the above technical problems, the present invention innovatively proposes a multi-stage and multi-level code feature efficient matching method. By analyzing and matching code features of different granularities, the method can quickly and accurately locate the matching file-level and code fragment-level results, and fit the project-level matching results, significantly improving the efficiency and accuracy of code tracing.

[0032] Among them, "multi-stage" specifically refers to the scientific division of the entire matching process into several continuous and interconnected steps. Each stage focuses on processing a specific type of data or performing a specific analysis task to ensure the gradual refinement of the matching results. Specifically, it includes three stages: code snippet level matching, code file level matching, and code project level matching. Fragment-level code matching focuses on the matching of snippets or code segments in the code, file-level code matching focuses on the matching between single code files, and project-level code matching involves the analysis of code files and snippet data of the entire software project.

[0033] More importantly, the present invention adopts the concept of "multi-level", which is inspired by the distinction between Type 1 (line-by-line identical) and Type 2 (structurally similar) clones, and further refines the analysis dimension of code features. This strategy does not just stop at the surface character-level comparison, but goes deeper into the structural and semantic levels: first, at the Type 1 clone level, the system focuses on identifying exactly the same code segments to achieve basic duplicate code detection; then, at the more complex Type 2 clone level, the structural pattern, logical flow, and semantic intent of the code are deeply analyzed, and functionally equivalent or highly similar parts can be identified even when the code expression changes. Such a multi-level analysis framework ensures the depth and breadth of the match and improves the accuracy of code matching.

[0034] At the same time, in order to maximize processing efficiency, this system incorporates a multi-threaded processing mechanism, especially when performing computationally intensive operations such as file-level feature matching and code snippet-level feature matching. Through parallel processing, the system can efficiently utilize computing resources, significantly speed up the matching process even when facing large-scale code bases, maintain a high degree of responsiveness and processing speed, and improve the efficiency of the code matching process during tracing.

[0035] The technical path of the present invention is divided into three stages: Figure 1 As shown, the details are as follows:

[0036] Step 1: File-level code feature matching.

[0037] (1) Multi-thread allocation according to programming language type.

[0038] Input the code file to be matched. The code file to be matched contains two kinds of feature information data, including file feature information and file code fragment feature information, as shown in Table 1 and Table 2. Both kinds of feature information are represented by standard JSON files.

[0039] Table 1 File feature information

[0040] Characteristic elements illustrate File feature value hash By extracting features from the file content, a unique hash value sequence is generated Suffix value extension File suffix

[0041] Table 2 File code snippet feature information

[0042]

[0043] According to the input code file suffix value to be matched, the programming language type of the code file is identified. Table 3 shows the mapping table of 6 common programming languages ​​and file suffixes. Each programming language can be assigned a thread to handle the subsequent matching tasks of the corresponding language, such as Figure 2 As shown in the figure, the thread pool is used to manage multi-threaded matching tasks. According to the mapping of file suffixes, the matching tasks are distributed to threads in different thread pools for execution. Each thread is responsible for reading the file list of the corresponding language and performing file-level code feature matching and fragment-level code feature matching.

[0044] Table 3 Programming language and file suffix mapping table

[0045] programming language File suffix java .java Python .py C .ch cpp .cpp.cxx vhdl .vhd.vhdl Verilog .v,.sv,.vh

[0046] (2) File hash value matching.

[0047] For a JSON file with file feature information as input, extract the feature information in the file. Each JSON object represents a file and includes the following information:

[0048] hash: The hash value of a file, which is used to uniquely identify the content of the file.

[0049] extension: The file suffix.

[0050] According to the extension and hash as the screening conditions, the existing code feature information knowledge base is matched and retrieved. If the file with the same extension and hash is retrieved, that is, the code file to be matched completely matches the file suffix in the code feature information knowledge base and the feature value is the same, the file is considered to be 100% matched, and the matching file in the knowledge base is taken as the file-level matching result. The code feature matching at the code snippet level in step 2 will no longer be performed for this file. If the above result is not matched in the code feature information knowledge base, the code feature matching at the code snippet level in step 2 is performed. The code feature information knowledge base used for matching above stores the file feature information and file code snippet feature information of all open source projects, which are stored in the Elasticsearch (an open source distributed search and analysis engine) index. Each file uses its hash value as a unique identifier, and stores information such as the file path and file content in the corresponding document field. For the file to be matched, query its file hash value or code snippet hash value. Taking advantage of the Elasticsearch index, files with the same hash value can be quickly located, avoiding traversal searches in traditional file systems. It is suitable for large-scale file matching and file tracing, and can handle millions or even hundreds of millions of files. The hash value, as the characteristic value of the unique identifier, ensures the accuracy of the matching results.

[0051] Step 2: Code feature matching at the code snippet level.

[0052] For the files to be matched that have no matching results in the file-level code feature matching stage, code fragment-level code feature matching is performed.

[0053] (1) Fragment feature acquisition.

[0054] The code snippet feature information is obtained for the matching file.

[0055] Extract the code snippet feature information from the JSON file containing the information. Each JSON object represents a code snippet and includes the following information:

[0056] hash: The hash value of the code snippet, which is used to uniquely identify the code snippet.

[0057] linestart: The starting line number of the code snippet.

[0058] linend: The ending line number of the code snippet.

[0059] As shown in Table 4, an example of feature information of a code fragment (block) of a file to be matched is given.

[0060] 0Table 4 File code fragment (block) feature information example

[0061]

[0062] Since the code snippets in the file are composed of several code snippets (blocks) divided into the file, each snippet contains N lines of valid code. The snippet feature acquisition process is extracted through a sliding window method, N lines of valid code are 1 window, and the hash value of each window is used as the feature value of the snippet code.

[0063] (2) Fragment feature matching.

[0064] Code snippet-level matching is performed based on all code snippets in the file to be matched. The hash value of each window in the file to be matched is compared with the hash value of the file code snippet in the code feature information knowledge base. If the same hash value is matched, it means that the code snippet has the same feature as the code snippet in the knowledge base, and the window is slid down according to the size until all code snippets in the file to be matched are detected.

[0065] (3) Fragment feature aggregation.

[0066] The number of code snippets successfully matched in the knowledge base in step 2 (2) is counted, and the code snippets are sorted from high to low according to the number of features that match the same file. The file in the knowledge base with the largest number of features identical to the code snippets in the file to be matched is selected as the matching file (file-level matching result). If there are two or more consecutive code snippets in the file to be matched that have the same feature values ​​as two or more consecutive code snippets in the knowledge base matching file, then these consecutive snippets are aggregated and merged into a larger code snippet whose starting line number is the starting line number of the first aggregated code snippet and whose ending line number is the ending line number of the last aggregated code snippet. The longest matching code snippet consisting of multiple code snippets is obtained, which is the snippet-level matching result of the file to be matched.

[0067] Step 3: Project-level code feature matching.

[0068] (1) Matching files are aggregated by project.

[0069] The file-level matching results and fragment-level matching results of step 2 are aggregated with the files belonging to the same project in the knowledge base, and the number of files successfully matched in the knowledge base is counted by project and sorted from high to low.

[0070] (2) Project similarity calculation.

[0071] Select the first ranked project as the project similar to the code file to be matched, accumulate the number of code snippet matching lines in all file-level matching stages and code-level matching stages, and get the total number of code matching lines for the project. Calculate the ratio of the total number of project code matching lines to the total number of project code lines, as shown in the following formula 1, and get the project similarity. Project similarity reflects the similarity between the file to be matched and the project in the knowledge base.

[0072] The project-level matching results can be obtained by calculating the row count aggregation of all file-level matching results.

[0073] The specific calculation formula is:

[0074]

[0075] The present invention overcomes the shortcomings of the prior art and provides a multi-stage and multi-level code feature efficient matching method. By analyzing and matching code features of different granularities, it can quickly and accurately locate the matching file-level and code fragment-level results, and fit the project-level matching results, significantly improving the efficiency and accuracy of code tracing.

[0076] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A multi-stage and multi-level code feature efficient matching method, characterized by The steps include: S1 inputs the code file to be matched, including file feature information and file code fragment feature information; Identify the programming language type of the code file according to the file suffix value in the input file feature information; assign a thread to each programming language to process subsequent matching tasks of the corresponding language; In each thread, when the file suffix and feature value in the file feature information are the same as those in the code feature information knowledge base, the matching file in the knowledge base is taken as the file-level matching result, and the process directly proceeds to step S3; otherwise, the process proceeds to step S2 to perform segment-level code feature matching on the matching code file; S2 sequentially matches the feature values ​​of the feature information of each code snippet in the code file to be matched with the feature values ​​of all code snippets in the knowledge base. If the same feature values ​​are matched, the code snippet is successfully matched with the current code snippet in the knowledge base, and a snippet-level matching result is obtained; Count the number of successfully matched code snippets in different files in the knowledge base, sort the files in the knowledge base from high to low according to the number of matched code snippets in the files, and select the file with the largest number of matched code snippets as the file-level matching result; S3 counts the number of files in the file-level matching results under each project in the knowledge base, and sorts each project from high to low according to the number of files that can be matched; the project that ranks first is selected as the project that is similar to the code file to be matched as the project-level matching result.

2. The multi-stage and multi-level code feature efficient matching method according to claim 1, characterized in that: The code feature information knowledge base stores the file feature information and file code snippet feature information of all open source projects. This information is stored in the open source distributed search and analysis engine index.

3. The multi-stage and multi-level code feature efficient matching method according to claim 1 is characterized in that: The feature information of each code snippet in the to-be-matched code file includes a feature value of the code snippet, a starting line number of the code snippet, and an ending line number of the code snippet.

4. The multi-stage and multi-level code feature efficient matching method according to claim 1, characterized in that: The feature value when matching is a hash value.

5. The multi-stage and multi-level code feature efficient matching method according to claim 1 is characterized in that: When matching the feature values ​​of the feature information of each file code snippet in the code file to be matched with the feature values ​​of all file code snippets in the knowledge base: if there are two or more consecutive code snippet feature values ​​in the code file to be matched that are the same as two or more adjacent consecutive code snippet feature values ​​in the knowledge base matching file, then the adjacent consecutive snippets in the knowledge base matching file are aggregated.

6. The multi-stage and multi-level code feature efficient matching method according to claim 5, characterized in that: When aggregating adjacent continuous fragments, the aggregation method is sequential superposition, that is, the starting line number of the aggregated code fragment is the starting line number of the first aggregated code fragment, and the ending line number is the ending line number of the last aggregated code fragment. The aggregated matching code fragment composed of multiple code fragments is used as the fragment-level matching result.

7. The multi-stage and multi-level code feature efficient matching method according to claim 1, characterized in that: The method for calculating the similarity between the code file to be matched and the project-level matching result is as follows: obtain the total number of code matching lines for the project, and calculate the ratio of the total number of code matching lines in the project to the total number of project code lines.

8. The multi-stage and multi-level code feature efficient matching method according to claim 1, characterized in that: Both file feature information and file code snippet feature information are represented by standard JSON files.

9. The multi-stage and multi-level code feature efficient matching method according to claim 1, characterized in that: In each thread, a file list of the corresponding language is read to perform file-level code feature matching and fragment-level code feature matching.

Images