Tracing Method for Homologous Code

Abstract

The present invention relates to a source code tracing method, which includes the following steps: establishing a standardized vulnerability knowledge base by crawling, obtaining open source code that meets the specifications, and storing the open source code results after preprocessing; building a Neo4j graph database; and performing business processing. Thus, based on the graph library association relationship between the content hash value of the source code and the directory hash value, the detection speed can be improved. The full-scale problem of detection through binary file and file collision is solved, and the speed is increased by more than 10 times, thereby providing engine support for further open source software supply chain security detection. Through multi-dimensional clustering, it is possible to distinguish whether a certain module or its subclass is used based on the calculated result score, and solve the problem of inaccurate supply chain detection results caused by the inclusion relationship between open source project modules. It can provide more accurate open source projects and versions, so that vulnerability detection in the software industry security industry can be more accurate.

Description

Technical Field

[0001] The present invention relates to a source tracing method, in particular to a source tracing method for homologous source codes. Background Art

[0002] With the popularization of the Internet and the rapid development of software, security vulnerabilities are constantly emerging, resulting in the emergence of vulnerability information intelligence agencies, such as Snyk, NVD, CNVD, WhiteSource, blackDuck and other authoritative vulnerability agencies and security vendors. They have all announced software vulnerabilities. As for the use of software security supply chain, we usually introduce external dependencies (such as jar packages or modules) when writing programs. Other people's jar packages or modules have a lot of content, and most of them are indirect references. Some even directly use part of a certain open source project for packaging. How to confirm the specific open source software we use is a difficult problem. If we can confirm the open source software and version used by a specific project, we can promptly discover its vulnerabilities and update them in time, which can improve the security of the software supply chain.

[0003] To identify specific open source projects, we first need to obtain the data sources of the relevant open source projects, create a directory and content fingerprint feature library, and finally calculate weights based on the matching projects. Finally, we use similarity clustering to select the most correct project and version based on the detection rate. In view of the above shortcomings, the designers actively researched and innovated to create a source code traceability method with greater industrial value.

[0004] However, the existing technology uses a full-text fingerprint library, so each time a file is matched, the underlying layer must perform a full-text search. If the project is large, the search time will be too long. At the same time, this technology cannot solve the problem of project inclusion relationships (for example, if a Spark program has multiple modules, the above file detection will result in multiple projects such as spark-core, sparkStreaming, and spark, but the actual detection result should only be spark). Summary of the Invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide a source code tracing method.

[0006] The source tracing method of the present invention is characterized by comprising the following steps:

[0007] Step 1: Build a standardized vulnerability knowledge base using a crawler, obtain open source code that meets the specifications, and pre-process the open source code results before storing them. Step 2: Build a Neo4j graph database. Step 3: Perform business processing.

[0008] Furthermore, in the above-mentioned source code tracing method, the processing process of step 1 is as follows:

[0009] a. Using web crawlers, crawl open source code that meets the specifications on source code hosting platforms and communities, including GitHub, Gitee, and Linux. The specifications include determining star ratings and the number of branches.

[0010] b. Store the total values of the items involved in the source code file after preprocessing. The preprocessing includes removing spaces, special characters, and files that are not helpful for detection (such as ReadMe.txt). The total value includes the number of file lines and the number of files.

[0011] Furthermore, in the above-mentioned source code tracing method, in step one, the anti-crawler mechanism is bypassed through dynamic proxy, decoding, and random crawling, and the crawled files are stored locally.

[0012] Furthermore, in the above-mentioned source code tracing method, the processing process of step 2 is as follows:

[0013] a. Traverse the open source code, obtain all directories containing files and extract them. Convert the extracted file contents into MD5 hashes using MD5, deduplicate the directories, associate the directories with the MD5 hashes, convert the directories into MD5 hashes, associate the open source project name and version with the file content set, and obtain a set of hash directories, hash file contents, and projects.

[0014] b. Append the directory to neo4j to ensure uniqueness. If the directory exists in Neo4j, append the file contents. If not, create the directory.

[0015] c. Append the file content to ensure uniqueness. If the file exists in neo4j, append the project node. If not, create the file node.

[0016] d. Append the project node to ensure uniqueness. If the project exists in neo4j, add the relationship with the file. If not, create the project node.

[0017] Finally, a graph database is built that associates directory hashes, file hashes, projects, and versions.

[0018] Furthermore, in the above-mentioned tracing method of the same source code, in the step 2, all files are traversed, their paths are saved and their contents are read, the directory containing the file content is associated with the file, the content and directory are encrypted by md5, converted into hash, and the hash is stored as a rowkey in neo4j, and the directory hash associated file hash and the directory file hash associated project are established respectively.

[0019] Furthermore, in the above-mentioned tracing method for the same source code, in the step 2, if the same files belong to the same project, dynamic expansion is performed, and neo4j adds nodes and relationships to form a relationship library.

[0020] Furthermore, in the above-mentioned source code tracing method, the processing process of step three is as follows:

[0021] a. Read the project to be tested, traverse it, obtain the directory and files containing it, convert them into hash values through MD5 encryption, and store their basic information in the task library of the project to be tested to form the MySQL business library;

[0022] b. The directory in the Neo4j graph database is collided with the directory of the file to be detected. After the traversal is completed, the different versions of the same project are grouped in the project version set. Based on the weight value, the highest project and version are selected and stored in the new project version set. The basic information of the corresponding project version is searched in the MySQL database and associated, and the result is stored in the MySQL cache library;

[0023] c. Read the contents of the cache library and task library through clustering and grouping. Calculate the detection rate and number of matching lines based on the number of files, total lines, and file lines of both parties, and score them. Finally, select the project with the highest score in the same group as the result and return the result set.

[0024] Furthermore, in the above-mentioned tracing method for the same source code, the basic information includes the number of file lines, the number of files, and the total number of lines; during the collision, if a 100% result is obtained, the hash value of the file in the neo4j directory is obtained, and the hash is collided with the hash value of the file to be detected. If the result is consistent, the weight of the project and version corresponding to the content of this file is accumulated. If the weight of this project and version does not exist, this project and version is created and stored in the project version set. If it exists, the weight value of the project and version is accumulated as the key.

[0025] Furthermore, in the above-mentioned source code tracing method, the clustering grouping is performed by the dimensions of name similarity, directory similarity, and file names under the project, and the optimal set of data is selected by the detection rate and the number of matching rows.

[0026] Furthermore, in the above-mentioned source code tracing method, in step b, the number of rows and detection rate information of the selected results matching the project are stored in the MySQL business database.

[0027] By means of the above solution, the present invention has at least the following advantages:

[0028] 1. The gallery association relationship between the source code content hash value and the directory hash value can improve the detection speed.

[0029] 2. Solve the full problem of binary file-to-file collision detection, increasing the speed by more than 10 times, thereby providing engine support for further open source software supply chain security detection.

[0030] 3. Through multi-dimensional clustering, the calculated scores can be used to distinguish whether a module or its subclass is used, solving the problem of inaccurate supply chain detection results caused by the inclusion relationship between open source project modules. It can provide more accurate open source projects and versions, thereby making vulnerability detection in the software and security industries more accurate.

[0031] 4. The implementation process is simple. Clustering and grouping can be performed through the directory fingerprint library and dimensions such as name similarity and path inclusion relationship. The cache result set of dimensions such as the number of code line matches and file matching rate for each group of data is calculated to obtain the final result.

[0032] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a rough flowchart of the implementation of the source code traceability method. DETAILED DESCRIPTION

[0034] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0035] like Figure 1 The method for tracing the source code of the same source code is characterized by comprising the following steps:

[0036] Step 1: Use crawlers to build a standardized vulnerability knowledge base, obtain open source code that meets the specifications, and pre-process the open source code results before storing them. Specifically, the processing process includes the following steps:

[0037] a. Using web crawlers, crawl open source code that meets the specifications on source code hosting platforms and communities, including GitHub, Gitee, and Linux. The specifications include determining star ratings and the number of branches.

[0038] b. Store the total values of the items involved in the source code file after preprocessing. The preprocessing includes removing spaces, special characters, and files that are not helpful for detection (such as ReadMe.txt). The total value includes the number of file lines and the number of files.

[0039] At the same time, during the implementation period, the anti-crawler mechanism can be bypassed through dynamic proxy, decoding, and random crawling, and the crawled files are stored locally.

[0040] Step 2: Build a Neo4j graph database. The process is as follows:

[0041] First, the open source code is traversed to obtain all directories containing files and extract them. The extracted file contents are converted into MD5 hashes using MD5. The directories are deduplicated and associated with the MD5 hashes. The directories are converted into MD5 hashes and the open source project name and version are associated with the file content set to obtain a set of hash directories, hash file contents, and projects.

[0042] Afterwards, append the directory to neo4j to ensure uniqueness. If the directory exists in Neo4j, append the file contents; if not, create the directory.

[0043] Next, append the file content to ensure uniqueness. If neo4j already has this file, append the project node. If not, create this file node.

[0044] Then, append the project node to ensure uniqueness. If the project exists in neo4j, add the relationship associated with the file. If not, create the project node.

[0045] Finally, a graph database is built that associates directory hashes, file hashes, projects, and versions.

[0046] Step 3: Process the business. Specifically, the process is as follows:

[0047] Step a: Read the project to be detected, traverse it, obtain the directory and files containing the files, convert them into hash values through MD5 encryption, and store their basic information (such as the number of file lines, the number of files, the total number of lines, etc.) in the task library of the project to be detected to form the MySQL business library.

[0048] Step b: The directories in the Neo4j graph database are collided with the directories of the files to be checked. After the traversal is complete, the project versions are grouped together. The highest weighted projects and versions are selected and stored in a new project version set. The basic information of the corresponding project versions is searched and correlated in the MySQL database, and the results are stored in the MySQL cache. The basic information involved includes the number of file lines, the number of files, and the total number of lines.

[0049] Specifically, during the collision process, if a 100% match is achieved, the hash value of the file in the neo4j directory is obtained and collided with the hash value of the file to be detected. If the result matches, the weight of the project and version (in neo4j) corresponding to the file content is accumulated. If the weight of this project and version does not exist, it is created and stored in the project version set. If it does exist, the weight value of the project and version is accumulated using them as the key. Simultaneously, the second file is processed in the same manner as above. Furthermore, the number of rows matching the selected result and the project, as well as the detection rate information, can be stored in the MySQL business database.

[0050] Step c, through clustering and grouping, read the contents of the cache library and the task library, calculate the detection rate, the number of matching lines, etc. according to the number of files on both sides, the total number of lines and the number of file lines, and score them. Finally, select the project with the highest score in the same group as the result, and return the result set. For example, the scoring method adopted by the present invention is that the file matching detection rate accounts for 30 points. If the open source project of the gallery matches the project to be detected 100%, 30 points are obtained. If the file detection rate is 60%, then 18 points, and the line number detection rate is 30 points. In addition, the number of lines and the amount of files are scored as above. Therefore, the results can be clustered and scored through multiple dimensions to remove the inclusion relationship.

[0051] Considering that the implementation results may contain inclusion relationships, which can easily lead to inaccurate results, this paper adopts a clustering grouping method. Specifically, the clustering grouping process is to cluster and group by name similarity, directory similarity, and file names under the project, and select the optimal set of data based on the detection rate and the number of matching rows.

[0052] Taking the database framework (hibernate) as an example, its folder path is passed into the business processing interface.

[0053] At this point, the program will begin to read the specific files and directories in the path (converted to Hash), and then perform a directory query in Neo4j to obtain the hash value under the corresponding directory (the project file of all open source files corresponding to the directory).

[0054] Next, the file hash value of Hibernate is matched against the hash value of the project file content in the matching directory. During this process, if a match is successful, the project and version associated with the file and directory are obtained. The project and version form a unique identifier. If other files and directories with the unique identifier have been matched, the weight of this unique identifier is increased by 1. If it does not exist, the unique identifier is created and set to a default value of 1.

[0055] Finally, the names of the same items in the unique identifier are grouped and the version with the highest weight in the unique identifier is selected.

[0056] During this process, matching directories, files, and file line counts are stored in memory. This information is then linked to the final unique identifier, yielding basic information about the open source project matched by the unique identifier (e.g., number of lines, line ratio, file ratio, etc.). Finally, the results (the highest-weighted projects) are grouped based on their name and directory similarity. Each project in the same group is then scored based on its basic information, and the final results are merged (the best project in the group is selected and stored in the business database).

[0057] It can be seen from the above textual description and the accompanying drawings that the present invention has the following advantages:

[0058] 1. The gallery association relationship between the source code content hash value and the directory hash value can improve the detection speed.

[0059] 2. Solve the full problem of binary file-to-file collision detection, increasing the speed by more than 10 times, thereby providing engine support for further open source software supply chain security detection.

[0060] 3. Through multi-dimensional clustering, the calculated scores can be used to distinguish whether a module or its subclass is used, solving the problem of inaccurate supply chain detection results caused by the inclusion relationship between open source project modules. It can provide more accurate open source projects and versions, thereby making vulnerability detection in the software and security industries more accurate.

[0061] 4. The implementation process is simple. Clustering and grouping can be performed through the directory fingerprint library and dimensions such as name similarity and path inclusion relationship. The cache result set of dimensions such as the number of code line matches and file matching rate for each group of data is calculated to obtain the final result.

[0062] In addition, the indicated orientations or positional relationships described in the present invention are all based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or structure referred to must have a specific orientation or operate with a specific orientation structure. Therefore, it cannot be understood as a limitation on the present invention.

[0063] The terms "primary" and "secondary" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features designated as "primary" or "secondary" may explicitly or implicitly include one or more of such features. In the description of this invention, "several" means two or more, unless otherwise specifically defined.

[0064] Similarly, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0065] In the present invention, unless otherwise expressly specified or limited, terms such as "connected" and "disposed" should be understood in a broad sense. For example, they can be fixedly connected, detachably connected, or integrated; they can be mechanically connected or electrically connected; they can be directly connected or indirectly connected through an intermediate medium; they can be internal communication between two components or an interactive relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. And it can be directly on another component or indirectly on the other component. When a component is said to be "connected to" another component, it can be directly connected to the other component or indirectly connected to the other component.

[0066] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. The source code tracing method is characterized by The following steps are involved: Step 1: Establish a standardized vulnerability knowledge base through crawling, obtain open source code that meets the specifications, and store the open source code results after preprocessing. The processing process is as follows: a. Using web crawlers, crawl open source code that meets the specifications on source code hosting platforms and communities, including GitHub, Gitee, and Linux. The specifications include determining star ratings and the number of branches. b. Storing the total value of the items involved in the source code file after preprocessing. The preprocessing includes removing spaces, special characters, and files that are not helpful for detection. The total value includes the number of file lines and the number of files. Dynamic proxy, decoding, and random crawling are used to bypass anti-crawler mechanisms. The crawled files are stored locally. Step 2: Build a Neo4j graph database. The process is as follows: a. Traverse the open source code, obtain all directories containing files and extract them. Convert the extracted file contents into MD5 hashes using MD5, deduplicate the directories, associate the directories with the MD5 hashes, convert the directories into MD5 hashes, associate the open source project name and version with the file content set, and obtain a set of hash directories, hash file contents, and projects. b. Append the directory to neo4j to ensure uniqueness. If the directory exists in Neo4j, append the file contents. If not, create the directory. c. Append the file content to ensure uniqueness. If neo4j has this file, append the project node. If not, create this file node. d. Append the project node to ensure uniqueness. If the project exists in neo4j, add the relationship with the file. If not, create the project node. Finally, a graph database is built that associates directory hashes, file hashes, projects, and versions. If the same files belong to the same project, dynamic expansion is performed, and neo4j adds nodes and relationships to form a relationship database. Step 3: Process the business. The process is as follows: a. Read the project to be tested, traverse it, obtain the directory and files containing it, convert them into hash values through MD5 encryption, and store their basic information in the task library of the project to be tested to form the MySQL business library; b. The directory in the Neo4j graph database is collided with the directory of the file to be detected. After the traversal is completed, the different versions of the same project are grouped in the project version set. Based on the weight value, the highest project and version are selected and stored in the new project version set. The basic information of the corresponding project version is searched in the MySQL database and associated, and the result is stored in the MySQL cache database. The number of rows matching the selected result and the project and the detection rate information are stored in the MySQL business database. c. Read the contents of the cache and task libraries through clustering and grouping. Calculate the detection rate and matching line count based on the number of files, total lines, and file lines in both parties. Finally, select the project with the highest score in the same group as the result and return the result set. The basic information includes the number of file lines, the number of files, and the total number of lines. During the collision, if a 100% result is obtained, the hash value of the file in the neo4j directory is obtained, and the hash is collided with the hash value of the file to be detected. If the results are consistent, the weight of the project and version corresponding to the content of this file is accumulated. If the weight of this project and version does not exist, the project and version are created and stored in the project version set. If they exist, the weight value of the project and version is accumulated as the key. The clustering grouping is performed by the dimensions of name similarity, directory similarity, and file names under the project, and the optimal set of data is selected by the detection rate and the number of matching rows.

Images