A compilation-related file identification method and device based on machine learning

Abstract

The present invention discloses a compilation-related file identification method and device based on machine learning, including: clustering all versions of an open source library based on the vector representation of the version; compiling each version in each category, and performing machine learning training based on the corresponding source file information of the successfully compiled version, respectively obtaining the file identification model of each category; using the file identification model, classifying each source file of the unsuccessfully compiled version, and obtaining the compilation-related file list of the unsuccessfully compiled version. Based on the compilation-related files in the compiled open source library version, the present invention automatically and relatively accurately predicts the compilation-related files in the version similar to the source file of the version, thereby improving the compilation-related file identification capability.

Description

Technical Field

[0001] The present invention relates to the field of software composition analysis, and in particular to a compilation-related file identification method and device based on machine learning. Specifically, in the process of source code level feature extraction, a source code folder contains source files compiled into a target binary program (compilation-related files) and source files that will not be compiled into the binary program (compilation-irrelevant files). A compilation-related file identification method based on machine learning is proposed and implemented. Background Art

[0002] Software component analysis is to identify third-party libraries, open source software, and hardware components contained in the software, and analyze whether there are potential risks in using these components. Software component analysis is very important for enterprise-level security risk control, and there are a large number of software component analysis tools at home and abroad. The common technical route of these tools is to extract the features of the binary file to be detected and the features of the candidate open source components, calculate the similarity between the source code and the binary code through feature matching, and analyze whether the binary code reuses the candidate open source components based on the calculated similarity.

[0003] The source code folder of the open source candidate component contains many source files, including files that are compiled into the target binary, which are defined as compilation-related files, and files that are not compiled into the target binary, which are defined as compilation-irrelevant files. For example, when compiling the third-party library zlib-v1.2.11, the proportion of compilation-related files in the candidate component source folder to the total source files (ending with c / cpp) is about 56%. After compiling 56 mainline versions of zlib, it was found that the average proportion of compilation-related files to the total source files was only about 38%.

[0004] When existing tools extract features from open source components, some tools do not distinguish between compilation-related files and compilation-irrelevant files, and extract features from all source files in the candidate open source component folder. This extraction scheme introduces a large number of noise features, which reduces the similarity between the binary and its corresponding open source library, leading to misjudgment. On the other hand, the more features there are, the longer the matching time will be, which will introduce a lot of unnecessary overhead in large-scale analysis scenarios.

[0005] At present, there are still some tools that hook the compilation process of open source components to obtain the files compiled into the target binary. The article "B2SFinder: Detecting open-source software reuse in COTS software" points out that the success rate of automatic compilation solutions is relatively low, generally only about 25%. In order to successfully compile most open source libraries, it is necessary to manually configure the environment required for compilation, such as the open source libraries that the compilation process depends on and the parsing tools required in the compilation process. The features extracted using this solution are relatively accurate, but it requires a lot of manual participation and has certain limitations in scalability.

[0006] Therefore, a heuristic method for obtaining compilation-related files is needed. This solution needs to reduce manual involvement as much as possible, while being able to obtain compilation-related files as accurately as possible, thereby taking into account both accuracy and scale requirements. Summary of the invention

[0007] In order to overcome the limitations of the existing solutions in terms of accuracy and scalability in obtaining compilation-related files of open source components, the present invention provides a compilation-related file identification method and device based on machine learning. For each open source library, this method first uses the kmeans method to cluster the various versions of the open source library, and clusters the versions with similar source files into one category. In each category, a decision tree is first constructed using the compilation-related files in the version that is automatically compiled successfully, and then the decision tree is used to classify the versions that are not successfully compiled in the category, and obtain the compilation-related file lists corresponding to these versions.

[0008] The technical solution of the present invention includes:

[0009] A compilation-related file identification method based on machine learning, the steps of which include:

[0010] 1) Clustering all versions of an open source library based on the vector representation of the versions;

[0011] 2) Compile each version in each category, and perform machine learning training based on the corresponding source file information of the successfully compiled version to obtain file recognition models of each category;

[0012] 3) Using the file recognition model, classify each source file of the unsuccessfully compiled version, and obtain a compilation-related file list of the unsuccessfully compiled version.

[0013] Furthermore, the vector representation of the version is obtained by the following steps:

[0014] 1) Traverse all source files in each version of an open source library, and use the file names of the source files to sort them in ascending alphabetical order to form a vocabulary;

[0015] 2) Generate a vector representation of each version based on the vocabulary and the names of all source files in the version.

[0016] Furthermore, clustering methods include: kmeans algorithm, DBSCAN algorithm, OPTICS algorithm, hierarchical clustering, graph-based clustering or graph neural network-based clustering.

[0017] Furthermore, when using the kmeans algorithm, clustering is stopped when the minimum distance distance < 1, where distance = min i∈[1,n],j∈[1,n],i≠j distEclud(centroid i ,centroid j ), n is the total number of classes, distEclud is the Euclidean distance calculation, centroid i With centroid j Divided into the centroid of the i-th class and the j-th class.

[0018] Furthermore, the compiling method includes: using make to parse the build script.

[0019] Further, the method of building the script includes: Makefile

[0020] Furthermore, the machine learning method includes: a cart decision tree, an ID3 decision tree, a C4.5 decision tree or a C5.0 decision tree.

[0021] Furthermore, the source file information includes: the source file name and the folder where the source file name is located.

[0022] Furthermore, the compilation-related file list includes: the names of the compiled files and the folders where the names of the compiled files are located.

[0023] A storage medium stores a computer program, wherein the computer program is configured to execute the above method when running.

[0024] An electronic device includes a memory and a processor, wherein the memory stores a program for executing the above method.

[0025] The beneficial effects of the present invention are as follows: the present method can automatically and relatively accurately predict the compilation-related files in the version similar to the source files of the compiled open source library version based on the compilation-related files in the compiled open source library version. Through experimental verification, the compilation-related file recognition capability of the present method can achieve an average accuracy of 85.9% on the open source library set of 210 mainline versions including zlib, libxml2, and freetype. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 T-SNE diagram of zlib compilation-related file version vectors.

[0027] Figure 2 Illustration of clustering results of zlib, libxml2, and freetype.

[0028] Figure 3 Architecture diagram of the classification method for compilation-related files based on machine learning.

[0029] Figure 4 Version vector illustration. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the objects, features and advantages of the present invention more obvious and easy to understand, the core technical parts of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0031] The technical solution adopted by the present invention to solve the technical problem mainly includes the following two aspects: first, through large-scale experiments, it is verified that the open source library versions with similar source file lists have similar compilation-related file lists, and then a specific identification method for compilation-related files based on machine learning is given. The specific introduction is as follows:

[0032] (1) The present invention constructs an open source library set S_candidate containing 210 versions, including 30 versions of zlib, 67 versions of freetype and 113 versions of libxml2. The open source libraries in the set are compiled using default compilation options. The present invention summarizes the association between the compilation-related source file sets and the entire source file set on different versions by performing large-scale analysis on the test set. The analysis mainly includes the following two steps.

[0033] 1. Visualization of compilation-related files: First, for each version of the open source library zlib in S_candidate, a version vector constructed by the compilation-related files of that version is generated, and the T-SNE high-dimensional vector visualization tool is used for visualization. The corresponding visualization results are as follows: Figure 1 As shown in the figure, some versions are grouped together, which proves that the compilation-related files of these versions are very similar.

[0034] 2. Summarize the relationship between source files and compilation-related files: For each open source library in S_candidate, the present invention generates a version vector based on all source files and compilation-related files of the version, and clusters these version vectors. The present invention uses the following formula to calculate the clustering results of the above two vectors. The clustering results of the version vectors generated by all source files are used Each cluster is represented by Si, that is, The clustering result of the version vector generated by compiling the relevant source files is represented by T. For Si and Ti, we use the following formula to calculate the similarity of the two sets, where the symbol # represents the total number of elements in the set.

[0035]

[0036] At the same time, the two clustering results as well as The similarity is calculated using the following formula. The calculation results for zlib, freetype and libxml2 are as follows: Figure 2 As shown in the figure, it can be found that the similarity of each group of clustering results exceeds 80%.

[0037]

[0038] 3. Conclusion: The compilation-related files between some versions of the open source library are relatively similar, and the source files of these versions with similar compilation files are also relatively similar. Therefore, the source files can be used to construct version vectors for clustering. The compilation-related files in each cluster are very similar.

[0039] (2) The present invention summarizes the results of the aforementioned large-scale analysis and designs and implements a set of compilation-related file identification methods based on machine learning. The workflow diagram is shown in FIG. Figure 3 As shown, the details are as follows:

[0040] Step 1: Given a specific open source library, assume that the open source library has K versions. In order to facilitate the recording of each open source library version and the source files it contains, the present invention first numbers the version of the open source library. Taking the open source library openssl that implements the Secure Sockets Layer protocol as an example, a version number list is formed as shown in {'1':'openssl_1_0_2t', '2':'openssl_1_0_k', '3':'openssl_1_0_0i'}, which is used to record the version name in string form as a digital form, which is convenient for subsequent clustering calculations and indexing of specific versions. The present invention obtains all source files of each version, and then as shown in Figure 4As shown in the figure, a version vector corresponding to each version is formed. Specifically, this method first traverses all source files in the open source library, and uses the file names of these source files to sort in ascending alphabetical order to form a vocabulary. For each version, first obtain the names of all source files in this version, and then fill 1 in the corresponding position of the vector according to the vocabulary. The position corresponding to the file name that does not exist in this version but exists in the vocabulary is filled with 0. The specific filling process is as follows Figure 4 As shown, let the vocabulary be a, b, c, d. For version A that only has files a and c, the version vector of this version is [1, 0, 1, 0].

[0041] Step 2: Obtain a set of similar versions of compilation-related files based on the clustering algorithm: The clustering goal of this method is to make the similarity of the source files of the versions within the class as high as possible, and the similarity of the source files of the versions outside the class as low as possible. Therefore, this method selects the clustering algorithm kmeans to cluster each version vector and obtain a set of similar versions of compilation-related files. Each class clustered by the kmeans algorithm has a corresponding centroid. In this paper, the centroid of each class is represented by centroid_i, where i represents the i-th class clustered and n represents the total number of clustered classes. In order to use the kmeans algorithm, the distance between centroids in this scenario and the conditions for stopping clustering need to be defined as follows: This paper uses the distance in the following formula to represent the minimum distance between centroids. For the centroid of each clustered class, this method calculates the Euclidean distance between the centroids (represented by the symbol distEclud) and takes the minimum distance between the centroids as the distance. When distance is less than 1, clustering stops.

[0042] distance=min i∈[1,n],j∈[1,n],i≠j distEclud(centroid i ,centroid j )

[0043] Step 3: Obtain compilation-related files based on decision tree algorithm: For each category obtained after the clustering algorithm, the method first uses automatic construction tools such as make to parse the makefile and other construction scripts in each version, and automatically compiles each version. For the version that can be successfully compiled in this category, its compilation-related files can be obtained by recording the compilation process log and extracting the list of source files actually involved in the compilation; for the version that is not successfully compiled in this category, the present invention is based on the compilation-related file list of the successfully compiled version in this category, specifically the source file name and the folder where the file name is located, so as to form an internal node on the decision tree to construct a cart decision tree, and based on the cart decision tree, classify each source file in the unsuccessfully compiled version to determine whether the file is a compilation-related file, and the file list with the classification result of yes is the compilation-related file list of the unsuccessfully compiled version, specifically the compiled file name and the folder where the file name is located. 。

[0044] Other Implementations

[0045] 1. Obtain a similar version set of compilation-related files based on clustering algorithms

[0046] Clustering is a data analysis technique that divides similar objects into different groups or more subsets through static analysis. After clustering, objects in each class have corresponding similar parts. Common clustering methods include partition-based clustering, which aims to make points within the class closer and points outside the class farther away. And density-based clustering: when the density of the neighboring area exceeds a certain threshold, clustering continues, such as DBSCAN and OPTICS. There are also hierarchical clustering, graph-based clustering, graph neural network-based clustering, and so on.

[0047] Partition-based clustering meets the requirements for clustering open source library versions, because we hope that the file similarity of versions clustered in the same class is as high as possible, and the file similarity between classes is as low as possible. The K-means algorithm in this step can be replaced by other partition-based clustering methods.

[0048] 2. Obtain compilation-related files based on the decision tree algorithm

[0049] Decision tree is a machine learning method. It is a tree structure in which each internal node represents a judgment on an attribute, each branch represents the output of a judgment result, and finally each leaf node represents a classification result. The decision tree algorithm used in this step can be replaced by other decision tree generation algorithms with the same purpose, such as ID3, C4.5 and C5.0.

[0050] Experimental data

[0051] This method constructs a dataset containing 67 manually compiled mainline versions of freetype, 30 mainline versions of zlib, and 113 mainline versions of libxml2, and obtains the compilation-related files of these files during the construction process as the experimental evaluation benchmark.

[0052] Given an open source library S, this method assumes that the open source library is manually analyzed, and the file list of true compilation-related files is true_compiled_files = tf1, tf2..., tfn. This solution uses the decision tree to predict the list of compilation-related files is predicted_compiled_files = pf1, pf2..., pfn. This solution mainly uses the following indicators to evaluate the effectiveness of the method.

[0053] 1.TP: TP means True Positive. Only when predicted_compiled_files and true_compiled_files are completely consistent, this solution is considered TP, that is, the predicted result is completely correct.

[0054] 2. FP: FP means False Positive. When there are files in predicted_compiled_files and true_compiled_files that are different, this solution considers this to be a FP.

[0055] Using the above evaluation indicators, this method can achieve 100%, 98.8% and 58.9% TP on zlib, libxml2 and freetype respectively.

[0056] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should be included within the scope of the present invention.

Claims

1. A compilation-related file identification method based on machine learning, the steps of which include: 1) Given an open source library with multiple versions, each version of which contains corresponding source files, traverse all source files in each version of the open source library, and use the file names of the source files to sort them in ascending alphabetical order to form a vocabulary; 2) generating a vector representation of each version based on the vocabulary and the names of all source files in the version; 3) Based on the vector representation of the version, a clustering algorithm is used to cluster all versions of the open source library to obtain a set of similar versions of compilation-related files; 4) Compile each version in each category, and perform machine learning training based on the corresponding source file information of the successfully compiled version to obtain the file recognition model of each category respectively; wherein the successfully compiled version is obtained by recording the compilation process log and extracting the list of source files actually involved in the compilation, and the source file information includes: the source file name and the folder where the source file name is located, and the file recognition model is a decision tree; 5) Using the file recognition model, classify each source file of the unsuccessfully compiled version, determine whether the source file is a compilation-related file, and obtain a compilation-related file list of the unsuccessfully compiled version; wherein the compilation-related file list includes: the name of the compiled file and the folder where the compiled file name is located.

2. The method according to claim 1, characterized in that The clustering algorithms include: kmeans algorithm, DBSCAN algorithm, OPTICS algorithm, hierarchical clustering, graph-based clustering or graph neural network-based clustering.

3. The method according to claim 2, characterized in that When using the kmeans algorithm, clustering stops when the minimum distance distance < 1, where distance = min i∈[1，n]，j∈[1，n]，i≠j distEclud(centroid i , centroid j ), n is the total number of classes, distEclud is the Euclidean distance calculation, centroid i With centroid j Divided into the centroid of the i-th class and the j-th class.

4. The method according to claim 1, characterized in that The compilation method includes: using make to parse the build script.

5. The method according to claim 1, characterized in that Methods for building scripts include: makefile.

6. The method according to claim 1, characterized in that Machine learning methods include: cart decision tree, ID3 decision tree, C4.5 decision tree or C5.0 decision tree.

7. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images