A PyPI malware package detection method based on code embedding and clustering

Abstract

This invention discloses a PyPI malware package detection method based on code embedding and clustering. The steps include: 1) constructing an APIEM model to obtain standardized API call sequences corresponding to Python scripts; 2) using the APIEM model to extract the standardized API call sequences corresponding to each Python script in the training set, and using them as training samples to train a standardized control flow graph embedding model (CCEM); 3) extracting the standardized API call sequences corresponding to the installation scripts of each Python package to be detected and benign Python packages, and embedding them into vectors using CCEM; 4) clustering the packages according to the vectors to identify outliers; 5) determining whether a package is a malware package based on the outlier degree of the outlier and the distance between the outlier and known malicious samples.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to a method for detecting malicious software packages in a PyPI software repository based on code embedding and clustering. Background Art

[0002] As the most popular programming language, Python has been widely used in recent years due to its simplicity, versatility and ease of use. Python has a large and active developer community, and a large number of packages have promoted the participation of a wide range of projects. Software repositories have played a vital role in the modern software development process, enabling developers to effortlessly download and reuse the third-party code they need, thereby greatly improving development efficiency. The Python Package Index (PyPI) is Python's official third-party package repository, providing a total of more than 450,000 third-party packages for relevant software developers. The PyPI open source community provides developers with a convenient platform to publish their own packages, allowing users to easily download and reuse packages from them. However, this openness exposes PyPI to serious malware threats. For example, malware packages uploaded by attackers will quickly spread to downstream projects through the software supply chain, causing serious damage to countless users. According to Sonatype's "Open Source Supply, Demand, and Security" report in 2022, the number of attacks on the software supply chain has surged by 742% in the past three years. In order to prevent the spread of malware packages before they cause serious consequences, a malware package detection tool is urgently needed to quickly and accurately detect newly uploaded packages in PyPI.

[0003] The current malware package detection methods for software supply chains mainly include Malware Checks, Application-Inspector, Buildwatch, and MalOSS. These methods identify malicious packages by applying various analysis techniques to the package code, metadata, and dynamic behavior. These existing analysis methods rely on explicit detection knowledge, including security-sensitive functions (e.g., source taints, sink points, and dangerous methods), malicious code patterns, and behavior patterns. However, obtaining explicit detection knowledge is a very expensive task, and researchers mainly collect the required knowledge by detecting known malware through manual analysis. More importantly, this knowledge-required detection method is passive in nature. Attackers can use techniques beyond known detection rules to implement attack vectors to evade detection. It is unrealistic to prepare a comprehensive set of detection rules in advance. Therefore, there is an urgent need for an efficient malware detection method that does not rely on explicit prior knowledge. Summary of the invention

[0004] The purpose of this invention is to fill the gap in the current detection of malware packages that do not rely on explicit prior knowledge, and propose a PyPI malware package detection method based on code embedding and clustering. This method can rank the installation scripts of the software packages (i.e., setup.py scripts) according to their degree of outliers and distances from known malicious instances without relying on explicit knowledge, highlight potential malware packages, assist other detection methods in detection, help auditors quickly discover malware packages, and supplement the arsenal of software supply chain security detection.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] The PyPI malware package detection method based on code embedding and clustering includes the following steps:

[0007] 1) Build an APIEM model for Python code. The APIEM model can learn the word embedding vector of the function to obtain the standardized API call sequence of the Python script.

[0008] 2) The standardized API call sequences of the training set are extracted, and the API call sequences are used as training samples to train the Fastext-PVDM model, thereby obtaining the standardized control flow graph embedding model CCEM (Canonical CFG Embedding Model). This model is used to embed the API call sequences of the installation scripts of the Python software packages into low-dimensional dense vectors for easy clustering and distance comparison.

[0009] 3) Using the APIEM model, the standardized API call sequences of the installation scripts of the Python software package to be detected and the benign Python software package are extracted, and the standardized API call sequences are embedded into vectors using CCEM.

[0010] 4) Cluster the samples to be tested according to the obtained vectors and identify outliers, that is, samples that are far away from the benign installation script.

[0011] 5) The outliers are ranked based on their degree of outlier and the distance from known malicious samples, and several top-ranked Python packages are considered malware packages to highlight potential malware packages.

[0012] Further, step 1) includes the following steps:

[0013] a) Randomly obtain a certain number of Python packages from PyPI and use all the Python scripts in them to form a training set.

[0014] b) For each Python script in the training set, the present invention extracts the API function calls one by one according to the physical order of the function API in the Python script to obtain the API call sequence within the process. A Python software package contains multiple Python scripts (.py files), one of which is an installation script, and each Python script corresponds to an API call sequence.

[0015] c) In order to introduce sufficient context information, the present invention uses the API function call graph (CG) of the Python software package to inline splice the API call sequence of the callee within the process into the API call sequence of the caller within its process, and the generated inter-process call sequence is used as a training sample for the APIEM model.

[0016] d) Using the API call sequence between processes as training samples, a FastText word embedding model is trained using the FastText word embedding technology to obtain an APIEM model. FasText word embedding technology is a well-known model training technology. The FastText model is a word embedding model that is trained using the FastText word embedding technology. The model can be used to embed words into vectors. The APIEM model is essentially a customized FastText model (that is, trained using our customized training samples).

[0017] Further, in step 2), the following steps are included:

[0018] a) For each Python script in the training set, extract their function call graph (CG) and control flow graph (CFG).

[0019] b) Depth-first search and breadth-first search strategies are used to traverse each basic block in the control flow graph of the function, and under the guidance of the APIEM model, the successor node to be visited is determined based on the standardized traversal priority of the successor node, thereby forming a standardized API call sequence within the process of the target function.

[0020] c) For each standardized API call sequence within a process in the control flow graph, the callee's sequence is inlined into the caller's sequence according to the corresponding function call graph to generate a standardized API call sequence between processes.

[0021] d) Using the standardized API call sequences between processes as training samples, FastText PVDM document embedding technology is used to train a document embedding model to obtain the CCEM model.

[0022] Furthermore, the CCEM model training involved in step 2) may use the same training set as that in step 1).

[0023] Further, in step 3), the following steps are included:

[0024] a) Obtain the target software package set and the benign software package set from PyPI, where a benign software package is defined as a software package that has survived in the software repository for more than a certain number of days.

[0025] b) Merge the sample set to be tested and the benign sample set, and remove the duplicate installation scripts.

[0026] c) Under the guidance of the APIEM model, extract the standardized API call sequences between the procedures of the installation scripts of the merged set.

[0027] d) Using CCEM, the standardized API call sequences are embedded as vectors.

[0028] Further, in step 4), the following steps are included:

[0029] a) Use the HDBSCAN method to cluster the merged set. A stricter threshold should be set during clustering to avoid clustering similar noise points in sparse areas.

[0030] b) For outliers in the clustering results, the present invention identifies them as potential malicious samples. For clusters in the clustering results, the present invention determines them as benign clusters.

[0031] Preferably, in step 4), the distance calculation involved may be selected as cosine distance.

[0032] Further, in step 5), the following steps are included:

[0033] a) For each outlier point x, calculate its distance d from the benign cluster b (x)

[0034] b) For each outlier point x, calculate the distance d between it and the malicious sample m (x).

[0035] c) For each outlier point x, according to the distance d b (x) and d m (x) Calculate the ranking score RScore(x) of x and rank according to the score. The formula involved in the above calculation is detailed in the description of the specific implementation method.

[0036] d) The top-ranked samples, for example, the top ten samples, will be considered as suspicious malware packages and reported to the user.

[0037] Preferably, in step 5), in order to obtain better effect, the distance calculation involved is selected as cosine distance.

[0038] The present invention also provides a server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the above method.

[0039] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0040] The advantages of the present invention are as follows:

[0041] The present invention uses code embedding and clustering technology to achieve automatic detection of PyPI malware packages without relying on explicit knowledge. This method can be migrated to third-party software repositories, which is a good supplement to the software supply chain security detection arsenal. The present invention also designs a standardized API call sequence generation method to obtain high-quality code representation. In addition, the present invention designs a new distance calculation formula for ranking outliers, so that the system has higher accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the PyPI malware package detection method based on code embedding and clustering.

[0043] Figure 2 It is a flowchart for building the APIEM model.

[0044] Figure 3 It is a flowchart for constructing CCEM model.

[0045] Figure 4 It uses the APIEM model to obtain the standardized API call sequence and uses CCEM to embed the target installation script into a vector flowchart.

[0046] Figure 5 It is a flowchart for identifying outliers and benign clusters through clustering.

[0047] Figure 6 It is a flowchart for ranking outliers based on the degree of outliers of the sample points and the distance between them and known malicious samples. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0049] This embodiment is a PyPI malware package detection method based on code embedding and clustering. The overall process is as follows: Figure 1 As shown, the method mainly includes the following steps:

[0050] 1) Build an APIEM model for Python code. The APIEM model can learn the word embedding vector of the function to obtain the standardized API call sequence of the Python script. The process of building the APIEM model is as follows: Figure 2 The specific instructions are as follows:

[0051] 1a) Randomly obtain a certain number of Python packages from PyPI, use all the Python scripts in them to form a training set, and go to 1b).

[0052] 1b) For each Python script in the training set, the present invention extracts function calls one by one according to the calling order of the function API in the Python script to obtain the API call sequence within the process. A Python software package contains multiple Python scripts (.py files), one of which is an installation script, and each Python script corresponds to an API call sequence. In order to introduce sufficient context information, the present invention inlines the API call sequence of the callee within the process into the API call sequence of the caller within its process based on the function call graph. The generated API call sequence between processes is used as a training sample for the APIEM model, and then go to 1c).

[0053] 1c) Using the API call sequence between processes as training samples, a FastText word embedding model is trained using the FastText word embedding technology to obtain an APIEM model. FasText word embedding technology is a well-known model training technology. The FastText model is a word embedding model that is trained using the FastText word embedding technology. The model can be used to embed words into vectors. The APIEM model is essentially a customized FastText model (that is, trained using our customized training samples).

[0054] 2) Extract the standardized API call sequence of the training set, and use the standardized API call sequence as a training sample to train the Fastext-PVDM model, thereby obtaining a standardized control flow graph embedding model CCEM, which is used to embed the API call sequence of the installation script of the Python software package into a low-dimensional dense vector, which is convenient for clustering and distance comparison. Among them, the process of building the CCEM model is as follows Figure 3 The specific instructions are as follows:

[0055] 2a) For each Python script in the training set, extract their function call graph and control flow graph, and go to 2b).

[0056] 2b) Use DFS and BFS strategies to traverse each basic block in the control flow graph of the function to obtain the global and local features of the code respectively, and under the guidance of the APIEM model, determine the successor node to be visited based on the standardized traversal priority of the successor node, thereby forming a standardized API call sequence within the process of the target function, and go to 2c).

[0057] 2c) For each standardized API call sequence within a process in the control flow graph, inline the callee's sequence into the caller's sequence according to the corresponding function call graph to generate a standardized API call sequence, and go to 2d).

[0058] 2d) The standardized API call sequences between processes are used as training samples, and the FastText PVDM document embedding technology is used to train the CCEM model.

[0059] 3) Under the guidance of the APIEM model, the standardized API call sequences of the installation scripts of the Python software package to be detected and the benign Python software package are extracted. And the standardized API call sequence is embedded into a vector using the CCEM model. The standardized API call sequence has the following properties: the standardized API sequence obtained from the code with homogeneous semantics should be consistent. The traditional control flow graph traversal method generates a non-standardized API sequence. In fact, the traditional traversal method may produce significantly different traversal results when the code structure has only slight differences. In theory, it is extremely challenging, if not impossible, to generate a perfect standardized API call sequence for a variety of codes. In the present invention, the following rule is used to obtain a nearly perfect standardized API call sequence: basic blocks containing similar API calls should have similar traversal priorities. Among them, the process of using the APIEM model to obtain the standardized API call sequence and using the CCEM model to embed the target installation script into a vector is as follows. Figure 4 The specific instructions are as follows:

[0060] 3a) Obtain the target software package set and the benign software package set from PyPI. Benign software packages are defined as software packages that have survived in the software repository for more than a certain number of days, for example, software packages that have survived for more than 90 days, and then go to 3b).

[0061] 3b) The sample set to be tested and the benign sample set are merged to provide a reliable statistical basis for the subsequent clustering, and the duplicate installation scripts are removed, and then go to 3c).

[0062] 3c) Use the APIEM model to extract the standardized API call sequence of the installation script of the merged set. Specifically, for the control flow graph of the target program, it is assumed that when traversing, the present invention will access the order of subsequent basic blocks based on the traversal priority of the basic block. The present invention uses word embedding technology to determine the traversal priority. Each function can be encoded into a vector using the APIEM model. The geometric center of the corpus of the APIEM model is selected as the basis vector. The priority of a function can be obtained by calculating the cosine distance between the vector and the basis vector. The priority of a basic block is determined by the minimum priority of the functions it contains. The present invention uses the above-mentioned standardized control flow graph traversal method to obtain a standardized API call sequence, and goes to 3d).

[0063] 3d) Using the CCEM model, the standardized API call sequences are embedded into vectors for clustering and distance comparison.

[0064] 4) Cluster the samples to be tested according to the obtained vectors to identify outliers. The present invention is based on two basic and profound observations: First, the number of malicious packages is far less than the number of benign packages. Second, the functions of the installation scripts of malicious software packages are very different from those of benign software packages, and the latter often form clusters. Therefore, the present invention uses clustering technology to cluster the installation scripts in PyPI software packages into clusters to identify outliers, i.e., potential malicious samples. The process of identifying outliers and benign clusters by clustering is as follows: Figure 5 The specific instructions are as follows:

[0065] 4a) Cluster the merged set using the HDBSCAN clustering method. The distance calculation involved in clustering can be the cosine distance. A stricter threshold should be set during clustering to avoid clustering similar noise points in sparse areas, and then go to 4b).

[0066] 4b) For outliers in the clustering results, the present invention identifies them as potential malicious samples. For clusters in the clustering results, the present invention determines them as benign clusters.

[0067] 5) Ranking outliers based on the degree of outliers and the distance between them and known malicious samples, and identifying several top-ranked Python packages as malicious software packages to highlight potential malicious software packages. The process of ranking outliers based on the degree of outliers and the distance between them and known malicious samples is as follows: Figure 6 The specific instructions are as follows:

[0068] 5a) For each outlier point x, calculate its alienation distance d from the benign cluster b(x). Specifically, we calculate the distances between the outlier and all members of a benign cluster and take the minimum value as the distance between the outlier and the cluster. Calculate the average distance d between x and the nearest k benign clusters (e.g., k = 3) b (x), the specific formula is as follows, where B is the set of benign clusters, B ′ is a subset of size k, b is a benign cluster, y is a sample vector, and distance(x,y) is the similarity distance calculation formula between x and y, which can be the cosine distance formula:

[0069]

[0070] 5b) For each outlier point x, calculate the distance d between it and the malicious sample m (x). Specifically, the calculation is the minimum distance d between x and the nearest known malicious sample script m m (x), the specific formula is as follows, where M is a benign cluster set, m is a sample vector, and distance(x,m) is the similarity distance calculation formula between x and m, which can be the cosine distance formula: d m (x) = min m∈M distance(x,m)

[0071] 5c) For each outlier point x, according to the distance d b (x) and d m (x), calculate the ranking score RScore(x), and rank according to the score. It is worth mentioning that some malicious installation scripts may only contain a few function calls, while benign installation scripts often have more semantics. In this case, d m (x) contains more information. The present invention introduces a bias β(x) based on the size of x to better balance d b (x) and d m (x). The specific formula for calculating RScore(x) is as follows, where sizeof(x) is the number of APIs in the API call sequence corresponding to the x vector:

[0072] RScore(x)=(0.5-β(x))d b (x)+(0.5+β(x))d m (x)

[0073]

[0074] 5d) For samples that rank high, for example, the top ten samples, they are considered as suspicious malware packages and are handed over to professional security personnel for further analysis.

[0075] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A PyPI malware package detection method based on code embedding and clustering, the steps of which include: 1) Building an API embedding model APIEM for Python code, wherein the APIEM model is used to obtain a standardized API call sequence corresponding to a Python script, and obtain an API call sequence corresponding to an installation script of a Python software package; each of the Python software packages includes multiple Python scripts, one of which is an installation script; The method for constructing the APIEM model is as follows: 1a) selecting multiple Python software packages from the PyPI software repository, and forming a training set A with each Python script in the selected Python software packages; 1b) For each Python script in the training set A, extract API function calls one by one according to the order of appearance of each function API in the Python script to obtain the API call sequence within the process; 1c) Based on the function call graph of the Python software package, the API call sequence of the callee within the process is inlined and spliced ​​into the API call sequence of the caller within its process to generate the API call sequence between processes; 1d) Using the API call sequence between processes as training samples, the FastText word embedding technology is used to build an APIEM model; 2) Select several Python packages from the PyPI software repository and use the Python scripts in the selected Python packages to construct a training set B; The APIEM model is used to extract the standardized API call sequence corresponding to each Python script in the training set B, and the standardized API call sequence is used as a training sample to train the FastText-PVDM model to obtain a standardized control flow graph embedding model CCEM; wherein, the standardized API call sequence has the following properties: the standardized API sequence obtained from the code with homogeneous semantics should be consistent; the method for obtaining the standardized control flow graph embedding model CCEM is as follows: 2a) for each Python script in the training set B, the function call graph and function control flow graph of the Python script are extracted; 2b) each basic block in the function control flow graph is traversed, and under the guidance of the APIEM model, the successor node to be accessed is determined based on the standardized traversal priority of the successor node, thereby forming a standardized sequence of the target function; 2c) for each standardized sequence in the function control flow graph, the sequence of the callee is inlined to the sequence of the caller according to the corresponding function call graph, and a standardized API call sequence between processes is generated; 2d) using the standardized API call sequence as a training sample, the document embedding model is trained using the FastText PVDM document embedding technology to obtain the standardized control flow graph embedding model CCEM; 3) Using the APIEM model to extract the API call sequence corresponding to the installation script of each Python software package to be detected, and using the standardized control flow graph embedding model CCEM to embed it into a vector; Using the APIEM model to extract the API call sequence corresponding to the installation script of each benign Python software package, and using the standardized control flow graph embedding model CCEM to embed it into a vector; 4) Clustering the set of Python software packages to be detected and the set of benign Python software packages according to the vectors obtained in step 3) to identify outliers; 5) Calculate the ranking score of the outlier based on the degree of outlier and the distance between the outlier and the known malicious samples; Rank the outliers according to their ranking scores, and identify the top-ranked Python packages as malicious software packages.

2. The method according to claim 1, characterized in that For each outlier point x, calculate its distance d from each cluster. b (x), the distance d between it and the malicious sample m (x), and then according to the distance d b (x) and d m (x) Calculate the ranking score RScore(x) of the outlier point x.

3. The method according to claim 2, characterized in that The method for obtaining the ranking score RScore(x) of the outlier point x is as follows: 5a) For the outlier point x, select the k clusters closest to the outlier point x; calculate the distance between the outlier point x and all members in each selected cluster, and take the minimum value as the distance between the outlier point x and the selected cluster; then take the average value of the distance between the outlier point x and the selected k clusters as the distance d b (x); 5b) For the outlier point x, the minimum distance between it and each known malicious sample is taken as the distance d m (x); 5c) According to the formula RScore(x)=(0.5-β(x))d b (x)+(0.5+β(x))d m (x) Calculate the ranking score RScore(x) of the outlier point x; where, sizeof(x) is the number of APIs in the corresponding API call sequence of the outlier point x.

4. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images