Dependence confusion vulnerability detection method based on multi-dimensional dependency analysis

Abstract

The invention relates to the technical field of software security, in particular to a dependency confusion vulnerability detection method based on multi-dimensional dependency analysis, aiming at a project source code file, scanning a configuration file and obtaining a project software package dependency set through dependency analysis; the multi-layer indirect dependency in the project software package dependency set is recursively analyzed through a pruning strategy and breadth-first search, all software packages in a project serve as nodes, the dependency relationship between the software packages serves as edges, and a project dependency tree is constructed; the project dependency tree is used for describing a direct dependency relationship and an indirect dependency relationship between software packages in a project; and according to the private software package referenced by the path in the project dependency tree, performing cross comparison on versions of the private software package and the public warehouse homonymous software package so as to identify a potential dependency confusion package in the project. According to the method, dependency confusion attacks are systematically and comprehensively identified and defended through multi-dimensional dependency analysis, and reliable guarantee is provided for software supply chain security.

Description

Technical Field

[0001] This invention relates to the field of software security technology, and in particular to a method for detecting dependency obfuscation vulnerabilities based on multi-dimensional dependency analysis. Background Technology

[0002] The booming development of modern software relies heavily on a rich ecosystem of third-party libraries. These ecosystems provide developers with immense convenience through centralized repositories (such as PyPI for Python and npm for JavaScript). Developers can simply declare dependencies in configuration files and automate installation and management through package managers like pip, significantly improving development efficiency. However, this convenience has also introduced serious software supply chain security challenges. Among these, dependency obfuscation attacks, as an emerging and highly stealthy threat, have received widespread attention in recent years.

[0003] Dependency obfuscation attacks were first disclosed by Alex Birsan in 2021. Attackers exploit the version priority mechanism of package managers in multi-repository environments (pip defaults to selecting the highest version package) to upload malicious packages with the same name but a higher version to public repositories. When developers configure both private and public repositories using `--extra-index-url`, pip may prioritize pulling the higher version malicious package from the public repository, leading to sensitive data leaks, backdoor implantation, or persistent data control. Such attacks have affected well-known companies like Yelp and PyTorch. More seriously, the widespread reuse of software further complicates this problem. In the Python ecosystem, projects typically form complex dependency chains through direct and indirect dependencies, and dependency obfuscation attacks are not limited to direct dependencies but can also spread through indirect dependency chains. For example, a seemingly harmless third-party library might depend on a malicious package in a public repository, and developers often lack sufficient awareness and control over this. Attackers can exploit this blind spot to implant malicious packages into deep dependency chains.

[0004] Existing detection tools, such as Confused and Combobulator, while capable of identifying potential risks from dependency obfuscation to some extent, have significant limitations. These tools typically focus only on analyzing direct dependencies, lacking comprehensive coverage of indirect dependencies, and perform poorly when handling various configuration file formats. This results in incomplete extraction of dependency information, leading to the overlooking of potential security vulnerabilities. This lack of detection capability provides attackers with an opportunity to exploit blind spots in indirect dependencies to launch attacks. Summary of the Invention

[0005] The purpose of this invention is to address the problem that existing detection tools are unable to comprehensively detect potential dependency obfuscation vulnerabilities. It provides a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis, which systematically and comprehensively identifies and defends against dependency obfuscation attacks through multi-dimensional dependency analysis, thus providing reliable protection for software supply chain security.

[0006] According to the design scheme provided by this invention, on the one hand, a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis is provided, comprising:

[0007] For the project source code files, the configuration files are scanned and dependency analysis is performed to obtain the project package dependency set. The configuration files contain several configuration files of different formats.

[0008] By recursively parsing the multi-level indirect dependencies in the project's software package dependency set using pruning strategies and breadth-first search, and constructing a project dependency tree with each software package in the project as a node and the dependencies between software packages as edges, the project dependency tree is used to describe the direct and indirect dependencies between software packages in the project.

[0009] Based on the private packages referenced by the paths in the project dependency tree, the versions of the private packages and the packages with the same name in the public repository are cross-compared to identify potential obfuscated dependency packages in the project. The private packages are those that exist in the project's enterprise private repository.

[0010] As a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis in this invention, further, the method scans the configuration file and obtains the project package dependency set through dependency analysis, including:

[0011] Obtain text configuration files and executable files from the project by scanning the project files;

[0012] For text configuration files and executable files, a differentiated parsing strategy is used to extract the names of dependent software packages in each file and to build a project software package dependency set. The differentiated parsing strategy includes a line-by-line parsing strategy, a parser-based parsing strategy, and an abstract syntax tree-based parsing strategy.

[0013] As a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis in this invention, further, the dependent package names in the file are extracted using a parsing strategy based on abstract syntax trees, including:

[0014] Parse the project source code into an abstract syntax tree;

[0015] By transforming the source code into an Abstract Syntax Tree (AST), it is possible to bypass code execution, traverse each node in the AST, extract dependencies from the string literals corresponding to the nodes, and record variable references, so as to capture all potential dependencies in the source code through recursive processing.

[0016] As a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis in this invention, it further includes, through pruning strategies and breadth-first search recursively parsing multi-level indirect dependencies in the project package dependency set, including:

[0017] Obtain the project's declared direct dependencies based on the project's package dependency set;

[0018] Starting with the directly dependent packages, the sub-dependencies of each package are resolved layer by layer.

[0019] As a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis in this invention, it further parses the sub-dependencies of each software package layer by layer, including:

[0020] Use a queue to store all root dependency packages in the package dependency set;

[0021] Iterate through each root dependency package in the queue and determine the source repository type of the root dependency package;

[0022] If the source repository type is a private repository, extract the direct dependencies of the root dependency package, put each sub-dependency into a queue, and append the root dependency package and its direct dependencies to the dependency tree.

[0023] If the source repository type is a public repository, then the root dependency package will be used as the leaf node of the dependency tree.

[0024] As a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis of this invention, further, a cross-comparison is performed between the versions of the private software package and the software package with the same name in the public repository, including:

[0025] Retrieve the names and currently highest available version numbers of private packages in the project dependency tree;

[0026] Query and retrieve information about the same package from the public repository, extract a list of all release version numbers of the same package from the information, and calculate the highest version number;

[0027] If the highest version number is greater than the current highest available version number, then the current private software package is marked as having a dependency obfuscation risk.

[0028] As part of the dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis of this invention, further, the method obtains the names of private software packages and the currently highest available version number in the project dependency tree, including:

[0029] The highest available version number is obtained from the specified metadata file according to the distribution format. The distribution format includes source code distribution packages and binary distribution packages. The specified metadata file is a distribution package metadata file that stores private package version information.

[0030] Furthermore, this invention also provides a dependency obfuscation vulnerability detection system based on multi-dimensional dependency analysis, comprising: a scanning and analysis module, a dependency tree construction module, and a dependency obfuscation detection module, wherein,

[0031] The scanning and analysis module is used to scan the configuration files of the project source code files and obtain the project package dependency set through dependency analysis. The configuration files contain several configuration files of different formats.

[0032] The dependency tree building module is used to recursively parse the multi-level indirect dependencies in the project package dependency set through pruning strategies and breadth-first search. It constructs a project dependency tree by treating each package in the project as a node and the dependency relationship between packages as an edge. The project dependency tree is used to describe the direct and indirect dependency relationships between packages in the project.

[0033] The dependency obfuscation detection module is used to cross-compare the versions of private software packages and public repositories with the same-named software packages referenced by paths in the project dependency tree in order to identify potential obfuscated dependency packages in the project. The private software packages are software packages that exist in the project's enterprise private repository.

[0034] The beneficial effects of this invention are:

[0035] This invention comprehensively identifies potential dependency obfuscation vulnerabilities through key steps such as multi-format configuration file parsing, dependency tree construction, and version comparison. Multi-format configuration file parsing ensures compatibility with various configuration file formats, guaranteeing the integrity of dependency information extraction. A complete dependency tree is constructed using a breadth-first search (BFS) algorithm and pruning strategies, covering multi-level dependency transitive scenarios. Version information of packages in public repositories is obtained using the PyPI official API to determine if the version of packages in private repositories is lower than that of packages with the same name in public repositories, accurately identifying dependency packages with obfuscation risks. Experimental results show that on seven real-world Python project datasets, the proposed solution, DCFinder, significantly outperforms existing tools in both dependency identification and risk detection. Specifically, DCFinder achieves a direct dependency identification recall rate of 96.5% and an F1-Score of 98.2%, representing improvements of 23.1% and 13.5% respectively compared to the existing Confused solution. More importantly, DCFinder achieves a 100% indirect dependency identification recall rate across all projects, while Confused's indirect dependency identification results are all 0. In terms of risk detection, DCFinder successfully identified all potential dependency obfuscation packages, and the detection results were completely consistent with the actual situation. Meanwhile, the existing Confused and Combobulator failed to detect any risks in any projects (outputting 0). These results fully validate that the DCFinder solution in this case, through multi-configuration file parsing, BFS dependency tree construction, and version comparison mechanisms, can comprehensively cover project dependencies and effectively detect dependency obfuscation vulnerabilities, providing reliable protection for software supply chain security. Attached image description:

[0036] Figure 1 This is a schematic diagram of the dependency obfuscation vulnerability detection process based on multi-dimensional dependency analysis in the embodiment;

[0037] Figure 2 This is an example of a dependency obfuscation attack in the embodiments;

[0038] Figure 3 This is a schematic diagram of the DCFinder architecture, which relies on the obfuscation vulnerability detection algorithm in this embodiment.

[0039] Figure 4 This is a schematic diagram illustrating the distribution of Python project configuration file types in the example.

[0040] Figure 5 This example illustrates a comparison of direct and indirect dependencies for different configuration file types. Detailed implementation method:

[0041] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0042] As the software ecosystem becomes increasingly complex, dependency obfuscation attacks have become a serious threat to software supply chain security. These attacks inject code into the target system by uploading a malicious package with the same name but a higher version to a public repository, leveraging the package manager's default mechanism of prioritizing the installation of higher-version packages.

[0043] The core mechanism of dependency obfuscation is to exploit version resolution flaws in package managers within a multi-repository environment. When a public package with the same name as an organization's private package exists in the official PyPI repository, the package manager becomes "obfuscated," unable to determine whether to acquire, use, and trust the package on the internal private registry or the package on the official default public registry. Packages on the public registry are highly likely to contain security vulnerabilities, potentially including malicious code, or even be specifically designed to steal data, implant backdoors, or achieve persistent data control for a particular organization.

[0044] The root of the problem lies in the default behavior of package managers. Taking Python's package manager PIP as an example, it defaults to fetching packages with higher versions. When using PIP, there are two key options:

[0045] `--index-url`: This option replaces the default PyPI index with the specified URL. This approach is more secure because it only checks packages in the specified index. Use this option when you need to ensure that packages are installed only from trusted private sources (such as the GitLab PyPI registry).

[0046] `--extra-index-url`: This option adds an extra index to the default PyPI index for searching. However, this method is less secure and more vulnerable to dependency obfuscation attacks because it checks packages in both the default PyPI and the additionally specified index, and it checks packages in the PyPI repository before checking custom repositories. `--extra-index-url` simply adds the provided URL as an extra registry entry, and the client checks if the package exists under that URL; while `--index-url` explicitly tells the client to only search for packages under the provided URL.

[0047] To more intuitively understand the principles and process of dependency obfuscation attacks, Figure 2 This demonstrates how attackers can exploit the default behavior of package managers to launch attacks. The specific steps are as follows:

[0048] (1) Attacker uploads malicious package: The attacker uploads a malicious package with the same name as the private package but with a higher version number (such as pkg-A v2.0) to the public repository.

[0049] (2) User request package: The user requests to install a package (such as pkg-A) through the package manager.

[0050] (3) Package Manager selects higher version package: The package manager checks the private repository and the public repository. It finds that the malicious package in the public repository has a higher version number, so it selects the malicious package first.

[0051] (4) Malicious package is executed: The user's system downloads and installs the malicious package, and executes the malicious code in it.

[0052] Attackers first upload a malicious package (e.g., pkg-A v2.0) with the same name but a higher version number than the target company's private package to a public repository. When a user requests to install a package through a package manager (e.g., Python's PIP), the package manager checks both the private and public repositories. Because the malicious package in the public repository has a higher version number, the package manager will prioritize and download it, causing the user's system to execute malicious code, potentially leading to security issues such as data breaches, backdoor implantation, or persistent data access control problems.

[0053] Existing detection tools such as Confused and Combobulator only support parsing the single requirements.txt configuration file, lack the ability to analyze multi-level indirect dependencies, and do not introduce a version cross-comparison mechanism, making it difficult to fully detect potential dependency obfuscation vulnerabilities.

[0054] The technical terms and symbols used in this plan are explained below:

[0055] A private repository contains a set of packages: P priv ={p i |p i =(name i ,ver i ),source(p i = private}. A public repository (such as PyPI) contains a set of packages: P pub ={p j |p j =(name j ,ver j ),source(p j =public}.

[0056] Dependency obfuscation attack success conditions: Same name: There exists a packet pair with the same name. name priv =name pub .

[0057] Version priority: Public package version is higher than private package version.pub >ver priv .

[0058] Configuration flaw: Use --extra-index-url.

[0059] Dependency reachability: Path references to private packages exist in the dependency tree. p ath:p root →...→p priv Where T is the project dependency tree, p root This is the root dependency of the project.

[0060] To detect potential dependency obfuscation vulnerabilities, embodiments of the present invention are described below. Figure 1 As shown, a dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis is provided, which specifically includes the following:

[0061] S101. For the project source code files, scan the configuration files and obtain the project package dependency set through dependency analysis. The configuration files contain several configuration files of different formats.

[0062] Specifically, scanning configuration files and performing dependency analysis to obtain the project's package dependency set can be designed to include:

[0063] Obtain text configuration files and executable files from the project by scanning the project files;

[0064] For text configuration files and executable files, a differentiated parsing strategy is used to extract the names of dependent software packages in each file and to build a project software package dependency set. The differentiated parsing strategy includes a line-by-line parsing strategy, a parser-based parsing strategy, and an abstract syntax tree-based parsing strategy.

[0065] The extraction of dependent package names from a file using an abstract syntax tree-based parsing strategy may include:

[0066] Parse the project source code into an abstract syntax tree;

[0067] The visitor pattern is used to traverse each node in the abstract syntax tree, extract dependencies from the string literals corresponding to the nodes and record variable references, so as to capture all potential dependencies in the source code through recursive processing.

[0068] Python project dependencies can be declared in various ways, including through multiple configuration files. These files can be categorized as text configuration files (requirements.txt, Pipfile, pyproject.toml, setup.cfg) and executable files (setup.py). Existing detection tools only support the single requirements.txt configuration file, leading to the omission of critical dependency information.

[0069] like Figure 3 The DCFinder algorithm architecture shown incorporates a multi-format configuration file parsing method. Different parsing strategies are formulated based on the format characteristics and declaration logic of different types of configuration files to achieve accurate extraction of dependency package names. The obtained dependency set will serve as the core input for subsequent dependency tree construction, laying the data foundation for indirect dependency parsing and risk detection.

[0070] For requirements.txt files in simple text format, a line-by-line parsing method can be used. This method skips comment lines starting with a hash symbol and blank lines when reading the file content, and removes any inline comments that may exist after the actual dependency declarations on each line.

[0071] For Pipfiles based on the TOML format, runtime and development dependencies can be declared in the [packages] and [dev-packages] sections, respectively. The parser loads the TOML content and extracts the key names from the key-value pairs in these sections. This ensures that runtime and development dependencies are captured, preventing the omission of packages needed during the development phase.

[0072] For the pyproject.toml file, TOML parsing enables support for PEP 621 compliant metadata and Poetry-specific configurations. The parser extracts dependencies, including dependencies and optional-dependencies, from the [project] table for PEP 621 compliant projects; for Poetry projects, it extracts them from the [tool.poetry.dependencies] and [tool.poetry.group.dev.dependencies] tables.

[0073] When processing setup.cfg files in INI file format, the standard Python configparser module is utilized. The key implementation is extracting dependencies from the [options] section, specifically the install_requires and extras_require keys. For the options.extras_require section, the parser iterates through key-value pairs to extract dependency groups for different extrafeatures.

[0074] Since setup.py is a dynamically executable Python script, directly extracting dependency information from it faces several challenges. First, setup.py may contain conditional branches, loops, or function calls, making traditional regular expression methods ineffective at capturing dependency declarations. Second, directly executing an unknown Python script may pose potential security risks, including unexpected network access, file system modification, or malicious code execution. To address these challenges, AST (Abstract Syntax Tree) parsing is used to statically analyze setup.py to extract dependencies. Based on Python packaging conventions, the dependency specification is mainly defined through four key parameters: install_requires, tests_require, extras_require, and setup_requires. The dependency set can be extracted by identifying the assignment statements related to these parameters and tracing their data flow relationships. The implementation of the dependency extraction process is shown in Algorithm 1.

[0075]

[0076] Given a setup.py file, the algorithm first parses the source code into an AST, then initializes the following data structures (lines 1-6): a data stream set DF containing target dependency parameters (such as install_requires), a condition context stack C for managing nested condition contexts, a function registry FS for recording return values, and two empty sets UnresolvedNames and DepSet for storing unresolved identifiers and final dependencies, respectively. Next, the algorithm traverses each node in the AST (line 7), processing assignment statements (processing the assigned expression when the target is a dependency parameter, lines 8-10), conditional statements (evaluating true and false branches by pushing and popping conditions onto and off the context stack, lines 11-18), and function definitions (registering function names and their return values ​​for later resolution, lines 19-22). When processing assignment statements, the ProcessExpression function recursively processes different AST node types: extracting dependencies from string literals, processing each element in lists and dictionaries, recording variable references for later resolution, and handling binary operations like list concatenation. For iterative resolution, the algorithm iteratively processes the UnresolvedNames set, resolving each reference by querying the registry function FS, and recursively processing the resolved values ​​until no new names can be resolved, thus ensuring that all potential dependencies are captured (lines 23-29).

[0077] Through the above multi-dimensional analysis, DCFinder can extract dependencies from various types of Python configuration files. The resulting dependency set (including runtime, development, testing, and optional dependencies) provides basic data for subsequent indirect dependency resolution and dependency tree construction stages.

[0078] S102. By recursively parsing the multi-level indirect dependencies in the project's software package dependency set through pruning strategies and breadth-first search, each software package in the project is treated as a node, and the dependency relationships between software packages are treated as edges, thus constructing a project dependency tree. The project dependency tree is used to describe the direct and indirect dependency relationships between software packages in the project.

[0079] Specifically, by using pruning strategies and breadth-first search to recursively resolve multi-level indirect dependencies in the project's package dependency set, it can be designed to include:

[0080] Obtain the project's declared direct dependencies based on the project's package dependency set;

[0081] Starting with the directly dependent packages, the sub-dependencies of each package are resolved layer by layer.

[0082] The process of resolving the sub-dependencies of each package layer by layer can include:

[0083] Use a queue to store all root dependency packages in the package dependency set;

[0084] Iterate through each root dependency package in the queue and determine the source repository type of the root dependency package;

[0085] If the source repository type is a private repository, extract the direct dependencies of the root dependency package, put each sub-dependency into a queue, and append the root dependency package and its direct dependencies to the dependency tree.

[0086] If the source repository type is a public repository, then the root dependency package will be used as the leaf node of the dependency tree.

[0087] Constructing a complete dependency tree is a crucial step in detecting potential security vulnerabilities. Traditional detection methods often only cover direct dependencies, neglecting the complex propagation chains of indirect dependencies. Untracked indirect dependencies can introduce potential dependency obfuscation, providing attack paths for malicious packets. DCFinder combines pruning strategies with the breadth-first search (BFS) algorithm to construct a complete dependency tree covering both direct and indirect dependencies, ensuring comprehensive dependency relationships.

[0088] The dependency tree construction method primarily relies on the breadth-first search (BFS) algorithm, starting from the project's declared direct dependencies and resolving the sub-dependencies of each package layer by layer. Furthermore, to optimize this algorithm, considering that indirect dependencies of non-private packages cannot exist in the internal repository, DCFinder employs a pruning strategy to optimize the dependency tree construction process, as shown in Algorithm 2.

[0089]

[0090] Root dependency set R = {r1, r2, ..., r n}, which is the set of direct dependency packages and private repository URLs declared in the project, P. priv The algorithm initializes a queue Q (enqueueing all root dependencies), an empty dependency tree T, and a visited set Visited (to prevent looping, lines 1-5). The core loop processes each packet p dequeued from Q. j If you have not visited p j Then add it to Visited and checkSource(p j ,P private The algorithm categorizes the source repositories as private or public (lines 6-10). For packages in private repositories (whose subsequent dependencies may exist internally), the algorithm extracts p... j direct dependency Each sub-dependency Added to the list for future expansion, and Append to T (lines 11-15). Conversely, packages from the public repository are added to T as leaf nodes (when the dependency set is empty). Since its transitive dependency cannot exist in a private repository, it is pruned to avoid redundant processing (lines 16-17). This pruning strategy optimizes performance by terminating the exploration of public branches, ensuring that the algorithm only focuses on possible private dependency chains. The process terminates when Q is empty (indicating that all reachable dependencies have been processed), and returns the final dependency tree T.

[0091] S103. Based on the private software packages referenced by the paths in the project dependency tree, cross-compare the versions of the private software packages and the software packages with the same name in the public repository to identify potential dependency obfuscation packages in the project. The private software packages are software packages existing in the project's enterprise private repository.

[0092] Specifically, cross-referencing the versions of private software packages and public repositories with the same name can be designed to include:

[0093] Retrieve the names and currently highest available version numbers of private packages in the project dependency tree;

[0094] Query and retrieve information about the same package from the public repository, extract a list of all release version numbers of the same package from the information, and calculate the highest version number;

[0095] If the highest version number is greater than the current highest available version number, then the current private software package is marked as having a dependency obfuscation risk.

[0096] Retrieving the names and currently highest available version numbers of private packages in the project dependency tree may include:

[0097] The highest available version number is obtained from the specified metadata file according to the distribution format. The distribution format includes source code distribution packages and binary distribution packages. The specified metadata file is a distribution package metadata file that stores private package version information.

[0098] The core condition for a successful dependency obfuscation attack is that a package with the same name exists in a public repository and its version is higher than that in a private repository.

[0099] p pub ∈P pub ,

[0100] name priv =name pub ∧ver pub >ver priv

[0101] For dependency tree T, marked as source(p) priv ) = each private package p privThe system directly retrieves the highest available version number (ver_priv_max) from the metadata of the private repository (or by parsing its package file). Version information for private packages is typically found in the distribution package's metadata file and is processed according to the distribution format. For source code distribution packages, the package file (.tar.gz / .zip) is unzipped, and the metadata directory (.dist-info, .egg-info) is recursively scanned to locate it. The Version field in the PKG-INFO file is then parsed. For binary distribution packages, the Version field in the .whl file is read directly via ZipPath. <package>The `.dist-info / METADATA` section calls PathDistribution to extract versions. If multiple versions of the same package exist in the private repository, the maximum value is extracted from all versions. Next, the PyPI official API is used to query information about the public package `p_pub` with the same name as `p_priv`. From the returned JSON data, a list of all its release version numbers is extracted, and its highest version number `ver_pub_max` is calculated. Finally, a strict version comparison is performed on each pair of packages with the same name. If `ver_pub_max` > `ver_priv_max`, then `p_priv` is marked as having potential dependency obfuscation risks. The final result is a list of all private packages with risks, detailing the package name, the highest version `ver_priv_max` in the private repository, the highest version `ver_pub_max` in the public repository, and its level in the dependency tree (direct / indirect dependencies).

[0102] Furthermore, based on the above method, this embodiment of the invention also provides a dependency obfuscation vulnerability detection system based on multi-dimensional dependency analysis, comprising: a scanning analysis module, a dependency tree construction module, and a dependency obfuscation detection module, wherein,

[0103] The scanning and analysis module is used to scan the configuration files of the project source code files and obtain the project package dependency set through dependency analysis. The configuration files contain several configuration files of different formats.

[0104] The dependency tree building module is used to recursively parse the multi-level indirect dependencies in the project package dependency set through pruning strategies and breadth-first search. It constructs a project dependency tree by treating each package in the project as a node and the dependency relationship between packages as an edge. The project dependency tree is used to describe the direct and indirect dependency relationships between packages in the project.

[0105] The dependency obfuscation detection module is used to cross-compare the versions of private software packages and public repositories with the same-named software packages referenced by paths in the project dependency tree in order to identify potential obfuscated dependency packages in the project. The private software packages are software packages that exist in the project's enterprise private repository.

[0106] To verify the effectiveness of this solution, the following explanation is based on experimental data:

[0107] To comprehensively evaluate the performance of the DCFinder algorithm in real-world Python projects, an initial dataset was selected from the top 1000 Python projects with the most stars on GitHub as of May 2025. After filtering and excluding projects that did not contain configuration files (requirements.txt, Pipfile, pyproject.toml, setup.cfg), 901 valid projects were retained for analysis of configuration file distribution and dependency characteristics.

[0108] To further verify DCFinder's dependency obfuscation detection capabilities, and considering the lack of publicly available dependency obfuscation datasets and access restrictions on private repositories, an indirect data collection scheme was designed for the experiment:

[0109] (1) GitHub code repository crawling: Search for Python projects using --extra-index-url as the keyword, and filter out code repositories that explicitly contain private repository URLs.

[0110] (2) Repository reachability verification: The validity of the private repository URL is confirmed by HTTP status code detection (200 OK), and invalid or unreachable repositories are excluded.

[0111] (3) Attack scenario simulation: Packets that meet the same name condition are considered potential attack targets, even if they come from the same organization.

[0112] Following the above process, seven repositories that meet the criteria were selected from the top 1000 Python projects on GitHub as an experimental dataset, as shown in Table 1, to verify DCFinder's dependency obfuscation detection capabilities.

[0113] Table 1 Dependency Obfuscation Dataset

[0114]

[0115] 1. Statistics on the distribution of Python project configuration files

[0116] To delve into the usage patterns and distribution of configuration files in Python projects, this study scanned and analyzed 901 open-source Python projects, identifying various types of configuration files contained within them. The statistical results of the experiment are as follows: Figure 4 As shown in the diagram, the experimental results indicate that the lightweight text configuration file `requirements.txt` remains dominant in the current Python ecosystem, with 76% of the scanned projects using this format for dependency management. Meanwhile, `pyproject.toml` files and `setup.py` script files are also prevalent, with adoption rates of 52% and 53%, respectively. Furthermore, analysis of the experimental results reveals a common phenomenon of multiple configuration files coexisting in projects, with 32% of projects using two or more types of configuration files simultaneously. For example, the well-known project AutoGPT includes `requirements.txt`, `pyproject.toml`, and `setup.py` files.

[0117] Experimental results demonstrate the importance of comprehensively analyzing multiple configuration files. Different configuration files play crucial roles in different stages and environments of a project, and a single file cannot adequately cover all project dependencies. To ensure the accuracy and completeness of dependency obfuscation detection, these configuration files must be considered simultaneously to obtain comprehensive dependency information for the project.

[0118] 2. Direct and Indirect Dependency Analysis

[0119] To deeply analyze the dependency structure characteristics of Python projects, this study extracted dependencies from the configuration files of 901 projects. Based on the breadth-first search (BFS) algorithm, a layer of indirect dependency resolution was performed on the project's dependency tree (i.e., only direct transitive dependencies of direct dependencies were analyzed). Experimental results are as follows: Figure 5 As shown in the figure, the number of indirect dependencies is significantly higher than that of direct dependencies in all configuration files. Taking requirements.txt as an example, the average number of direct dependencies is 31.89, while the average number of indirect dependencies is as high as 145.24, a ratio of approximately 1:4.6. Other configuration file types show similar trends.

[0120] Experimental data shows that the number of indirect dependencies is significantly higher than that of direct dependencies. This phenomenon reveals an important security issue: attackers can exploit dependency obfuscation attacks to upload malicious packages with the same names as indirect dependencies to public repositories. Because existing tools generally lack sufficient recursion depth (considering only direct dependencies), such malicious packages can easily bypass detection, further expanding the attack surface of the software supply chain and making deep dependency chains a weak link in security governance.

[0121] 3. Dependency Obfuscation Detection and Analysis

[0122] To evaluate the performance of DCFinder in the dependency obfuscation detection task, this study selected the mainstream tools Confused and Combobulator as comparison objects, and carried out experimental analysis from two dimensions: dependency identification ability and potential dependency obfuscation detection ability.

[0123] (1) Experimental setup

[0124] Dataset: From the top 1000 Python projects with stars on GitHub as of May 2025, 7 repositories were selected that simultaneously meet the criteria of "containing --extra-index-url and accessible from private repositories", as detailed in Table 1.

[0125] Truth Value Construction: In the truth value construction for dependency obfuscation detection, for the dependency package identification and detection stage, a strategy combining manual review and automated parsing was adopted for truth value annotation of direct and indirect dependencies in the Python project dependency graph. For direct dependencies, the dependency packages explicitly declared in various configuration files within the project were analyzed manually to ensure that the explicitly declared dependency relationships were recorded completely and accurately. For indirect dependencies (i.e., transitive dependencies), a dedicated parsing program was written to automatically obtain and analyze dependency packages introduced by private package binary distribution or source code release. For the truth value determination of packages with potential dependency obfuscation risks, the highest available version of each private dependency package is first extracted from the package metadata (such as .dist-info / METADATA, PKG-INFO files) or package files (.whl, .tar.gz format) in the private repository. Then, the information of public packages with the same name as the private package is queried through the PyPI official API, and the highest release version of the public package is extracted. If there is a package with the same name as the private package in the public repository and its highest version is strictly higher than the highest available version of the private package, then the private package is marked as a positive sample with "dependency obfuscation risk". Otherwise, it is judged as a negative sample without risk.

[0126] (2) Reliance on recognition capabilities

[0127] To quantitatively evaluate dependency identification methods, this study uses three commonly used performance metrics—precision, recall, and F1 score—to quantitatively analyze the performance of Confused and DCFinder in direct dependency identification tasks.

[0128] Among these metrics, Precision (TP) = (TP + FP) measures the proportion of correctly identified dependencies among those identified by the tool; Recall (TP) = (TP + FN) evaluates the tool's coverage of true dependencies; and F1-Score is the harmonic mean of precision and recall, used to comprehensively evaluate identification performance. In these metrics, TP (TruePositive) represents the number of correctly identified true dependencies, FP (False Positive) represents the number of incorrectly identified dependencies, FN (False Negative) represents the number of missed true dependencies, and Ground Truth represents the total number of manually verified true dependencies.

[0129] Since Combobulator lacks native support for Python projects (requires manual input of dependency information), the experiment only compared the dependency resolution capabilities of Confused and DCFinder. The results are shown in Table 2, which covers the number of packages resolved by the two methods at the "direct dependency" and "indirect dependency" levels.

[0130] Table 2. Identification results depending on different methods

[0131]

[0132] In terms of direct dependency identification, DCFinder significantly outperformed Confused (73.4% and 84.7%) in both recall (96.5%) and F1-Score (98.2%), indicating that DCFinder can more comprehensively capture the actual direct dependencies in a project. Both tools achieved 100% precision, meaning there were no false positives in their identification results. DCFinder's high recall is mainly due to its support for parsing multiple types of configuration files (such as setup.py, pyproject.toml, etc.), while Confused only supports the single format of requirements.txt, causing it to miss a large number of dependencies declared through other means.

[0133] In terms of indirect dependency identification, DCFinder's results in all test projects were completely consistent with the manually verified ground truth set, indicating that its breadth-first search (BFS) algorithm can effectively traverse and parse the complete dependency tree, recursively obtaining all transitive dependencies, thus ensuring a very high identification recall rate. In contrast, the Confused tool failed to identify any indirect dependencies in any project, exposing its fundamental deficiency in lacking indirect dependency analysis capabilities and its inability to construct a complete project dependency graph. DCFinder can effectively handle complex dependency chains introduced by private packages (including both binary distributions and source code releases).

[0134] The main reason DCFinder missed identifying direct dependencies in the axolotl project is that the `extras_require` field in the `setup.py` configuration file is not declared as a literal constant, but rather as an external dependency variable that can only be calculated at runtime. This means the final value cannot be derived through static AST analysis and the "recursive resolution of variable references and simple function calls" strategy, leading to identification failure. However, DCFinder can achieve comprehensive identification of indirect dependencies. This is because, of the 12 direct dependencies identified in `setup.py`, only `auto-gptq` is a private dependency package; the others, such as `mlflow`, `galore`, and `apollo`, exist in public repositories. When constructing the dependency tree for `auto-gptq`, subsequent branches that only exist in public repositories (e.g., `rouge`, `gekko`, `safetensors`, `ruff`, `parameterized`) are pruned. Dependencies (such as `datasets`) that overlap with existing direct dependencies in the project are merged into the dependency tree, thus optimizing the resolution process and ensuring complete identification.

[0135] (3) Risk detection capability

[0136] In terms of relying on obfuscation risk detection, Confused, Combobulator, and DCFinder show significant differences in performance, as shown in Table 3.

[0137] Table 3 shows the number of potential dependency obfuscators identified by each method.

[0138]

[0139] Confused and Combobulator failed to identify any potential obfuscated packages in any direct or indirect dependencies across all experimental projects (outputting 0 for all), while DCFinder's detection results were completely consistent with the manually verified ground truth. This discrepancy stems primarily from fundamentally different detection strategies: Confused and Combobulator can only check the existence of dependent packages in public repositories (such as PyPI) based on their names, lacking the ability to analyze version cross-relationships between private and public repositories, and also failing to introduce version comparison functionality. Therefore, they can only generate alerts for packages that clearly exist in private repositories, but cannot identify the potential obfuscation risks caused by version differences of packages with the same name in different repositories.

[0140] DCFinder employs a multi-dimensional version comparison mechanism, utilizing the PyPI official API to obtain version information of packages with the same name in public repositories and precisely comparing it with package versions in private repositories. This allows for accurate identification of potential obfuscation vulnerabilities with version differences. For example, in the axolotl project, DCFinder successfully identified potential obfuscated packages in 5 direct dependencies and 37 indirect dependencies; in the kohya_ss project, it also detected obfuscation risks in 4 direct dependencies and 46 indirect dependencies, perfectly matching the truth set. This result demonstrates that DCFinder has extremely high recall and precision in detecting obfuscation risks in both direct and indirect dependencies, and is particularly adept at handling complex dependency chains and indirect dependency attack scenarios.

[0141] DCFinder's advantage lies in its multi-format configuration file parsing and dependency tree construction technology, ensuring comprehensive coverage of dependency information, and combined with an efficient version comparison algorithm to improve detection accuracy. In contrast, Confused and Combobulator, lacking indirect dependency analysis capabilities and version comparison functions, struggle to cope with the complex dependency obfuscation scenarios in modern software supply chains, resulting in extremely high false negative rates. The research results clearly demonstrate that DCFinder has significant advantages in dependency obfuscation risk detection, providing a more reliable technical guarantee for software supply chain security.

[0142] The experimental results demonstrate that the multi-dimensional dependency obfuscation detection scheme presented in this case effectively addresses the shortcomings of existing tools. By recursively parsing direct and indirect dependency chains, it constructs a complete dependency tree model, providing a more comprehensive defense against supply chain attacks. It significantly outperforms existing tools such as Confused and Combobulator in dependency coverage and detection accuracy, ensuring the complete extraction of dependency information. Utilizing a breadth-first search algorithm and pruning strategies, the constructed dependency tree effectively covers multi-layered dependency propagation scenarios. The innovative version comparison mechanism accurately determines version differences through the PyPI official API, further improving detection accuracy and providing new ideas and methods for research and practice in the field of software supply chain security.

[0143] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0144] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0145] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0146] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0147] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / package>

Claims

1. A dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis, characterized in that, Include: For the project source code files, the configuration files are scanned and dependency analysis is performed to obtain the project package dependency set. The configuration files contain several configuration files of different formats. By recursively parsing the multi-level indirect dependencies in the project's software package dependency set using pruning strategies and breadth-first search, and constructing a project dependency tree with each software package in the project as a node and the dependencies between software packages as edges, the project dependency tree is used to describe the direct and indirect dependencies between software packages in the project. Based on the private packages referenced by the paths in the project dependency tree, the versions of the private packages and the packages with the same name in the public repository are cross-compared to identify potential obfuscated dependency packages in the project. The private packages are those that exist in the project's enterprise private repository.

2. The dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis according to claim 1, characterized in that, Scan the configuration file and perform dependency analysis to obtain the project's package dependency set, including: Obtain text configuration files and executable files from the project by scanning the project files; For text configuration files and executable files, a differentiated parsing strategy is used to extract the names of dependent software packages in each file and to build a project software package dependency set. The differentiated parsing strategy includes a line-by-line parsing strategy, a parser-based parsing strategy, and an abstract syntax tree-based parsing strategy.

3. The dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis according to claim 2, characterized in that, Extract the names of dependent packages from the file using an abstract syntax tree-based parsing strategy, including: Parse the project source code into an abstract syntax tree; Traverse each node in the abstract syntax tree, extract dependencies from the string literals corresponding to the nodes, and record variable references, so as to capture all potential dependencies in the source code through recursive processing.

4. The dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis according to claim 1, characterized in that, The project's package dependency set is recursively resolved using pruning strategies and breadth-first search, including multiple levels of indirect dependencies: Obtain the project's declared direct dependencies based on the project's package dependency set; Starting with the directly dependent packages, the sub-dependencies of each package are resolved layer by layer.

5. The dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis according to claim 4, characterized in that, The sub-dependencies of each package are resolved layer by layer, including: Use a queue to store all root dependency packages in the package dependency set; Iterate through each root dependency package in the queue and determine the source repository type of the root dependency package; If the source repository type is a private repository, extract the direct dependencies of the root dependency package, put each sub-dependency into a queue, and append the root dependency package and its direct dependencies to the dependency tree. If the source repository type is a public repository, then the root dependency package will be used as the leaf node of the dependency tree.

6. The dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis according to claim 1, characterized in that, Cross-compare the versions of private packages and packages with the same name in public repositories, including: Retrieve the names and currently highest available version numbers of private packages in the project dependency tree; Query and retrieve information about the same package from the public repository, extract a list of all release version numbers of the same package from the information, and calculate the highest version number; If the highest version number is greater than the current highest available version number, then the current private software package is marked as having a dependency obfuscation risk.

7. The dependency obfuscation vulnerability detection method based on multi-dimensional dependency analysis according to claim 6, characterized in that, Retrieve the names and currently highest available version numbers of private packages in the project dependency tree, including: The highest available version number is obtained from the specified metadata file according to the distribution format. The distribution format includes source code distribution packages and binary distribution packages. The specified metadata file is a distribution package metadata file that stores private package version information.

8. A dependency obfuscation vulnerability detection system based on multi-dimensional dependency analysis, characterized in that, It includes: a scanning and analysis module, a dependency tree construction module, and a dependency obfuscation detection module. The scanning and analysis module is used to scan the configuration files of the project source code files and obtain the project package dependency set through dependency analysis. The configuration files contain several configuration files of different formats. The dependency tree building module is used to recursively parse the multi-level indirect dependencies in the project package dependency set through pruning strategies and breadth-first search. It constructs a project dependency tree by treating each package in the project as a node and the dependency relationship between packages as an edge. The project dependency tree is used to describe the direct and indirect dependency relationships between packages in the project. The dependency obfuscation detection module is used to cross-compare the versions of private software packages and public repositories with the same-named software packages referenced by paths in the project dependency tree in order to identify potential obfuscated dependency packages in the project. The private software packages are software packages that exist in the project's enterprise private repository.

9. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 7.

Images