A dependency analysis and visualization method for a Java open source software supply chain

Abstract

The application discloses a kind of Java open source software supply chain-oriented dependency analysis and visualization method, including that software package metadata set is parsed, obtains all software package characteristic attribute and dependency information, uses software package characteristic attribute and dependency information data to construct graph database, according to the demand, the dependency relationship of software package is parsed using graph database, including direct dependency and indirect dependency, obtain the dependency relationship data of software package, etc. Step, Java software package dependency relationship analysis based on the graph database of the present application, compared with Maven command line parsing, greatly improve the flexibility and efficiency of dependency analysis. Its simplified library table design and the use of version iteration algorithm, further improve the efficiency of graph database dependency analysis. Through a variety of different statistical analysis schemes, the overall system of Java language ecology package dependency relationship and characteristics is comprehensively and systematically represented.

Description

Technical Field

[0001] This invention relates to the field of open-source software supply chain security, specifically, a dependency analysis and visualization method for Java open-source software supply chains. Background Technology

[0002] With the rapid development of the software industry, the complexity of software structures and functions is constantly increasing. Componentization and open-source development have become the mainstream in the industry, driving the vigorous development of third-party libraries in the open-source software supply chain. Statistics show that at least 45% of the code in open-source Java projects comes from open-source software libraries, and only 10% of projects do not use code from open-source libraries. While the increasing complexity and diversification of the open-source software supply chain has significantly reduced the cost and time of software development, it has also led to an increase in cyberattacks targeting weak links in the supply chain, exacerbating software supply chain security risks and becoming a key factor affecting software security. In recent years, security incidents targeting the software supply chain have occurred frequently worldwide, with significant impact. For example, the Apache Log4j2 vulnerability disclosed in December 2021 and the Spring Framework vulnerability disclosed in March 2022, due to their low exploitation difficulty, high harm, and wide impact, have had a huge impact on the security of the entire Java ecosystem and the open-source software supply chain. After the vulnerabilities appear, in addition to the easily analyzable direct impact, the supply chain transmission impact caused by dependencies between software packages should not be underestimated. According to the 2021 Open Source Software Supply Chain Security Risk Research Report, the impact of primary propagation was increased by 125 times due to software package dependencies. Secondary propagation's impact was increased by 173 times compared to the original sample! Open source software supply chain security has become a widespread and far-reaching global security issue, making it imperative to prioritize and research solutions.

[0003] Due to the complexity of modern software dependency trees, clarifying the dependencies between open-source software packages is crucial for effectively preventing the spread of vulnerabilities. Currently, dependency analysis for Java software packages primarily relies on the Maven package management tool, resolving package dependencies by traversing the Maven repository or using the Maven command line. While this method can parse a dependency tree with a single package as the root node, it cannot reverse-engineer the entire dependency tree from the leaf nodes. This approach suffers from inflexibility and inefficiency when studying the dependencies of the entire Java ecosystem. Furthermore, the parsing results are typically displayed as text datasets, lacking intuitiveness and simplicity.

[0004] Therefore, a method is needed to overcome the above problems in Java language ecosystem dependency analysis. Summary of the Invention

[0005] The purpose of this invention is to address the security issues in the open-source software supply chain by providing a dependency analysis and visualization method for the Java open-source software supply chain. Based on graph databases and design algorithms, this method can flexibly and efficiently perform dependency analysis of the Java language ecosystem. Furthermore, by utilizing Gephi software, the analysis results are clearly and aesthetically visualized, which is beneficial for further research and analysis.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a dependency analysis and visualization method for the Java open-source software supply chain, characterized by the following steps:

[0008] 1) Parse the software package metadata dataset to obtain all software package feature attributes and dependency information;

[0009] 2) Construct a graph database using package feature attributes and dependency information data;

[0010] 3) Based on the requirements, use a graph database to parse the dependencies of the software packages, including direct and indirect dependencies, to obtain the dependency data of the software packages;

[0011] 4) Based on the dependency data of software packages, we will study the overall picture and characteristics of the software package dependencies in the Java language ecosystem through different statistical analysis methods;

[0012] 5) Use drawing tools to visualize the dataset containing the overall picture and characteristics of package dependencies.

[0013] As a further improvement, step 2) of the present invention, which uses package feature attributes and dependency information data to construct a graph database, includes the following sub-steps:

[0014] 2.1) Design the database structure table, which contains software package characteristic attributes and dependency information;

[0015] 2.2) Design the database index structure;

[0016] 2.3) Store the package attribute data (as nodes in the graph database) and dependency information data (as edges in the graph database) obtained in step 1) into the graph database one after the other;

[0017] 2.4) Design an algorithm for inserting iterative relationships between software package versions in a graph database.

[0018] As a further improvement, the feature attributes in the sub-steps of this invention include: GroupID, ArtifactID, Version, Timestamp, Packaging, and License.

[0019] As a further improvement, the dependency information in the sub-steps of this invention includes: dependency object, dependency type, and dependency attribute.

[0020] As a further improvement, step 4) of this invention, based on the dependency data of software packages, studies the overall picture and characteristics of the software package dependencies in the Java language ecosystem through different statistical analysis methods, including the following sub-steps:

[0021] 4.1) Construct an automated dependency query interface to resolve the dependency of any software package;

[0022] 4.2) Based on step 4.1), the dependencies of all software packages are analyzed to understand the overall picture and characteristics of the dependencies in the Java language ecosystem.

[0023] As a further improvement, sub-step 4.2) of the present invention includes the following methods:

[0024] 4.2.1) The dependencies between package versions were abstracted and aggregated to obtain the direct dependencies between all packages, thus obtaining a complete picture of the dependencies in the Java ecosystem;

[0025] 4.2.2) Statistically rank the number of dependencies of all packages in the Java language ecosystem, and analyze the mutual dependencies among the top 50 packages;

[0026] 4.2.3) The dependencies of a key software package are abstracted hierarchically by propagation distance, and the direct and indirect dependent software packages (including all versions) are counted at each level to analyze the hierarchical influence of the key software package.

[0027] 4.2.4) Count the number of dependencies of a key software package that are referenced by other software packages, and analyze the weight distribution of the influence of the key software package.

[0028] As a further improvement, step 5) of this invention uses a drawing tool to visualize the dataset containing the overall picture and features of package dependencies. The visualized dependency graph includes:

[0029] 5.1) Use the data from step 4.2.1) to draw a complete picture and macro-scale diagram of the Java ecosystem dependencies;

[0030] 5.2) Use the data from step 4.2.2) to draw a graph of dependencies between popular software packages;

[0031] 5.3) Using the data from step 4.2.3), plot the propagation distance of a key software package dependency.

[0032] 5.4) Using the data from step 4.2.4), draw a weight distribution diagram of the propagation influence of a key software package.

[0033] The beneficial effects of this invention are:

[0034] This invention addresses the transitive impact of software package dependencies on open-source software supply chain security. Maven, a Java package management library with a large number of packages and a wide impact, is selected as the research object. Based on a constructed database and designed analysis methods, a multi-dimensional systematic analysis and visualization of dependency paths between Maven packages is conducted. Compared to Maven command-line parsing, graph database-based Java package dependency analysis significantly improves the flexibility and efficiency of dependency resolution. Its streamlined table design and version iteration algorithm further enhance the efficiency of graph database dependency resolution. Through various statistical analysis schemes, the overall picture and characteristics of package dependencies in the Java language ecosystem are comprehensively and systematically presented. Simultaneously, the use of Gephi drawing tools greatly increases the amount of data processed for visualization, while the diverse and easily extensible layout algorithms make the visualization clearer and more aesthetically pleasing, providing favorable conditions for further research and analysis. This leads to comprehensive conclusions, providing a basis for subsequent vulnerability propagation threat analysis. Attached Figure Description

[0035] Figure 1 This is the data table structure of a graph database;

[0036] Figure 2 This is the pseudocode for the version iteration relationship insertion algorithm;

[0037] Figure 3 It is a method flowchart. Detailed Implementation

[0038] This invention, based on Maven data parsing and extraction, constructs a native dependency graph database for software supply chain analysis, accurately depicting the dependency, inheritance, and version iteration relationships between software packages across the entire time span and all version scales. Relying on existing graph algorithms, it optimizes the database's performance in node creation, relationship insertion, and path querying under massive dependency paths. Ultimately, it enables diverse analysis of the current dependency status in the Java ecosystem and provides visualization using graphing software. The invention is described in detail below with reference to the accompanying drawings. The purpose and effects of this invention will become clearer as a result. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] This invention provides a method for dependency analysis and visualization in the Java open-source software supply chain. Figure 3 The main implementation flow of the method is shown, which specifically includes the following steps:

[0040] 1) Parse the software package metadata dataset to obtain all software package feature attributes and dependency information:

[0041] Based on crawling all packages from the Maven repository, the characteristics and dependency information of the packages are analyzed. For each package, its metadata includes attributes such as group, artifact, version, and dependencies. This metadata resides in the corresponding pom file. The downloaded pom file is then processed by a pom file parser to extract all the obtained package characteristics, attributes, and dependency information, which is stored in a JSON file.

[0042] 2) Construct a graph database using package feature attributes and dependency information data:

[0043] 2-1) Design a graph database structure table containing package characteristic attributes and dependency information. Since constructing dependency edges in a graph database requires both ends of the edge to exist in the database, it's necessary to first obtain all package information through traversal and store it as nodes in the database. The table structure design needs to balance the completeness of node information with efficient storage. To improve storage efficiency and analytical effectiveness, some characteristic attributes of the Maven database were selected to design the database table structure. For example... Figure 1As shown, `coordinate` is used to uniquely identify the package version and consists of `{groupID:artifactID:version}`. It also includes `GroupID` (group number), `ArtifactID` (package number), `Version`, `Timestamp`, `Packaging` (package type), and `License` (open-source license). Dependency types include: `DEPENDS_ON`, `CHILD_OF`, and `NEXT_VER`. `DEPENDS_ON` corresponds to direct dependencies, `CHILD_OF` is used for package version attribute inheritance, allowing child nodes to inherit dependencies and other attributes from their parent nodes, and `NEXT_VER` is primarily used for version iteration relationships within the same package. `scope` identifies six dependency attributes to distinguish different dependency usage scenarios, including: `Compile`, `Runtime`, `Provided`, `Test`, `System`, and `Import`.

[0044] 2-2) Design an index structure using the node's coordinate attribute as the key. To avoid duplicate node imports and effectively improve subsequent query efficiency, a new attribute, coordinate, is constructed from the three characteristic attributes {groupID:artifactID:version}, serving as a unique identifier for the software package in the database.

[0045] 2-3) Using the parsed dataset as input, the software package feature attribute data (as nodes in the graph database) and dependency information data (as edges in the graph database) are stored in the database sequentially through the reserved programming interface of the graph database.

[0046] 2-4) Design and implement a version iteration relationship insertion algorithm. Since the package information in the pom file is too scattered, an algorithm for inserting version iteration relationships of packages of the same type was designed and implemented using the Neo4j database. Figure 2 This is the pseudocode for the version iteration relationship insertion algorithm. The algorithm is as follows: First, iterate through all nodes in the database, extract the ID {GroupID:ArtifactID} of each node, and count the frequency of each ID. Second, filter IDs with a frequency greater than one. Third, query the database for packages with the same ID as the key. Then, sort the packages according to their timestamps. Finally, insert the version iteration relationship NEXT_VER one by one in ascending order.

[0047] 3) Based on the requirements, use a graph database to parse the dependencies of the software packages, including direct and indirect dependencies, to obtain the dependency data of the software packages:

[0048] An automated dependency query interface for software packages is built to resolve dependencies of any software package. After data storage is completed, the dependency subgraph associated with the input software package information can be queried. By combining existing classic graph database algorithms, actual data characteristics, and application requirements, query requirements are customized to improve the retrieval performance of nodes, direct dependency edges, and transitive dependency paths. The specific customization method is as follows: First, select the starting software package, which can be a specific node or a type of node. For example, input "org.apache.logging.log4j:log4j-core:2.0" as the starting software package. Second, select the relationship type. By default, all types of relationships will be traversed. This option needs to be used to limit the query when querying dependencies. For example, if you need to parse all dependencies of a software package, you can select the "DEPENDS_ON" type. Third, select the traversal direction. By default, bidirectional relationships will be traversed, which will affect the performance of the analysis system. This parameter is used to limit the relationship direction when traversing the subgraph. If '>' is entered, the forward traversal will be performed; if '<' is entered, the reverse traversal will be performed. For example, to parse all dependencies of a package, select '>' for forward traversal; to parse all dependencies of the package, select '<' for backward traversal. Here, we'll use forward traversal as an example. Next, select the traversal depth; by default, it will traverse according to the maximum depth. If the minimum traversal parameter is 0, the returned subgraph will have the input node as its root; if it's 1, the returned subgraph will have the direct relation nodes of the input node as their root nodes. If the minimum traversal parameter is -1, it will traverse according to the maximum depth. You can choose the traversal depth according to your specific needs, for example, 14. Then select the algorithm; the default is to use the DFS algorithm, but you can also choose the BFS algorithm. The traversal algorithm can be chosen arbitrarily, or you can use the default options. Finally, select the end package; by default, the traversal depth is used as the end marker. If node information is input, the traversal will stop when the node is encountered and the subgraph will be returned. If you are parsing all dependencies or dependents of a package, you generally end the traversal by the traversal depth. If you need to verify whether a package is a dependency or dependent, you can set the last package as the end package. Finally, the dependency data of the input software package is obtained.

[0049] 4) Based on package dependency data, various statistical analysis methods are used to study the overall picture and characteristics of package dependencies in the Java language ecosystem:

[0050] 4-1) Building upon step 3, the direct dependencies of all software packages (i.e., level 1 traversal depth) were analyzed. Then, the dependencies of all versions of each software package were aggregated to obtain the direct dependencies between all software packages, thus obtaining a complete picture of the Java language ecosystem and the distribution of its dependencies. Step 5) then draws a comprehensive map showing the macro-scale proportions of the Java language ecosystem. Each node represents a Java software package, each edge represents a dependency between two software packages, and each color represents a software package and its affected dependencies. This map visually displays the entire Java language ecosystem and the distribution of its dependency network. We can clearly see from the map that widely relied-upon software packages in the Java ecosystem include Scala-library, Slf4j-api, and Guava. These software packages and their associated dependencies occupy a large area in the map, indicating their significant ecosystem influence. If vulnerabilities are exposed, they will have a serious impact.

[0051] 4-2) Building upon step 3), the dependencies of all software packages were parsed (through reverse traversal), and the dependencies of all versions of each package were aggregated. Then, the direct influence of all packages in the Java ecosystem, i.e., the number of dependencies, was statistically analyzed. The Top 50 packages and their dependencies were selected in descending order, and a dependency graph of popular packages was drawn in step 5). Popular packages in the Java ecosystem include scala-library, guava, slf4j-api, etc., and these popular packages have interdependencies. For example, the graph shows that the Guava package depends on the popular package Jsr305, and is also depended on by another popular package, Guice, forming an interconnected dependency group. This means that if a vulnerability exists in these packages, its propagation will have an amplified effect, resulting in a wider reach and greater impact.

[0052] 4-3) Building upon step 3), the dependencies of a critical software package are abstracted hierarchically based on propagation distance. Direct and indirect dependencies (including all versions) are statistically analyzed by level. The root node represents the critical package, the first level consists of packages directly dependent on the root node, and other packages at different levels indirectly depend on the root node. This allows analysis of the hierarchical influence of the critical package. For example, parsing the dependencies of all versions of Log4j (14 levels traversed in reverse) yields all packages dependent on Log4j from levels 1 to 14. Then, these dependencies are statistically analyzed hierarchically by propagation distance to obtain the packages and dependencies at each level. Step 5) is then used to plot the propagation distance of the Log4j package dependencies. The Log4j package affects over a million package versions, indicating a wide impact. The plot clearly shows that most dependencies are concentrated within five levels, indicating a shallow reference level to Log4j and a short code path, making vulnerabilities easily triggered. Furthermore, the prevalence of shallow dependencies facilitates vulnerability blocking and remediation.

[0053] 4-4) Building upon step 3), count the number of dependencies of a critical software package that are in turn depended upon by other software packages, thereby analyzing the weight distribution of the critical software package's influence. For example, perform reverse parsing on all dependencies of Log4j to obtain the corresponding number of dependencies. Then, sort them according to the number of dependencies from highest to lowest, and finally select the top 30 software packages. Step 5) then plots and displays the weight distribution of the Log4j software package's propagation influence. The plot clearly shows the distribution of the top 30 software packages with the highest influence weights among those that depend on Log4j. The critical software packages with the highest propagation influence rankings, due to their large number of dependencies, are key to further expanding the impact of vulnerability propagation, and they play a crucial role in blocking and patching vulnerabilities.

[0054] 5) Use plotting tools to visualize the dataset containing the overall picture and characteristics of package dependencies:

[0055] 5-1) The software package data (nodes) and dependency information data (edges) obtained from the graph database are processed using different statistical analysis methods to obtain a dataset containing the overall picture and characteristics of the software package dependencies. This data is then imported into the Gephi software as a file.

[0056] 5-2) Then, use Gephi software to set the display attributes of the nodes, such as color, size, border, and edge color and thickness. The color and size can be uniformly rendered according to a certain attribute value based on the display requirements. For example, in the Log4j dependency propagation distance graph, the size and color of the root node Log4j are set separately, while the size of other nodes is uniformly set, and the color is uniformly set according to the node's level, with different colors rendered at different levels.

[0057] 5-3) In terms of layout, various layout algorithms such as ForceAtlas, NetworkSplitter 3D, and Circular layout can be selected to render the dependency network graph according to different display needs. For example, in the Log4j dependency propagation distance graph, the Network Splitter 3D layout algorithm is selected to display the vertical hierarchy effect, and the hierarchy distance from the node to the root node is set as the vertical layout parameter in the algorithm.

[0058] Furthermore, the visualization of dependency graphs in this method includes, but is not limited to:

[0059] 5-3-1) Overview and macro-scale diagram of Java ecosystem dependencies;

[0060] 5-3-2) Dependency graph of popular software packages;

[0061] 5-3-3) Propagation distance diagram of critical software package dependencies;

[0062] 5-3-4) Distribution chart of the influence of key software packages.

[0063] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A method for dependency analysis and visualization in the Java open-source software supply chain, characterized in that, Includes the following steps: 1) Parse the software package metadata dataset to obtain all software package feature attributes and dependency information; 2) Construct a graph database using package feature attributes and dependency information data; 3) Based on the requirements, use a graph database to parse the dependencies of the software packages, including direct and indirect dependencies, to obtain the dependency data of the software packages; 4) Based on the dependency data of software packages, we will study the overall picture and characteristics of the software package dependencies in the Java language ecosystem through different statistical analysis methods; 5) Use drawing tools to visualize the dataset containing the overall picture and characteristics of package dependencies; Step 2) involves constructing a graph database using package feature attributes and dependency information data, which includes the following sub-steps: 2.1) Design the database structure table, which contains software package characteristic attributes and dependency information; 2.2) Design the database index structure; 2.3) Store the package attribute data and dependency information data obtained in step 1) into the graph database in sequence. The package attribute data is used as a node in the graph database, and the dependency information data is used as an edge in the graph database. 2.4) Design an algorithm for inserting iterative relationships between software package versions in a graph database; Step 4) involves studying the overall picture and characteristics of package dependencies in the Java language ecosystem based on package dependency data and using different statistical analysis methods. This includes the following sub-steps: 4.1) Construct an automated dependency query interface to resolve the dependency of any software package; 4.2) Based on step 4.1), by parsing the dependencies of all software packages, we analyze the overall picture and characteristics of the dependencies in the Java language ecosystem; Sub-step 4.2) includes the following methods: 4.2.1) The dependencies between package versions were abstracted and aggregated to obtain the direct dependencies between all packages, thus obtaining a complete picture of the dependencies in the Java ecosystem; 4.2.2) Statistically rank the number of dependencies of all packages in the Java language ecosystem, and analyze the mutual dependencies among the top 50 packages; 4.2.3) The dependencies of a key software package are abstracted hierarchically by propagation distance, and the direct and indirect dependent software packages are counted by level. The software packages include all versions, thereby analyzing the hierarchical influence of the key software packages. 4.2.4) Count the number of dependencies of a key software package that are referenced by other software packages, and analyze the weight distribution of the influence of the key software package.

2. The dependency analysis and visualization method for the Java open-source software supply chain according to claim 1, characterized in that, The characteristic attributes in the sub-step include: GroupID, ArtifactID, Version, Timestamp, Packaging, and License.

3. The dependency analysis and visualization method for the Java open-source software supply chain according to claim 1, characterized in that, The dependency information in the sub-step includes: dependency object, dependency type, and dependency attribute.

4. The dependency analysis and visualization method for the Java open-source software supply chain according to claim 1, characterized in that, In step 5), a plotting tool is used to visualize the dataset containing the overall picture and features of package dependencies. The visualized dependency graph includes: 5.1) Use the data from step 4.2.1) to draw a complete picture and macro-scale diagram of the Java ecosystem dependencies; 5.2) Use the data from step 4.2.2) to draw a graph of dependencies between popular software packages; 5.3) Using the data from step 4.2.3), plot the propagation distance of a key software package dependency. 5.4) Using the data from step 4.2.4), draw a weight distribution diagram of the propagation influence of a key software package.

Images