An open source component provenance analysis method and system

Abstract

The application discloses an open source component bloodline analysis method and system, relates to the technical field of software supply chain security, and comprises the following steps: S1, automatically discovering open source components; S2, analyzing software components; S3, open source component fingerprint analysis; S4, generating an SBOM bill of materials; and S5, open source component bloodline analysis. The application realizes comprehensive tracking of a source code data processing process through a cryptographic hash function and a cosine similarity algorithm. When a component is modified, the component can also be quickly identified through a similarity analysis technology. The platform regularly collects file fingerprint features of common open source components, extracts characteristic values in combination with extracted open source software dependency packages, saves the characteristic values for comparison, identifies tampered code details in a jar package, locates open source component risks, and improves controllability and security management of a software deployment package.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of software supply chain security, in particular to an open source component blood relationship analysis method and system. BACKGROUND

[0002] In recent years, under the Internet environment, data leakage, network extortion and other events occur frequently, and these events are related to software and hardware vulnerabilities. According to relevant data, the scanning results of the Source Check platform show that 53.8% of the software projects contain at least one high-risk vulnerability, which increases the risk of software use. If the vulnerability is not repaired in time, it will not be able to provide a good experience for users. The software project involves delivery, operation, maintenance, update and other links. Compared with the delivery and operation links, the security protection of the development link is relatively lacking, and the development as an upstream part of the supply chain will affect the downstream link and cause losses once a security event occurs in this link. In the patent with the application number CN202211737067.4, a power industry software supply chain security management platform is disclosed, which includes a supplier evaluation module, a code component analysis module, an automatic intrusion and attack simulation module, and an automatic intrusion and attack simulation module. Based on the knowledge graph artificial intelligence algorithm, combined with business component identification, an attack simulation tool is automatically generated to help enterprises real-time understand software supply chain security risks. The present application is based on the whole life cycle security management of software supply chain, and introduces security management for the power industry business system software in the procurement, online, operation and other stages, improves the overall security prevention, and effectively handles the balance between agile development and security cost.

[0003] However, the difficulty of software supply chain security check is quite large, and the following difficulties are mainly faced in the software delivery process: firstly, attackers control the upstream software supply chain by polluting development tools, reusing open source code and other methods, and take advantage of the characteristics of software development fragmentation to promote the rapid spread of polluted code and affect the downstream users. Secondly, due to the lack of security analysis and evaluation of delivery channels, delivery environment and other aspects, attackers use methods such as bundled software, download hijacking and dependency confusion to implement attacks. Thirdly, due to the lack of strict inspection of users on upgrade programs, components and codes, attackers implant malicious codes to control user systems in the process of updating and patching. SUMMARY

[0004] The purpose of the present application is to provide an open source component blood relationship analysis method and system to solve the problems raised in the background art.

[0005] To achieve the above object, the application provides the following technical scheme: an open source component blood analysis method, comprising the following steps:

[0006] S1, automatically discovering open source components: through scanning and detecting system source code and compiling deployment packages, automatically discovering system referenced open source components through component fingerprint characteristics, dependency and correlation analysis, automatically discovering the framework used by the system through the referenced open source components and open source component feature recognition, and discovering the threats and risks of the framework through correlation threat intelligence;

[0007] S2, analyzing software components: after discovering the open source components, accurately recognizing the open source components used in the software, and supporting the recognition of open source component versions developed in multiple languages such as Java, Python, JavaScript,.NET, PHP,.Net, C++, and Perl;

[0008] S3, open source component fingerprint analysis: using open source component fingerprint recognition tools for dynamic scanning, collecting the list of all open source components used in the software, extracting and utilizing information data in the open source components;

[0009] S4, generating an SBOM bill of materials: generating a complete SBOM bill of materials, analyzing the information data used by the developer, and accurately recognizing known security vulnerabilities existing in the system;

[0010] S5, open source component blood analysis: unpacking and retrieving the file to be traced, extracting attribute data according to the retrieval condition, obtaining manufacturer, component name and version number information, finding the corresponding original jar file in the feature warehouse according to the manufacturer, component name and version number information, analyzing the file to be traced and the corresponding original jar file to find the host, extracting feature values through similarity analysis technology and saving them for comparison, and identifying tampered code details in the jar package.

[0011] Preferably, the step S3 specifically comprises the following steps:

[0012] S301, using OWASPDependency-Check, Nexpose, BlackDuck and Snyk open source component fingerprint recognition tools to dynamically scan the source code of the software open source components, dependency list and binary files;

[0013] S302, collecting the list of all open source components used in the software, including the version, license information, potential vulnerability library, framework and auxiliary dependency of the open source component;

[0014] S303, extract the specific code fragments, version numbers and file names of the open source components, and generate a unique fingerprint and hash value using the extracted data and content of the open source components through a cryptographic hash function.

[0015] Preferably, the step S4 specifically comprises the following steps:

[0016] S401, generate a complete SBOM bill of materials through software component analysis, and establish a software composition map at the same time;

[0017] S402, analyze various source codes, modules, frameworks and libraries used by developers to identify and inventory components of open source software and their composition and dependency relationships;

[0018] S403, accurately identify known security vulnerabilities and potential license authorization problems existing in the system, and protect end users from security vulnerabilities.

[0019] Preferably, the step S5 specifically comprises the following steps:

[0020] S501, use the zip decompression algorithm to unpack the file1.jar file to be traced, and search for the pom.xml file, if the pom file does not exist, continue to search for the meta-inf.mf file;

[0021] S502, extract the data of the three attributes of groupId, artifactId and version of the Pom.xml file, which respectively correspond to the manufacturer, component name and version number information;

[0022] S503, extract the data of the three attributes of Specification-Title, Implementation-Title and Implementation-Version of the meta-inf.mf file, which respectively correspond to the manufacturer, component name and version number information;

[0023] S504, find the corresponding original jar file file2.jar in the library according to the manufacturer, component name and version number information.

[0024] Preferably, the step S5 specifically further comprises the following steps:

[0025] S505, calculate the sha256 value of file1.jar and file2.jar, if the sha256 values are the same, it means that they are the same file, the tracing is ended, and the program is exited, otherwise, continue to execute S506;

[0026] S506, decompressing file1.jar and file2.jar respectively to obtain class file lists lst1 and lst2 respectively, calculating the sha256 values of the class files in the lst1 and lst2 directories one by one, and removing the files with the same sha256 values;

[0027] S507, generating a tampering list lst3 by using a cosine similarity algorithm on lst1, traversing the lst3 directory, and comparing the contents of each file in lst3 with the file list in lst2 one by one, if the content similarity exceeds 90%, it indicates that the host search is successful.

[0028] Preferably, the step S5 includes pre-prepared data and feature comparison, the pre-prepared data is downloaded jar file through maven official website, and the jar file is decompressed, and feature data is generated after decompression, and is saved into a feature warehouse, the feature comparison first decompresses the jar package in the project, then generates feature data, and then compares in the feature warehouse.

[0029] An open source component blood relationship analysis system, comprising an open source component automatic discovery module, a software component analysis module, an open source component fingerprint analysis module, an SBOM bill of materials module, and an open source component blood relationship analysis module;

[0030] The open source component automatic discovery module automatically discovers the open source components referenced by the system through scanning and detecting the system source code and the compiled deployment package, through component fingerprint characteristics, dependency and correlation analysis, automatically discovers the framework used by the system through the referenced open source components and the feature recognition of the open source components, and discovers the threats and risks of the framework through correlation threat intelligence;

[0031] The software component analysis module accurately identifies the open source components used in the software after discovering the open source components, and supports the identification of open source components developed in multiple languages such as Java, Python, JavaScript,.NET, PHP,.Net, C++, and Perl;

[0032] The open source component fingerprint analysis module uses an open source component fingerprint recognition tool for dynamic scanning, collects the list of all open source components used in the software, extracts and utilizes the information data in the open source components;

[0033] The SBOM bill of materials module generates a complete SBOM bill of materials, analyzes the information data used by the developer, and accurately identifies known security vulnerabilities existing in the system;

[0034] The open source component blood analysis module unpacks and retrieves the file to be traced, extracts attribute data according to the retrieval condition, obtains factory, component name and version number information, finds the corresponding original jar file in the feature warehouse according to the factory, component name and version number information, analyzes the file to be traced and the corresponding original jar file to find the host, extracts the characteristic value through the similarity analysis technology and saves it for comparison, and identifies the tampered code details in the jar package.

[0035] Compared with the prior art, the beneficial effects of the present application are:

[0036] 1、The present application realizes the comprehensive tracking of the source code data processing process through the password hash function and the cosine similarity algorithm, so as to find all related metadata objects with a certain data object as the starting point and the relationship between these metadata objects, and the relationship between these metadata objects is specifically represented as the data flow input and output relationship of these metadata objects. The system will collect the file fingerprint features of common open source components, and through the similarity analysis technology, the modified components can also be quickly identified. The platform will regularly collect the file fingerprint features of common open source components, and combine the extracted open source software dependency package;

[0037] 2、Through the similarity analysis technology, the characteristic value is extracted and saved for comparison, the tampered code details in the jar package are identified, the risk of the open source component is located, the controllability and security management of the software deployment package are improved, the components, libraries, frameworks and dependencies used in the software product are identified, and their sources, versions, licenses and known security vulnerabilities are determined, which helps administrators better understand the security risks and influence scope of the software product and take corresponding measures to reduce the security risks. Through the component fingerprint features, the dependency and correlation analysis mode is used to automatically discover the framework used by the system, and through the correlation threat intelligence, the threats and risks of the framework are discovered. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The schematic diagram of the overall method flow provided by the embodiment of the present application is shown in the figure;

[0039] Figure 2 The step flow schematic diagram of the open source component fingerprint analysis provided by the embodiment of the present application is shown in the figure;

[0040] Figure 3 The step flow schematic diagram of the generation of the SBOM bill of materials provided by the embodiment of the present application is shown in the figure;

[0041] Figure 4 The step flow schematic diagram of the open source component blood analysis provided by the embodiment of the present application is shown in the figure;

[0042] Figure 5 The system unit block diagram provided by the embodiment of the present application is shown in the figure;

[0043] Figure 6 The schematic diagram of open source component automatic discovery provided for the embodiment of the present application is shown in the figure.

[0044] Figure 7 The flowchart of pre-prepared data and feature comparison provided for the embodiment of the present application is shown in the figure.

[0045] In the figure: 1, open source component automatic discovery module; 2, software component analysis module; 3, open source component fingerprint analysis module; 4, SBOM bill of materials module; 5, open source component blood relationship analysis module. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0047] Please refer to Figures 1-7 The present application provides a technical solution: an open source component blood relationship analysis method, comprising the following steps:

[0048] S1, automatically discovering open source components: through scanning and detecting system source code and compiling deployment package, automatically discovering system-referred open source components through component fingerprint features, dependency and correlation analysis, solving the problems of manual difficulty in identification and inaccurate identification, automatically discovering the framework used by the system, and discovering the threats and risks of the framework through correlation threat intelligence;

[0049] S2, analyzing software components: after discovering open source components, accurately identifying the open source components used in the software, and supporting the identification of open source component versions developed in multiple languages such as Java, Python, JavaScript,.NET, PHP,.Net, C++, and Perl;

[0050] S3, open source component fingerprint analysis: using open source component fingerprint identification tools for dynamic scanning, collecting the list of all open source components used in the software, extracting and utilizing information data in the open source components;

[0051] S4, generating SBOM bill of materials: generating a complete SBOM bill of materials, analyzing the information data used by the developer, and accurately identifying known security vulnerabilities existing in the system;

[0052] S5, open source component blood analysis: unpack and retrieve the file to be traced, extract attribute data according to the retrieval condition, obtain manufacturer, component name and version number information, find the corresponding original jar file in the feature warehouse according to the manufacturer, component name and version number information, analyze the file to be traced and the corresponding original jar file to find the host, extract the characteristic value through the similarity analysis technology and save it for comparison, and identify the tampered code details in the jar package.

[0053] Step S3 specifically includes the following steps:

[0054] S301, using OWASPDependency-Check, Nexpose, BlackDuck and Snyk open source component fingerprint identification tools to dynamically scan the source code, dependency list and binary files of the software open source components of the application;

[0055] S302, collect the list of all open source components used in the software, including the version, license information, potential vulnerability library, framework and auxiliary dependency of the open source component;

[0056] S303, extract the specific code fragment, version number and file name of the open source component, and generate a unique fingerprint and hash value using the extracted open source component data and content through a cryptographic hash function, through open source component fingerprint analysis, the development team can better manage the dependency relationship of its software, reduce potential vulnerability risks, and ensure software compliance;

[0057] Step S4 specifically includes the following steps:

[0058] S401, generate a complete SBOM bill of materials through software component analysis, and establish a software composition map at the same time;

[0059] S402, analyze various source codes, modules, frameworks and libraries used by developers to identify and inventory the components of open source software and their composition and dependency relationship;

[0060] S403, accurately identify known security vulnerabilities and potential license authorization problems in the system, and protect end users from security vulnerabilities;

[0061] Step S5 specifically includes the following steps:

[0062] S501, use the zip decompression algorithm to unpack the file1.jar file to be traced, and retrieve the pom.xml file, if the pom. file does not exist, continue to find the meta-inf.mf file;

[0063] S502, extract the data of the three attributes of groupId, artifactId and version of the Pom.xml file, and the data of the attributes correspond to the manufacturer, component name and version number information respectively;

[0064] S503, extract the data of the three attributes of Specification-Title, Implementation-Title and Implementation-Version of the meta-inf.mf file, and the data of the attributes correspond to the manufacturer, component name and version number information respectively;

[0065] S504, find the corresponding original jar file file2.jar in the feature warehouse according to the manufacturer, component name and version number information;

[0066] Step S5 further includes the following steps:

[0067] S505, calculate the sha256 values of file1.jar and file2.jar, if the sha256 values are the same, it means that they are the same file, the traceability ends, and the program exits, otherwise, continue to execute S506;

[0068] S506, decompress file1.jar and file2.jar to obtain respective class file lists lst1 and lst2, calculate the sha256 values of the class files in the directories lst1 and lst2 respectively, and remove the files with the same sha256 values;

[0069] S507, use the cosine similarity algorithm to tamper with the list lst3, traverse the directory lst3, compare the contents of each file in lst3 with the file list of lst2 one by one, if the content similarity is more than 90%, it means that the host search is successful;

[0070] The step S5 includes pre-prepared data and feature comparison, the pre-prepared data is downloaded jar file through maven official website, and the jar file is decompressed, and the feature data is generated after decompression, and is saved into the feature warehouse, the feature comparison first decompresses the jar package in the project, then generates the feature data, and then compares in the feature warehouse;

[0071] An open source component blood relationship analysis system, comprising an open source component automatic discovery module 1, a software component analysis module 2, an open source component fingerprint analysis module 3, an SBOM bill of materials module 4 and an open source component blood relationship analysis module 5;

[0072] The open source component automatic discovery module 1 automatically discovers the open source components referenced by the system through scanning and detecting the system source code and the compiled deployment package, through the component fingerprint characteristics, dependency and correlation analysis, automatically discovers the framework used by the system through the referenced open source components and the characteristics of the open source components, and discovers the threats and risks of the framework through correlation threat intelligence;

[0073] The software component analysis module 2 accurately identifies the open source components used in the software after discovering the open source components, and supports the identification of open source component versions developed in multiple languages such as Java, Python, JavaScript,.NET, PHP,.Net, C++, and Perl;

[0074] The open source component fingerprint analysis module 3 performs dynamic scanning using an open source component fingerprint identification tool, collects a list of all open source components used in the software, extracts and utilizes information data in the open source components;

[0075] The SBOM bill of materials module 4 generates a complete SBOM bill of materials, analyzes the information data used by the developer, and accurately identifies known security vulnerabilities existing in the system;

[0076] The open source component blood analysis module 5 unpacks and retrieves the file to be traced, extracts attribute data according to the retrieval condition, obtains factory, component name and version number information, finds the corresponding original jar file in the feature warehouse according to the manufacturer, component name and version number information, analyzes the file to be traced and the corresponding original jar file to find the host, extracts the characteristic value through the similarity analysis technology and saves it for comparison, and identifies the tampered code details in the jar package.

[0077] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for analyzing the lineage of open source components, characterized by The following steps are involved: S1. Automatic discovery of open source components: Scans and detects system source code and compiled deployment packages. It automatically discovers open source components referenced by the system through component fingerprinting, dependency, and association analysis. It automatically discovers the frameworks used by the system through identifying the referenced open source components and their features. It then identifies threats and risks associated with the frameworks by correlating them with threat intelligence. S2. Analyze software components: After discovering open source components, accurately identify the open source components used in the software, and support the identification of open source component versions developed in multiple languages; S3. Open Source Component Fingerprint Analysis: Use open source component fingerprint recognition tools to perform dynamic scanning and collect a list of all open source components used in the software, extracting and utilizing information data from the open source components; S4. Generate SBOM Bill of Materials: Generate a complete SBOM bill of materials, analyze the information data used by developers, and accurately identify known security vulnerabilities in the system; S5. Open Source Component Lineage Analysis: Unpack and search the file to be traced. Extract attribute data based on the search results to obtain the manufacturer, component name, and version number information. Search the feature repository for the corresponding original JAR file based on the manufacturer, component name, and version number information. Analyze the file to be traced and the corresponding original JAR file to find the host. Use similarity analysis technology to extract feature values ​​and save them for comparison to identify the details of the tampered code in the JAR package. The step S5 specifically includes the following steps: S501. Unpack the file1.jar file to be traced and retrieve the pom.xml file. If the pom.xml file does not exist, continue to search for the meta-inf.mf file. S502: Extract data corresponding to three specific attributes of the pom.xml file and the meta-inf.mf file, including the manufacturer, component name, and version number information; S503, searching the feature repository for the corresponding original jar file file2.jar based on the manufacturer, component name, and version number information; S504. Calculate the sha256 values ​​of file1.jar and file2.jar. If the sha256 values ​​are the same, they are the same file. The traceback process ends and the program exits. Otherwise, the program continues with S505. S505, decompress file1.jar and file2.jar to obtain their respective class file lists lst1 and lst2, calculate the sha256 values ​​of the class files in the lst1 and lst2 directories one by one, and eliminate files with the same sha256 values; S506. Use the cosine similarity algorithm in lst1 to obtain the list lst3, traverse the lst3 directory, and compare the content of each file in lst3 with the file list of lst2 one by one. If the content similarity exceeds the threshold, it means that the host search is successful.

2. The open source component lineage analysis method according to claim 1, characterized in that: The step S3 specifically includes the following steps: S301. Use OWASP Dependency-Check, Nexpose, BlackDuck, and Snyk open source component fingerprinting tools to dynamically scan the application's source code, dependency lists, and binary files for open source software components. S302. Collect an inventory of all open source components used in the software, including versions, license information, potentially vulnerable libraries, frameworks, and auxiliary dependencies of the open source components; S303: Extract the specific code snippet, version number, and file name of the open source component, and generate a unique fingerprint and hash value using the extracted data and content of the open source component through a cryptographic hash function.

3. The open source component lineage analysis method according to claim 1, characterized in that: The step S4 specifically includes the following steps: S401. Generate a complete SBOM bill of materials through software component analysis and establish a software composition map; S402. Analyze the various source codes, modules, frameworks, and libraries used by developers to identify and inventory the components of the open source software and their composition and dependencies; S403. Accurately identify known security vulnerabilities and potential license authorization issues in the system to protect users from security vulnerabilities.

4. The open source component lineage analysis method according to claim 1, characterized in that: The step S5 includes prefabricated data and feature comparison. The prefabricated data is downloaded from the Maven official website as a jar file, and the jar file is decompressed. After decompression, feature data is generated and saved in the feature warehouse. The feature comparison first decompresses the jar package in the project, then generates feature data, and then compares it in the feature warehouse.

5. A system based on the open source component lineage analysis method according to any one of claims 1 to 4, characterized in that It includes an open source component automatic discovery module (1), a software component analysis module (2), an open source component fingerprint analysis module (3), an SBOM bill of materials module (4), and an open source component lineage analysis module (5); The open source component automatic discovery module (1) scans and detects the system source code and compiled deployment packages, automatically discovers the open source components referenced by the system through component fingerprint characteristics, dependency and association analysis, automatically discovers the framework used by the system through the referenced open source components and the characteristics of the open source components, and discovers the threats and risks of the framework through the association of threat intelligence; After discovering the open source components, the software component analysis module (2) accurately identifies the open source components used in the software and supports the identification of open source component versions developed in multiple languages; The open source component fingerprint analysis module (3) uses an open source component fingerprint recognition tool to perform dynamic scanning and collect a list of all open source components used in the software, extracting and utilizing information data in the open source components; The SBOM Bill of Materials module (4) generates a complete SBOM Bill of Materials, analyzes the information data used by developers, and accurately identifies known security vulnerabilities in the system; The open source component lineage analysis module (5) unpacks and searches the file to be traced, extracts attribute data based on the search results, obtains information about the manufacturer, component name and version number, searches for the corresponding original jar file in the feature warehouse based on the manufacturer, component name and version number information, analyzes the file to be traced and the corresponding original jar file to find the host, extracts feature values ​​through similarity analysis technology and saves them for comparison, and identifies the details of the tampered code in the jar package.

Images