Automatic generation system for third-party component fingerprint database

Abstract

The invention discloses an automatic generation system for a third-party component fingerprint database, and belongs to the technical field of software development. Comprising a dependency analysis module which is used for analyzing a construction configuration file of a project and obtaining third-party dependency information of the project; the method fingerprint generation module is used for analyzing and extracting a third-party method in a project code through a static code and generating a unique method fingerprint; the JAR packet analysis module is used for analyzing a class file in the JAR packet, extracting a method signature, calculating a fingerprint and matching the fingerprint with a fingerprint in a database; the data storage and query module is used for storing fingerprint information and providing a query interface for other systems to use; according to the method, automatic analysis is achieved, and repeated work is reduced; the analysis time is shortened; the pertinence is high, and the matching efficiency is improved; when the version is upgraded, a new calling method fingerprint can still be automatically extracted, so that the maintenance difficulty is reduced; a huge third-party component fingerprint database does not need to be maintained, and occupation of storage resources is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software development, and particularly relates to a third-party component fingerprint library automatic generation system. BACKGROUND

[0002] Software Composition Analysis (SCA) tools can help developers and enterprises identify, manage and reduce the use risks of open source software and third-party components, and ensure software security, compliance and quality. In the SCA tool, there are usually two methods for third-party component detection; the first method is to scan the source code and dependency configuration (pom.xml file, etc.), and the second method is to scan the compiled version package. The detection principle of the second method is: first, calculate the fingerprint information of the method used in the version package, and then compare it with the data in the fingerprint library, to accurately identify the used third-party components and versions; therefore, the fingerprint library of the third-party component plays a key role, which is the core of identifying and managing open source components and related risks. The fingerprint library is a database containing a large number of open source components, which is constructed by collecting file hash values, metadata (such as component name, version) and associated vulnerability information of the components. Through the fingerprint library, the tool can not only efficiently identify the components, but also provide detailed information about the vulnerabilities and compliance risks related to these components.

[0003] In the SCA tool, the generation and update of the component fingerprint database face significant challenges: this needs to scan and collect the component source code and metadata of various open source code repositories (such as GitHub, GitLab) and package management platforms (such as Maven, Gradle), and generate unique fingerprints through file hash, block hash, syntax feature analysis, etc. This process is a huge workload, involving millions of open source component multi-version data, with high update frequency, wide coverage, and the need to deal with the complexity of different languages and ecosystems.

[0004] In addition, the existing generation and update mechanism also has the following disadvantages: Large resource consumption: scanning and processing full data requires a large amount of computing resources, storage space and bandwidth, which is costly for small and medium-sized organizations or tool providers; Unbalanced coverage: mainstream and high-frequency components are usually given priority, while cold or long-tail components may be ignored, leading to the incompleteness of the database; High maintenance cost: the fingerprint library needs long-term continuous update, verification and deduplication, and the maintenance workload increases rapidly with the growth of the number of components.

[0005] These shortcomings make the generation and maintenance of third-party component fingerprint database a bottleneck for the development of SCA tools, and also indicate the need for more efficient collection mechanisms to meet the needs of modern software supply chains. SUMMARY

[0006] The purpose of the present application is to provide a method for SCA tools to automatically generate a third-party component fingerprint database during source code scanning, and to automatically generate a weight index for the component fingerprint data based on user confirmation of the scanning results. By automatically generating component fingerprint data, the maintenance cost of the fingerprint database can be greatly reduced. By using the weight index, the accuracy of the scanning results can be greatly improved.

[0007] To achieve the above purpose, the present application adopts the following technical solutions: A third-party component fingerprint library automatic generation system, comprising: A dependency parsing module: parsing the project's build configuration file to obtain the project's third-party dependency information; A method fingerprint generation module: extracting third-party methods from the project code through static code analysis and generating unique method fingerprints; A JAR package analysis module: parsing the class files in the JAR package, extracting method signatures and calculating fingerprints, and matching them with the fingerprints in the database; A data storage and query module: storing fingerprint information and providing a query interface for other systems to use.

[0008] In some embodiments, the dependency parsing module generates fingerprint information only involving the actual third-party methods called by the current function code by analyzing these actual calling methods; Comprising the following steps: A1, loading and parsing pom.xml file Load pom.xml and extract the following key information: Dependency list: groupId, artifactId, version Dependency scope: scope A2, locate the dependent JAR file A3, record dependency information Cache all dependency information for subsequent method source judgment.

[0009] In some embodiments, the method fingerprint generation module extracts third-party methods and generates fingerprints: Comprising the following steps: B1, extract the methods actually called in the source code; B2, match third-party dependencies: according to the dependency information extracted from pom.xml, judge whether the method called in the source code belongs to the third-party dependency library; B3, extract method information of third-party dependencies; B4, generate method fingerprints; B5, store results: save the method fingerprint information and its MD5 value in the database.

[0010] In some embodiments, in step B1, extracting the actual method called in the source code: using JavaParser or Spoon to perform static analysis on the source code file, extracting the syntax tree of all method calls; for each method call, parse its complete path, including package name, class name, method name; In addition, filter internal methods and JDK native methods: according to the package name of the method path, exclude the package name belonging to the current project and the package name belonging to JDK.

[0011] In some embodiments, in step B3, extracting method information of third-party dependencies: B31, locate the class and JAR file of the target method: according to the complete path of the method, determine its belonging class and package, and extract the target class from the corresponding dependency JAR file; B32, parse the class file and extract the target method information: parse the.class file; find the method definition in the target class, and extract: method name, return type, parameter quantity, parameter type.

[0012] In some embodiments, in step B4, generating method fingerprints: B41, generate fingerprints according to the format of package name.class name.method name: return type: (parameter type 1, parameter type 2,...); B42, calculate the MD5 value of the method fingerprint.

[0013] In some embodiments, in step B5, store the results: Including: dependencies table and method_fingerprints table; The dependencies table includes fields: id, group_id, artifact_id, version; The method_fingerprints table includes id: dependency_id, fingerprint, method, count; Each record in the dependencies table represents a third-party dependency; each record in the method_fingerprints table represents a specific method fingerprint of a certain dependency; the dependencies table and the method_fingerprints table are in a one-to-many relationship through dependency_id.

[0014] In some embodiments, the JAR package analysis module determines the third-party components used by the target JAR package through the fingerprint information; The method comprises the following steps: C1, parsing a JAR package The target JAR package is parsed to obtain all.class files in the JAR package; C2, extracting method signatures in the target JAR package Each.class file is read to extract method signature information, including: method name, return value type, number of input parameters, and input parameter type; C3, generating method fingerprints According to each method signature information extracted, MD5 hash values are generated as method fingerprints using the same rules as in the previous step; C4, matching with data in the fingerprint library.

[0015] In some embodiments, in step C4, matching with data in the fingerprint library: C41, traversing method fingerprints in the target JAR package: For each generated fingerprint, a matching item is found in the fingerprint library; C42, counting the matching results: If a matching item is found, the corresponding component information is recorded.

[0016] In some embodiments, it further comprises C43, processing multiple matching cases: if a method fingerprint matches multiple components, the components are sorted according to the number of calls, and the component with the most calls is selected.

[0017] Compared with the prior art, the present application provides a third-party component fingerprint library automatic generation system, which has the following beneficial effects.

[0018] 1. The present application has high efficiency in automatic generation of methods, small fingerprint library, fast query, and low maintenance cost after version upgrade.

[0019] 2. The present application has automatic analysis, reduces repetitive work, shortens analysis time, is highly targeted, improves matching efficiency, can still automatically extract new call method fingerprints after version upgrade, reduces maintenance difficulty, does not need to maintain a large third-party component fingerprint library, and reduces storage resource occupation.

[0020] Additional advantages, objects, and features of the application will be apparent from the following description; BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 Process diagram for third-party component fingerprint library generation.

[0022] Figure 2 Flowchart for obtaining third-party component information used in jar package through fingerprint library. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the application will be described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all.

[0024] Reference Figures 1-2 A third-party component fingerprint library automatic generation system, the architecture design is divided into four modules, each module is responsible for different functions, and the modules interact with each other through interfaces or databases.

[0025] The four modules are as follows: Dependency analysis module: parse the project's build configuration file (such as pom.xml or gradle.build) to obtain the project's third-party dependency information; Method fingerprint generation module: extract third-party methods in project code through static code analysis and generate unique method fingerprints; JAR package analysis module: parse class files in JAR packages, extract method signatures and calculate fingerprints, and match them with fingerprints in the database; Data storage and query module: store fingerprint information and provide query interface for other systems.

[0026] Correspondingly, the technology stack: SpringBoot: as the core framework, provides dependency injection, configuration management, RESTful API, etc. JavaParser: used for static code analysis to extract method signatures; ASM / ByteBuddy: used for bytecode analysis to extract method signatures in JAR packages; MySQL: used for storing fingerprint information and dependency data; Maven / Gradle: used for parsing project build configuration files; MD5: Hash algorithm used to generate method fingerprints.

[0027] Instance effects of the system: The jar package scanning result report content using the automatically generated third-party component library is as follows.

[0028] { "jarName": "example.jar", "usedComponents": [ { {"groupld": "org.springframework", "artifactld": "spring-core", "version": "5.3.20" }, { "groupld": "com.google.guava", "artifactld": "guava", "version": "31.0.1-jre" }, { "groupld": "com.fasterxml.jackson.core", "artifactld": "jackson-databind", "version": "2.12.3" }, { "groupld": "org.apache.logging.log4j", "artifactld": "log4j-core", "version": "2.14.1" }, { "groupld": "org.junit.jupiter", "artifactld": "junit-jupiter-api", "version": "5.7.0" } ] }

[0029] The dependency analysis module generates fingerprint information only involving the actual third-party methods called by the current function code (Java source code), rather than all methods of the entire dependency library.

[0030] The method comprises the following steps: A1, loading and parsing the pom.xml file: Load pom.xml using Maven's parsing tool or XML parsing library, and extract the following key information: Dependency list: group ID, artifact ID, version Dependency scope: scope (such as compile, runtime, ignore test and provided).

[0031] Example: <dependency> <groupid>org.apache.commons< / groupid> <artifactid>commons-lang3< / artifactid> <version> 3.12.0< / version> < / dependency>

[0032] A2, locate the JAR file of the dependency: Find the corresponding JAR file of the dependency through the Maven local repository path; The default path of the local repository is ~ / .m2 / repository The dependency path rule is: ~ / .m2 / repository / <groupid> / <artifactid> / <version> / <artifactid> - <version>.jar.

[0033] Note that: If the JAR file does not exist locally, it can be downloaded via Maven.

[0034] A3, record dependency information: Cache all dependency information (JAR path and package name, class name, method name) for subsequent method source judgment.

[0035] The method fingerprint generation module extracts third-party methods and generates fingerprints, including the following steps.

[0036] B1, extract the method actually called in the source code Parse the Java source code file: use JavaParser or Spoon to perform static analysis on the source code file, extract the syntax tree (AST) of all method calls; Extract the complete path of the method call: for each method call, parse its complete path (package name, class name, method name).

[0037] org.apache.commons.lang3.Stringutils.isEmpty.

[0038] Filter internal methods and JDK native methods: According to the package name of the method path, the following cases are excluded: The package name belongs to the current project (internal method); the package name belongs to JDK (such as java.*, javax.*, sun.*).

[0039] B2, match third-party dependencies: According to the dependency information (package name, class name, method name) extracted from pom.xml, judge whether the method called in the source code belongs to the third-party dependency library.

[0040] B3, extract the method information of the third-party dependency.

[0041] B31, locate the class and JAR file to which the target method belongs: According to the complete path of the method, determine the class and package to which it belongs; extract the target class from the corresponding dependent JAR file.

[0042] B32, parse the class file and extract the target method information: Use ASM or javassist to parse the.class file; find the method definition in the target class, and extract the following information: method name, return value type, parameter quantity, and parameter type.

[0043] B4, generate method fingerprints.

[0044] B41, generate the fingerprint in the format of package.name.method: return type: (param type1, param type2,...), for example: org.apache.commons.lang3.StringUtils.isEmpty:boolean:(String).

[0045] B42, calculate the MD5 value of the method fingerprint: Convert the fingerprint string to a UTF-8 byte array; use the MessageDigest tool in Java to calculate the MD5 hash value.

[0046] B5, store the result: save the method fingerprint information and its MD5 value in the database.

[0047] Among them, the database table design dependencies table (dependency information table), method_fingerprints table (method fingerprint table).

[0048] dependencies table (dependency information table), as follows. Field Name Data Type Description id BIGINT (primary key, auto-increment) Unique identifier group_id VARCHAR (255) Dependent organization ID artifact_id VARCHAR (255) Dependent module ID version VARCHAR (50) Dependent version number

[0049] method_fingerprints table (method fingerprint table), as follows. id BIGINT (primary key, auto-increment) Unique identifier dependency_id BIGINT (foreign key) Foreign key association to dependencies.id fingerprint CHAR (32) MD5 value of method fingerprint method TEXT Method complete description count INT Number of times the method is called

[0050] Relationship between tables: Each record in the dependencies table represents a third-party dependency; Each record in the method_fingerprints table represents a specific method fingerprint of a dependency; dependencies table and method_fingerprints table through dependency_id establish a one-to-many relationship.

[0051] Database insertion example.

[0052] Dependency insertion example (dependencies table).

[0053] INSERT INTO dependencies (group_id, artifact_id, version) VALUES ('org.apache.commons', 'commons-lang3', '3.12.0'), ('com.google.guava', 'guava', '31.1-jre').

[0054] Method fingerprints insertion example (method_fingerprints table).

[0055] INSERT INTO method_fingerprints (dependency_id, fingerprint, method, count) VALUES (1, '5d41402abc4b2a76b9719d911017c592', 'org.apache.commons.lang3.StringUtils.isEmpty:boolean:(String)', 3), (1, '8e58aab82738b38d5d243a84a2a8341d', 'org.apache.commons.lang3.StringUtils.isNotEmpty:boolean:(String)', 2), (2, 'd41d8cd98f0b204e9800998ecf8427e', 'com.google.common.base.Strings, nullToEmpty:String:(String)', 5).

[0056] The JAR package analysis module determines the third-party components used by the target JAR package through the fingerprint information; The goal is to analyze a target JAR package through the generated fingerprint information (including groupId, artifactId, version, and method fingerprints, etc.), so as to determine whether the methods contained in the target JAR package match the method fingerprints of known third-party components.

[0057] The method comprises the following steps.

[0058] C1, parsing the JAR package Use the java.util.jar package of Java or similar libraries (such as ASM or ByteBuddy) to parse the target JAR package; obtain all.class files in the JAR package.

[0059] C2, extracting method signatures in the target JAR package Use the ASM library to read each.class file and extract method signature information, including: method name, return value type, number of input parameters, and input parameter type.

[0060] C3, Generating fingerprints Based on the extracted method signature information, an MD5 hash value is generated as the method fingerprint using the same rules as in the previous step.

[0061] C4. Match the data with the fingerprint database.

[0062] C41. Traverse the method fingerprints in the target JAR package: For each generated fingerprint, search for a match in the fingerprint database.

[0063] C42. Statistics on matching results: If a match is found, record the corresponding component information (groupId, artifactId, version).

[0064] C43. Handling multiple matching cases: If a method fingerprint matches multiple components, sort them by the number of times they are called and select the component with the most calls.

[0065] Example of querying the method with the highest fingerprint weight (most frequently called): SELECT d.group_id, d.artifact_id, d.version, mf.fingerprint, MAX(mf.count))ASmax_calls FROM dependenciesd JOIN method_fingerprintsmf ON d.Id=mf.dependency_id MHERE mf.fingerprint='specified fingerprint value' GROUPBY mf.fingerprint, d.group_id, d.artifact_id, d.version ORDERBY max_callsDESC LIMIT1.

[0066] Conduct tests for comparison.

[0067] The test design is as follows.

[0068] Test data.

[0069] 1. 5 business modules, each module depends on 10 third-party components, totaling 50 components (some components are repeated in multiple modules).

[0070] 2. Code repository package; third-party component list, as follows. ordinal Component name groupId artifactId Version 1 SpringFramework org.springframework spring-context 5.3.10 2 Hibernate org.hibernate hibernate-core 5.6.15.Final 3 Jackson com.fasterxml.jackson.core jackson-databind 2.13.2 4 ApacheCommonsIO commons-io commons-io 2.11.0 5 ApacheHttpClient org.apache.httpcomponents httpclient 4.5.13 6 SLF4J org.slf4j slf4j-api 1.7.32 7 Logback ch.qos.logback logback-classic 1.2.10 8 Guava com.google.guava guava 31.0.1-jre 9 JUnit org.junit.jupiter junit-jupiter-api 5.8.2 10 MyBatis org.mybatis mybatis 3.5.7

[0071] 3. Number of component methods: each component contains about 1000 methods on average.

[0072] 4. Total number of methods: 50 components x 1000 methods = 50000 methods.

[0073] Experimental method: Traditional method: scan all component libraries to generate full method fingerprints; The automatic generation method of the present application: static analysis of business code calls to generate actual method fingerprints.

[0074] Comparison index: Fingerprint generation time; fingerprint library size; subsequent query efficiency.

[0075] Test results: Metrics Traditional method Automatically generated method Total number of dependent components 50 50 Total number of methods 50000 50000 Actual number of called methods N / A 2500 Fingerprint generation time 1, 200 seconds (about 20 minutes) 50 seconds (about 1 minute) Fingerprint library size 50000 × 100 bytes = 5 MB 2500 × 100 bytes = 250 KB Average single query time 20 milliseconds 5 milliseconds Re-generation cost after version upgrade Full re-generation, time-consuming 20 minutes Only analyze new calls, time-consuming 10 seconds Results.

[0076] 1. Generation efficiency: The efficiency of the automatic generation method is 24 times that of the traditional method (50 seconds vs. 1200 seconds).

[0077] 2. Fingerprint library size: The fingerprint library size of the automatic generation method is 1 / 20 of that of the traditional method (250KB vs. 5MB).

[0078] 3. Query efficiency: The query time of the automatic generation method is 4 times faster than that of the traditional method (5 milliseconds vs. 20 milliseconds).

[0079] 4. Maintenance cost after version upgrade: The traditional method needs full re-scanning, which takes a long time; the automatic generation method only needs to analyze new calls, and the maintenance cost is low.

[0080] In the present application, by performing static analysis on the actual business code (including the current project code and its dependent third-party components), the fingerprints of the called third-party component methods are automatically generated, without the need to scan the entire third-party component library separately. Compared with the traditional method (manually scanning components or generating component fingerprints one by one), the present application has at least the following significant advantages.

[0081] 1) Automatic analysis, reduce repetitive work Traditional method: need to manually scan each third-party component, time-consuming and laborious, especially for large project dependent components, the number of versions is large and iterative, generating full component fingerprints is extremely tedious; The automatic generation method of the application: only analyze the methods called in the actual code, directly generate fingerprints, without processing the uncalled part, reduce invalid work.

[0082] 2) Shorten the analysis time The traditional method of generating fingerprints of the entire component library may need to scan millions of lines of code, consuming a lot of time and resources; the automatic generation of the application only analyzes the methods called in the business code, the analysis range is greatly reduced, which can significantly reduce the time and calculation cost.

[0083] 3) Strong pertinence, improve matching efficiency The automatically generated method fingerprint is directly derived from the actual call, with high pertinence, the generated fingerprint library is smaller and more concise, and the subsequent query is more efficient; after reducing the size of the fingerprint library, the matching complexity is reduced, thereby improving the matching performance at runtime.

[0084] 4) Adaptability of version upgrade The full-scan component fingerprint in the traditional method needs to be regenerated after version change, with high maintenance cost; the automatic generation method of the application can automatically extract new call method fingerprints by analyzing the code call relationship in real time, even if the component is upgraded, thereby reducing the maintenance difficulty.

[0085] 5) Efficient use of resources No need to maintain a large third-party component fingerprint library, reducing storage resource occupation; directly generate fingerprints through existing code analysis, avoiding repeated scanning.

[0086] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can make equivalent replacement or change within the technical range disclosed by the application according to the technical solution and inventive concept of the application, which should be covered within the protection scope of the application.

[0087] Although the embodiments of the application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the application.< / version> < / artifactid> < / version> < / artifactid> < / groupid>

Claims

1. A third party component fingerprint library auto-generation system, characterized in that, Comprise: Dependency resolution module: parse the project's build configuration file, get the project's third-party dependency information; Method fingerprint generation module: extract third-party methods in project code through static code analysis, and generate unique method fingerprints; JAR package analysis module: parse class files in JAR package, extract method signatures and calculate fingerprints, and match with fingerprints in database; Data storage and query module: store fingerprint information and provide query interface for other systems.

2. The third party component fingerprint library automatic generation system of claim 1, wherein, The dependency resolution module generates fingerprint information only involving the actual calling methods by analyzing the third-party methods actually called by the current function code; Comprise the following steps: A1, load and parse pom.xml file Load pom.xml and extract the following key information: Dependency list: group id, artifact id, version Dependency scope: scope A2, locate dependent JAR files A3, record dependency information Cache all dependency information for subsequent method source judgment.

3. The third party component fingerprint repository automatic generation system of claim 1, wherein, The method fingerprint generation module extracts third-party methods and generates fingerprints: Comprise the following steps: B1, extract the methods actually called in the source code; B2, match third-party dependencies: according to the dependency information extracted from pom.xml, judge whether the method called in the source code belongs to the third-party dependency library; B3, extract method information of third-party dependencies; B4, generate method fingerprints; B5, store results: save method fingerprint information and its MD5 value in the database.

4. The third party component fingerprint library automatic generation system of claim 3, wherein, In step B1, extract the methods actually called in the source code: use JavaParser or Spoon to perform static analysis on the source code file to extract the syntax tree of all method calls; for each method call, parse its complete path, including package name, class name, and method name; In addition, filter internal methods and JDK native methods: exclude package names belonging to the current project and package names belonging to JDK according to the package name of the method path.

5. The third party component fingerprint library automatic generation system of claim 3, wherein, In step B3, extract method information of third-party dependencies: B31, locate the class and JAR file to which the target method belongs: determine the class and package to which the method belongs according to the complete path of the method, and extract the target class from the corresponding dependent JAR file; B32, parse class file and extract target method information: parse.class file; Find the method definition in the target class and extract: method name, return type, number of input parameters, and input parameter types.

6. The third party component fingerprint library automatic generation system of claim 3, wherein, In step B4, generate method fingerprints: B41, generate fingerprints in the format of package name.class name.method name:return type:(input parameter type1, input parameter type2,...); B42, calculate the MD5 value of the method fingerprint.

7. The third party component fingerprint library automatic generation system of claim 3, wherein, In step B5, store results: Include dependencies table and method_fingerprints table; The dependencies table includes fields: id, group_id, artifact_id, version; The method_fingerprints table includes id: dependency_id, fingerprint, method, count; Each record in the dependencies table represents a third-party dependency; each record in the method_fingerprints table represents a specific method fingerprint of a certain dependency; the dependencies table and the method_fingerprints table establish a one-to-many relationship through dependency_id.

8. The third party component fingerprint repository automatic generation system of claim 1, wherein, The JAR package analysis module judges the third-party components used by the target JAR package through the fingerprint information; The method comprises the following steps: C1, parsing the JAR package Parse the target JAR package to obtain all.class files in the JAR package; C2, extracting method signatures in the target JAR package Read each.class file and extract method signature information, including: method name, return value type, number of input parameters, and input parameter type; C3, generating method fingerprints According to the extracted method signature information of each method, use the same rule as the previous step to generate an MD5 hash value as the method fingerprint; C4, matching with data in the fingerprint library.

9. The third party component fingerprint library automatic generation system of claim 8, wherein, In step C4, matching with data in the fingerprint library: C41, traversing the method fingerprints in the target JAR package: For each generated fingerprint, find matching items from the fingerprint library; C42, statistics matching results: If a matching item is found, record the corresponding component information.

10. The third party component fingerprint library automatic generation system of claim 9, wherein, Further comprising: C43, handling multiple matching cases: if a method fingerprint matches multiple components, sort them according to the number of calls and select the component with the most calls.

Images