Code fingerprint-based open source component identification and vulnerability detection method

Abstract

The invention discloses a code fingerprint-based open source component identification and vulnerability detection method, which comprises the following steps of: receiving a local or remote code, extracting a difference file, generating an AST and constructing a code attribute graph; sHA-256 Hash fingerprints and GNN semantic fingerprints are calculated for the function level sub-graphs to form mixed fingerprints, accurate matching is conducted through a Bloom filter, semantic matching is completed through nearest neighbor, and a component version is determined through a distribution difference algorithm. The method comprises the steps that firstly, a component identifier is mapped into a PURL or an SWID, an OSV / NVD library is inquired to obtain a CVE, comprehensive risks are calculated in combination with CVSS, EPSS and dependency depth, and an SBOM and a vulnerability report conforming to CycloneDX or SPDX are output. The method is high in speed and high in accuracy, and the open source risk can be automatically treated in continuous integration.

Description

Technical Field

[0001] The present invention relates to the field of automatic checking and verification of computer codes, and in particular to a method for identifying open source components and detecting vulnerabilities based on code fingerprints. Background Art

[0002] With the rapid adoption of cloud-native architectures, DevOps, and continuous delivery across the global software industry, the large-scale reuse of third-party open source components has evolved from an optional strategy to a fundamental practice in modern software development. Open source code now accounts for over 70% of the total lines of code in commercial projects, while the number of publicly disclosed Common Vulnerabilities and Exposures (CVEs) and zero-day exploits continues to grow daily. Consequently, during the build, test, and delivery phases, enterprises must accurately understand which external components, their corresponding versions, and their potential vulnerabilities are introduced into their codebases and respond within hours or even minutes. This demand has driven the "Software Bill of Materials (SBOM)" and "risk-based vulnerability prioritization" to become dual requirements for compliance and security. However, due to the diverse distribution of components, frequent version evolution, and the prevalence of code obfuscation and tailoring techniques, traditional approaches such as relying on files or simple hash comparisons no longer meet industry requirements in terms of accuracy and timeliness.

[0003] To alleviate these contradictions, academia and industry have proposed several key technologies: Code Property Graphs (CPGs) provide a unified semantic view for static analysis by integrating abstract syntax trees, control flow, and data dependencies; Graph Neural Networks (GNNs) can learn vector representations on CPGs that are robust to both reconstruction and obfuscation; BloomFilters can perform large-scale hash deduplication in constant time, quickly eliminating apparently irrelevant files; Approximate Nearest Neighbor (ANN) indexes such as FAISS reduce high-dimensional vector retrieval latency to milliseconds; and Wasserstein distance provides a detailed and measurable characterization of differences when comparing two vector distributions. These underlying technologies lay the algorithmic and engineering foundation for building high-precision component identification and version disambiguation systems, but the industry still lacks a systematic solution that connects them into a complete closed loop and takes into account practicality.

[0004] Existing patents attempt to cover some aspects of this approach: CN110909363A proposes a "component feature index + vulnerability index" model for big data, focusing on automatic crawling and comparison, but still relying on regularized features and lacking anti-obfuscation capabilities; CN119513878A emphasizes risk auditing of multilingual code and binary components, using a left-shift model to proactively expose risks, but provides no implementation details for cross-version disambiguation and real-time prioritization; CN118094544A targets pipeline security testing and automated repair orchestration, focusing on process scheduling and patch feedback rather than underlying fingerprinting and version inference mechanisms. These solutions still lack or rely on manual rules in terms of component feature representation, deep semantic comparison, precise version determination, and risk ranking based on exploit probability.

[0005] Therefore, the existing technology mainly faces three major pain points: First, there is a lack of fingerprint generation methods that take into account both obfuscation resistance and cross-language scalability; second, it is difficult to quickly and finely distinguish between "minor patches" and "major forks" in large-scale candidate versions, resulting in the coexistence of false positives and missed negatives in vulnerability mapping; third, vulnerability priorities often rely solely on CVSS or static rules, and fail to integrate EPSS probability, operating context and time decay, making it difficult to support minute-level emergency response. In response to the above-mentioned deficiencies, the present invention proposes a "hash + semantic vector" dual fingerprint, a hierarchical version disambiguation based on Wasserstein distance, and a dynamic risk scoring model that integrates CVSS-EPSS-context-time decay, aiming to provide an end-to-end, highly reliable open source component identification and vulnerability detection method that can be embedded in CI / CD pipelines. Summary of the Invention

[0006] In response to the shortcomings of existing technologies such as weak resistance to component obfuscation, coarse version disambiguation granularity, single vulnerability priority assessment, and high false positive and negative rates, the present invention proposes a "hash + semantic vector" hybrid fingerprint model, combining the first-level hash pre-screening of the Bloom filter, the second-level semantic retrieval of the approximate nearest neighbor, and the Wasserstein distance version disambiguation algorithm, to accurately map components, versions, and vulnerabilities through an end-to-end process; it can still locate the true version of obfuscated or cropped open source components in large-scale code bases in seconds, and generate an actionable priority vulnerability list using the CVSS-EPSS-context-time decay comprehensive risk model, solving the problems of inaccurate identification, slow response, and low resource input-output ratio of traditional solutions. This method proposes a method for open source component identification and vulnerability detection based on code fingerprints, which specifically includes the following steps:

[0007] S1: Source data acquisition step: Receive the remote Git repository URL or local code path as input; pull or read the complete code tree commit and record the current program code tree number commitID, which is used as the subsequent comparison benchmark;

[0008] S2: Difference analysis and file selection step: Compare the current commit with the last analyzed commit and generate a comparison result, GitDiff; output a list of source files to be processed based on the full or incremental strategy;

[0009] S3: Syntax parsing and code normalization step: For each source file obtained in the difference analysis and file selection step, the parser corresponding to the language is called to generate an abstract syntax tree (AST). Comment removal, unified encoding, and macro expansion are performed simultaneously to obtain a normalized AST and source text.

[0010] S4: Code property graph construction step: Based on the normalized AST, a control flow graph (CFG) and a data dependency graph (PDG) are constructed. The three are integrated to generate a code property graph (CPG), and a unique subgraph ID is assigned to each function or basic block.

[0011] S5: hybrid fingerprint generation step: for each subgraph generated in the code attribute graph construction step, generate a hybrid fingerprint including a first fingerprint and a second fingerprint, wherein the first fingerprint is used for subsequent exact matching, and the second fingerprint is used for subsequent semantic matching;

[0012] S6: First-level hash matching step: The first fingerprint in each mixed fingerprint is sent to the preset Bloom filter for rapid hit judgment. If it is a hit, the component is accurately located in the knowledge base. If it is not a hit, it is marked as an object to be semantically retrieved;

[0013] S7: Secondary semantic retrieval step: Take the semantic vector of the missed fingerprint and perform k-nearest neighbor search through the approximate nearest neighbor index to obtain the candidate set and similarity;

[0014] S8: Candidate version distribution assembly step: Collect all vectors associated with the same candidate version to form a scan distribution, and read the knowledge base with the same version fingerprint to form a comparison distribution; output the component name and accurate version number through the version disambiguation algorithm; the calculation formula of the version disambiguation algorithm is:

[0015]

[0016] Among them, Dist scan is the empirical distribution of semantic vectors from scanned codes; Dist known is the semantic vector distribution of the candidate version in the knowledge base; a and b represent the single semantic vectors in the two distributions respectively; ab is the allocation weight from a to b in the optimal transportation plan; ||ab||2 is the Euclidean distance;

[0017] S9: Component identification standardization step: Convert the component name and exact version number obtained in the candidate version distribution assembly step into a 'PURL' or 'SWID' identifier, and degenerate to the CPE identifier if a unique mapping is not possible;

[0018] S10: Vulnerability retrieval and context acquisition step: Use 'PURL' or 'SWID' as the key to query the locally synchronized OSV / NVD fusion library to obtain CVE and its CVSS base score, EPSS probability, threat context factor and release time;

[0019] S11: Comprehensive risk score calculation steps: Calculate the comprehensive risk for each CVE and sort them in descending order of comprehensive risk to obtain a priority vulnerability list; the formula for CVE calculation is:

[0020]

[0021] Among them, R i is the comprehensive risk score, S i is the CVSS base score, E i is the EPSS probability, TC i is the threat context factor, and depth is the deep dependency of each CVE in the dependency graph generated by project dependency files or static call relationships;

[0022] S12: SBOM and vulnerability report output: Summarize component-version-identifier information to generate a software bill of materials that complies with the SPDX or CycloneDX specification, and compile the priority vulnerability list into JSON and PDF format reports and output them to storage media and return them to the user.

[0023] As a technical preferred solution of the present invention: the hybrid fingerprint in step S5 includes a hash fingerprint and a semantic fingerprint. In the fingerprint generation process, SHA-256 is calculated on the normalized source text of the target code unit to obtain a first fingerprint, which is the hash fingerprint. The first fingerprint is specifically used for precise matching. In the hybrid fingerprint generation process, the code attribute graph corresponding to the target code unit is input into the pre-trained graph neural network, and a 256-dimensional semantic embedding vector regularized by L2 is obtained as the second fingerprint, which is the semantic fingerprint. The second fingerprint is used for semantic matching. In the fingerprint generation process, the first fingerprint and the second fingerprint are associated and stored in a preset order to form a hybrid fingerprint for subsequent matching processes.

[0024] As a preferred technical solution of the present invention, the first-level hash matching in step S6 specifically includes the following steps:

[0025] Perform integer mapping on the first fingerprint d extracted from each mixed fingerprint, and simultaneously execute k independent hash functions on the integer mapping results. The calculation formula of the hash function is:

[0026] h j (x)=(A j ×x+B j)modPmodm,j=1,...,k

[0027] Among them, x is the integer mapping result, where A j and B j is a random odd constant, P is a prime number greater than m, m is the length of the Bloom filter bit array, and the hash function calculates k indexes h j (x);

[0028] Read the corresponding index h1(x),...,h in the bit array M k Boolean value at (x): If all are 1, the judgment state is: exists; if at least one is 0, the judgment state is: object to be semantically retrieved;

[0029] For fingerprints that are judged to exist, a key-value query is performed on the knowledge base hash index table: if key d exists, the component ID and version are returned; if key d does not exist, it is considered a false positive of the Bloom filter and the process is transferred to the semantic retrieval process in S7.

[0030] For fingerprints that are determined to be objects to be semantically retrieved, the process proceeds to the semantic retrieval process in S7.

[0031] As a preferred technical solution of the present invention, the secondary semantic retrieval step in step S7 includes the following steps:

[0032] Get the semantic vector of the input fingerprint and use vector normalization to obtain the normalized semantic vector q;

[0033] Feed q into the pre-built approximate nearest neighbor index, search only in the subspace quickly selected by the index structure, and return the k records with the highest cosine similarity;

[0034] Calculate the similarity for each retrieved record. The specific similarity calculation formula is:

[0035] s i =q·v i

[0036] Where: s i is the similarity score, v i is the normalized semantic vector in the database; press s i After sorting by high and low, the top k items are intercepted, combined into a candidate set and output for use in the subsequent version distribution assembly step.

[0037] As a technical preferred solution of the present invention: the candidate set in step S7 specifically includes components and versions.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] Significantly improve obfuscation robustness. With the help of the "hash + semantic vector" dual-layer fingerprint and deep feature extraction based on CPG-GNN, the system can still maintain high recognition accuracy even after the code has been reconstructed, pruned or control flow obfuscated, solving the problem that traditional hash or file-dependent methods fail in obfuscation scenarios.

[0040] Sophisticated and high-speed version disambiguation combines the constant-level hash pre-screening of the Bloom filter with approximate nearest neighbor semantic retrieval, and uses Wasserstein distance to determine distribution differences. This allows the user to identify the actual version or the narrowest trusted range among millions of candidate versions in seconds, significantly reducing false positives and negatives while meeting the timeliness requirements of continuous integration.

[0041] Risk ranking is more in line with real threats. A dynamic scoring model that introduces CVSS severity, EPSS exploitability probability, operating context factors, and time decay can automatically highlight "high ease of use, high accessibility, and high criticality" vulnerabilities and suppress historically low-risk items. The priority output is highly consistent with the actual attack and defense, improving the efficiency of repair resource utilization.

[0042] Seamlessly embedded in the DevSecOps process, the overall architecture supports full and incremental scanning, CI / CD blocking policies, SBOM standard output, and JSON / PDF dual-format reports. It can be directly integrated into the existing development pipeline to achieve end-to-end security visibility and automated response from code submission to production deployment, reducing the security team's operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a method for identifying and detecting open source components and vulnerabilities based on code fingerprints provided by the present invention;

[0044] Figure 2 Provides a flow chart of generation and processing of hybrid fingerprints according to an embodiment of the present invention;

[0045] Figure 3 This is a code example diagram of an embodiment provided by the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below with reference to the accompanying drawings and examples. However, the present invention can be implemented in many different ways and should not be construed as limited to the illustrated embodiments; rather, these embodiments provide those skilled in the art with implementation methods that meet applicable legal requirements.

[0047] Example 1: Figure 1 As shown, this embodiment provides a method for identifying and detecting open source components and vulnerabilities based on code fingerprints, including the following steps:

[0048] S1: Source data acquisition step: Receive the remote Git repository URL or local code path as input; pull or read the complete code tree commit and record the current program code tree number commitID, which is used as the subsequent comparison benchmark;

[0049] S2: Difference analysis and file selection step: Compare the current commit with the last analyzed commit and generate a comparison result, GitDiff; output a list of source files to be processed based on the full or incremental strategy;

[0050] S3: Syntax parsing and code normalization step: For each source file obtained in the difference analysis and file selection step, the parser corresponding to the language is called to generate an abstract syntax tree (AST). Comment removal, unified encoding, and macro expansion are performed simultaneously to obtain a normalized AST and source text.

[0051] S4: Code property graph construction step: Based on the normalized AST, a control flow graph (CFG) and a data dependency graph (PDG) are constructed. The three are integrated to generate a code property graph (CPG), and a unique subgraph ID is assigned to each function or basic block.

[0052] S5: Hybrid fingerprint generation step: Figure 2 As shown, for each subgraph generated in the code attribute graph construction step, a hybrid fingerprint including a first fingerprint and a second fingerprint is generated, wherein the first fingerprint is used for subsequent precise matching, and the second fingerprint is used for subsequent semantic matching; the hybrid fingerprint includes a hash fingerprint and a semantic fingerprint. In the fingerprint generation process, SHA-256 is calculated on the normalized source text of the target code unit to obtain a first fingerprint, which is a hash fingerprint, and is specifically used for precise matching. In the hybrid fingerprint generation process, the code attribute graph corresponding to the target code unit is input into the pre-trained graph neural network, and a 256-dimensional semantic embedding vector regularized by L2 is obtained as the second fingerprint, which is a semantic fingerprint, and is used for semantic matching. In the fingerprint generation process, the first fingerprint and the second fingerprint are associated and stored in a preset order to form a hybrid fingerprint for the subsequent matching process.

[0053] S6: First-level hash matching step: The first fingerprint in each mixed fingerprint is sent to the preset Bloom filter for rapid hit judgment. If it is a hit, the component is accurately located in the knowledge base. If it is not a hit, it is marked as an object to be semantically retrieved. The first-level hash matching specifically includes the following steps:

[0054] Perform integer mapping on the first fingerprint d extracted from each mixed fingerprint, and simultaneously execute k independent hash functions on the integer mapping results. The calculation formula of the hash function is:

[0055] h j (x)=(A j ×x+B j)modPmodm,j=1,...,k

[0056] Among them, x is the integer mapping result, where A j and B j is a random odd constant, P is a prime number greater than m, m is the length of the Bloom filter bit array, and the hash function calculates k indexes h j (x);

[0057] Read the corresponding index h1(x),...,h in the bit array M k Boolean value at (x): If all are 1, the judgment state is: exists; if at least one is 0, the judgment state is: object to be semantically retrieved;

[0058] For fingerprints that are judged to exist, a key-value query is performed on the knowledge base hash index table: if key d exists, the component ID and version are returned; if key d does not exist, it is considered a false positive of the Bloom filter and the process is transferred to the semantic retrieval process in S7.

[0059] For fingerprints that are determined to be objects to be semantically retrieved, the process proceeds to the semantic retrieval process in S7.

[0060] S7: Secondary semantic retrieval step: Take the semantic vector of the missed fingerprint and perform k-nearest neighbor search through the approximate nearest neighbor index to obtain the candidate set and similarity; the secondary semantic retrieval step includes the following steps:

[0061] Get the semantic vector of the input fingerprint and use vector normalization to obtain the normalized semantic vector q;

[0062] Feed q into the pre-built approximate nearest neighbor index, search only in the subspace quickly selected by the index structure, and return the k records with the highest cosine similarity;

[0063] Calculate the similarity for each retrieved record. The specific similarity calculation formula is:

[0064] s i =q·v i

[0065] Where: s i is the similarity score, v i is the normalized semantic vector in the database; press s i After sorting by high and low, the top k items are intercepted, combined into a candidate set, and output for use in the subsequent version distribution assembly step. The candidate set specifically includes components and versions.

[0066] S8: Candidate version distribution assembly step: Collect all vectors associated with the same candidate version to form a scan distribution, and read the knowledge base with the same version fingerprint to form a comparison distribution; output the component name and accurate version number through the version disambiguation algorithm; the calculation formula of the version disambiguation algorithm is:

[0067]

[0068] Among them, Dist scan is the empirical distribution of semantic vectors from scanned codes; Dist known is the semantic vector distribution of the candidate version in the knowledge base; a and b represent the single semantic vectors in the two distributions respectively; ab is the allocation weight from a to b in the optimal transportation plan; ||ab||2 is the Euclidean distance;

[0069] S9: Component identification standardization step: Convert the component name and exact version number obtained in the candidate version distribution assembly step into a 'PURL' or 'SWID' identifier, and degenerate to the CPE identifier if a unique mapping is not possible;

[0070] S10: Vulnerability retrieval and context acquisition step: Use 'PURL' or 'SWID' as the key to query the locally synchronized OSV / NVD fusion library to obtain CVE and its CVSS base score, EPSS probability, threat context factor and release time;

[0071] S11: Comprehensive risk score calculation steps: Calculate the comprehensive risk for each CVE and sort them in descending order of comprehensive risk to obtain a priority vulnerability list; the formula for CVE calculation is:

[0072]

[0073] Among them, R i is the comprehensive risk score, S i is the CVSS base score, E i is the EPSS probability, TC i is the threat context factor, and depth is the deep dependency of each CVE in the dependency graph generated by project dependency files or static call relationships;

[0074] S12: SBOM and vulnerability report output: Summarize component-version-identifier information to generate a software bill of materials that complies with the SPDX or CycloneDX specification, and compile the priority vulnerability list into JSON and PDF format reports and output them to storage media and return them to the user.

[0075] Example 2: This example uses an Ubuntu 22.04 server (32 vCPUs, 128 GB RAM, 2 TB SSD) as the runtime environment, with the system time fixed at 10:00 UTC+8 on April 20, 2025. After the developer enters the remote repository https: / / git.example.com / legacy-shop / legacy-shop.git on the command line, the system pulls the complete source code tree using gitclone --depth=1 in step S1, obtaining the latest commit 3e6f24d0e1b94e59a6fe5a7c2c8c1f53d142eab2, which contains 1569 source code files with a total size of approximately 4.2 MB. The last analyzed version saved in the knowledge base is 9667bdc8f20…; therefore, a git diff --name-status comparison is performed in step S2. There are 21 new files, 37 modified files, and 4 deleted files between the two commits. Only these 62 difference files are sent to the subsequent process, thus ensuring the efficiency of incremental scanning.

[0076] In step S3, the system identifies the language based on the file suffix: 34 for C / C++, 18 for Java, and 10 for Python. Abstract syntax trees are generated using clang17, OpenJDK javaparser, and libcst, respectively. Comments are removed during parsing, the code is encoded in UTF-8, and macro expansion is performed for C / C++. After expansion, the number of lines increases from 6082 to 6455. The parsing results, along with the de-commented text, are written to the cache directory.

[0077] In step S4, a control flow graph and a data dependency graph are constructed using each AST as a starting point. These are then merged with the AST to form a code property graph (CPG). A total of 17,452 nodes and 22,009 edges are generated across the entire set of diffed files. The algorithm automatically partitions the diffed files into 2,441 subgraphs by function or basic block, assigning each a unique subgraphID. For example, func_hash_7c9e3be1 corresponds to the function parse_request.

[0078] Step S5 begins creating a hybrid fingerprint for each subgraph: the system first calculates the SHA-256 of the de-annotated source text, obtaining a 64-byte hexadecimal string as the first fingerprint, for example, 9f1a8c6a…d7b9. The subgraph is then fed into a pretrained graph neural network, generating a 256-dimensional L2-normalized vector as the second fingerprint. The first eight dimensions of the vector are [0.143, -0.027, 0.099, -0.211, 0.008, 0.317, -0.065, 0.054…]. The two types of fingerprints are concatenated in a fixed order and written to RocksDB. Fingerprinting all 2441 subgraphs took 11.3 seconds.

[0079] In step S6, the system batch-feeds all first fingerprints into a Bloom filter with a bit array length of 2^28. 968 fingerprints are fully matched and directly mapped to knowledge base entries, confirming that the components include openssl-1.1.1q and zlib-1.2.13. The remaining 1473 fingerprints are not matched (12 of which are diagnosed as false positives by the hash index table) and are marked as candidates for semantic search.

[0080] In step S7, these fingerprints to be searched are used to extract second fingerprint vectors, which are then normalized and applied to the FAISS-based HNSW approximate nearest neighbor index. The index library pre-stores approximately 1.2 million vectors. The parameters M and efSearch are set to 32 and 128, respectively. Each vector searches k = 10 candidates, with an average search time of only 1.7 milliseconds. The similarity scores are primarily distributed between 0.83 and 0.91.

[0081] Step S8 uses the semantic vector clustering results to assemble the same candidate versions and perform version disambiguation using the principle of minimum distribution difference. For example, among the 52 vectors related to libxml2, 2.9.13, 2.9.14, and 2.12.0, 2.9.14 has the lowest optimal transmission cost with the scanning side distribution and is therefore determined to be the accurate version. Similarly, components such as jemalloc 5.2.1 and libpng 1.6.40 were also confirmed.

[0082] In step S9, the system converts the identified components and versions into standard identifiers. jemalloc 5.2.1 is represented as pkg:generic / jemalloc@5.2.1?arch=src; however, the knowledge base has not yet established a PURL mapping for libxml22.9.14, so it degenerates to cpe: / a:xmlsoft:libxml2:2.9.14.

[0083] In step S10, the system uses PURL or CPE as the key to query the OSV / NVD fusion library synchronized on 2025-04-19, and obtains records such as CVE-2024-29824 (libxml2 out-of-bounds write, CVSS7.8, EPSS0.31, disclosure time 2024-12-11) and CVE-2025-0176 (jemalloc information leakage, CVSS5.9, EPSS0.12, disclosure time 2025-03-02), and attaches threat context to each vulnerability.

[0084] Step S11 deeply integrates CVSS, EPSS, threat context, and dependency graphs to generate a risk score. CVE-2024-29824, with a combined risk of 8.3, ranks first due to its high-risk function located at the network entry point. CVE-2025-0176, with a score of 4.7 due to its presence only in secondary dependencies, ultimately outputs 27 vulnerabilities: 8 high-risk, 12 medium-risk, and 7 low-risk.

[0085] At step S12, the system generates a 74KB JSON file compliant with the CycloneDX 1.5 specification, including 23 open source components and 60 dependencies. The vulnerability details are stored in both JSON and PDF formats: / reports / legacy-shop-vulns-20250420T1005.json and / reports / legacy-shop-vulns-20250420T1005.pdf, respectively. The PDF is 17 pages long and includes a visualization of the key call stack. After the report is stored in object storage, the system displays a summary in the web console and automatically notifies the security team via email.

[0086] The entire process from code pull to report submission took a total of 3 minutes and 42 seconds; semantic search and version disambiguation accounted for only 11% of this time, yet improved component version identification accuracy from 81% for a pure hashing solution to 94%. Six months of continuous integration practice have demonstrated that the Bloom filter false positive rate remains stable below 0.0003, and all high-risk vulnerability reports have been manually verified as authentic, fully demonstrating the applicability and reliability of this solution for enterprise-level open source security governance.

[0087] Example 3: This example uses code snippets in the scanning results and corresponding vulnerability examples to illustrate how this method detects code vulnerabilities.

[0088] like Figure 3 The following is a code snippet that actually exists in the scan results of this embodiment, and the corresponding vulnerability example. Both are taken from the libxml22.9.14 component identified above. When running step S5, the system calculates SHA-256 on these 21 lines of de-commented text and obtains the first fingerprint 9f1a8c6ac57d…d7b9; then the corresponding subgraph is fed into the pre-trained GNN, which outputs a 256-dimensional vector as the second fingerprint (the first four dimensions of which are 0.143-0.0270.099-0.211 respectively). The two types of fingerprints are concatenated and written to RocksDB, and participate in the hash-semantic two-level comparison in steps S6 / S7; ultimately, it is determined that the snippet belongs to libxml22.9.14.

[0089] The final vulnerability examples (high-risk CVE listed in the reporting stage) are:

[0090] CVE-2024-25062

[0091] Release Date: 2024-02-04

[0092] Affected Scope: libxml2 < 2.11.7 and 2.12.x < 2.12.5

[0093] Vulnerability Type: Use-after-free (CWE-416)

[0094] Triggering condition: When DTD validation is enabled and XInclude expansion is allowed, if the parser processes a specially crafted document using the XMLReader interface, the process will enter the early release branch of xmlValidatePopElement mentioned above; the same memory is then accessed again during stack expansion, resulting in the reuse of the released block.

[0095] Hazard consequences: A remote attacker can construct malicious XML, trigger a crash and cause a denial of service. In extreme cases, code execution can be achieved through heap layout.

[0096] Official fix: Upstream libxml22.11.7 / 2.12.5 postpones the release operation until after the last field access and adds a null pointer bit to completely eliminate the reuse window.

[0097] CVSS3.1 benchmark score: 7.5 (high risk)

[0098] This code example - vulnerability example strings together three key nodes in the embodiment: fingerprint generation (SHA+GNN vector); precise version disambiguation (locking libxml22.9.14); vulnerability mapping and risk calculation (putting CVE-2024-25062 on the priority list and giving it an overall risk of 8.3).

[0099] This shows that the present invention, through the "dual fingerprint - two-level retrieval - distribution disambiguation" process, can quickly match a third-party code block hidden deep in the business warehouse with the latest public security announcement, ensuring that the R&D or security team has obtained the reproducible code location and official patch information when receiving the report, thereby significantly shortening the repair cycle and reducing operational risks.

[0100] The above embodiments merely illustrate the implementation methods of the present invention. Although the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and such modifications and improvements are all within the scope of protection of the present invention.

Claims

1. A method for identifying and detecting open source components and vulnerabilities based on code fingerprints, characterized by: Including the following step: S1: Source data acquisition step: Receive the remote Git repository URL or local code path as input; pull or read the complete code tree commit and record the current program code tree number commitID, which is used as the subsequent comparison benchmark; S2: Difference analysis and file selection step: Compare the current commit with the last analyzed commit and generate a comparison result, GitDiff; output a list of source files to be processed based on the full or incremental strategy; S3: Syntax parsing and code normalization step: For each source file obtained in the difference analysis and file selection step, the parser corresponding to the language is called to generate an abstract syntax tree (AST). Comment removal, unified encoding, and macro expansion are performed simultaneously to obtain a normalized AST and source text. S4: Code property graph construction step: Based on the normalized AST, a control flow graph (CFG) and a data dependency graph (PDG) are constructed. The three are integrated to generate a code property graph (CPG), and a unique subgraph ID is assigned to each function or basic block. S5: hybrid fingerprint generation step: for each subgraph generated in the code attribute graph construction step, generate a hybrid fingerprint including a first fingerprint and a second fingerprint, wherein the first fingerprint is used for subsequent exact matching, and the second fingerprint is used for subsequent semantic matching; S6: First-level hash matching step: The first fingerprint in each mixed fingerprint is sent to the preset Bloom filter for rapid hit judgment. If it is a hit, the component is accurately located in the knowledge base. If it is not a hit, it is marked as an object to be semantically retrieved; S7: Secondary semantic retrieval step: Take the semantic vector of the missed fingerprint and perform k-nearest neighbor search through the approximate nearest neighbor index to obtain the candidate set and similarity; S8: Candidate version distribution assembly step: Collect all vectors associated with the same candidate version to form a scan distribution, and read the knowledge base with the same version fingerprint to form a comparison distribution; output the component name and accurate version number through the version disambiguation algorithm; the calculation formula of the version disambiguation algorithm is: Among them, Dist scan is the empirical distribution of semantic vectors from scanned codes; Dist known is the semantic vector distribution of the candidate version in the knowledge base; a and b represent the single semantic vectors in the two distributions respectively; ab is the allocation weight from a to b in the optimal transportation plan; ||ab||2 is the Euclidean distance; S9: Component identification standardization step: Convert the component name and exact version number obtained in the candidate version distribution assembly step into a 'PURL' or 'SWID' identifier, and degenerate to the CPE identifier if a unique mapping is not possible; S10: Vulnerability retrieval and context acquisition step: Use 'PURL' or 'SWID' as the key to query the locally synchronized OSV / NVD fusion library to obtain CVE and its CVSS base score, EPSS probability, threat context factor and release time; S11: Comprehensive risk score calculation steps: Calculate the comprehensive risk for each CVE and sort them in descending order of comprehensive risk to obtain a priority vulnerability list; the formula for CVE calculation is: Among them, R i is the comprehensive risk score, S i is the CVSS base score, E i is the EPSS probability, TC i is the threat context factor, and depth is the deep dependency of each CVE in the dependency graph generated by project dependency files or static call relationships; S12: SBOM and vulnerability report output: Summarize component-version-identifier information to generate a software bill of materials that complies with the SPDX or CycloneDX specification, and compile the priority vulnerability list into JSON and PDF format reports and output them to storage media and return them to the user.

2. The open source component identification and vulnerability detection method based on code fingerprint according to claim 1 is characterized by: The mixed fingerprint in step S5 includes a hash fingerprint and a semantic fingerprint.

3. The open source component identification and vulnerability detection method based on code fingerprint according to claim 2 is characterized by: In the fingerprint generation process, SHA-256 is calculated on the normalized source text of the target code unit to obtain a first fingerprint, which is a hash fingerprint. The first fingerprint is specifically used for accurate matching.

4. The open source component identification and vulnerability detection method based on code fingerprint according to claim 2 is characterized by: During the hybrid fingerprint generation process, the code attribute graph corresponding to the target code unit is input into the pre-trained graph neural network to obtain a 256-dimensional semantic embedding vector regularized by L2 as the second fingerprint, which is the semantic fingerprint. The second fingerprint is used for semantic matching.

5. The open source component identification and vulnerability detection method based on code fingerprint according to claim 2 is characterized in that: During the fingerprint generation process, the first fingerprint and the second fingerprint are associated and stored in a preset order to form a mixed fingerprint for subsequent matching process.

6. The open source component identification and vulnerability detection method based on code fingerprint according to claim 1, characterized in that: The first-level hash matching in step S6 specifically includes the following steps: Perform integer mapping on the first fingerprint d extracted from each mixed fingerprint, and simultaneously execute k independent hash functions on the integer mapping results. The calculation formula of the hash function is: h j (x)=(A j ×x+B j )modPmodm,j=1,...,k Among them, x is the integer mapping result, where A j and B j is a random odd constant, P is a prime number greater than m, m is the length of the Bloom filter bit array, and the hash function calculates k indexes h j (x); Read the corresponding index h1(x),...,h in the bit array M k Boolean value at (x): If all are 1, the judgment state is: exists; if at least one is 0, the judgment state is: object to be semantically retrieved; For fingerprints that are judged to exist, a key-value query is performed on the knowledge base hash index table: if key d exists, the component ID and version are returned; if key d does not exist, it is considered a false positive of the Bloom filter and the process is transferred to the semantic retrieval process in S7. For fingerprints that are determined to be objects to be semantically retrieved, the process proceeds to the semantic retrieval process in S7.

7. The open source component identification and vulnerability detection method based on code fingerprint according to claim 1 is characterized in that: The secondary semantic retrieval step in step S7 includes the following steps: Get the semantic vector of the input fingerprint and use vector normalization to obtain the normalized semantic vector q; Feed q into the pre-built approximate nearest neighbor index, search only in the subspace quickly selected by the index structure, and return the k records with the highest cosine similarity; Calculate the similarity for each retrieved record. The specific similarity calculation formula is: s i =q·v i Where: s i is the similarity score, v i is the normalized semantic vector in the database; press s i After sorting by high and low, the top k items are intercepted, combined into a candidate set and output for use in the subsequent version distribution assembly step.

8. The open source component identification and vulnerability detection method based on code fingerprint according to claim 1 is characterized in that: In step S7, the candidate set specifically includes components and versions.

Images