A software component analysis optimization method based on hash cache

Abstract

The application belongs to the technical field of electric digital data processing, and particularly relates to a software component analysis optimization method based on a hash cache, which comprises the following steps: S1, first scanning of multi-dimensional features and fusion of dynamic behavior features to calculate and store component hashes: when performing first SCA scanning on software, dynamic behavior features of components during running are collected through a lightweight sandbox environment, and basic features, content features and dependency features of software packages and components are extracted; a hash value is introduced as a unique identifier of the software package and the component, the hash value of the component and the corresponding detection result are calculated and stored during the first scanning; during subsequent scanning, the hash value of the component is preferentially verified, if the hash value already exists in the cache library, the historical detection result is directly reused, and it is not necessary to repeatedly perform a full analysis process, under the premise of ensuring analysis accuracy, repeated calculation is reduced through a cache reuse mechanism, and the improvement of SCA analysis efficiency and the reduction of resource consumption are realized.

Description

Technical Field

[0001] This invention belongs to the field of electronic digital data processing technology, specifically a software component analysis and optimization method based on hash caching. Background Technology

[0002] Hash caching is a caching system implemented with a hash table as the underlying data structure. It combines the "fast access" feature of hash tables with the core goal of caching to "temporarily store hot data and reduce backend load". It is a classic technical solution to solve the problem of "fast access to high-frequency data".

[0003] In the application of SCA (Software Composition Analysis) technology, traditional analysis methods require a full scan analysis of software packages and components to identify composition information; however, in real-world scenarios, software projects often have a large number of duplicate software packages or components (such as different projects introducing the same version of open-source components, or components not changing when the same project is scanned multiple times).

[0004] Traditional methods still perform complete scanning and analysis on these repetitive components, resulting in a significant waste of computing resources and low scanning efficiency. This problem is particularly prominent in large-scale software projects or high-frequency scanning scenarios, and cannot meet the needs of efficient software supply chain quality control.

[0005] Therefore, this invention provides a software component analysis optimization method based on hash caching. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0007] The technical solution adopted by this invention to solve its technical problem is: a software component analysis optimization method based on hash caching, which includes the following steps:

[0008] S1. Initial Scanning of Multi-Dimensional Features and Fusion of Dynamic Behavioral Features to Calculate Storage Component Hash: During the initial SCA scan of the software, dynamic behavioral features of components during runtime are collected through a lightweight sandbox environment. The basic features, content features, and dependency features of the software package and each component are extracted. After calculating the hash of each dimension feature, dynamic weights are fused, and the hash value of the dynamic behavioral feature is incorporated with dynamic weights to obtain its globally unique hash value. At the same time, a complete component analysis process is executed to obtain the component information of the software package or component, and the global hash value and its associated data are stored in the hash cache library.

[0009] S2. Subsequent scanning and verification hash calculation: When performing an SCA scan on the software, first extract the multi-dimensional features of the software package and components to be detected and integrate the dynamic behavior features to calculate the hash value. Then, perform a feature variability threshold judgment. If it is determined to be a minor non-core change, the cached result is directly reused; otherwise, a Bloom filter is used for pre-verification, and then the hash cache is queried to see if there is a record corresponding to the hash value.

[0010] S3. Reuse of identical hash results and incremental analysis: If a matching hash value record exists in the hash cache, the component analysis result corresponding to that record is directly called without repeated scanning; if the hash value does not hit the cache but a historical version record of the component exists, incremental analysis is performed through the cross-version component information inheritance mechanism; if the hash value does not hit the cache and there is no historical version record, a full scan analysis is performed on the component to generate new component information and store the new data in the cache.

[0011] S4. Cache library management and updates: Regularly maintain the hash cache library, including cleaning up expired records, updating dynamic information related to components, and adjusting the cleanup priority based on business association weights to ensure the cache retention of core components and the validity and storage efficiency of cached data.

[0012] Preferably, when extracting features of software packages and component files in S1, the collection and fusion of dynamic behavior features of component runtime are added. The specific operation includes: when extracting features of software packages and component files, running the component in a lightweight sandbox environment, collecting dynamic behavior data within a preset time period, and using a time-series hash algorithm to calculate the hash value of dynamic behavior features.

[0013] The hash of the storage component calculated in the first scan of S1 consists of the following four characteristics:

[0014] Multi-dimensional feature extraction: Extracting three layers of features from software packages and components: basic features, content features, and dependency features;

[0015] Layered hash calculation: After calculating the hash of the three-layer features separately, they are merged into a globally unique hash through dynamic weights. The dynamic weight factor is dynamically adjusted according to the stability factors such as component functional characteristics and security risks to accurately reflect the true attributes of the component and effectively distinguish components with consistent static characteristics but different dynamic behaviors with security risks, thereby reducing the false judgment rate of hash caching.

[0016] Full component analysis: Execute the complete SCA process to obtain component information including: component unique identifier, version number, license type, vulnerability list, and open source community activity.

[0017] Related storage: Construct a dual-index structure for the hash cache library: use the global hash as the primary key, and simultaneously establish an auxiliary index (component identifier - hash list) to support reverse lookup of hash records of different versions of the same component;

[0018] The global hash value is calculated using the following formula:

[0019] ;

[0020] in The hash value of the i-th layer feature. Basic characteristics Content characteristics Dependency features;

[0021] For dynamic weights, ( ,and , );

[0022] To merge hash functions;

[0023] The weights are dynamic behavioral features, ranging from 0.1 to 0.2, and satisfy the following conditions: At the same time, maintain the original Weighted sorting;

[0024] The storage model for component information follows the following formula:

[0025] , , , , , ;

[0026] in, Used as a unique identifier for components;

[0027] Version number;

[0028] License type;

[0029] List of vulnerabilities;

[0030] Rate activity level;

[0031] For cache creation time;

[0032] By incorporating the dynamic behavior of components at runtime into the feature system, dynamic data is collected through a lightweight sandbox, and dynamic feature hashes are generated by combining time-series hashing algorithms. Weights are dynamically allocated according to component type and incorporated into the global hash calculation. At the same time, dynamic behavior description tags are associated and stored. This mechanism, through the fusion of dynamic and static features, makes the hash value more accurately reflect the true attributes of the component and reduces the false judgment rate.

[0033] Preferably, the basic features include, but are not limited to, file size, modification time metadata, and standard directory structure of identifiable components in the file path; the content features include: extracting key fragment hashes from the code file, skipping comments and blank lines, and calculating hashes only for function definitions, class inheritance relationships, and core logic; the dependency features include: parsing the component's dependency description file, generating a dependency graph hash, and calculating hashes after sorting the dependencies by name.

[0034] The dynamic weights are dynamically adjusted based on basic features, content features, and dependency features, with content features having the highest weight, dependency features having the second highest weight, and basic features having the lowest weight.

[0035] The full component analysis requires a complete SCA analysis to be performed on the first scan.

[0036] Preferably, before S2 calculates the hash value of the component to be detected and queries the cache, a feature variability threshold judgment step is added. This aims to solve the problem of cache misses caused by minor non-core feature changes such as annotation modifications and format adjustments. The specific operations include:

[0037] Feature variability calculation: the three-layer basic features of the component to be detected ( , , ) and the historical features with the highest similarity in the cache library ( , , The characteristic variability is calculated using the following formula. :

[0038] ;

[0039] in For the first Similarity of layer features Consistent with the dynamic weights of S1;

[0040] Variation threshold determination: Preset feature variation threshold ;

[0041] like The change was determined to be a "minor non-core change". The component analysis results in the cache were directly reused without any further hash lookup.

[0042] like Perform Bloom filter pre-verification and hash cache lookup according to the original S2 process;

[0043] Variance record storage: This stores the variability data from each calculation. The judgment result is stored in the "Feature Mutation Log" field of the cache. This is used as a historical data reference for subsequent S4 cache cleanup.

[0044] In S2, the subsequent scanning calculation of the verification hash introduces an incremental hash generation mechanism and a Bloom filter pre-verification. The Bloom filter is only used as a fast filtering tool, and its judgment result needs to be combined with the feature variability threshold judgment result for the final decision.

[0045] Incremental feature extraction: For subsequent software packages to be detected, prioritize extracting features that are most likely to differ from the first scan. If the feature hash of this layer matches the historical records, then verify features of other layers to reduce computational load.

[0046] Bloom filter pre-validation: Before querying the hash cache, a Bloom filter is used to quickly determine whether a hash value "might exist", filtering out more than 99% of invalid queries and reducing database I / O pressure.

[0047] Sharded index query: The hash cache library shards according to hash value prefixes. When querying, the target shard is located directly, and then an exact match is made using the primary key;

[0048] The pre-validation function of a Bloom filter is calculated according to the following formula:

[0049] ;

[0050] The formula for locating the fragment index follows the following calculation method:

[0051] ;

[0052] in Before retrieving the hash value The bit is used as a shard identifier to implement the mapping from hash to shard;

[0053] Introducing feature variability The concept involves calculating the similarity between the component to be detected and historical cached features using Hamming distance, and dynamically setting a threshold based on the component's security level. This mechanism directly reuses cached results for minor, non-core changes with variability below a threshold, eliminating the need for a full hash lookup. In contrast, traditional methods trigger cache lookups or full scans for any feature differences, leading to non-core changes such as comment modifications and format adjustments being misjudged as requiring re-analysis and wasting computational resources. This mechanism precisely filters out invalid changes through variability quantification and threshold determination.

[0054] Preferably, the incremental feature extraction is performed using a (probability of change) priority: first, the content feature hash is calculated; when the content feature hash matches a record in the cache, the dependent feature hash is calculated; finally, the basic feature hash is calculated.

[0055] When the Bloom filter first stores the hash, it will Multiple bits in the filter are mapped to multiple hash functions and marked as (1); during subsequent queries, if the corresponding bits in the filter are not all (1), then... It is definitely not in the cache; if all are (1), then it may exist.

[0056] Preferably, when S3 handles hash misses or partial matches, a new cross-version component information inheritance mechanism is added, analyzing only the core dimensions involved in version changes. Specific operations include:

[0057] Version association query: When the hash of the component to be detected does not match, the unique identifier of the component is used for the query. Query the historical version records of this component in the cache library and extract the component information of the historical versions;

[0058] Inheritable information determination: Based on the component version change type, determine the inheritable component information:

[0059] Patch version / minor version: Inherited license type Open source community activity Only re-analyze the version number Vulnerability List ;

[0060] Major version: Inherit only the component's unique identifier Re-analyze the version number License type Vulnerability List Open source community activity ;

[0061] Inheritance Information Marker: A new "Inheritance Source Field" is added to the newly generated component information. With "Reanalyze Fields" To ensure the traceability of ingredient information

[0062] In S3, the reuse of hash results with the same hash and the incremental analysis are distinguished between (complete matching) and (partial matching) scenarios;

[0063] Exact match reuse: when Completely consistent with the cache library records, and The results of component analysis can be directly reused, but it is necessary to verify in real time whether the dynamic information is updated.

[0064] Incremental analysis of partial matching: when Similar to a record in the cache library Then, only the component dimensions corresponding to the differential features are re-analyzed;

[0065] Full analysis of the new component: If a hash miss occurs, the full analysis process of S1 is executed, and the newly generated... Component information is stored in a cache library;

[0066] The formula for calculating hash similarity follows the following method:

[0067] ;

[0068] in, For indicator functions , The dynamic weights are the same as those in S1. 1 indicates a perfect match. Indicates partial matching;

[0069] The formula for the incremental analysis range is calculated according to the following formula:

[0070] ;

[0071] That is, analysis is performed only on the component dimensions corresponding to the mismatched feature layers;

[0072] By analyzing version number semantics, the inheritable component information of a component across version iterations is determined. Only the core dimensions involved in version changes are re-analyzed, and the inheritance source is marked to ensure traceability. Compared with traditional methods that perform full analysis on all new versions of components and ignore the characteristic of "reusable historical information for non-destructive changes" in version iterations, which leads to redundant analysis, this mechanism improves the analysis efficiency in version iteration scenarios while ensuring the accuracy of information.

[0073] Preferably, in the case of the perfect match reuse, when The cached records are completely consistent ( The component analysis results (component ID, version, license) can be directly reused, but the "dynamic information" needs to be verified in real time.

[0074] Vulnerability Information: Call the vulnerability database API to check for any newly added CVEs;

[0075] License Status: Verify whether the license has been updated by the open-source community;

[0076] In the incremental analysis of the partial matching, when (Partial feature matching) only analyzes the component dimensions corresponding to the differential features to avoid full rescanning.

[0077] Preferably, when S4 performs cache maintenance and cleanup, it adds the calculation and fusion of business-related weights to solve the problem of "accidental cleanup of core business components due to relying solely on access frequency and vulnerability risk". Specific operations include:

[0078] Business Relevance Weight Calculation: By connecting to the component dependency graph of the enterprise's business systems, the "business relevance" of each component is obtained. Components that the core business modules depend on Components that the auxiliary module depends on Components that redundant modules depend on .

[0079] The S4 cache library management and update adopts a dynamic expiration mechanism and a two-factor cleanup strategy based on popularity and value.

[0080] Dynamic expiration time calculation adjusts the expiration time dynamically based on the component's access frequency and vulnerability risk level.

[0081] The incremental update mechanism periodically updates the dynamic information of components in the cache library, and connects to the external database through the unique identifier ID of the component, without the need to recalculate the hash.

[0082] The two-factor cleanup strategy ensures that when the cache library capacity reaches a threshold, it prioritizes cleaning up the records with the lowest access frequency and vulnerability risk level, while retaining a certain proportion of historical version hashes for backtracking analysis.

[0083] The dynamic expiration time formula is calculated according to the following formula:

[0084] ;

[0085] in Based on the expiration time;

[0086] Access frequency;

[0087] Vulnerability risk level;

[0088] , As weight;

[0089] The cleanup priority is calculated using the following formula:

[0090] ;

[0091] in, , To adjust the coefficient, Records with lower values ​​are cleaned up first;

[0092] This refers to the business relevance weighting coefficient; The lower the value, the higher the cleanup priority;

[0093] Core component protection mechanism: Set a "cleanup protection threshold" for the core business components. If the component Even if the cache library capacity reaches the threshold, it will not be included in the cleanup scope; it will only be included if the component has no access records for 180 consecutive days and the vulnerability risk level is [not specified]. Only then will the protection be lifted and cleaning allowed;

[0094] By connecting to the enterprise's business dependency graph, the "business relevance" of components is calculated. This mechanism incorporates cache cleanup priority formulas and sets "cleanup protection thresholds" for core components to avoid unnecessary cleanup. Compared to traditional cache cleanup that relies solely on access frequency and vulnerability risk, which may mistakenly delete caches of core business components such as payment and authentication, leading to the need for re-analysis in subsequent scans and impacting business continuity, this mechanism improves the cache retention rate of core components by weighting based on business value, while optimizing resource release for non-core components, thus balancing cache efficiency and business stability.

[0095] Preferably, the dynamic expiration time adopts differential retention, where different components have different cache values, through... Dynamically adjust retention time:

[0096] High-frequency access components: The access frequency is close to 1. Set to 90 days;

[0097] High-risk components: The risk level is close to 1. It can be set to 180 days;

[0098] Low-frequency and low-risk components: and All are low. It can be set to 30 days;

[0099] The static information in the incremental update database does not need to be updated frequently, but the dynamic information needs to be updated periodically.

[0100] Batch query external databases using component unique identifiers to update vulnerability lists and license status;

[0101] Only the changed fields are updated; there is no need to recalculate the hash or perform a full overwrite.

[0102] During the two-factor cleanup process, when the cache library capacity reaches its limit, it is sorted by priority. Clean up low-value records:

[0103] Prioritize cleaning The record with the lowest value;

[0104] Retain historical version hash pool: For core components, even if the current version has been updated, the hash records of the last 3 versions are still retained for backtracking analysis.

[0105] The beneficial effects of this invention are as follows:

[0106] 1. The software component analysis optimization method based on hash caching described in this invention introduces hash values ​​as unique identifiers for software packages and components. During the first scan, the hash values ​​of components and corresponding detection results are calculated and stored. In subsequent scans, the component hash values ​​are verified first. If the hash values ​​already exist in the cache, the historical detection results are directly reused without repeating the full analysis process. Under the premise of ensuring the accuracy of the analysis, the cache reuse mechanism reduces redundant calculations, thereby improving the efficiency of SCA analysis and reducing resource consumption.

[0107] 2. The software component analysis optimization method based on hash caching described in this invention breaks through the limitations of traditional single-file hashing, achieves component uniqueness identification through multi-dimensional feature fusion hashing, enhances the sensitivity of hashing to component mutations by combining dynamic weight factors, and establishes a structured storage model for component information. Attached Figure Description

[0108] The invention will now be further described with reference to the accompanying drawings.

[0109] Figure 1 This is a flowchart of the multi-dimensional feature hash calculation and associated storage process for the first scan in this invention;

[0110] Figure 2 This is a flowchart of the incremental hash verification and fast query process for subsequent scans in this invention;

[0111] Figure 3 This is a flowchart of the result reuse and intelligent incremental analysis based on hash matching in this invention;

[0112] Figure 4 This is a flowchart of the lifecycle-based adaptive management process for the cache library in this invention. Detailed Implementation

[0113] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0114] like Figures 1 to 4As shown in the embodiment of the present invention, a software component analysis optimization method based on hash caching includes the following steps:

[0115] S1. Initial Scanning of Multi-Dimensional Features and Fusion of Dynamic Behavioral Features to Calculate Storage Component Hash: During the initial SCA scan of the software, dynamic behavioral features of components during runtime are collected through a lightweight sandbox environment. The basic features, content features, and dependency features of the software package and each component are extracted. After calculating the hash of each dimension feature, dynamic weights are fused, and the hash value of the dynamic behavioral feature is incorporated with dynamic weights to obtain its globally unique hash value. At the same time, a complete component analysis process is executed to obtain the component information of the software package or component, and the global hash value and its associated data are stored in the hash cache library.

[0116] S2. Subsequent scanning and verification hash calculation: When performing an SCA scan on the software, first extract the multi-dimensional features of the software package and components to be detected and integrate the dynamic behavior features to calculate the hash value. Then, perform a feature variability threshold judgment. If it is determined to be a minor non-core change, the cached result is directly reused; otherwise, a Bloom filter is used for pre-verification, and then the hash cache is queried to see if there is a record corresponding to the hash value.

[0117] S3. Reuse of identical hash results and incremental analysis: If a matching hash value record exists in the hash cache, the component analysis result corresponding to that record is directly called without repeated scanning; if the hash value does not hit the cache but a historical version record of the component exists, incremental analysis is performed through the cross-version component information inheritance mechanism; if the hash value does not hit the cache and there is no historical version record, a full scan analysis is performed on the component to generate new component information and store the new data in the cache.

[0118] S4. Cache library management and updates: Regularly maintain the hash cache library, including cleaning up expired records, updating dynamic information related to components, and adjusting the cleanup priority based on business association weights to ensure the cache retention of core components and the validity and storage efficiency of cached data.

[0119] like Figures 1 to 4 As shown, when extracting features of software packages and component files in S1, the collection and fusion of dynamic behavior features of component runtime are added. Specific operations include:

[0120] When extracting features from software packages and component files, the collection and fusion of dynamic behavior features at runtime are added: Dynamic behavior feature collection: The component is run in a lightweight sandbox environment (such as a Docker lightweight container) to collect dynamic behavior data within a preset time period, including: the frequency of component calls to system interfaces (such as file I / O, number of network requests), peak and fluctuation range of memory usage, and thread creation and destruction patterns; a sliding window method is used to sample dynamic data to avoid data redundancy, and the sampling window duration is set to 5-10 minutes (which can be adaptively adjusted according to the component type).

[0121] Dynamic feature hash calculation: The collected dynamic behavior data is standardized (e.g., the interface call frequency is normalized to the [0,1] interval), and the dynamic behavior feature hash value is calculated using a time-series hash algorithm (e.g., TSH-128). .

[0122] The hash of the storage component calculated in the first scan of S1 consists of the following four characteristics:

[0123] Multi-dimensional feature extraction: Extracting three layers of features from software packages and components: basic features: file metadata, filenames and path structures; content features: code snippet hashes, feature segments of binary files; dependency features: component dependency graph;

[0124] Hierarchical hash calculation: After calculating the hash of each of the three layers of features, the hash is merged into a globally unique hash through dynamic weights; the weight factors are dynamically adjusted according to the stability of the features.

[0125] Full component analysis: Perform the complete SCA process to obtain component information including: component unique identifier, version number, license type, vulnerability list, open source community activity, and structure it;

[0126] Related storage: Construct a dual-index structure for the hash cache library: using a global hash as the primary key, and simultaneously creating a secondary index (component identifier - hash list), supporting reverse lookups of hash records for different versions of the same component; add a dynamic behavior feature description field. (e.g., "high network request components", "low memory fluctuation components") and , These fields are stored together in a hash cache library to provide a basis for risk prediction in subsequent scans.

[0127] Global hash value calculation formula: Referencing dynamic behavior feature hash:

[0128] ;

[0129] in The hash value of the i-th layer feature. Basic characteristics Content characteristics Dependency features;

[0130] For dynamic weights, ( ,and , );

[0131] To merge hash functions;

[0132] The weights are dynamic behavioral features, ranging from 0.1 to 0.2 (the upper limit for core business components and the lower limit for utility components), and satisfy the following conditions: At the same time, maintain the original Weighted sorting

[0133] Formula for ingredient information storage model:

[0134] , , , , , ;

[0135] in, Used as a unique identifier for components;

[0136] Version number;

[0137] License type;

[0138] List of vulnerabilities;

[0139] Rate activity level;

[0140] For cache creation time;

[0141] This mechanism incorporates component runtime dynamic behavior (system interface call frequency, memory fluctuations, thread patterns, etc.) into a feature system. Dynamic data is collected through a lightweight sandbox and combined with a time-series hashing algorithm (TSH-128) to generate dynamic feature hashes. Weights are dynamically assigned according to component type and integrated into the global hash calculation, while simultaneously storing dynamic behavior description tags. Traditional methods, relying solely on static features (basic, content, dependencies), cannot distinguish components with consistent static features but differing dynamic behaviors that pose security risks (e.g., malicious components masquerading as static features but frequently initiating abnormal network requests), leading to hash cache misjudgments. This mechanism, through dynamic-static feature fusion, makes hash values ​​more accurately reflect the true attributes of components, reducing the false positive rate.

[0142] Breaking through the limitations of traditional single-file hashing, this method achieves unique component identification through multi-dimensional feature fusion hashing, enhances the sensitivity of hashing to component mutations by combining dynamic weight factors, and establishes a structured storage model for component information.

[0143] like Figures 1 to 4 As shown, the basic features include not only metadata such as file size and modification time, but also the standard directory structure of identifiable components in the file path; the content features extract hashes of key segments from the code file, skipping comments and blank lines, and only calculating hashes for function definitions, class inheritance relationships, and core logic; the dependency features parse the component's dependency description file, generate a dependency graph hash, and calculate the hash after sorting the dependencies by name; avoiding feature differences caused by different dependency declaration orders.

[0144] The dynamic weights are dynamically adjusted based on basic features, content features, and dependency features. Among them, content features have the highest weight because modifications to the code and binary content directly reflect changes in component functionality and are the core identifier of component uniqueness. Dependency features have the next highest weight because changes in dependency relationships may introduce new sub-components or vulnerabilities. Basic features have the lowest weight because filenames and modification times are easily tampered with manually and have poor stability.

[0145] The full component analysis requires a complete SCA analysis during the initial scan: unique identifiers are determined by matching the component's feature library; license file is parsed to determine the license type and verify for license conflicts; external vulnerability databases are accessed to obtain associated CVE vulnerabilities and their risk levels; and the activity level of the open-source community is statistically analyzed to assess the component's maintenance status.

[0146] like Figures 1 to 4 As shown, before S2 calculates the hash value of the component to be detected and queries the cache, a new feature variability threshold judgment step is added to solve the problem of "cache misses caused by minor non-core feature changes" and reduce unnecessary full scans. Specific operations include:

[0147] Feature variability calculation: the three-layer basic features of the component to be detected ( , , ) and the historical features with the highest similarity in the cache library ( , , The characteristic variability is calculated using the following formula. :

[0148] ;

[0149] in For the first Similarity of layer features (calculated using Hamming distance, with a value range of [0,1], where 1 is when they are completely identical and 0 is when they are completely different). Consistent with the dynamic weights of S1;

[0150] Variation threshold determination: Preset feature variation threshold (Default 0.05, can be adjusted according to component security requirements: high security level components) ordinary components );

[0151] like If the change is determined to be a "minor non-core change", the component analysis results in the cache can be directly used without further querying the global hash.

[0152] like Perform Bloom filter pre-verification and hash cache lookup according to the original S2 process;

[0153] Variance record storage: This stores the variability data from each calculation. The judgment result is stored in the "Feature Mutation Log" field of the cache. This is used as a historical data reference for subsequent S4 cache cleanup.

[0154] In S2, the subsequent scan calculation of the verification hash introduces an incremental hash generation mechanism and a Bloom filter pre-verification.

[0155] Incremental feature extraction: For subsequent software packages to be detected, prioritize extracting features that are most likely to differ from the first scan. If the feature hash of this layer matches the historical records, then verify features of other layers to reduce computational load.

[0156] Bloom filter pre-validation: Before querying the hash cache, a Bloom filter is used to quickly determine whether a hash value "might exist", filtering out more than 99% of invalid queries and reducing database I / O pressure.

[0157] Sharded index query: The hash cache library shards according to hash value prefixes. When querying, the target shard is located directly, and then an exact match is made using the primary key;

[0158] Bloom filter pre-validation function:

[0159] ;

[0160] Fragmented index location formula:

[0161] ;

[0162] in Before retrieving the hash value The bit is used as a shard identifier to implement the mapping from hash to shard;

[0163] Introducing feature variability The concept involves calculating the similarity between the component to be detected and historical cached features using Hamming distance, and dynamically setting a threshold based on the component's security level. This mechanism directly reuses cached results for minor, non-core changes with variability below a threshold, eliminating the need for a full hash lookup. In contrast, traditional methods trigger cache queries or full scans for any feature difference, leading to non-core changes like comment modifications and formatting adjustments being misjudged as requiring re-analysis and wasting computational resources. This mechanism precisely filters invalid changes through variability quantification and threshold determination, reducing unnecessary scans by 30%-50% and improving efficiency.

[0164] An incremental hash generation mechanism and Bloom filter pre-verification are introduced to reduce the cost of repeated calculations, while the query efficiency of a large-scale cache library is improved through hash sharding index.

[0165] like Figures 1 to 4 As shown, the incremental feature extraction adopts a (change probability) priority: the content feature hash is calculated first (because content changes have the greatest impact on component functionality and the change probability is lower than that of the basic features); when the content feature hash matches a record in the cache, the dependency feature hash is calculated (to verify whether the dependency has changed); finally, the basic feature hash is calculated (verified only in the first two layers of matching to avoid misjudgment due to minor changes in metadata).

[0166] When the Bloom filter first stores the hash, it will Multiple bits in the filter are mapped to multiple hash functions and marked as (1); during subsequent queries, if the corresponding bits in the filter are not all (1), then... It is definitely not in the cache (perform a full analysis directly); if all are (1), it may exist (further database verification is required);

[0167] Through incremental computation and efficient query mechanisms, the cost of repeated feature extraction and hash calculation is reduced, allowing subsequent scans to focus only on "potentially changing parts".

[0168] like Figures 1 to 4 As shown, when S3 handles hash misses or partial matches, it adds a cross-version component information inheritance mechanism to solve the problem of "redundant analysis caused by component version iteration." It only analyzes the core dimensions involved in version changes. Specific operations include:

[0169] Version association query: When the hash of the component to be detected does not match, the unique identifier of the component is used for the query. Query the historical version records of this component in the cache library and extract the component information of the historical versions;

[0170] Inheritable information determination: Based on the component version change type (through version number semantic analysis: such as...) Upgrade to Upgrade to "Patch Version" Upgrade to "minor version" For the "major version"), determine the inheritable component information:

[0171] Patch version / minor version: Inherited license type Open source community activity Only re-analyze the version number Vulnerability List ;

[0172] Major version: Inherit only the component's unique identifier Re-analyze the version number License type Vulnerability List Open source community activity ;

[0173] Inheritance Information Marker: A new "Inheritance Source Field" is added to the newly generated component information. (Record inherited historical version number) and "Reanalysis Field" (Record the dimensions of this reanalysis) to ensure the traceability of component information;

[0174] In S3, the reuse of hash results with the same hash and the incremental analysis are distinguished between (complete matching) and (partial matching) scenarios;

[0175] Exact match reuse: when Completely consistent with the cache library records, and The component analysis results can be directly reused, but it is necessary to verify in real time whether dynamic information (such as the vulnerability list) is updated (by querying the external vulnerability database API).

[0176] Incremental analysis of partial matching: when Similar to a record in the cache library (If the content feature hash matches but the dependency feature changes), then only the component dimensions corresponding to the difference feature are re-analyzed (e.g., only the dependency relationship and associated vulnerability are updated).

[0177] Full analysis of the new component: when hash misses ( If so, execute the full analysis process of S1, and then process the newly generated data. Component information is stored in a cache library;

[0178] Hash similarity calculation formula:

[0179] ;

[0180] in, For indicator functions , The dynamic weights are the same as those in S1. 1 indicates a perfect match. Indicates partial matching;

[0181] Incremental analysis range formula:

[0182] ;

[0183] That is, analysis is only performed on the component dimensions corresponding to the mismatched feature layers (e.g., if the dependent features do not match, only the dependency relationship is analyzed);

[0184] By using version number semantic analysis (distinguishing between patch versions, minor versions, and major versions), this mechanism determines the inheritable component information (such as license type and community activity) across version iterations. It only re-analyzes the core dimensions involved in version changes (such as vulnerability lists) and marks the inheritance source to ensure traceability. Compared to traditional methods that perform full analysis on all new versions of components, ignoring the characteristic of "reusable historical information for non-destructive changes" in version iterations (such as patch versions that only fix vulnerabilities while the license type remains unchanged), leading to redundant analysis, this mechanism improves analysis efficiency in version iteration scenarios by 40%-60% while ensuring information accuracy.

[0185] like Figures 1 to 4 As shown, during the complete matching multiplexing, when The cached records are completely consistent ( The component analysis results (component ID, version, license) can be directly reused, but the "dynamic information" needs to be verified in real time.

[0186] Vulnerability Information: Call the vulnerability database API to check for any newly added CVEs;

[0187] License Status: Verify whether the license has been updated by the open-source community;

[0188] In the incremental analysis of the partial matching, when (Partial feature matching) Only analyzes the component dimensions corresponding to the differential features to avoid full rescanning:

[0189] If only the basic features do not match: reuse all component information and only update the file path related records.

[0190] If only the feature mismatch is considered: simply re-analyze the dependencies, and new vulnerabilities will emerge.

[0191] If only the content features do not match: re-analyze the component's functional fingerprint and vulnerability information, but retain the original license and other unchanged information;

[0192] When the hash is not hit ( This indicates that it is a new component being scanned for the first time, requiring the execution of the full analysis process in S1. After the analysis is completed, it will be... The ingredient information is stored in a cache for reuse in the next scan.

[0193] By using hash matching results, we can differentiate between three scenarios: "completely identical", "partially changed", and "completely new component", and process them accordingly to reduce repetitive work while ensuring the accuracy of the analysis results.

[0194] like Figures 1 to 4 As shown, when S4 performs cache maintenance and cleanup, it adds the calculation and fusion of business-related weights to solve the problem of "accidental cleanup of core business components due to relying solely on access frequency and vulnerability risk". Specific operations include:

[0195] Business Relevance Weight Calculation: By connecting to the component dependency graph of the enterprise's business systems, the "business relevance" of each component is obtained. (Value range [0,1]): Components that core business modules (such as payment, user authentication) depend on. Components that auxiliary modules (such as logging and monitoring) depend on. Components that redundant modules depend on ;

[0196] The S4 cache library management and update adopts a dynamic expiration mechanism and a two-factor cleanup strategy based on popularity and value.

[0197] Dynamic expiration time calculation adjusts the expiration time dynamically based on the component's access frequency and vulnerability risk level; high-frequency access or high-risk components are retained longer, while low-frequency and low-risk components expire quickly.

[0198] The incremental update mechanism periodically updates the dynamic information of components in the cache library, and connects to the external database through the unique identifier ID of the component, without the need to recalculate the hash.

[0199] The two-factor cleanup strategy ensures that when the cache library capacity reaches a threshold, it prioritizes cleaning up the records with the lowest access frequency and vulnerability risk level, while retaining a certain proportion of historical version hashes for backtracking analysis.

[0200] Dynamic expiration time formula:

[0201] ;

[0202] in Based on the expiration time;

[0203] Access frequency;

[0204] Vulnerability risk level;

[0205] , As weight;

[0206] Cleanup Priority: Incorporate Business Relevance Weights

[0207] ;

[0208] in, , To adjust the coefficient, Records with lower values ​​are cleaned up first;

[0209] The business association weighting coefficient (default 0.3, and...) , satisfy ); The lower the value, the higher the cleanup priority;

[0210] Core component protection mechanism: Set a "cleanup protection threshold" for the core business components. (Default 0.5): If the component's Even if the cache library capacity reaches the threshold, it will not be included in the cleanup scope; it will only be included if the component has no access records for 180 consecutive days. And the vulnerability risk level Only then will the protection be lifted and cleaning allowed.

[0211] By connecting to the enterprise's business dependency graph, the "business relevance" of components is calculated. (The highest value is taken for components that depend on core business modules), and this is incorporated into the cache cleanup priority formula to set a "cleanup protection threshold" for core components to avoid unnecessary cleanup. Compared to traditional cache cleanup that only relies on access frequency and vulnerability risk, which may mistakenly delete caches of components that core businesses such as payment and authentication depend on, causing subsequent scans to need to be re-analyzed and affecting business continuity, this mechanism improves the cache retention rate of core components by more than 80% through business value weighting, while optimizing the resource release of non-core components, balancing cache efficiency and business stability.

[0212] This solves the problems of "falsely deleted valid data" or "accumulation of invalid data" caused by traditional fixed-time expiration.

[0213] like Figures 1 to 4 As shown, the dynamic expiration time uses differential retention, with different components having different cache values, through... Dynamically adjust retention time:

[0214] High-frequency access components: The access frequency is close to 1. Set to 90 days;

[0215] High-risk components: The risk level is close to 1. It can be set to 180 days;

[0216] Low-frequency and low-risk components: and All are low. It can be set to 30 days;

[0217] The static information in the incremental update database does not need to be updated frequently, but the dynamic information needs to be updated periodically.

[0218] Batch query external databases using component unique identifiers to update vulnerability lists and license status;

[0219] Only the changed fields are updated; there is no need to recalculate the hash or perform a full overwrite.

[0220] During the two-factor cleanup process, when the cache library capacity reaches its limit, it is sorted by priority. Clean up low-value records:

[0221] Prioritize cleaning The record with the lowest value;

[0222] Retain historical version hash pool: For core components, even if the current version has been updated, the hash records of the last 3 versions are still retained for backtracking analysis;

[0223] By dynamically expiring, incrementally updating, and intelligently cleaning up, the storage efficiency and data validity of the cache library are balanced, avoiding "invalid data accumulation" or "false deletion of valid data".

[0224] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A software component analysis optimization method based on hash caching, characterized in that: The optimization method includes the following steps: S1. Initial Scanning of Multi-Dimensional Features and Fusion of Dynamic Behavioral Features to Calculate Storage Component Hash: During the initial SCA scan of the software, dynamic behavioral features of components during runtime are collected through a lightweight sandbox environment. The basic features, content features, and dependency features of the software package and each component are extracted. After calculating the hash of each dimension feature, dynamic weights are fused, and the hash value of the dynamic behavioral feature is incorporated with dynamic weights to obtain its globally unique hash value. At the same time, a complete component analysis process is executed to obtain the component information of the software package or component, and the global hash value and its associated data are stored in the hash cache library. S2. Subsequent Scanning and Verification Hash Calculation: During subsequent SCA scanning of the software, multi-dimensional features of the software package and components to be detected are extracted and dynamic behavioral features are fused to calculate hash values. Feature variability threshold is then determined. The feature variability is calculated by comparing the three-layer basic features of the component to be detected with the historical features with the highest similarity in the cache. If it is determined to be a minor non-core change, the cache result is directly reused. Otherwise, a Bloom filter is used for pre-verification, and then the hash cache is queried to see if there is a record corresponding to the hash value. S3. Reuse of Hash Result and Incremental Analysis: If a matching hash value record exists in the hash cache, the component analysis result corresponding to that record is directly called without repeated scanning. If the hash value does not hit the cache but a historical version record of the component exists, incremental analysis is performed through the cross-version component information inheritance mechanism. Through version number semantic analysis, the inheritable component information in cross-version iterations of the component is determined. Only the core dimensions involved in version changes are re-analyzed, and the inheritance source is marked to ensure traceability. Version Association Query: When the hash of the component to be detected does not hit, the component's unique identifier is used. Query the historical version records of the component in the cache library and extract the component information of the historical version; if the hash value does not hit the cache and there is no historical version record, perform a full scan analysis on the component, generate new component information, and store the new data in the cache library; S4. Cache library management and updates: Regularly maintain the hash cache library, including cleaning up expired records, updating dynamic information related to components, and adjusting the cleanup priority based on business association weights to ensure the cache retention of core components and the validity and storage efficiency of cached data.

2. The software component analysis optimization method based on hash caching according to claim 1, characterized in that: When extracting features of software packages and component files in S1, the collection and fusion of dynamic behavior features of component runtime are added. The specific operations include: when extracting features of software packages and component files, the component is run in a lightweight sandbox environment, dynamic behavior data is collected within a preset time period, and the dynamic behavior feature hash value is calculated using a time-series hash algorithm. The hash of the storage component calculated in the first scan of S1 consists of the following four characteristics: Multi-dimensional feature extraction: Extracting three layers of features from software packages and components: basic features, content features, and dependency features; Layered hash calculation: After calculating the hash of the three layers of features separately, they are merged into a globally unique hash through dynamic weights. The dynamic weight factor is dynamically adjusted according to the component stability factors to accurately reflect the true attributes of the component and effectively distinguish components with consistent static features but different dynamic behaviors with security risks, thereby reducing the false judgment rate of hash caching. The stability factors include component functional characteristics and security risks. Full component analysis: Execute the complete SCA process to obtain component information including: component unique identifier, version number, license type, vulnerability list, and open source community activity. Related storage: Construct a dual-index structure for the hash cache library: using a global hash as the primary key, while also creating secondary indexes to support reverse queries of hash records of different versions of the same component; in the S1 component information storage model, add a dynamic behavioral feature description field. and , The fields are jointly stored in a hash cache library to provide a basis for risk prediction in subsequent scans; In step S1, the global hash value is calculated using the following formula: ； in The hash value of the i-th layer feature. Basic characteristics Content characteristics Dependency features; For dynamic weights, ,and , ; To merge hash functions; The weights are dynamic behavioral features, ranging from 0.1 to 0.2, and satisfy the following conditions: At the same time, maintain the original Weighted sorting; The dynamic behavior feature hash value is calculated by a time-series hash algorithm to collect dynamic behavior data of components during runtime in a lightweight sandbox. In S1, the component information is stored, and its storage model follows the following formula: ， ， ， ， ， ； in, Used as a unique identifier for components; Version number; License type; List of vulnerabilities; Rate activity level; For cache creation time; The dynamic behavior of components at runtime is incorporated into the feature system. Dynamic data is collected through a lightweight sandbox, and dynamic feature hashes are generated by combining time-series hashing algorithms. Weights are dynamically allocated according to component type and incorporated into global hash calculations. At the same time, dynamic behavior description tags are associated and stored.

3. The software component analysis optimization method based on hash caching according to claim 2, characterized in that: The basic features include file size, modification time metadata, and a standard directory structure of identifiable components in the file path; the content features involve extracting hashes of key segments from code files, skipping comments and blank lines, and calculating hashes only for function definitions, class inheritance relationships, and core logic; Dependency characteristics: Parse the component's dependency description file, generate a dependency graph hash, and calculate the hash after sorting the dependencies by name; The dynamic weights are dynamically adjusted based on the basic features, content features, and dependency features, and are sorted in descending order of content feature weights, dependency feature weights, and basic feature weights. The full component analysis requires a complete SCA analysis to be performed on the first scan.

4. The software component analysis optimization method based on hash caching according to claim 1, characterized in that: Before S2 calculates the hash value of the component to be detected and queries the cache, a new feature variability threshold judgment step is added. The specific operations include: Feature variability calculation: the three-layer basic features of the component to be detected ( , , ) and the historical features with the highest similarity in the cache library ( , , The characteristic variability is calculated using the following formula. : ； in For the first Similarity of layer features Consistent with the dynamic weights of S1; Variation threshold determination: Preset feature variation threshold ; like The change was determined to be a "minor non-core change". The component analysis results in the cache were directly reused without any further hash lookup. like Perform Bloom filter pre-verification and hash cache lookup according to the original S2 process; Variance record storage: This stores the variability data from each calculation. The judgment result is stored in the "Feature Mutation Log" field of the cache. This is used as a historical data reference for subsequent S4 cache cleanup. In S2, the subsequent scanning calculation of the verification hash introduces an incremental hash generation mechanism and a Bloom filter pre-verification. The Bloom filter is only used as a fast filtering tool, and its judgment result needs to be combined with the feature variability threshold judgment result for the final decision. Incremental feature extraction: For subsequent software packages to be detected, prioritize extracting the features that differ the most from the first scan. If the feature hash of this layer matches the historical records, then verify the features of other layers. Bloom filter pre-validation: Before querying the hash cache, a Bloom filter is used to quickly determine if the hash value exists, filtering out invalid queries. Sharded index query: The hash cache library shards according to hash value prefixes. When querying, the target shard is located directly, and then an exact match is made using the primary key; In S2, the Bloom filter pre-validation function is calculated according to the following formula: ； In S2, the calculation method for the fragment index positioning formula follows the following formula: ； in Before retrieving the hash value The bit serves as a fragment identifier; Introducing feature variability The concept involves calculating the similarity between the component to be detected and historical cached features using Hamming distance, and dynamically setting a threshold based on the component's security level. For "minor non-core changes with variability below the threshold", the cached results can be reused directly without performing a full hash lookup.

5. The software component analysis optimization method based on hash caching according to claim 4, characterized in that: The incremental feature extraction adopts a change probability priority method: first, the content feature hash is calculated; when the content feature hash matches a record in the cache, the dependent feature hash is calculated; finally, the basic feature hash is calculated. When the Bloom filter first stores the hash, it will Multiple bits in the filter are mapped to multiple hash functions and marked as (1); during subsequent queries, if the corresponding bits in the filter are not all (1), then... It is definitely not in the cache; if all are (1), then it may exist.

6. The software component analysis optimization method based on hash caching according to claim 1, characterized in that: When S3 handles hash misses, a new cross-version component information inheritance mechanism is added, which only analyzes the core dimensions involved in version changes; The specific operations include: Inheritable information determination: Based on the component version change type, determine the inheritable component information: Patch Version: Inherited License Type Open source community activity Only re-analyze the version number Vulnerability List ; Major version: Inherit only the component's unique identifier Re-analyze the version number License type Vulnerability List Open source community activity ; Inheritance information marker: A new "Inheritance Source Field" is added to the newly generated component information. With "Reanalyze Fields" To ensure the traceability of ingredient information; In S3, the reuse of identical hash results and incremental analysis are distinguished between full matching and partial matching scenarios. Exact match reuse: when Completely consistent with the cache record, i.e. The results of component analysis can be directly reused, but it is necessary to verify in real time whether the dynamic information is updated. Incremental analysis of partial matching: when Similar to a record in the cache library Then, only the component dimensions corresponding to the differential features are re-analyzed; Full analysis of the new component: If a hash miss occurs, the full analysis process of S1 is executed, and the newly generated... Component information is stored in a cache library; In S3, the hash similarity calculation formula follows the following formula: ； in, For indicator functions , The dynamic weights are the same as those in S1. 1 indicates a perfect match. Indicates partial matching; In S3, the formula for the incremental analysis range is calculated according to the following formula: ； That is, analysis is only performed on the component dimensions corresponding to the mismatched feature layers.

7. The software component analysis optimization method based on hash caching according to claim 6, characterized in that: When the complete matching reuse is used, The results are completely identical to the cached records, allowing direct reuse of component analysis results, but real-time verification of dynamic information is required. Vulnerability Information: Call the vulnerability database API to check for any newly added CVEs; License Status: Verify whether the license has been updated by the open-source community; In the incremental analysis of the partial matching, when Partial feature matching only analyzes the component dimensions corresponding to the differential features, avoiding full rescanning.

8. The software component analysis optimization method based on hash caching according to claim 1, characterized in that: When S4 performs cache maintenance and cleanup, it adds the calculation and fusion of business-related weights. The specific operations include: Business Relevance Weight Calculation: By connecting to the component dependency graph of the enterprise's business systems, the "business relevance" of each component is obtained. Components that the core business modules depend on The range of values ​​is Components that the auxiliary module depends on The range of values ​​is Components that redundant modules depend on The range of values ​​is ; The S4 cache library management and update adopts a dynamic expiration mechanism and a two-factor cleanup strategy based on popularity and value. Dynamic expiration time calculation adjusts the expiration time dynamically based on the component's access frequency and vulnerability risk level; The incremental update mechanism periodically updates the dynamic information of components in the cache library, and connects to the external database through the unique identifier ID of the component, without the need to recalculate the hash. The two-factor cleanup strategy ensures that when the cache library capacity reaches a threshold, it prioritizes cleaning up the records with the lowest access frequency and vulnerability risk level, while retaining a certain proportion of historical version hashes for backtracking analysis. In S4, the dynamic expiration time formula is calculated according to the following formula: ； in Based on the expiration time; Access frequency; Vulnerability risk level; , As weight; In step S4, the cleanup priority is calculated according to the following formula: ； in, , To adjust the coefficient, Records with lower values ​​are cleaned up first; This refers to the business relevance weighting coefficient; The lower the value, the higher the cleanup priority; Core component protection mechanism: Set a "cleanup protection threshold" for the core business components. If the component Even if the cache library capacity reaches the threshold, it will not be included in the cleanup scope; it will only be included if the component has no access records for 180 consecutive days and the vulnerability risk level is [not specified]. Only then will the protection be lifted and cleaning allowed; By connecting to the enterprise's business dependency graph, the "business relevance" of components is calculated. This is incorporated into the cache cleanup priority formula, setting a "cleanup protection threshold" for core components to avoid unnecessary cleanup.

9. The software component analysis optimization method based on hash caching according to claim 8, characterized in that: The dynamic expiration time uses differential retention, with different components having different cache values. Dynamically adjust retention time: High-frequency access components: The access frequency is close to 1. Set to 90 days; High-risk components: The risk level is close to 1. Set to 180 days; Low-frequency and low-risk components: and All are low. Set to 30 days; The static information in the incremental update database does not need to be updated frequently, but the dynamic information needs to be updated periodically. Batch query external databases using component unique identifiers to update vulnerability lists and license status; Only the changed fields are updated; there is no need to recalculate the hash or perform a full overwrite. During the two-factor cleanup process, when the cache library capacity reaches its limit, it is sorted by priority. Clean up low-value records: Prioritize cleaning The record with the lowest value; Retain historical version hash pool: For core components, even if the current version has been updated, the hash records of the last 3 versions are still retained for backtracking analysis.

Images