A component version detection method and system based on source code snippet fingerprints

Abstract

The application relates to a component version detection method and system based on source code segment fingerprints, and the detection method comprises the following steps: scanning engineering source codes by a scanning system, converting the engineering source codes into source code segment fingerprints, and uploading the source code segment fingerprints to a platform system; querying a knowledge base according to the source code segment fingerprints, screening out component versions by matching, if there are multiple component versions, querying the fingerprints of each component version by backtracking of the knowledge base, analyzing the index intersection of the component versions and the engineering source code segment fingerprints, establishing the component version and the matched fingerprint set relationship, eliminating false alarm component versions according to the fingerprint set relationship rules between the component versions, and obtaining expected component versions. When the SCA detection system detects multiple versions of components, the application can query the matched fingerprints in the knowledge base according to the detected component versions, analyze the fingerprint set relationship between the versions, eliminate false alarm component versions, and improve the precision of the source code segment detection component version.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data security, and in particular to a component version detection method and system based on source code segment fingerprints. BACKGROUND

[0002] Software Composition Analysis (SCA) is a method for managing the security of open source component applications. Through an SCA detection system, open source components introduced by engineering software can be quickly tracked and analyzed, all related components, support libraries, and their direct and indirect dependencies can be found, the SCA detection system can also detect software licenses, deprecated dependencies, and vulnerabilities and potential threats, and the SCA detection system generates a Software Bill of Materials (SBOM) during the detection process, thereby providing a complete list of open source components for project software assets. An open source component has a life cycle and needs to be constantly updated and maintained, so there are many different versions of the components. In the field of SCA, the vulnerabilities of a component are associated with the specific version of the component. For example, Apache Log4j 1.0 has 194 versions, some of which have vulnerabilities, and some of which do not. Therefore, only by detecting the component and the version can users be prompted whether the components introduced in the project have vulnerability risks, and users can repair vulnerability risks according to the specific component version, such as upgrading the component version to a version without vulnerabilities.

[0003] Currently, the detection methods for analyzing open source components introduced by engineering software by SCA detection systems mainly include package manager detection, file directory detection, binary detection, and source code segment detection. Source code segment detection is based on a sliding window rule to extract source code segments from source code files and generate source code segment fingerprints, and the fingerprints are used to query a knowledge base to detect open source components and versions introduced in engineering software. The existing source code segment detection method detects components based on the fingerprints of part of the source code files, can detect the names of open source components, but cannot accurately detect the versions of open source components, and there are many false positives of component versions. Therefore, how to improve the accuracy of detecting component versions by source code segment detection based on the traditional source code segment detection method, filter out matching fingerprint data by backtracking query of the knowledge base, analyze the set relationship of the fingerprint data between component versions, and eliminate false positive component versions is a problem that needs to be considered at the present stage.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a component version detection method and system based on source code fragment fingerprints, which solves the deficiencies of the existing SCA detection method.

[0006] The objective of this invention is achieved through the following technical solution: a component version detection method based on source code fragment fingerprints, the detection method comprising:

[0007] The scanning system scans the project source code, converts the source code into source code fragment fingerprints, and uploads them to the platform system.

[0008] The platform system queries the knowledge base based on the fingerprint of the source code fragment, matches and filters the component version. If there are multiple component versions, it backtracks through the knowledge base to find the fingerprint of each component version, analyzes the intersection of indicators between the component version and the fingerprint of the project source code fragment, establishes the relationship between the component version and the matching fingerprint set, and eliminates false alarm component versions according to the fingerprint set relationship rules between component versions to obtain the desired component version.

[0009] The version of the component for eliminating false alarms based on fingerprint set relationship rules includes:

[0010] If the fingerprint set of the current component version is included in the fingerprint sets of other component versions, that is, the fingerprint set of the current component version... If the component's fingerprint set is from other versions, then the current component version is a false positive.

[0011] If the fingerprint set of the current component version is not included in the fingerprint sets of other component versions, that is, the fingerprint set of the current component version... If the component fingerprint set is from other versions, then the current component version is the expected target version.

[0012] The analysis of the intersection of metrics between component versions and project source code fragment fingerprints, and the establishment of the relationship between component versions and matching fingerprint sets, specifically includes the following:

[0013] A1. Match the fingerprint and component version of the project source code using the knowledge base API;

[0014] A2. The knowledge base matches and filters the fingerprints of the component version with the fingerprints of the input project source code fragments, and outputs the intersection of the component version fingerprint and the project source code fragment fingerprint.

[0015] A3. Based on steps A1 and A2, detect the version of each component in sequence and establish the relationship between the component version and the matching fingerprint set.

[0016] The fingerprint set relationship is calculated using the matrix multiplication method, specifically including the following:

[0017] The matrix is ​​represented as T = [T1, T2, T3, T4, T5, ..., Tn ], T represents the set of matched fingerprints, T n represents the fingerprints, if component version A exists fingerprints T1, T3, T n , the fingerprint matrix of component version A can be represented as A = [1, 0, 1, 0, 0, …, 1]; if component version B exists fingerprints T1, T3, T4, T n , the fingerprint matrix of component version B can be represented as B = [1, 0, 1, 1, 0, …, 1];

[0018] If AB T = AA T , A is a subset of B, A is a false positive version, wherein T represents the transpose;

[0019] If AB T < AA T , A is not a subset of B, A is an expected version,

[0020]

[0021] The source code fragment fingerprint generated from the engineering source code specifically includes the following contents:

[0022] Recursively traverse the engineering directory and files, and filter the source code files according to the file suffix of the files;

[0023] Parse the source code file text content, and sequentially read N lines in a sliding window manner;

[0024] Filter the file content by line, remove comments, spaces and special characters, and retain the source code content;

[0025] Compress the N lines of parsed text into one line, generate a fingerprint according to the hash algorithm sha1, and generate JSON data.

[0026] A component version detection system based on source code fragment fingerprints, comprising a scanning system, a platform system and a knowledge base;

[0027] The scanning system is used for scanning engineering source code, and converting the engineering source code into source code fragment fingerprints, and uploading to the platform system;

[0028] The platform system is used for querying the knowledge base according to the source code fragment fingerprint, matching and screening out component versions, if there are multiple component versions, then querying the fingerprints of each component version through the knowledge base, analyzing the intersection of the component version and the engineering source code fragment fingerprint, establishing the component version and the matched fingerprint set relationship, and eliminating false positive component versions according to the fingerprint set relationship rule between the component versions to obtain the expected component version;

[0029] The knowledge base is used to store the file fingerprints and source code segment fingerprint data of component versions.

[0030] The analysis of the intersection of the component version and the engineering source code segment fingerprint index establishes a component version and matching fingerprint set relationship, and specifically includes the following contents:

[0031] A1, the fingerprints of the engineering source code and the component version are queried for matching through the knowledge base API;

[0032] A2, the knowledge base filters and outputs the intersection of the component version fingerprints and the engineering source code segment fingerprints according to the matching of the corresponding fingerprints of the component version and the input engineering source code segment fingerprints;

[0033] A3, each component version is detected in turn according to steps A1 and A2, and a component version and matching fingerprint set relationship is established.

[0034] The false positive component version is eliminated according to the fingerprint set relationship rules, which includes:

[0035] If the fingerprint set of the current component version is contained in the fingerprint set of other component versions, that is, the fingerprint set of the current component version is a subset of the fingerprint set of other component versions, the current component version is a false positive version; The fingerprint set of the other version component;

[0036] If the fingerprint set of the current component version is not contained in the fingerprint set of other component versions, that is, the fingerprint set of the current component version is not a subset of the fingerprint set of other component versions, the current component version is the expected target version. The fingerprint set of the other version component;

[0037] The present application has the following advantages: a component version detection method and system based on source code segment fingerprints can detect multiple versions of components based on an SCA detection system, query matching fingerprints in the knowledge base according to the detected component versions, analyze the fingerprint set relationship between versions, eliminate false positive component versions, and improve the accuracy of source code segment detection component version. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 It is a flowchart of the method of the present application;

[0039] Figure 2 It is a matching fingerprint number diagram of the detected engineering source code fingerprint and the component version source code fingerprint in the knowledge base;

[0040] Figure 3 It is a fingerprint intersection distribution diagram of the component version and the detected engineering fingerprint data set;

[0041] Figure 4 It is a fingerprint set containing relationship distribution diagram between component versions;

[0042] Figure 5 The figure is a structural schematic diagram of the system of the present application. DETAILED DESCRIPTION

[0043] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in combination with the drawings herein is not intended to limit the protection scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application. The present application will be further described below in combination with the drawings.

[0044] As shown in Figure 1 one of the embodiments of the present application relates to a component version detection method based on source code snippet fingerprints, the detection method comprising:

[0045] The scanning system scans the engineering source code and converts the engineering source code into source code snippet fingerprints, and uploads them to the platform system;

[0046] Further, the following contents are specifically included:

[0047] The engineering directory and files are recursively traversed, and the source code files are identified and filtered according to the file suffixes such as.java,.cpp,.c,.js, etc.

[0048] The source code file text content is parsed, and N lines are sequentially read in a sliding window manner;

[0049] The file content is filtered by line, and comments, spaces and special characters are removed, and source code content is retained;

[0050] The parsed N lines of text are compressed into one line, a fingerprint is generated according to the three-circle hash algorithm sha1, and JSON data is generated.

[0051] The platform system queries the knowledge base according to the source code snippet fingerprints, matches and filters out the component versions, if there are multiple component versions, the fingerprints of each component version are queried back through the knowledge base, the index intersection of the component versions and the engineering source code snippet fingerprints is analyzed, the component version and the matched fingerprint set relationship is established, and the false positive component versions are eliminated according to the fingerprint set relationship rules between the component versions, to obtain the expected component version.

[0052] Further, eliminating false positive component version according to the fingerprint set relationship rule comprises:

[0053] If the fingerprint set of the current component version is contained in the fingerprint set of other component versions, that is, the fingerprint set of the current component version is contained in the fingerprint set of other component versions other version component fingerprint set, the current component version is a false positive version;

[0054] If the fingerprint set of the current component version is not contained in the fingerprint set of other component versions, that is, the fingerprint set of the current component version is not contained in the fingerprint set of other component versions other version component fingerprint set, the current component version is a desired target version.

[0055] Further, analyzing the intersection of the component version and the engineering source code segment fingerprint index, and establishing the component version and the matching fingerprint set relationship comprises the following contents:

[0056] A1, querying the matching of the engineering source code fingerprint and the component version through the knowledge base API;

[0057] Specifically, the engineering source code fingerprint and the component version are encapsulated into the request data part of the HTTP message, and an HTTP request message is initiated to the knowledge base, and the engineering source code fingerprint and the component version data are pushed to the knowledge base.

[0058] A2, the knowledge base filters according to the matching of the corresponding fingerprint of the component version and the input engineering source code segment fingerprint, and outputs the intersection of the component version fingerprint and the engineering source code segment fingerprint;

[0059] Specifically, the knowledge base receives the HTTP request message, parses and obtains the engineering source code fingerprint and the component version; the knowledge base queries the source code fingerprint of the component version from the local database of the knowledge base according to the received component version; the knowledge base analyzes the received source code fingerprint data and the source code fingerprint data queried from the local database, and takes out the intersection part or the common part (already matched) of the source code fingerprint data.

[0060] A3, detecting each component version according to steps A1 and A2 in turn to establish the component version and the matching fingerprint set relationship.

[0061] Specifically, according to steps A1 and A2, each component version is detected and analyzed in turn, and then each component version and the matched source code fingerprint data (component version: source code fingerprint data) can be obtained.

[0062] The matching fingerprint number of the detected engineering source code fingerprint and the component version source code fingerprint in the knowledge base is as shown in Figure 2 The X-axis represents different versions of components, and the Y-axis represents the number of fingerprint matches, that is, the number of intersections of the fingerprint of the X-axis component version and the engineering source code fingerprint. It can be analyzed that the intersection and matching degree of the fingerprint of the component version and the fingerprint of the engineering source code are different.

[0063] The source code fingerprint of the detected project is quantified as a coordinate value, and specific fingerprint quantification is shown in the following table:

[0064] Source fingerprint Quantized value dc279a62bf40465e52c205d4b2b9e802 0 5056f0755d6be2fa3061030d88caeb52 1 5febffdf404a06a39e0cf113b129005b 2 7d44d41efd0048e2b04e2d6189e1154d 3 85ca771495cf8eeb471ce9d0933f557d 4 8f89371ba93bf4c419924751b9dd1ed9 5 …… N d6f53e73c10f0cec863992e66f13aa1f N+1

[0065] The source code fingerprint data is sorted, and after sorting, the first source code fingerprint is quantized as 0, the second source code fingerprint is quantized as (0+1) that is 1, the third source code fingerprint is quantized as (0+2) that is 2, and the Nth source code fingerprint is quantized as (0+N-1) that is N-1.

[0066] By quantizing as a coordinate, a set operation, that is, a linear algebra operation, is facilitated. The set operation can analyze the relationship of the source code fingerprint data set, and further the set relationship can determine whether the component version is a false positive version.

[0067] The intersection fingerprint distribution of the component version fingerprint and the detected project fingerprint data set is shown in Figure 3 , the X-axis represents the version of the component, the Y-axis represents the quantized fingerprint coordinate, and the dots in the figure represent the fingerprint distribution of the component version. It can be analyzed from the figure that the fingerprint set of some component versions belongs to the subset of the fingerprint set of other versions, and there exists a "version V1 fingerprint version V2 fingerprint" scenario, such as

[0068] The fingerprint set inclusion relationship distribution between component versions is shown in Figure 4 , the X-axis and Y-axis represent the specific versions of the components, and the dots represent that the component version fingerprint set of the X-axis belongs to the subset of the component version fingerprint set of the Y-axis, that is, "X-axis version fingerprint set Y-axis version fingerprint set", according to the rule between sets, false positive component versions can be eliminated.

[0069] The fingerprint set relationship is calculated by a matrix product method, which specifically includes the following contents:

[0070] The matrix is represented as T=[T1, T2, T3, T4, T5, …, T n ], T represents the set of matched fingerprints, T n represents a fingerprint, if the component version A has fingerprints T1, T3, T n , then the fingerprint matrix of the component version A can be represented as A=[1, 0, 1, 0, 0, …, 1]; if the component version B has fingerprints T1, T3, T4, T n , then the fingerprint matrix of the component version B can be represented as B=[1, 0, 1, 1, 0, …, 1];

[0071] If AB T = AAT Then A belongs to the subset of B, A is a false positive version, wherein T represents transposition;

[0072] If AB T <AA T Then A does not belong to the subset of B, A is a desired version,

[0073] As Figure 5 Another embodiment of the present application relates to a component version detection system based on source code snippet fingerprints, which comprises a scanning system, a platform system and a knowledge base;

[0074] The scanning system is used for scanning engineering source code and converting the engineering source code into source code snippet fingerprints and uploading the source code snippet fingerprints to the platform system;

[0075] The platform system is used for querying the knowledge base according to the source code snippet fingerprints, screening out component versions, if there are multiple component versions, querying the fingerprints of each component version through the knowledge base, analyzing the intersection of the component versions and the engineering source code snippet fingerprints, establishing the set relationship between the component versions and the matched fingerprints, eliminating false positive component versions according to the set relationship rules between the component versions, and obtaining desired component versions;

[0076] The knowledge base is used for storing file fingerprints and source code snippet fingerprint data of component versions.

[0077] The analysis of the intersection of the component versions and the engineering source code snippet fingerprints and the establishment of the set relationship between the component versions and the matched fingerprints specifically include the following contents:

[0078] A1, querying and matching the fingerprints of the engineering source code and the component versions through the knowledge base API;

[0079] A2, the knowledge base screens out the intersection of the component version fingerprints and the engineering source code snippet fingerprints according to the matching of the corresponding fingerprints of the component versions and the input engineering source code snippet fingerprints;

[0080] A3, detecting each component version according to steps A1 and A2 in turn, and establishing the set relationship between the component versions and the matched fingerprints.

[0081] The elimination of false positive component versions according to the set relationship rules includes:

[0082] If the fingerprint set of the current component version is contained in the fingerprint set of other component versions, that is, the fingerprint set of the current component version is the fingerprint set of other component versions, the current component version is a false positive version;

[0083] If the current component version's fingerprint set is not contained in other component version fingerprint sets, i.e. the current component version fingerprint set other version component fingerprint sets, the current component version is the desired target version.

[0084] The above description is only preferred embodiments of the present application, it should be understood that the present application is not limited to the forms disclosed herein, should not be seen as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein, by the above teachings or related art or knowledge. The modifications and changes made by those skilled in the art without departing from the spirit and scope of the present application shall be within the scope of the appended claims of the present application.

Claims

1. A component version detection method based on source code fragment fingerprints, characterized in that: The detection method includes: The scanning system scans the project source code, converts the source code into source code fragment fingerprints, and uploads them to the platform system; The platform system queries the knowledge base based on the fingerprint of the source code fragment, matches and filters the component version. If there are multiple component versions, it backtracks through the knowledge base to find the fingerprint of each component version, analyzes the fingerprint intersection between the component version and the fingerprint of the project source code fragment, establishes the fingerprint set relationship between component versions, and eliminates false alarm component versions according to the fingerprint set relationship rules between component versions to obtain the desired component version. The component versions used to eliminate false alarms based on the fingerprint set relationship rules between component versions include: If the fingerprint set of the current component version is included in the fingerprint sets of other component versions, that is, the fingerprint set of the current component version... If other component version fingerprint sets are used, then the current component version is a false positive version; If the fingerprint set of the current component version is not included in the fingerprint sets of other component versions, that is, the fingerprint set of the current component version... If there are other component version fingerprint sets, then the current component version is the expected component version; The analysis of the intersection of metrics between component versions and project source code fragment fingerprints, and the establishment of fingerprint set relationships between component versions, specifically includes the following: A1. Match the fingerprint and component version of the project source code using the knowledge base API; A2. The knowledge base matches and filters the fingerprints of the component version with the fingerprints of the input project source code fragments, and outputs the intersection of the component version fingerprint and the project source code fragment fingerprint. A3. Based on steps A1 and A2, detect the versions of each component in sequence and establish the fingerprint set relationship between component versions; The fingerprint set relationship is calculated using the matrix multiplication method, specifically including the following: The matrix is ​​represented as T = [ , , , , , …, ], where T represents the set of matched fingerprints. This indicates a fingerprint; if component version A has a fingerprint... If component version A has a fingerprint matrix, then the fingerprint matrix of component version A is represented as A = [1, 0, 1, 0,0, …, 1]; if component version B has a fingerprint... Then the fingerprint matrix of component version B is represented as B = [1, 0, 1, 1, 0, …, 1]. if If A is a subset of B, then A is a false positive version, where T represents the transpose; if If A is not a subset of B, then A is the desired version. , .

2. The component version detection method based on source code fragment fingerprints according to claim 1, characterized in that: The process of converting project source code into source code fragment fingerprints specifically includes the following: Recursively traverse the project directory and files, and filter source code files based on their file extensions; Parse the text content of the source code file and read N lines sequentially using a sliding window. Filter the file content line by line, removing comments, spaces, and special characters while preserving the source code content; The parsed N lines of text are compressed into one line, a fingerprint is generated based on the SHA1 hash algorithm, and JSON data is generated.

3. A component version detection system based on source code fragment fingerprints, characterized in that: This includes scanning systems, platform systems, and knowledge bases; The scanning system is used to scan the project source code, convert the project source code into source code fragment fingerprints, and upload them to the platform system; The platform system is used to query the knowledge base based on the fingerprint of the source code fragment, match and filter the component version. If there are multiple component versions, the fingerprint of each component version is retrieved by backtracking through the knowledge base, and the fingerprint intersection between the component version and the fingerprint of the project source code fragment is analyzed to establish the fingerprint set relationship between component versions. False alarm component versions are eliminated according to the fingerprint set relationship rules between component versions to obtain the desired component version. The knowledge base is used to store file fingerprints and source code fragment fingerprint data for component versions; The component versions used to eliminate false alarms based on the fingerprint set relationship rules between component versions include: If the fingerprint set of the current component version is included in the fingerprint sets of other component versions, that is, the fingerprint set of the current component version... If other component version fingerprint sets are used, then the current component version is a false positive version; If the fingerprint set of the current component version is not included in the fingerprint sets of other component versions, that is, the fingerprint set of the current component version... If there are other component version fingerprint sets, then the current component version is the expected component version; The analysis of the intersection of metrics between component versions and project source code fragment fingerprints, and the establishment of fingerprint set relationships between component versions, specifically includes the following: A1. Match the fingerprint and component version of the project source code using the knowledge base API; A2. The knowledge base matches and filters the fingerprints of the component version with the fingerprints of the input project source code fragments, and outputs the intersection of the component version fingerprint and the project source code fragment fingerprint. A3. Based on steps A1 and A2, detect the versions of each component in sequence and establish the fingerprint set relationship between component versions; The fingerprint set relationship is calculated using the matrix multiplication method, specifically including the following: The matrix is ​​represented as T = [ , , , , , …, ], where T represents the set of matched fingerprints. This indicates a fingerprint; if component version A has a fingerprint... Then the fingerprint matrix of component version A is represented as A = [1, 0, 1, 0,0, …, 1]; If component version B has a fingerprint Then the fingerprint matrix of component version B is represented as B = [1, 0, 1, 1,0, …, 1]; if If A is a subset of B, then A is a false positive version, where T represents the transpose; if If A is not a subset of B, then A is the desired version. , .

Images