A method and system for signature detection of files and file tree identification of open source components

Abstract

The present invention relates to a method and system for detecting open source components by signature detection and file tree identification, comprising: encrypting and calculating a signature of feature information extracted from an open source component using a signature encryption algorithm to obtain multiple feature values, and using the multiple feature values ​​as the unique signature information of each open source component; associating the multiple feature values ​​calculated for each open source component with the corresponding open source component and storing them in a database as a knowledge base; parsing and scanning a project to be scanned to obtain feature information results of all nodes of the project; matching each node to obtain a matching result; and screening out accurate open source components from the matching results. The present invention has a high recognition rate, as reflected in the fact that even open source components that are not fully referenced can be identified and scanned, and as long as the referenced content contains node feature information of the open source component, it will be identified. At the same time, the identified components also have a matching rate display and can be flexibly configured.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a method and system for signature detection of files and file tree identification of open source components. Background Art

[0002] Currently, during project development, third-party open source components are sometimes referenced using JAR packages or by copying the open source project code. In this scenario, existing SCA (software composition analysis) tools analyze the components of open source software through package manager scanning, but cannot identify open source components introduced through JAR packages or copied code. Some methods that use the previous matching of open source components have low recognition rates, resulting in insufficient matching accuracy and the possibility of false positives or incorrect versions. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a method and system for signature detection file and file tree identification of open source components, which solves the problems existing in the existing open source component analysis and identification.

[0004] The object of the present invention is achieved by the following technical solution: a method for signature detection file and file tree identification open source component, the method comprising:

[0005] S1. Using a signature encryption algorithm, the signature of the feature information extracted from the open source component is encrypted and calculated to obtain multiple feature values. These multiple feature values ​​are used as the unique signature information of each open source component, and the directory node of the open source component is converted into signature information.

[0006] S2. Map the multiple feature values ​​calculated for each open source component to the corresponding open source component to form a one-to-one correspondence relationship, and store the relationship in the database as a knowledge base.

[0007] S3. Analyze and scan the project to be scanned to obtain feature information results of all nodes of the project;

[0008] S4. Scan and analyze the entire project's feature information data node set, and match each node one by one or simultaneously match multiple nodes to obtain a matching result.

[0009] S5. Filter out accurate open source components from the matching results using the first screening logic and / or the second screening logic.

[0010] The signature encryption algorithm extracts features from five aspects to obtain five characteristic values ​​as the unique signature information of the open source component. The five characteristic values ​​include directory without SIZE type, directory with SIZE type, directory structure without file type, file's precise SHA1 value and characteristic information of jar package.

[0011] The result of obtaining the characteristic information of all nodes of the project includes: the directory without SIZE type, the directory with SIZE type and the directory structure without file type information value of each directory node, as well as the number of directories and files under the node, including the precise SHA1 value of the file for each file node. If there is a jar package in the project, the FILE_MD5 value of the jar package is output and compiled into a scanning characteristic information list.

[0012] The matching of each node one by one or multiple nodes at the same time to obtain the matching results includes: matching each target point one by one or multiple nodes at the same time, the knowledge base returns the matching status of each corresponding node, and for the successfully matched nodes, returns the matching open source component information, stores the number of nodes corresponding to each open source component and the number of matching directory nodes of the matching identification results in the knowledge base, and calculates the matching rate of a directory of open source components based on the number of matching directory nodes and the total number of directory nodes of the open source components, and controls the accuracy of open source component identification through the matching rate.

[0013] The method of screening out accurate open source components from the matching results through the first screening logic includes: setting a threshold value of the number of directory nodes to N, screening out open source components through the matching rate of the matching results, if the number of directory nodes that meet the matching requirements in the matching results reaches the threshold value N of the number of directory nodes of the open source components, then the identified open source components are accurate open source components, screening out open source components through file nodes, if the file node matches successfully, then it is considered that the file is exactly the same as the file of the open source component; and screening out accurate open source components through the matching rate of directory nodes and file nodes.

[0014] The method of filtering out accurate open source components from the matching results through the second filtering logic includes: improving accuracy through jar package analysis, analyzing the specific version number of the jar package by parsing the content of the component identification information file contained in the jar package, and if the jar package has not been modified, directly identifying the open source component through the FILE_MD5 value of the entire jar package.

[0015] A system for detecting files and identifying open source components through signatures and file trees, comprising a signature encryption algorithm calculation module, a knowledge base establishment module, a scanning and analysis module, a matching and identification module, and a logic screening module; the signature encryption calculation module is used to encrypt and calculate the feature information signature extracted from the open source component through a signature encryption algorithm to obtain multiple feature values, and use these multiple feature values ​​as the unique signature information of each open source component, and convert the directory nodes of the open source component into signature information; the knowledge base establishment module is used to map the multiple feature values ​​calculated for each open source component with the corresponding open source component to form a one-to-one corresponding association relationship, and store the relationship in a database as a knowledge base; the scanning and analysis module is used to parse and scan the project to be scanned, and obtain feature information results of all nodes of the project; the matching and identification module is used to match each node one by one or simultaneously match multiple nodes based on the feature information data node set of the entire project obtained through scanning and analysis, to obtain a matching result; and the logic screening module is used to screen out accurate open source components from the matching results through a first screening logic and / or a second screening logic.

[0016] The logic screening module includes a first screening logic unit and a second screening logic unit; the first screening logic unit sets the directory node number threshold to N, and screens the open source components according to the matching rate of the matching results. If the number of directory nodes that meet the matching requirements in the matching results reaches the directory node number threshold N of the open source component, the identified open source component is an accurate open source component, and the open source component is screened by the file node. If the file node matches successfully, it is considered that the file is exactly the same as the file of the open source component; the accurate open source component is screened according to the matching rate of the directory node and the file node; the second screening logic unit improves the accuracy by jar package analysis, and analyzes the specific version number of the jar package by parsing the content of the component identification information file contained in the jar package. If the jar package has not been modified, the open source component is directly identified by the FILE_MD5 value of the entire jar package.

[0017] The result of obtaining the characteristic information of all nodes of the project includes: the directory without SIZE type, the directory with SIZE type and the directory structure without file type information value of each directory node, as well as the number of directories and files under the node, including the precise SHA1 value of the file for each file node. If there is a jar package in the project, the FILE_MD5 value of the jar package is output and compiled into a scanning characteristic information list.

[0018] The result of obtaining the characteristic information of all nodes of the project includes: the directory without SIZE type, the directory with SIZE type and the directory structure without file type information value of each directory node, as well as the number of directories and files under the node, including the precise SHA1 value of the file for each file node. If there is a jar package in the project, the FILE_MD5 value of the jar package is output and compiled into a scanning characteristic information list.

[0019] The present invention has the following advantages:

[0020] 1. The present invention has a high recognition rate, which is reflected in the fact that even if the open source components are not fully referenced, the method of the present invention can identify and scan them. As long as the referenced content contains the node feature information of the open source component, it will be identified. At the same time, the identified components also have a matching rate display and can be flexibly configured.

[0021] 2. The present invention has a low false alarm rate, which is reflected in the dual detection mechanism. When multiple open source components are matched for the same node, the accurate open source components are screened out through precise matching detection of the MD5 value of the jar package or by parsing the identification content inside the jar package. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the process of the present invention;

[0023] Figure 2 A flowchart for establishing a knowledge base;

[0024] Figure 3 Schematic diagram of the scanning analysis matching process. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided below in conjunction with the drawings is not intended to limit the scope of protection of the present application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. The present invention is further described below in conjunction with the drawings.

[0026] like Figure 1As shown, the present invention can accurately identify the jar package components referenced in the project and the open source components directly copied and referenced in the project. It includes five parts: signature encryption algorithm, knowledge base establishment, scanning analysis, matching identification and logical screening, specifically including the following contents:

[0027] 1. Signature encryption algorithm: After signature encryption calculation of some characteristic information extracted from open source components using signature encryption algorithms (MD5 and SHA1), five characteristic values ​​are obtained. These five characteristic values ​​will serve as the unique signature information of the open source component. By comparing and matching the relationship between the signature information and the open source component, it can be determined which open source component it belongs to. Among them, the five characteristic value information includes directory without SIZE type, directory with SIZE type, directory structure without file type, file's exact SHA1 value and characteristic information of JAR package.

[0028] (1) Directory with no size (NO_SIZE): First, sort the nodes in lexicographical order, then concatenate all file names under the node, and then concatenate the current node name + the NO_SIZE of the directory node under the node. This string is encrypted into a 32-bit value using MD5.

[0029] For example: under the directory fileDir, there are files aaa.java and bbb.java and an empty directory ccc;

[0030] First calculate the NO_SIZE value of the directory ccc: d41d8cd98f00b204e9800998ecf8427e;

[0031] The original information is: @aaa.java@@bbb.java@>filedir-d41d8cd98f00b204e9800998ecf8427e<;

[0032] The value after execution should be: b8d27a43277ac0f7dbd592de52a915b2;

[0033] The NO_SIZE value of the final directory fileDir node is b8d27a43277ac0f7dbd592de52a915b2;

[0034] An open source component is composed of many nodes. The NO_SIZE values ​​of all nodes in the entire open source component can be combined in the same way.

[0035] (2) Directory with SIZE type (WITH_SIZE): This type has the file size in addition to NO_SIZE. The purpose of this is to identify the component more accurately. At the same time, it can judge whether the copied source code has been modified by the hit situation of NO_SIZE and WITH_SIZE. Specifically, when WITH_SIZE is hit, the directory and files of the node have all the same sizes as the open source component, then it is considered that the directory node must have referenced the corresponding open source component. On the contrary, when only the NO_SIZE feature value is matched and WITH_SIZE is not hit, it means that the files and directories under the directory node are exactly the same as the open source software, but the file sizes are inconsistent, which means that the files with inconsistent file sizes under the directory must have been modified.

[0036] For example: under the directory fileDir, there are files aaa.java with a file size of 100 bytes and bbb.java with a file size of 200 bytes, and an empty directory ccc;

[0037] First calculate the WITH_SIZE value of the directory ccc: d41d8cd98f00b204e9800998ecf8427e;

[0038] The original information before signing is: @aaa.java:100@@bbb.java:200@>filedir-d41d8cd98f00b204e9800998ecf8427e<;

[0039] The WITH_SIZE value of the final directory fileDir node is 982aa7317f994a8e4795ca155c00ea41;

[0040] An open source component is composed of many nodes. The WITH_SIZE value of all nodes in the entire open source component can be combined in the same way.

[0041] (3) Directory structure does not contain file types (STRUCTURE_ONLY): This type of rule only needs to splice the directories and can sign the directory structure of the open source component, so that the specified open source component can be located through the directory structure.

[0042] For example, under the directory fileDir, there are files aaa.java with a file size of 100 bytes, bbb.java with a file size of 200 bytes, and an empty directory ccc;

[0043] First calculate the STRUCTURE_ONLY value of the directory ccc: d41d8cd98f00b204e9800998ecf8427e;

[0044] The original information before signing is: >filedir-d41d8cd98f00b204e9800998ecf8427e<;

[0045] The STRUCTURE_ONLY value of the final directory fileDir node is 66f98837f32ec41a267cd28abb0652a9;

[0046] An open source component is composed of many nodes. The STRUCTURE_ONLY value of all nodes in the entire open source component can be combined in the same way.

[0047] (4) File exact SHA1 value (FILE_SHA1): This type of value is at the file level. It stores the SHA1 values ​​of all files in the open source component and parses them one by one using the SHA1 algorithm through the byte array of the file.

[0048] (5) Characteristic information of jar package (FILE_MD5): The value of this type belongs to the jar package of the open source component. The data stream of the open source component jar is parsed into a byte array and then parsed using the MD5 algorithm.

[0049] 2. Establish a knowledge base: such as Figure 2 As shown, through the signature encryption algorithm of the above 5 characteristic values, all nodes of the open source component can be parsed into a fixed node tree and file identification information and stored in the knowledge base; for the open source component jar package, there is a FILE_MD5 value. For the open source component source code, the entire directory node of the source code needs to be converted into signature information. The directory node has three types of information: NO_SIZE, WITH_SIZE, and STRUCTURE_ONLY. For the file, there is FILE_SHA1 type information. For the open source component, the 5 characteristic values ​​of each open source component can be calculated through the signature encryption algorithm. By calculating the corresponding 5 characteristic values ​​for all open source components, a relationship is established and stored in the database as a knowledge base for screening and matching.

[0050] 3. Scan analysis: Through the signature encryption algorithm, the project to be scanned is parsed and scanned to obtain the feature information results of all nodes of the project, including the NO_SIZE, WITH_SIZE, and STRUCTURE_ONLY information values ​​of each directory node, the number of directories and files under the node, and the FILE_SHA1 value of each file node. If there is a jar package in the project, the FILE_MD5 value of the jar package is output and compiled into a scan feature value list.

[0051] like Figure 3As shown, the scanning analysis and matching process is: scan the project to be scanned through the scanning end program, and transmit all the feature information of the scan to the matching recognition program. The matching recognition program queries the matching knowledge base, and the knowledge base returns the matching results to the matching recognition program. The matching recognition program analyzes the results, and after the two screening logics mentioned above, it outputs a list of open source component identification results for the project.

[0052] 4. Matching identification: The feature information data node set of the entire project is output through scanning and analysis. The node set is divided into directory nodes and file nodes. Directory nodes are each directory in the project corresponding to a directory node, and file nodes are each file corresponding to a file node. Each node is matched one by one. Of course, this process can also match multiple nodes in batches at the same time. The knowledge base returns the matching status of each node. The unmatched content is ignored. The node corresponding to the matched content returns the matching open source component information. For each open source component, the corresponding number of nodes can be stored in the knowledge base. The number of matching directory nodes of the matching identification results can be used to calculate the component directory matching rate (the number of matching directory nodes divided by the total number of directory nodes of the component). The matching rate can be used to control the accuracy of open source component identification.

[0053] 5. Logical screening: Filtering out accurate components from matching results can be done from two aspects:

[0054] (1) First, we filter open source components by the matching rate in the matching results. We can freely set the threshold. For example, if we set the threshold to 50, then the number of directory nodes that meet the matching results must reach half of the number of directory nodes of the open source component to identify the correct open source component. If the file node can be successfully matched in the knowledge base, then the matching rate is definitely 100%, because the file node is successfully matched and it is considered that the file is exactly the same as the open source component file. Therefore, we can flexibly filter out the correct open source components by the matching rate of directory nodes and file nodes.

[0055] (2) The second aspect is mainly for the reference of jar packages. Since the matching of directory nodes and file nodes is identified by the directory structure characteristics of an open source component, when the versions of two open source components are very close, the directory structures of the two versions may be basically the same. At this time, false positives may occur. For example, the directory structures of log4j version 2.7 and 2.6 are basically the same, which will lead to the matching results of the two components reaching a matching rate of more than 80%. In this special case, it is impossible to accurately identify the exact open source component. At this time, the accuracy can be improved by analyzing the jar package. Most jar packages contain the component identification information file MANIFEST.MF. By parsing the content of this file, the specific version number of the jar package can be analyzed. Moreover, if the jar package has not been modified, the open source component can be directly identified by the FILE_MD5 value of the entire jar package.

[0056] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A method for detecting files and identifying open source components using signatures in a file tree, characterized by: The method comprises: S1. Using a signature encryption algorithm, the signature of the feature information extracted from the open source component is encrypted and calculated to obtain multiple feature values. These multiple feature values ​​are used as the unique signature information of each open source component, and the directory node of the open source component is converted into signature information. S2. Mapping the multiple feature values ​​calculated for each open source component with the corresponding open source component to form a corresponding association relationship, and storing the relationship in the database as a knowledge base; S3. Analyze and scan the project to be scanned to obtain feature information results of all nodes of the project; S4. Scan and analyze the entire project's feature information data node set, and match each node one by one or simultaneously match multiple nodes to obtain a matching result. S5. Filtering out accurate open source components from the matching results using the first screening logic and / or the second screening logic; The signature encryption algorithm extracts features from five aspects to obtain five characteristic values ​​as the unique signature information of the open source component. The five characteristic values ​​include the directory without SIZE type, the directory with SIZE type, the directory structure without file type, the file's precise SHA1 value, and the characteristic information of the jar package; The directory has no size type, that is, NO_SIZE: first, sort the nodes in lexicographical order, then concatenate all the file names under the node, and then concatenate the current node name + the NO_SIZE of the directory node under the node. This string is then encrypted with MD5 to a 32-bit value. The directory contains the SIZE type, namely WITH_SIZE: This type contains the file size in addition to NO_SIZE. The hit situation of NO_SIZE and WITH_SIZE can be used to determine whether the copied source code has been modified; The directory structure does not contain file types, i.e. STRUCTURE_ONLY: This type of rule only needs to splice the directories and can sign the directory structure of the open source component, so that the specified open source component can be located through the directory structure; FILE_SHA1: This type of file-level SHA1 value stores the SHA1 values ​​of all files in the open source component and parses them one by one using the SHA1 algorithm through the byte array of each file. The characteristic information of the jar package is FILE_MD5: This type of value belongs to the jar package of the open source component. The data stream of the open source component jar is parsed into a byte array and then parsed using the MD5 algorithm.

2. The method for detecting files and identifying open source components using signatures in a file tree according to claim 1, characterized in that: The result of obtaining the characteristic information of all nodes of the project includes: the directory without SIZE type, the directory with SIZE type and the directory structure without file type information value of each directory node, as well as the number of directories and files under the node, including the precise SHA1 value of the file for each file node. If there is a jar package in the project, the FILE_MD5 value of the jar package is output and compiled into a scanning characteristic information list.

3. The method for detecting files and identifying open source components using signatures in a file tree according to claim 1, characterized in that: The matching of each node one by one or multiple nodes at the same time to obtain the matching result includes: matching each node one by one or multiple nodes at the same time, the knowledge base returns the matching status of each corresponding node, and for the successfully matched node, returns the matching open source component information, stores the number of nodes corresponding to each open source component and the number of matching directory nodes of the matching identification result in the knowledge base, and calculates the matching rate of a directory of open source components based on the number of matching directory nodes and the total number of directory nodes of the open source components, and controls the accuracy of open source component identification through the matching rate.

4. The method for detecting files and identifying open source components using signatures in a file tree according to claim 1, characterized in that: The method of screening out accurate open source components from the matching results through the first screening logic includes: setting a threshold value of the number of directory nodes to N, screening out open source components through the matching rate of the matching results, if the number of directory nodes that meet the matching requirements in the matching results reaches the threshold value N of the number of directory nodes of the open source components, then the identified open source components are accurate open source components, screening out open source components through file nodes, if the file node matches successfully, then it is considered that the file is exactly the same as the file of the open source component; and screening out accurate open source components through the matching rate of directory nodes and file nodes.

5. The method for detecting files and identifying open source components using signatures in a file tree according to claim 1, characterized in that: The method of filtering out accurate open source components from the matching results through the second filtering logic includes: improving accuracy through jar package analysis, analyzing the specific version number of the jar package by parsing the content of the component identification information file contained in the jar package, and if the jar package has not been modified, directly identifying the open source component through the FILE_MD5 value of the entire jar package.

6. A system for signature detection of files and file tree identification of open source components, characterized by: It includes a signature encryption algorithm calculation module, a knowledge base establishment module, a scanning analysis module, a matching recognition module, and a logic screening module; the signature encryption calculation module is used to encrypt the feature information signature extracted from the open source component through the signature encryption algorithm to obtain multiple feature values, and use these multiple feature values ​​as the unique signature information of each open source component, and convert the directory node of the open source component into signature information; The knowledge base establishment module is used to map the multiple feature values ​​calculated by each open source component with the corresponding open source component to form a corresponding association relationship, and store it in the database as a knowledge base; the scanning analysis module is used to analyze and scan the project to be scanned to obtain the feature information results of all nodes of the project; The matching identification module is used to scan and analyze the feature information data node set of the entire project, match each node one by one or match multiple nodes simultaneously to obtain a matching result; the logic screening module is used to filter out accurate open source components from the matching results through the first screening logic and / or the second screening logic; The feature information results of all nodes of the project are obtained, including: the directory without SIZE type, the directory with SIZE type, and the directory structure without file type information value of each directory node, as well as the number of directories and files under the node, including the precise SHA1 value of the file of each file node. If there is a jar package in the project, the FILE_MD5 value of the jar package is output, and the scan feature information list is compiled; The directory has no size type, that is, NO_SIZE: first, sort the nodes in lexicographical order, then concatenate all the file names under the node, and then concatenate the current node name + the NO_SIZE of the directory node under the node. This string is then encrypted with MD5 to a 32-bit value. The directory contains the SIZE type, namely WITH_SIZE: This type contains the file size in addition to NO_SIZE. The hit situation of NO_SIZE and WITH_SIZE can be used to determine whether the copied source code has been modified; The directory structure does not contain file types, i.e. STRUCTURE_ONLY: This type of rule only needs to splice the directories and can sign the directory structure of the open source component, so that the specified open source component can be located through the directory structure; FILE_SHA1: This type of file-level SHA1 value stores the SHA1 values ​​of all files in the open source component and parses them one by one using the SHA1 algorithm through the byte array of each file. The characteristic information of the jar package is FILE_MD5: This type of value belongs to the jar package of the open source component. The data stream of the open source component jar is parsed into a byte array and then parsed using the MD5 algorithm.

7. The system for detecting signatures of files and identifying open source components in file trees according to claim 6, characterized in that: The logic screening module includes a first screening logic unit and a second screening logic unit; the first screening logic unit sets the directory node number threshold to N, and screens the open source components according to the matching rate of the matching results. If the number of directory nodes that meet the matching requirements in the matching results reaches the directory node number threshold N of the open source component, the identified open source component is an accurate open source component, and the open source component is screened by the file node. If the file node matches successfully, it is considered that the file is exactly the same as the file of the open source component; the accurate open source component is screened according to the matching rate of the directory node and the file node; the second screening logic unit improves the accuracy by jar package analysis, and analyzes the specific version number of the jar package by parsing the content of the component identification information file contained in the jar package. If the jar package has not been modified, the open source component is directly identified by the FILE_MD5 value of the entire jar package.

Images