Software detection method and device based on software bill of material file

Abstract

The invention relates to the technical field of computer software, in particular to a software detection method and device based on a software bill of material file, and is used for solving the problem of inaccurate detection of the software bill of material file in related technologies. Data cleaning and data re-clustering processing are carried out on original description information, obtained by analyzing the software bill of material file of the target software, of each component, target description information of each component is obtained, and the component knowledge set comprises a plurality of standard components and standard description information of each standard component; based on the target description information and the component knowledge set of each component, performing software detection on the target software to obtain a detection result of the target software; thus, through data cleaning and data re-clustering processing, fine granularity data of the components in the software bill of material file of the target software are mined in the obtained target description information of each component, the data precision is improved, and the detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of computer software technology, and in particular to a software detection method and device based on a software bill of materials file. Background Art

[0002] With the rapid iteration of software supply chain security, open source components have exploded. In the system development cycle built by DevSecOps, the proportion of open source components introduced is increasing. In order to ensure the security of the software supply chain and reduce the cost of development, procurement and maintenance, the Software Bill of Materials (SBOM) file is usually used to collect and share information, and manage vulnerabilities, licenses and authorizations on this basis, which can reveal the weaknesses in the entire software supply chain, improve the transparency of the software supply chain, enhance mutual trust between upstream and downstream, and effectively control the potential risks in the software supply chain.

[0003] An SBOM file is a standardized, machine-readable set of metadata that describes the dependency tree of a software package, including key information such as vendor, version number, and component name. By definition, an SBOM file is a list file that contains all components, libraries, and other dependencies used in the software. Therefore, by detecting the SBOM file, we can learn about the security threats of the software. However, in the related art, many methods of detecting SBOM files have the problem of inaccurate detection, and the corresponding software still has potential risks. Summary of the invention

[0004] Some embodiments of the present application provide a software detection method and device based on a software bill of materials file, which are used to mine fine-grained features in a software bill of materials (SBOM) file, thereby improving the detection accuracy from the root, and further improving the SBOM file detection accuracy.

[0005] The specific technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a software detection method based on a software bill of materials file, comprising:

[0007] Parsing a software bill of materials file of the target software to obtain original description information of each component used in the target software, wherein the original description information includes description information of at least one type of the corresponding component;

[0008] Based on the component knowledge set, data processing is performed on the original description information of each component to obtain the target description information of each component, the component knowledge set includes multiple standard components and standard description information of each standard component, and the data processing includes data cleaning and data re-clustering;

[0009] Based on the target description information of each component and the component knowledge set, software detection is performed on the target software to obtain a detection result of the target software.

[0010] In some possible implementations, parsing the software bill of materials file of the target software to obtain original description information of each component used in the target software includes:

[0011] Identifying the file format of the software bill of materials file of the target software to obtain the target file format of the software bill of materials file;

[0012] Based on the data structure of the target file format, the software bill of materials file is parsed to obtain original description information of each component used in the target software.

[0013] In some possible implementations, the performing data processing on the original description information of each component based on the component knowledge set to obtain the target description information of each component includes:

[0014] Based on the standard description information of the multiple standard components, data cleaning is performed on the original description information of at least one component among the components to obtain intermediate description information of the components, wherein the data cleaning includes at least one of normalization processing and data completion processing;

[0015] Data re-clustering processing is performed on the intermediate description information of each component to obtain target description information of each component.

[0016] In some possible implementations, the step of performing data cleaning on original description information of at least one of the components based on the standard description information of the multiple standard components to obtain intermediate description information of the components includes:

[0017] For each of the components, perform the following operations:

[0018] Based on the original description information of any component among the components and the standard description information of the multiple standard components, matching a standard component matching the any component from the multiple standard components;

[0019] According to the preset types, the standard description information of the standard component is compared with the original description information of any component to determine whether data cleaning is required for the original description information of any component;

[0020] If so, based on the standard description information of the standard component, the original description information of any component is normalized and / or data-completed to obtain the intermediate description information of any component;

[0021] If not, the original description information of any component is determined as the intermediate description information of any component.

[0022] In some possible implementations, the performing normalization processing and / or data completion processing on the original description information of any component based on the standard description information of the standard component to obtain the intermediate description information of any component includes:

[0023] If it is determined based on the comparison that the format of the first information in the original description information of any component is not in the reference format, the first information in the original description information of any component is replaced with information in the reference format that is the same type as the first information in the standard description information of the standard component;

[0024] and / or,

[0025] If it is determined based on the comparison that the second information is missing from the original description information of any component, the second information in the original description information of any component is completed with information of the same type as the second information in the standard description information of the standard component;

[0026] The processed description information of any component is determined as the intermediate description information of any component.

[0027] In some possible implementations, performing data re-clustering processing on the intermediate description information of each component to obtain the target description information of each component includes:

[0028] Based on the intermediate description information of each component and the standard description information of the plurality of standard components, hierarchical clustering is performed on the intermediate description information of each component with a preset number of clusters as an iterative target of hierarchical clustering to obtain clustering results of each component;

[0029] Based on the clustering results of the components, target description information of the components is obtained respectively.

[0030] In some possible implementations, the types include some or all of component identification, component version, component developer, component license, component dependency, component encryption, and source address.

[0031] In some possible implementations, performing software detection on the target software based on the target description information of each component and the component knowledge set to obtain the detection result of the target software includes:

[0032] For each of the components, perform the following operations:

[0033] Extracting features of target description information of any component among the components;

[0034] Comparing the features with the features corresponding to the plurality of standard components respectively, to obtain a component matching relationship between any one of the components and a standard component matching the any one of the components;

[0035] According to the vulnerability information of the standard component in the component matching relationship, vulnerability detection is performed on any of the components to obtain vulnerability detection information of any of the components;

[0036] Based on the component matching relationship and vulnerability detection information of each component, a detection result of the target software is generated.

[0037] In a second aspect, an embodiment of the present application provides a software detection device based on a software bill of materials file, including:

[0038] A parsing module, used to parse the software bill of materials file of the target software to obtain original description information of each component used in the target software, wherein the original description information includes description information of at least one type of the corresponding component;

[0039] A data processing module, used for performing data processing on the original description information of each component based on a component knowledge set to obtain target description information of each component, wherein the component knowledge set includes a plurality of standard components and standard description information of each standard component, and the data processing includes data cleaning and data re-clustering;

[0040] The detection module is used to perform software detection on the target software based on the target description information of each component and the component knowledge set to obtain the detection result of the target software.

[0041] In some possible implementations, the parsing module is specifically used to:

[0042] Identifying the file format of the software bill of materials file of the target software to obtain the target file format of the software bill of materials file;

[0043] Based on the data structure of the target file format, the software bill of materials file is parsed to obtain original description information of each component used in the target software.

[0044] In some possible implementations, the data processing module is specifically used to:

[0045] Based on the standard description information of the multiple standard components, data cleaning is performed on the original description information of at least one component among the components to obtain intermediate description information of the components, wherein the data cleaning includes at least one of normalization processing and data completion processing;

[0046] Data re-clustering processing is performed on the intermediate description information of each component to obtain target description information of each component.

[0047] In some possible implementations, the data processing module is specifically used to:

[0048] For each of the components, perform the following operations:

[0049] Based on the original description information of any component among the components and the standard description information of the multiple standard components, matching a standard component matching the any component from the multiple standard components;

[0050] According to the preset types, the standard description information of the standard component is compared with the original description information of any component to determine whether data cleaning is required for the original description information of any component;

[0051] If so, based on the standard description information of the standard component, the original description information of any component is normalized and / or data-completed to obtain the intermediate description information of any component;

[0052] If not, the original description information of any component is determined as the intermediate description information of any component.

[0053] In some possible implementations, the data processing module is specifically used to:

[0054] If it is determined based on the comparison that the format of the first information in the original description information of any component is not in the reference format, the first information in the original description information of any component is replaced with information in the reference format that is the same type as the first information in the standard description information of the standard component;

[0055] and / or,

[0056] If it is determined based on the comparison that the second information is missing from the original description information of any component, the second information in the original description information of any component is completed with information of the same type as the second information in the standard description information of the standard component;

[0057] The processed description information of any component is determined as the intermediate description information of any component.

[0058] In some possible implementations, the data processing module is specifically used to:

[0059] Based on the intermediate description information of each component and the standard description information of the plurality of standard components, hierarchical clustering is performed on the intermediate description information of each component with a preset number of clusters as an iterative target of hierarchical clustering to obtain clustering results of each component;

[0060] Based on the clustering results of the components, target description information of the components is obtained respectively.

[0061] In some possible implementations, the types include some or all of component identification, component version, component developer, component license, component dependency, component encryption, and source address.

[0062] In some possible implementations, the detection module is specifically used to:

[0063] For each of the components, perform the following operations:

[0064] Extracting features of target description information of any component among the components;

[0065] Comparing the features with the features corresponding to the plurality of standard components respectively, to obtain a component matching relationship between any one of the components and a standard component matching the any one of the components;

[0066] According to the vulnerability information of the standard component in the component matching relationship, vulnerability detection is performed on any of the components to obtain vulnerability detection information of any of the components;

[0067] Based on the component matching relationship and vulnerability detection information of each component, a detection result of the target software is generated.

[0068] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0069] Memory, used to store computer programs or instructions;

[0070] A processor is used to execute the computer program or instructions in the memory so that any method described in the first aspect above is executed.

[0071] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any method described in the first aspect above.

[0072] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes any method as described in the first aspect above.

[0073] In an embodiment of the present application, a software bill of materials file of a target software is parsed to obtain original description information of each component used in the target software, the original description information includes description information of at least one type of the corresponding component; based on a component knowledge set, data processing is performed on the original description information of the above-mentioned components to obtain target description information of each component, the component knowledge set includes multiple standard components and standard description information of each standard component, and the data processing includes data cleaning and data re-clustering; then, based on the target description information of each component and the component knowledge set, software detection is performed on the target software to obtain a detection result of the target software; in this way, through data cleaning and data re-clustering processing, the target description information of each component obtained mines the fine-grained data of the components in the software bill of materials file of the target software, thereby improving data accuracy; then, based on the target description information of each component obtained, software detection is performed on the target software, which can improve the detection accuracy from the root, thereby improving the detection accuracy of the generated detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A schematic diagram of a software detection process based on a software bill of materials file in an embodiment of the present application;

[0075] Figure 2 This is a schematic diagram of a flow chart of obtaining the original description information of a component by direct translation and parsing in an embodiment of the present application;

[0076] Figure 3 A schematic diagram of components of target description information of a component in an embodiment of the present application;

[0077] Figure 4 This is a schematic diagram of a target description information flow for determining a component in an embodiment of the present application;

[0078] Figure 5 This is a schematic diagram of a data cleaning process in an embodiment of the present application;

[0079] Fig. 6A This is a schematic diagram of a data cleaning effect in an embodiment of the present application;

[0080] Figure 6B This is another schematic diagram of data cleaning effect in an embodiment of the present application;

[0081] Figure 7 This is a schematic diagram of a hierarchical clustering process in an embodiment of the present application;

[0082] Figure 8 This is a schematic diagram of a software bill of materials detection process in an embodiment of the present application;

[0083] Fig. 9This is a schematic diagram of an overall software detection process based on a software bill of materials file in an embodiment of the present application;

[0084] Fig.10 A schematic diagram of the logical architecture of a software detection device based on a software bill of materials file in an embodiment of the present application;

[0085] Fig.11 It is a schematic diagram of the physical structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0087] In order to help those skilled in the art to better understand the technical solution of the present application, some terms in the embodiments of the present application are explained below.

[0088] (1) In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0089] (2) The term "SBOM" in the embodiments of the present application is an abbreviation of Software Bill of Materials, which is translated into Chinese as "software bill of materials". It is a comprehensive, structured list that lists in detail all components, libraries, and dependencies used in a software product or application, including the name, version, license, and other detailed information of each component. It usually includes multiple text formats, such as SPDX, CycloneDX, DSDX, etc.

[0090] (3) The term "hierarchical clustering" in the embodiments of the present application is an unsupervised machine learning method that recursively merges (or splits) data to construct a tree-like clustering structure, called a "dendrogram"; it is usually used to explore the hierarchical structure of data.

[0091] (4) The terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in sequences other than those illustrated or described herein.

[0092] In order to solve the problem of inaccurate detection in software bill of materials file detection in the related art, an embodiment of the present application provides a software detection method based on a software bill of materials file. In the embodiment of the present application, the software bill of materials file of the target software is parsed to obtain original description information of each component used in the target software, and the original description information includes at least one type of description information of the corresponding component; based on the component knowledge set, the original description information of the above-mentioned components is data processed to obtain target description information of each component, the component knowledge set includes multiple standard components and standard description information of each standard component, and the data processing includes data cleaning and data re-clustering; then, based on the target description information of each component and the component knowledge set, software detection is performed on the target software to obtain the detection result of the target software.

[0093] In this way, by parsing the software bill of materials file of the target software, the original description information of each component used in the target software can be obtained, and then based on the pre-collected component knowledge set including multiple standard components and the standard description information of multiple standard components, the original description information of each component is processed, and the target description information of the above components can be obtained by data cleaning and data re-clustering. Since the target description information of each component is obtained through data cleaning and data re-clustering based on the component knowledge set, the target description information of each component mines the fine-grained data of the component in the software bill of materials file of the target software, thereby improving the data accuracy. Then, based on the obtained target description information of each component, software detection is performed on the target software, which can improve the detection accuracy from the root, thereby improving the detection accuracy of the generated detection results.

[0094] The preferred implementation modes of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0095] See also Figure 1 As shown, the embodiment of the present application provides a software detection method based on a software bill of materials file, and the specific process of the method may include but is not limited to the following steps:

[0096] Step 100: Parse the software bill of materials file of the target software to obtain original description information of each component used in the target software, wherein the original description information includes description information of at least one type of the corresponding component.

[0097] In the embodiment of the present application, when executing step 100, refer to Figure 2 As shown, the following steps can be performed to obtain the original description information of each component used in the target software:

[0098] Step 1001: Identify the file format of the software bill of materials file of the target software to obtain the target file format of the software bill of materials file.

[0099] In the related art, the SBOM detection of the software to be detected can be implemented through the software bill of materials (SBOM) platform. Usually, the user can upload the SBOM file of the software to be detected (referred to as the target software in this application) to the SBOM platform, triggering the SBOM platform to perform software detection on the target software based on the SBOM file.

[0100] Since SBOM files include multiple file formats, such as SPDX format, CDX format, CycloneDX format, DSDX format, etc., different file formats have different data structures and field contents.

[0101] For example, in the SPDX format, the standard software bill of materials (i.e., standard SBOM) includes: Document CreationInformation - SBOM list information, Package Information - component information, FileInformation - code file information, Snippet Information - code snippet information; OtherLicensing Information Detected - other detected license information, RelationshipsBetween SPDX Elements - SPDX element relationship information, Annotations - annotation information, etc. For another example, in the CDX format, the standard SBOM includes: BOM information - SBOM list information, Components - component information, Services - service information, external References - external reference information, Dependencies - component dependency information, Compositions - composition information, Vulnerabilities - vulnerability information, Annotations - annotation information, Formulation - manufacturing / deployment information, Properties - other additional information, Signature - signature information, etc.

[0102] Therefore, the SBOM platform usually first performs file format identification on the uploaded SBOM file, that is, executes step 1001, performs file format identification on the SBOM file, and determines the target file format of the SBOM file, so that the subsequent step 1002 can be executed to quickly extract the original description information of each component in the SBOM file based on the target file format of the SBOM file.

[0103] Step 1002: Based on the data structure of the target file format, the software bill of materials file is parsed to obtain original description information of each component used in the target software.

[0104] In the related art, when software is detected through the SBOM platform, after the file format of the SBOM file is identified, the description information of each component used in the target software is parsed from the SBOM file based on the file format. Then, based on the component knowledge base constructed by the detailed information of all standard components obtained by the crawler, the description information of each component obtained above is compared, and then compared with the vulnerability knowledge base, the vulnerability is matched, and the detection result is generated.

[0105] The inventors studied the above method in the related art and found that when performing SBOM detection, the above method has a high underreporting rate in data comparison for inconsistent formats, missing content, etc., which leads to a low detection accuracy of the related technology.

[0106] Based on this, in the present application, the SBOM file is directly parsed to obtain the original description information of each component used in the target software, that is, after the target file format of the SBOM file of the target software is identified in step 1001, when step 1002 is executed, the SBOM file of the target software is directly parsed based on the data structure of the target file format, so as to obtain the original description information of each component used in the target software, wherein the direct parsing characterization obtains all the component data obtained by parsing. Compared with the parsing and identification in the related art, in the present application, not only the component information of the function body can be extracted, but also the component information of the cross-domain function body can be extracted. In this way, in the present application, all the component data in the SBOM file of the target software can be retained, and by executing subsequent steps, fine-grained features can be mined in combination with the component knowledge set, so as to improve the detection accuracy of the SBOM file from the root.

[0107] In some embodiments, the original description information may include but is not limited to part or all of component name information, component version information, path information, vulnerability information, and the like.

[0108] Step 110: Based on the component knowledge set, data processing is performed on the original description information of the above components to obtain the target description information of the above components, wherein the component knowledge set includes multiple standard components and standard description information of each standard component, and the data processing includes data cleaning and data re-clustering.

[0109] In the present application, the component knowledge set may be stored in the form of the aforementioned component knowledge base, which may be based on detailed information of all standard components pre-acquired by the crawler, and the above standard components may include but are not limited to open source components.

[0110] In some embodiments of this application, refer to Figure 3 As shown, the target description information of any component may include, but is not limited to, part or all of: component identification information, component version information, component developer information, component license information, component dependency information, component encryption information, and source address information.

[0111] It should be noted that the component knowledge set is used in the present application to complete the original description information of each component used in the target software. Therefore, the standard description information of each standard component included in the component knowledge set at least includes the above information.

[0112] In some embodiments of the present application, when executing step 110, refer to Figure 4 As shown, the target description information corresponding to each component can be obtained by executing the following steps:

[0113] Step 1101: Based on the standard description information of multiple standard components, data cleaning is performed on the original description information of at least one component among the components to obtain intermediate description information of each component, wherein the data cleaning includes at least one of normalization processing and data completion processing.

[0114] In some embodiments of the present application, when executing step 1101, refer to Figure 5 As shown, for each of the above components, the intermediate description information of each component can be obtained by executing the following steps:

[0115] Step 500: Based on the original description information of any component among the above components and the standard description information of the above multiple standard components, a standard component matching the component is matched from multiple standard components.

[0116] In some embodiments of the present application, the standard description information of each standard component included in the component knowledge set corresponds to a characteristic value respectively or as a whole. Therefore, when executing step 500, the characteristic value of the component used in the target software can be extracted and the characteristic value can be compared with the characteristic value of the component knowledge set to match the standard component that matches the component from multiple standard components.

[0117] It should be noted that the above matching method is not limited to the feature value comparison method, and other matching methods of related technologies can also be used to achieve it, and this application does not make any specific limitations.

[0118] Step 510: According to the preset types, the standard description information of the standard component is compared with the original description information of the component to determine whether the original description information of the component needs to be cleansed; if so, that is, it is determined that the original description information of the component needs to be cleansed, then step 520 is executed; if not, that is, it is determined that the original description information of the component does not need to be cleansed, then step 530 is executed.

[0119] In some embodiments of the present application, when executing step 510, each type of sub-information included in the standard description information of the matched standard component can be compared with each type of sub-information included in the original description information of the component according to each preset type to determine whether data cleaning of the original description information of the component is required.

[0120] In the present application, the above types may include but are not limited to: component identification, component version, component developer, component license, component dependency, component encryption, and some or all of the source address.

[0121] Step 520: Based on the standard description information of the standard component, the original description information of the component is normalized and / or data-completed to obtain the intermediate description information of the component.

[0122] In some embodiments of the present application, when executing step 510, if it is determined based on the comparison that the format of the first information in the original description information of the component is not a benchmark format, the first information in the original description information of the component is replaced with information in the benchmark format of the same type as the first information in the standard description information of the matched standard component; and / or, if it is determined based on the comparison that the second information is missing in the original description information of the component, the second information of the component is completed with information in the standard description information of the matched standard component of the same type as the second information; finally, the processed description information of the component is determined as the intermediate description information of the component.

[0123] For example, suppose the original description information of any component (denoted as component A) is compared with the standard description information of each standard component in the component knowledge set, and the comparison result is as follows: Fig. 6A As shown, Fig. 6A The table shown shows component name (i.e. component identification information), component version information, component vulnerability information, component publisher information, etc. from top to bottom. Fig. 6A As shown, for the "fast-json-" of component A, the comparison result shows that the item is "not written in a standard way" and there is a problem, that is, the item is not in the benchmark format; for the "version number" of component A, the comparison result shows that the item is "not available" and there is no problem; for the "vulnerability" of component A, the comparison result shows that the item is "available" and there is no problem; for the "manufacturer" of component A, the comparison result shows that the item is "available" and there is no problem; ...

[0124] Based on the above comparison, it is determined that the "fast-json-" of component A needs to be standardized. Therefore, the "fast-json-" of component A is replaced with the information in the same type and benchmark format as the "fast-json-" of component A in the standard description information of the standard component that matches component A in the component knowledge set. Then, the description information of component A after data cleaning can be "Fastjson, XXX, yes, yes,...". It should be noted that for the version number item, since the version number information is the necessary description information of the component, this item is the content that needs to be completed and needs to be completed. Fig. 6A The "XXX" shown in is the completed content.

[0125] For another example, suppose that the original description information of any component (denoted as component B) is compared with the standard description information of each standard component in the component knowledge set, and the comparison result is as follows: Figure 6B As shown, the table contains component name (i.e. component identification information), component version information, component vulnerability information, component publisher information, etc. from top to bottom. Figure 6B As shown, the first information and the second information of component B are both "empty", and the comparison result indicates that both items are "none", and there is no problem. For the "vulnerability" of component B, the comparison result indicates that the item is "yes", and there is no problem. For the "manufacturer" of component B, the comparison result indicates that the item is "none", and there is a problem, etc.

[0126] Based on the above comparison, it is determined that the first two items of component B need to be completed. Therefore, the first two items of component B are completed with the information of the same type as the first two items of component B in the standard data information of the standard component matching component B in the component knowledge set. Then, the description information of component B after data cleaning is "Xxxxx, YYY, yes, yes, ..."

[0127] It should be noted that, usually in the process of software development, a part of the code is usually intercepted from the source code of the standard component into the target software. Therefore, when the components in the SBOM file of the target software are detected, there may be situations where the format is not standardized and / or the content is uncertain. Among them, the situation of uncertain content may be that the description information of the target software cannot be accurately determined. For example, the obtained function body is involved in component 1 and also in component 2. At this time, it is impossible to determine whether the component used in the target software is component 1 or component 2; in this application, a normalization processing method is used to unify the format for the non-standard format, and a data completion method is used to complete the uncertain content. In this way, based on the component knowledge set, the missing data in the SBOM file of the target software can be mined, thereby integrating it into fine-grained data, which provides a basis for the subsequent incremental mining of the messy and complex data relationships in the SBOM file through data reclustering, and obtaining accurate data information chains, so as to improve the accuracy of component vulnerability determination.

[0128] Step 530: Determine the original description information of the component as the intermediate description information of the component.

[0129] In some embodiments of the present application, if step 510 is executed to determine that data cleaning of the original description information of the component is not required, step 530 is executed. At this time, it is indicated that the original description information extracted from the SBOM file of the target software includes various types of description information (sub-information) contained in the target description information, and matches the standard description information of the standard component. Therefore, the original description information of the component is directly determined as the intermediate description information of the component, and subsequent data re-clustering processing is performed.

[0130] Step 1102: Perform data re-clustering processing on the intermediate description information of each component to obtain the target description information of each component.

[0131] In some embodiments of the present application, when executing step 1102, refer to Figure 7 As shown, this can be achieved by executing the following steps:

[0132] Step 700: Based on the intermediate description information of the above components and the standard description information of multiple standard components, hierarchical clustering is performed on the intermediate description information of each component with a preset number of clusters as the iterative target of hierarchical clustering to obtain clustering results of each component.

[0133] In some embodiments of the present application, a clustering model may be constructed based on a hierarchical clustering analysis algorithm, wherein the clustering model may be pre-trained and may be iteratively updated based on actually obtained data during implementation.

[0134] In some embodiments, when executing step 700, the constructed clustering model is initialized. Each sub-information included in the intermediate description information of each component is taken as a single discrete variable data, and then the discrete variable data set is traversed, and the clustering model is used to search from the top to the bottom with small node data such as component version, component vulnerability, manufacturer, path, etc., and the above number of clusters is used as the iteration target of the clustering model to generate a hierarchical structure of different clusters of points and surfaces, that is, to construct a dendrogram, thereby realizing hierarchical clustering of the intermediate description information of each component and obtaining the clustering results of each component.

[0135] In some embodiments of the present application, in each iteration, two clusters with the closest relationship distance are selected to be merged into one cluster, and after reaching a set number of clusters, the clustering result is output. In some embodiments of the present application, the following code can be used to represent the above hierarchical clustering process:

[0136] [d(C_i, C_j)=\min_{x\in C_i, y\in C_j}d(x, y)]

[0137] [d(x,y)=\sqrt{\sum_{k=1}^{p}(x_k-y_k)^2}]

[0138] Wherein, k represents the number of clusters, k∈[1,K]; K is the preset number of clusters; C_x and C_y represent two clusters respectively, and i and j are serial numbers respectively; d(x, y) represents the distance between C_x and C_y; in some embodiments of the present application, the single linkage method can be used in hierarchical clustering, but it should be noted that the present application is not limited to it, and other cluster connection methods in hierarchical clustering can also be used.

[0139] Step 710: Based on the clustering results of each component, obtain target description information of each component.

[0140] In some embodiments of the present application, when executing step 710, the following operations are performed for each component in each component: for the clustering result of any component in each component, the node data in the clustering result is used as a sub-information in the target description information of the component, thereby obtaining the target description information of the component.

[0141] In this application, due to the clustering model constructed based on the hierarchical clustering analysis algorithm, small node data such as component version, component vulnerability, manufacturer (i.e. component publisher), path (i.e. component source address) are searched from top to bottom from the top to generate a hierarchical structure of different clusters of points and surfaces. Compared with related technologies, this application realizes fine-grained and accurate mining of all data in the SBOM file of the target software, completes discrete mixed points, thereby reducing the false alarm rate of detection and improving the detection accuracy.

[0142] Step 120: Based on the target description information of each component and the component knowledge set, perform software detection on the target software to obtain a detection result of the target software.

[0143] In the embodiment of the present application, when executing step 120, the participants Figure 8 As shown, for each of the above components, the following steps can be specifically performed:

[0144] Step 1201: Perform the following operations for each of the above components:

[0145] Step 12011: extracting features of target description information of any of the above components;

[0146] Step 12012: respectively comparing the feature with the features corresponding to the aforementioned plurality of standard components to obtain a component matching relationship between the component and the standard component matching the component;

[0147] Step 12013: According to the vulnerability information of the standard component in the component matching relationship, vulnerability detection is performed on the component to obtain vulnerability detection information of the component;

[0148] Step 1202: Based on the component matching relationship and vulnerability detection information of the above components, a detection result of the target software is generated.

[0149] In some embodiments of the present application, by executing steps 1201 to 1202, a feature comparison can be performed between a full component knowledge base formed by component detailed information of multiple standard components pre-acquired through a crawler and target description information of each component extracted from the SBOM file of the target software, thereby identifying each component used in the target software and determining the security threat of the target software, i.e., the detection result, based on the vulnerability information of the standard components that match it.

[0150] The overall process of the software detection method based on the software bill of materials file in this application is described below with a specific implementation method.

[0151] like Fig. 9 As shown, the overall process may include but is not limited to the following steps:

[0152] Step 901: Receive the SBOM file of the target software.

[0153] In some embodiments of the present application, a user can upload an SBOM file of a target software readable by the SBOM platform, wherein the file can be a normative manifest file, and its text format can be any file format that complies with the SBOM standard, such as SPDX format, CycloneDX format, DSDX format, etc.

[0154] Step 902: Identify the file format.

[0155] In some embodiments of the present application, by executing step 902, the target file format of the SBOM file can be identified.

[0156] Step 903: Translate component information.

[0157] In some embodiments of the present application, when executing step 903, the aforementioned step 1002 may be specifically executed to obtain data with complete information by direct translation and parsing, such as but not limited to component name, component version, path, vulnerability information, etc.

[0158] Step 904: Use the component knowledge set to clean the data and re-cluster the data.

[0159] In some embodiments of the present application, when executing step 904, the standard description information of multiple standard components in the component knowledge set is used to clean the original description information of each component in the SBOM file obtained by direct translation and parsing, so as to obtain the target description information of each component. Specifically, component data with inconsistent formats, uncertain content forms, missing key information, etc. is obtained, and the component knowledge set is used to perform normalization processing and / or data completion processing, so as to complete the missing data and obtain the intermediate description information of each component.

[0160] Step 905: Perform data re-clustering processing on the intermediate description information of each component to obtain the target description information of each component.

[0161] In some embodiments of the present application, when executing step 905, a hierarchical clustering analysis algorithm is used to integrate the data after direct translation analysis + data cleaning, so as to obtain the complete data chain of each component. Among them, each node data in the complete data chain of each component is a sub-information in the target description information of the component, that is, a type of description information.

[0162] Step 906: Perform feature comparison based on the target description information of each component and the component knowledge set to obtain component matching relationships of each component, wherein each component matching relationship represents a matching relationship between the corresponding component and the standard component.

[0163] In some embodiments of the present application, when executing step 906, the component matching relationship of the above-mentioned components can be determined by using a feature comparison method based on the feature values ​​of the full component knowledge set formed by the detailed information of all standard components obtained by the crawler, and the feature values ​​of the target description information of each component extracted from the SBOM analysis.

[0164] Step 907: According to the vulnerability information of the standard component in the component matching relationship of each component, vulnerability detection is performed on each component to obtain vulnerability detection information.

[0165] Step 908: Generate a detection result of the target software based on the vulnerability detection information of each component.

[0166] In some embodiments of the present application, the detection result may be presented in the form of a report, which may include a component matching list (generated based on component matching relationships) and a vulnerability detection list (generated based on vulnerability detection results).

[0167] In this application, the missing data in the SBOM file is mined through data cleaning and data re-clustering processing, and the fine-grained data structure is integrated with the component knowledge set, the target description information of each component is traversed, and the complex data relationship is incrementally mined, so that an accurate and complete data information chain can be output after processing. This improves the accuracy of detection from the root, and lays a more solid foundation for the judgment of component vulnerabilities in the early stage, thereby improving the accuracy of SBOM detection, achieving an improvement in the accuracy of SBOM detection, and then reducing the false alarm rate of SBOM inventory analysis.

[0168] Based on the same inventive concept, see Fig.10 As shown, in an embodiment of the present application, a software detection device based on a software bill of materials file is provided, comprising:

[0169] A parsing module 1010 is used to parse the software bill of materials file of the target software to obtain original description information of each component used in the target software, wherein the original description information includes description information of at least one type of the corresponding component;

[0170] A data processing module 1020 is used to perform data processing on the original description information of each component based on a component knowledge set to obtain target description information of each component, wherein the component knowledge set includes multiple standard components and standard description information of each standard component, and the data processing includes data cleaning and data re-clustering;

[0171] The detection module 1030 is used to perform software detection on the target software based on the target description information of each component and the component knowledge set to obtain the detection result of the target software.

[0172] In some possible implementations, the parsing module 1010 is specifically used to:

[0173] Identifying the file format of the software bill of materials file of the target software to obtain the target file format of the software bill of materials file;

[0174] Based on the data structure of the target file format, the software bill of materials file is parsed to obtain original description information of each component used in the target software.

[0175] In some possible implementations, the data processing module 1020 is specifically used to:

[0176] Based on the standard description information of the multiple standard components, data cleaning is performed on the original description information of at least one component among the components to obtain intermediate description information of the components, wherein the data cleaning includes at least one of normalization processing and data completion processing;

[0177] Data re-clustering processing is performed on the intermediate description information of each component to obtain target description information of each component.

[0178] In some possible implementations, the data processing module 1020 is specifically used to:

[0179] For each of the components, perform the following operations:

[0180] Based on the original description information of any component among the components and the standard description information of the multiple standard components, matching a standard component matching the any component from the multiple standard components;

[0181] According to the preset types, the standard description information of the standard component is compared with the original description information of any component to determine whether data cleaning is required for the original description information of any component;

[0182] If so, based on the standard description information of the standard component, the original description information of any component is normalized and / or data-completed to obtain the intermediate description information of any component;

[0183] If not, the original description information of any component is determined as the intermediate description information of any component.

[0184] In some possible implementations, the data processing module 1020 is specifically used to:

[0185] If it is determined based on the comparison that the format of the first information in the original description information of any component is not in the reference format, the first information in the original description information of any component is replaced with information in the reference format that is the same type as the first information in the standard description information of the standard component;

[0186] and / or,

[0187] If it is determined based on the comparison that the second information is missing from the original description information of any component, the second information in the original description information of any component is completed with information of the same type as the second information in the standard description information of the standard component;

[0188] The processed description information of any component is determined as the intermediate description information of any component.

[0189] In some possible implementations, the data processing module 1020 is specifically used to:

[0190] Based on the intermediate description information of each component and the standard description information of the plurality of standard components, hierarchical clustering is performed on the intermediate description information of each component with a preset number of clusters as an iterative target of hierarchical clustering to obtain clustering results of each component;

[0191] Based on the clustering results of the components, target description information of the components is obtained respectively.

[0192] In some possible implementations, the types include some or all of component identification, component version, component developer, component license, component dependency, component encryption, and source address.

[0193] In some possible implementations, the detection module 1030 is specifically configured to:

[0194] For each of the components, perform the following operations:

[0195] Extracting features of target description information of any component among the components;

[0196] Comparing the features with the features corresponding to the plurality of standard components respectively, to obtain a component matching relationship between any one of the components and a standard component matching the any one of the components;

[0197] According to the vulnerability information of the standard component in the component matching relationship, vulnerability detection is performed on any of the components to obtain vulnerability detection information of any of the components;

[0198] Based on the component matching relationship and vulnerability detection information of each component, a detection result of the target software is generated.

[0199] See also Fig.11 As shown, an electronic device is provided in an embodiment of the present application, and the electronic device can implement the functions of the aforementioned method, referring to Fig.11 , the electronic device may include:

[0200] At least one processor 111, and a memory 112 connected to the at least one processor 111. The specific connection medium between the processor 111 and the memory 112 is not limited in the embodiment of the present application. Fig.11 In the example, the processor 111 and the memory 112 are connected via the bus 110. The bus 110 is Fig.11 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 110 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Fig.11 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 111 can also be called a controller, and there is no limitation on the name.

[0201] In the embodiment of the present application, the memory 112 stores instructions that can be executed by at least one processor 111. The at least one processor 111 can perform the method discussed above by executing the instructions stored in the memory 112. The processor 111 can implement the functions of the aforementioned device.

[0202] In one possible design, the processor 111 may include one or more processing units, and the processor 111 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 111. In some embodiments, the processor 111 and the memory 112 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0203] The processor 111 may be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0204] The memory 112 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 112 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 112 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 112 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0205] By designing and programming the processor 111, the code corresponding to the method described in the above embodiment can be fixed into the chip, so that the chip can execute the steps of the method of the above embodiment when running. How to design and program the processor 111 is a technology well known to those skilled in the art and will not be described here.

[0206] Based on the same inventive concept, an embodiment of the present application provides a computer-readable non-volatile storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above embodiments.

[0207] In some possible implementations, various aspects of the above methods provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a device, the program code is used to enable the device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification.

[0208] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0209] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0210] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0211] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0212] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A software detection method based on a software bill of materials file, characterized in that: include: Parsing a software bill of materials file of the target software to obtain original description information of each component used in the target software, wherein the original description information includes description information of at least one type of the corresponding component; Based on the component knowledge set, data processing is performed on the original description information of each component to obtain the target description information of each component, the component knowledge set includes multiple standard components and standard description information of each standard component, and the data processing includes data cleaning and data re-clustering; Based on the target description information of each component and the component knowledge set, software detection is performed on the target software to obtain a detection result of the target software.

2. The method according to claim 1, characterized in that The software bill of materials file of the target software is parsed to obtain original description information of each component used in the target software, including: Identifying the file format of the software bill of materials file of the target software to obtain the target file format of the software bill of materials file; Based on the data structure of the target file format, the software bill of materials file is parsed to obtain original description information of each component used in the target software.

3. The method according to claim 1, characterized in that The process of performing data processing on the original description information of each component based on the component knowledge set to obtain the target description information of each component includes: Based on the standard description information of the multiple standard components, data cleaning is performed on the original description information of at least one component among the components to obtain intermediate description information of the components, wherein the data cleaning includes at least one of normalization processing and data completion processing; Data re-clustering processing is performed on the intermediate description information of each component to obtain target description information of each component.

4. The method according to claim 3, characterized in that The step of performing data cleaning on original description information of at least one of the components based on the standard description information of the multiple standard components to obtain intermediate description information of the components includes: For each of the components, perform the following operations: Based on the original description information of any component among the components and the standard description information of the multiple standard components, matching a standard component matching the any component from the multiple standard components; According to the preset types, the standard description information of the standard component is compared with the original description information of any component to determine whether data cleaning is required for the original description information of any component; If so, based on the standard description information of the standard component, the original description information of any component is normalized and / or data-completed to obtain the intermediate description information of any component; If not, the original description information of any component is determined as the intermediate description information of any component.

5. The method according to claim 4, characterized in that The step of performing normalization processing and / or data completion processing on the original description information of any component based on the standard description information of the standard component to obtain the intermediate description information of any component includes: If it is determined based on the comparison that the format of the first information in the original description information of any component is not in the reference format, the first information in the original description information of any component is replaced with information in the reference format that is the same type as the first information in the standard description information of the standard component; and / or, If it is determined based on the comparison that the second information is missing from the original description information of any component, the second information in the original description information of any component is completed with information of the same type as the second information in the standard description information of the standard component; The processed description information of any component is determined as the intermediate description information of any component.

6. The method according to claim 3, characterized in that The performing data re-clustering processing on the intermediate description information of each component to obtain the target description information of each component includes: Based on the intermediate description information of each component and the standard description information of the plurality of standard components, hierarchical clustering is performed on the intermediate description information of each component with a preset number of clusters as an iterative target of hierarchical clustering to obtain clustering results of each component; Based on the clustering results of the components, target description information of the components is obtained respectively.

7. The method according to claim 3, characterized in that The types include some or all of component identification, component version, component developer, component license, component dependency, component encryption, and source address.

8. The method according to any one of claims 1 to 7, characterized in that: The performing software detection on the target software based on the target description information of each component and the component knowledge set to obtain the detection result of the target software includes: For each of the components, perform the following operations: Extracting features of target description information of any component among the components; Comparing the features with the features corresponding to the plurality of standard components respectively, to obtain a component matching relationship between any one of the components and a standard component matching the any one of the components; According to the vulnerability information of the standard component in the component matching relationship, vulnerability detection is performed on any of the components to obtain vulnerability detection information of any of the components; Based on the component matching relationship and vulnerability detection information of each component, a detection result of the target software is generated.

9. An electronic device, characterized in that: include: Memory, used to store computer programs or instructions; A processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 8 is performed.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 8.

Images