A method and system for identifying software open source code components

Abstract

This application provides a method and system for identifying open-source software code components, belonging to the field of open-source software code component detection technology. The method includes: acquiring the source code of the software under test to obtain a first source code file; extracting configuration file metadata and source code file metadata from the first source code file and performing a first preprocessing step to obtain a first dependent component directed acyclic graph and a second dependent component directed acyclic graph; then performing a second preprocessing step to obtain relational islands; using a depth-first traversal algorithm to fill in the relational islands to obtain giant islands; acquiring all open-source code and establishing a knowledge base; inputting the giant islands into an analysis engine of the corresponding computer language for reading and analysis; inputting the reading and analysis results into the knowledge base for filtering of related data; and generating an identification result report according to the SBOM format. This method can assist users in identifying the risk surface of their own software assets, controlling the impact scope of component vulnerabilities, and avoiding intellectual property risks associated with using third-party component libraries.

Description

Technical Field

[0001] This invention belongs to the field of software open source code component detection technology, and particularly relates to a method and system for identifying software open source code components. Background Technology

[0002] Software composition analysis is a method for managing the security of open-source components used in applications. Its purpose is to allow development teams to quickly track and analyze open-source components introduced into a project. Software composition analysis tools can also discover all relevant components, supporting libraries, and their direct and indirect dependencies. Furthermore, they can detect software licenses, deprecated dependencies, vulnerabilities, and potential threats. The scanning process generates a Bill of Materials (BOM), providing a complete inventory of the project's software assets. Software composition is a crucial indicator of software supply chain security; however, currently, there is no comprehensive and highly accurate statistical solution for software composition analysis products on the market.

[0003] A publicly available method and system for screening and identifying open-source components in source code based on simhash (application number: CN202210337119.2) includes: constructing a basic source code library; using the simhash algorithm to parse the source code in each open-source file in the basic source code library to obtain a data matching table; using the same parsing method as the open-source files to parse each source code file in the source code component under test to obtain several second digital groups; matching the strings in the second digital groups with the strings in each first digital group in the data matching table; determining whether there is a string in any first digital group that is the same as the current second digital group, and if so, defining the first digital group as a candidate digital group; and finding several open-source components related to the source code component under test in the basic source code library based on the candidate digital groups.

[0004] This solution is based on the simhash algorithm for identifying open-source components. The simhash algorithm is used to identify open-source code components copied from others. However, open-source components are not only identified by copying other people's source code, but also by direct references to configuration files. The simhash algorithm cannot analyze the metadata of configuration files. Therefore, the analysis method of this solution is not comprehensive enough and cannot perform a comprehensive analysis and identification of open-source components in the software, resulting in inaccurate identification results. Summary of the Invention

[0005] This application provides a method and system for identifying open-source software components, aiming to solve the problem mentioned above that the analysis of open-source software components is not comprehensive enough, and can only analyze source code metadata but not configuration file metadata.

[0006] To achieve the above objectives, the present invention adopts the following technical solution, including:

[0007] Obtain the source code of the software under test to get the first source code file. The acquisition methods include GIT, SVN, TFS and Mercurial.

[0008] Extract and preprocess the configuration file metadata and source code file metadata of the first source code file to obtain the first dependent component directed acyclic graph and the second dependent component directed acyclic graph.

[0009] A second preprocessing step is performed on the first dependent component directed acyclic graph and the second dependent component directed acyclic graph to obtain relation islands;

[0010] The depth-first search algorithm is used to fill in the gaps in the isolated relationships, resulting in a giant island.

[0011] Obtain all open-source source code and build a knowledge base. Input the giant island into the analysis engine of the corresponding computer language for reading and analysis, and input the reading and analysis results into the knowledge base for filtering of related data. Generate a recognition result report according to the SBOM format.

[0012] As preferred methods, Git can be obtained using either a personal access token or a username and password.

[0013] The personal access token method is as follows: access the homepage and subpages of the GIT repository main site address, extract the features of the homepage and subpages respectively, and use the SHA1 algorithm to form feature values. Substitute the feature values ​​into the knowledge base for SHA value comparison to obtain the type and version data of the GIT repository. Based on the type and version data, determine the correct interface of the access token, and then obtain the source code of the software under test in the GIT repository.

[0014] The username and password method is as follows: a web crawler is built based on the type and version data, and the source code of all software repositories to be tested in Git is obtained through the web crawler.

[0015] Preferably, the first source code file is subjected to extraction of configuration file metadata and source code file metadata, and a first preprocessing is performed to obtain a first dependent component directed acyclic graph and a second dependent component directed acyclic graph, including:

[0016] The first source code file is filtered for files with special characteristics using a directory traversal algorithm and a file type recognition algorithm. The metadata of the files is then subjected to feature extraction and content cleaning to obtain the first component information. Each component in the first component information is then used as a point in the graph, and the dependency relationship is a directed edge. A directed acyclic graph of the first dependent components is then constructed. The special characteristics include XML file format, JSON file format, INI file format, TEXT file format, and LOCK file format.

[0017] The code in the first source code file is divided into fixed granularity to obtain comparison code units. The granularity matrix is ​​constructed by extracting the metric values ​​from the comparison code units. The vector similarity algorithm is used to substitute the granularity matrix into the knowledge base to calculate the vector similarity. The components in the knowledge base whose similarity meets the threshold are summarized to obtain the second component information. Each component in the second component information is used as a point in the graph, and the dependency relationship is a directed edge. A directed acyclic graph of the second dependent components is constructed. The metric values ​​include code variables, keywords, variable loop counts, parameters, return values, constants and symbols.

[0018] As preferred, vector similarity algorithms include:

[0019]

[0020] Among them, Sim CM This is the similarity value between the granularity matrix and the corresponding matrix in the knowledge base. CM1[i] is the i-th vector in the matrix of the knowledge base, CM2[match(i)] is the i-th matching vector in the granularity matrix, A is the vector in CM1, B is the vector in CM2, α is the angle between vectors A and B, n and m are natural numbers, Similarity(A,B) is the similarity between vectors A and B, balance is the average, SimThres is the similarity input value, a is the measure value of the keyword, b is the measure value of the symbol, CompareResult is the calculation result, Sim... a Sim is the similarity value between the keyword matrix and the corresponding matrix in the knowledge base. b This represents the similarity value between the symbol matrix and the corresponding matrix in the knowledge base.

[0021] Preferably, a second preprocessing step is performed on the directed acyclic graph of the first dependent component and the directed acyclic graph of the second dependent component to obtain relation islands, including:

[0022] Determine whether there are duplicate components in the first dependent component directed acyclic graph and the second dependent component directed acyclic graph;

[0023] If so, the directed acyclic graph with repeating components is merged using the depth-first traversal algorithm, the recursive algorithm, and the edge shrinking algorithm to obtain the first processed graph;

[0024] If not, no processing is performed, and the second processed image is obtained directly;

[0025] By combining the first and second processing graphs, we obtain the relational islands.

[0026] As a preferred method, a depth-first search algorithm is used to fill in the gaps in the isolated relational islands, resulting in a giant island, which includes:

[0027] The root node in the relational island is determined, and the other nodes connected to the root node are identified using a depth-first traversal algorithm to obtain the first identifier. The first identifier is then substituted into the knowledge base to search for dependency data and obtain the third component information.

[0028] The nodes corresponding to the first identifier are supplemented according to the information of the third component to obtain new points. The user parameter value is obtained and it is determined whether to supplement the new points according to the user parameter value. If yes, the new points are traversed using the depth-first traversal algorithm. If no, the new points are removed to obtain the final giant island.

[0029] As a preferred method, the giant island is substituted into the analysis engine of the corresponding computer language for reading and analysis, specifically:

[0030] The first source code file is processed by removing strings, symbols, functions, and variables. The main language feature values ​​are generated according to declaration statements, member access symbols, pointer symbols, closure symbols, and package reference keywords. The main language feature values ​​are then compared with the knowledge base to obtain the main language type of the first source code file.

[0031] After determining the analysis engine based on the main language type, the giant island is substituted into the determined analysis engine for reading and analysis.

[0032] A system for identifying components in open-source software code, comprising:

[0033] Source code acquisition module: used to acquire the source code of the software under test and obtain the first source code file. Acquisition methods include GIT, SVN, TFS and Mercurial.

[0034] Directed Acyclic Graph (DAG) Module: This module is used to extract configuration file metadata and source code file metadata from the first source code file and perform the first preprocessing to obtain the first dependent component DAG and the second dependent component DAG.

[0035] Relationship island generation module: used to perform a second preprocessing on the first dependent component directed acyclic graph and the second dependent component directed acyclic graph to obtain relationship islands;

[0036] Giant Island Generation Module: Used to fill in the gaps in relational isolated islands using a depth-first traversal algorithm to obtain giant islands;

[0037] Open Source Code Component Identification Module: This module is used to acquire all open source code and build a knowledge base. It then inputs the giant island into the analysis engine of the corresponding computer language for reading and analysis, and inputs the reading and analysis results into the knowledge base for filtering of related data. Finally, it generates an identification result report in SBOM format.

[0038] A software open-source code component identification system includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a software open-source code component identification method as described in any one of the above.

[0039] A computer-readable storage medium storing a computer program, which, when executed by a computer, implements a method for identifying open-source software code components as described in any one of the above descriptions.

[0040] The present invention has the following beneficial effects:

[0041] (1) It supports multiple ways to obtain the first source code file of the software, which effectively reduces the workload before software component analysis and improves development efficiency;

[0042] (2) It can help users sort out the risk aspects of their own software assets, control the impact of component vulnerabilities, and avoid intellectual property risks brought about by using third-party component libraries;

[0043] (3) Provide a standard Software Bill of Materials (SBOM) to visualize the risks;

[0044] (4) Unique component analysis algorithms, directed graph generation algorithms, and multi-dimensional information recognition algorithms. Compared with existing technologies on the market, this solution can identify both the source code metadata and configuration file metadata of the software, making the identification of open-source components of the software more comprehensive, with a higher recognition rate, a lower false alarm rate, better accuracy, and richer information display;

[0045] (5) The metric extraction method and similarity algorithm involved in this solution can be extended to code clone detection, and can be used not only to identify open source components. Therefore, this solution is more valuable and practical.

[0046] (6) The calculation of similarity in this scheme involves a granularity issue. If the granularity is too large, the accuracy will be low, and if it is too small, the speed of value generation will be low. It is an algorithm that can control the balance between speed and accuracy. The conventional simhash algorithm generates hash values ​​very quickly, but the accuracy is not high. It cannot adjust the speed and accuracy. Therefore, the similarity algorithm used in this scheme can customize the control of speed and accuracy compared with the conventional one, and has better applicability. Attached Figure Description

[0047] Figure 1 This is a flowchart of a method for identifying components of open-source software code according to the present invention.

[0048] Figure 2 This is a flowchart of the component similarity calculation algorithm in this invention.

[0049] Figure 3 This is an example diagram of the giant island in the present invention.

[0050] Figure 4 This is a flowchart of the scheme in Embodiment 1 of the present invention.

[0051] Figure 5 This is a schematic diagram of the structure of a software open-source code component identification system according to the present invention.

[0052] Figure 6 This is a schematic diagram of an electronic device structure for a software open-source code component identification system according to the present invention. Detailed Implementation

[0053] Example 1

[0054] like Figure 1 As shown, a method for identifying components in open-source software code includes the following steps:

[0055] S11. Obtain the source code of the software under test to get the first source code file. The acquisition methods include GIT, SVN, TFS and Mercurial.

[0056] S12. Extract the configuration file metadata and the source code file metadata from the first source code file and perform the first preprocessing to obtain the first dependent component directed acyclic graph and the second dependent component directed acyclic graph.

[0057] S13. Perform a second preprocessing on the first dependent component directed acyclic graph and the second dependent component directed acyclic graph to obtain relation islands;

[0058] S14. Use the depth-first search algorithm to fill in the isolated relational islands to obtain the giant island;

[0059] S15. Obtain all open-source source code and establish a knowledge base. Input the giant island into the analysis engine of the corresponding computer language for reading and analysis, and input the reading and analysis results into the knowledge base for filtering of related data. Generate a recognition result report according to the SBOM format.

[0060] One implementation of the above scheme is as follows:

[0061] 1. Obtain the first source code file of the self-developed software being tested. File acquisition supports both proactive acquisition from code repositories and user-uploaded files. To support subsequent operations, a knowledge base is established. This knowledge base contains confidential data from each security company, a database created by security personnel using technical means. Data sources (not all): well-known source code repositories such as GitHub, Bitbuck, and Gitee; CVE (Common Vulnerability Disclosure), CWE (Common Weakness Enumeration Database), NVD (National Vulnerability Database), CNVD (China National Information Security Vulnerability Sharing Platform), and CNNVD (China National Information Security Vulnerability Database), etc. This solution aggregates all available open-source source code and components from existing databases, stores them in the database, and establishes a knowledge base to support the analysis of the open-source components of the self-developed software. The knowledge base established in this solution contains the following information: component feature vectors, component feature matrices, component-associated vulnerabilities, and component-associated licenses.

[0062] 2. Source code is obtained from code repositories, including methods such as Git, SVN, TFS, and Mercurial. SVN, TFS, and Mercurial utilize existing official commands for retrieval, which are standard technologies and will not be elaborated upon in this solution. For obtaining the full source code of a Git repository, there are two methods: personal access tokens and username / password. These two methods are unique to this solution. The method for obtaining the full source code of a Git repository using a personal access token is as follows: Since the methods for obtaining personal access tokens differ between Git repositories, this solution proceeds as follows: First, access the Git repository's main website homepage and subpages to extract page features. This feature extraction includes multi-dimensional information such as image extraction, keyword extraction, and NLP page content processing. The SHA1 algorithm is used to generate feature values, which are compared with the SHA value in the knowledge base to determine the type and version of the Git repository. This determines the correct interface for obtaining the access token, and then the source code of the software under test in the Git repository (this source code is the "first source code file") is obtained. Because the feature extraction is multi-dimensional, the accuracy rate reaches 98%. The method for obtaining the full source code of a Git repository using a username and password is as follows: Use the same method as above to determine the repository version to identify the accurate Git repository version. Use the corresponding version of the web crawler (i.e., "web crawler") to crawl the source code. Since the Git repository pages are different, it is necessary to develop a web crawler to solve the problem of obtaining various repositories. The source code obtained by the web crawler is also the "first source code file".

[0063] 3. After obtaining the first source code file, it is necessary to extract the configuration file metadata to form a component dependency graph. (Metadata is data that describes data, mainly information about the attributes of code-dependent components, used to support functions such as application name, dependent component version, compilation options, test cases, etc. Metadata is a kind of electronic catalog. To achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby assisting in data retrieval.) The method is as follows: Traverse the configuration files of the source code package management tool. This involves using directory traversal and file type recognition algorithms to find files with special characteristics, such as: xml, json, ini, text, lock, and other common configuration file formats. Perform feature extraction and content cleaning on the metadata of the above files to ensure that the metadata is all component information (the extracted and cleaned metadata is the "first component information"). Then, treat each component as a point in the graph, and the dependency relationship as a directed edge. Finally, transform all the data into a directed acyclic graph of dependent components (the method for handling cycles is rather brute-force; currently, brute-force deduplication is performed on the points on the paths where cycles occur). This directed acyclic graph is the "first dependent component directed acyclic graph".

[0064] 4. After obtaining the first source code file, extract the source code file metadata, generate source code metrics, and compare these metrics with the knowledge base to obtain dependent components, forming a component dependency graph. The method is as follows: Divide the code into comparison code units of fixed granularity (code functions), extract metrics from these units, and the metrics include: code variables, keywords, variable loop counts, parameters, return values, constants, symbols, etc. A granularity matrix is ​​formed based on these metrics. Then, a vector similarity value is calculated between this matrix and a pre-generated matrix in the knowledge base (the similarity algorithm uses the vector similarity algorithm from the Boreas algorithm). Components that reach the similarity threshold are considered to contain this component. The above method yields a complete set of dependent components (the complete set of dependent components is the "second component information"). Simultaneously, a directed acyclic graph of dependent components is generated according to method 3; this directed acyclic graph is the "second dependent component directed acyclic graph".

[0065] The process of processing the first source code file and establishing a directed acyclic graph of dependent components in steps 3 and 4 is the "first preprocessing" process.

[0066] The algorithm process for this step is as follows: Figure 2As shown, the code is divided into comparison unit code with fixed granularity (top left of the figure). The underlined part represents the metrics extracted from the comparison unit code. Metrics include: code variables, keywords, variable loop counts, parameters, return values, constants, symbols, etc. The tables in the figure are vector matrices, representing the quantity, keyword, and symbol matrices, respectively. The bottom right chart CM2 shows the generated metric matrix (i.e., the "granularity matrix"), and CM1 is the matrix pre-generated for the open-source component in the knowledge base (i.e., the "corresponding matrix in the knowledge base"). The similarity between the two matrices is calculated using the algorithm formula below the chart. The vector similarity algorithm includes:

[0067]

[0068] Among them, Sim CM This is the similarity value between the granularity matrix and the corresponding matrix in the knowledge base. CM1[i] is the i-th vector in the matrix of the knowledge base, CM2[match(i)] is the i-th matching vector in the granularity matrix, A is the vector in CM1, B is the vector in CM2, α is the angle between vectors A and B, n and m are natural numbers, Similarity(A,B) is the similarity between vectors A and B, balance is the average, SimThres is the similarity input value, a is the measure value of the keyword, b is the measure value of the symbol, CompareResult is the calculation result, Sim... a Sim is the similarity value between the keyword matrix and the corresponding matrix in the knowledge base. b This represents the similarity value between the symbol matrix and the corresponding matrix in the knowledge base.

[0069] CM1 and CM2 distinguish multidimensional matrices. The steps have been explained and are shown in the diagram. Each row mathematically represents a multidimensional vector, which we simply call V. Taking a row from CM1 is A, and taking a row from CM2 is B. This gives us two vectors. For example, a two-dimensional vector consists only of (x, y). A directed line from the origin to (x, y) is called a vector. Similarly, A and B are both vectors and are multidimensional. Since they both originate from the origin, the two lines must have an angle between them. This angle is α. If α is 0, the vectors are the same, i.e., cos(α) = cos(0) = 1. In multidimensional space, cos(α) is Similarity(A,B).

[0070] A i Let B be the i-th value of vector A. iLet CM1 and CM2 be the i-th value of vector B. Both CM1 and CM2 are multidimensional. Multiplying the Similarity(A,B) values ​​obtained from their matching vectors gives us SimCM. In other words, we substitute all the A and B vectors in the two matrices into the second formula above to find the Similarity(A,B) value. Then, we find all the Similarity(A,B) values ​​in the matrix and multiply these Similarity(A,B) values ​​together to get SimCM. This process is what the first formula above means.

[0071] Sim a and Sim b The numerical value is calculated using the same principle as SimCM; simply replace the relevant parameters in the first and second formulas.

[0072] Next, the remaining three formulas are used sequentially to substitute numerical values ​​and calculate the final (CompareResult) result, which is the vector similarity value. Then, it is determined whether the value reaches the similarity threshold (the threshold is a dynamically adjusted value; the larger the number of learning samples, the more reasonable the threshold becomes). If it does, it is considered that the component is included.

[0073] The components in the knowledge base that meet the calculated threshold are aggregated to obtain all components whose similarity to the source code file metadata in the first source code file meets the threshold. These components are called "second component information".

[0074] 5. Based on the methods in steps 3 and 4, two directed acyclic graphs (i.e., "directed acyclic graph of the first dependent component and directed acyclic graph of the second dependent component") are obtained. It is then determined whether there are duplicate points (components) in the two graphs. If there are duplicate points, the two graphs are merged. The algorithm mainly involves depth-first traversal, recursion, and edge contraction. If the two graphs do not have duplicate points, no further processing is performed. In this scheme, this phenomenon is called the multi-starting-point case, and each graph is referred to as a relational island.

[0075] For example: the directed graph data before merging is only obtained from the metadata, and there may be multiple data sources, such as three sources named A, B, and C.

[0076] Since graphs A and B have common points, they represent a relationship and need to be merged into a single graph, called D. This results in two graphs, C and D. D is the merged graph (the "first processed graph"), while C cannot be merged because it has no common points with other graphs (C is the "second processed graph"). Before the points are added, both C and D are considered relationship islands.

[0077] The process described above for processing a directed acyclic graph to obtain relational islands is the "second preprocessing" process.

[0078] 6. Next, fill in the gaps in all relational islands. The method is as follows: The relational islands in this step only contain directly dependent components. In this solution, the dependency tracing is customized according to user parameters, thus providing a maximum of 5 levels. The process of filling in indirect dependencies will be described next; other levels can be deduced similarly. Starting from the root node of the directed graph obtained above, a depth-first search algorithm is used. When each node is reached, its unique identifier (the unique identifier is the "first identifier") is extracted. Based on the identifier, the dependency data is searched in the knowledge base to obtain the set of directly dependent components of that node (the set of directly dependent components is the "third component information"). Based on this set, the node is filled in. Direct dependencies are represented as new nodes, and dependencies are represented as edges. The user parameter value determines whether to fill in the new node. If yes, the depth-first search algorithm is used to traverse the new node; otherwise, the new node is removed and no traversal is performed. Finally, a richer directed acyclic graph is obtained, which is called a giant island in this solution. The final image of the giant island is as follows. Figure 3 As shown.

[0079] Continuing with the example in step 5, the graphs obtained by adding points to C and D are called E and F. The structure of E and F looks like... Figure 3 In this diagram, E and F are referred to as giant islands.

[0080] 7. After obtaining the giant island (at least one), it is necessary to determine the primary language of the first source code file, and then submit the giant island data to the corresponding language analysis engine. The method is as follows: Extract the metadata of the first source code file, remove strings, symbols, functions, variables, etc., leaving declaration statements, member access symbols, pointer symbols, closure symbols, and package reference keywords. Generate feature values ​​(the "primary language feature values") based on the above five dimensions. The specific algorithm for generating these feature values ​​is as follows: sort the strings by letters, numbers, and special symbols; sort characters of the same type by ASCII code; finally, perform MD5 to obtain the feature values, compare them with the feature values ​​in the knowledge base, and determine the primary language of the first source code file. After obtaining the corresponding language, the giant island can be submitted to the corresponding language analysis engine for analysis.

[0081] While the analysis engines differ across languages, their underlying logic remains largely the same. This section focuses on Java as an example, but the same logic applies to other languages. After receiving the Giant Island analysis engine, it reads the information contained in the nodes of the directed graph. Each node contains the following information: unique component identifier, component version, and call location. Based on the unique component identifier and version, it searches the knowledge base for associated vulnerabilities, dependencies, and open-source license information. Vulnerability types include CVE, CNVD, and CNNVD vulnerability numbers; open-source licenses include over 1400 open-source license types; dependencies are the node's direct dependencies and the set of all parent components that depend on the node (the parent components that directly depend on the node are not just the set in Giant Island, but the entire open-source ecosystem). Based on all the data obtained, a report is generated according to the SBOM standard format; this report is the "Identification Result Report." The overall process is as follows... Figure 4 As shown.

[0082] This concludes the identification process, yielding a giant island diagram, vulnerability set, license set, component relationship diagram, and software inventory report.

[0083] Example 2

[0084] like Figure 5 As shown, a software open-source code component identification system includes:

[0085] Source code acquisition module 10: used to acquire the source code of the software under test and obtain the first source code file. Acquisition methods include GIT, SVN, TFS and Mercurial.

[0086] Directed Acyclic Graph (DAG) Module 20: This module is used to extract configuration file metadata and source code file metadata from the first source code file and perform a first preprocessing step to obtain a first dependent component DAG and a second dependent component DAG.

[0087] Relationship island generation module 30: used to perform a second preprocessing on the first dependent component directed acyclic graph and the second dependent component directed acyclic graph to obtain relationship islands;

[0088] Giant Island Generation Module 40: Used to fill in the gaps in relational islands using a depth-first traversal algorithm to obtain giant islands;

[0089] Open source code component identification result module 50: It is used to obtain all open source code and build a knowledge base, input the giant island into the analysis engine of the corresponding computer language for reading and analysis, input the reading and analysis results into the knowledge base for filtering of related data, and generate an identification result report according to the SBOM format.

[0090] One implementation of the above system is as follows: In the source code acquisition module 10, the source code of the software under test is acquired to obtain a first source code file. The acquisition methods include GIT, SVN, TFS, and Mercurial. In the directed acyclic graph (DAG) building module 20, the first source code file is subjected to extraction of configuration file metadata and source code file metadata and first preprocessing to obtain a first dependent component DAG and a second dependent component DAG. In the relationship island generation module 30, the first dependent component DAG and the second dependent component DAG are subjected to second preprocessing to obtain relationship islands. In the giant island generation module 40, the relationship islands are filled with points using a depth-first traversal algorithm to obtain giant islands. In the open source code component identification result module 50, all open source code is acquired and a knowledge base is built. The giant islands are substituted into the analysis engine of the corresponding computer language for reading and analysis. The reading and analysis results are substituted into the knowledge base for filtering of related data, and an identification result report is generated according to the SBOM format.

[0091] Example 3

[0092] Based on the above embodiments, this embodiment provides an electronic device, such as... Figure 6 As shown.

[0093] Example 4

[0094] Based on the above embodiments, this embodiment provides a storage medium.

[0095] The above description is only a specific embodiment of the present invention, but the technical features of the present invention are not limited thereto. Any changes or modifications made by those skilled in the art within the scope of the present invention are covered by the patent scope of the present invention.

Claims

1. A method for identifying components of open-source software code, characterized in that, include: Obtain the source code of the software under test to get the first source code file. The acquisition methods include GIT, SVN, TFS and Mercurial. Extract and preprocess the configuration file metadata and source code file metadata of the first source code file to obtain the first dependent component directed acyclic graph and the second dependent component directed acyclic graph. A second preprocessing step is performed on the first dependent component directed acyclic graph and the second dependent component directed acyclic graph to obtain relation islands; The depth-first search algorithm is used to fill in the gaps in the isolated relationships, resulting in a giant island. Obtain all open-source source code and build a knowledge base. Input the giant island into the analysis engine of the corresponding computer language for reading and analysis, and input the reading and analysis results into the knowledge base for filtering of related data. Generate a recognition result report according to the SBOM format. A second preprocessing step is performed on the directed acyclic graph of the first dependent component and the directed acyclic graph of the second dependent component to obtain relation islands, including: Determine whether there are duplicate components in the first dependent component directed acyclic graph and the second dependent component directed acyclic graph; If so, the directed acyclic graph with repeating components is merged using the depth-first traversal algorithm, the recursive algorithm, and the edge shrinking algorithm to obtain the first processed graph; If not, no processing is performed, and the second processed image is obtained directly; By combining the first and second processing graphs, we obtain relational islands; using a depth-first search algorithm, we fill in the gaps in these relational islands to obtain giant islands, including: The root node in the relational island is determined, and the other nodes connected to the root node are identified using a depth-first traversal algorithm to obtain the first identifier. The first identifier is then substituted into the knowledge base to search for dependency data and obtain the third component information. The nodes corresponding to the first identifier are supplemented according to the information of the third component to obtain new points. The user parameter value is obtained and it is determined whether to supplement the new points according to the user parameter value. If yes, the new points are traversed using the depth-first traversal algorithm. If no, the new points are removed to obtain the final giant island.

2. The method for identifying components of open-source software code according to claim 1, characterized in that, There are two ways to obtain Git: personal access token and username / password. The personal access token method is as follows: access the homepage and subpages of the GIT repository main site address, extract the features of the homepage and subpages respectively, and use the SHA1 algorithm to form feature values. Substitute the feature values ​​into the knowledge base for SHA value comparison to obtain the type and version data of the GIT repository. Based on the type and version data, determine the correct interface of the access token, and then obtain the source code of the software under test in the GIT repository. The username and password method is as follows: a web crawler is built based on the type and version data, and the source code of all software repositories to be tested in Git is obtained through the web crawler.

3. The method for identifying components of open-source software code according to claim 1, characterized in that, The first source code file undergoes extraction and preprocessing of both configuration file metadata and source code file metadata to obtain a first dependent component directed acyclic graph and a second dependent component directed acyclic graph, including: The first source code file is filtered for files with special characteristics using a directory traversal algorithm and a file type recognition algorithm. The metadata of the files is then subjected to feature extraction and content cleaning to obtain the first component information. Each component in the first component information is then used as a point in the graph, and the dependency relationship is a directed edge. A directed acyclic graph of the first dependent components is then constructed. The special characteristics include XML file format, JSON file format, INI file format, TEXT file format, and LOCK file format. The code in the first source code file is divided into fixed granularity to obtain comparison code units. The granularity matrix is ​​constructed by extracting the metric values ​​from the comparison code units. The vector similarity algorithm is used to substitute the granularity matrix into the knowledge base to calculate the vector similarity. The components in the knowledge base whose similarity meets the threshold are summarized to obtain the second component information. Each component in the second component information is used as a point in the graph, and the dependency relationship is a directed edge. A directed acyclic graph of the second dependent components is constructed. The metric values ​​include code variables, keywords, variable loop counts, parameters, return values, constants and symbols.

4. The method for identifying components of open-source software code according to claim 3, characterized in that, Vector similarity algorithms include: ； Among them, Sim CM This is the similarity value between the granularity matrix and the corresponding matrix in the knowledge base. CM1[i] is the i-th vector in the matrix of the knowledge base, CM2[match(i)] is the i-th matching vector in the granularity matrix, A is the vector in CM1, B is the vector in CM2, a is the angle between vector A and vector B, n and m are natural numbers, Similarity(A,B) is the similarity between vector A and vector B, balance is the average, SimThres is the similarity input value, a is the measure value of the keyword, b is the measure value of the symbol, CompareResult is the calculation result, Sim a Sim is the similarity value between the keyword matrix and the corresponding matrix in the knowledge base. b This represents the similarity value between the symbol matrix and the corresponding matrix in the knowledge base.

5. The method for identifying components of open-source software code according to claim 1, characterized in that, The giant island is then fed into the analysis engine of the corresponding computer language for reading and analysis, specifically as follows: The first source code file is processed by removing strings, symbols, functions, and variables. The main language feature values ​​are generated according to declaration statements, member access symbols, pointer symbols, closure symbols, and package reference keywords. The main language feature values ​​are then compared with the knowledge base to obtain the main language type of the first source code file. After determining the analysis engine based on the main language type, the giant island is substituted into the determined analysis engine for reading and analysis.

6. A system for identifying components of open-source software code, characterized in that, To implement a method for identifying software open-source code components as described in any one of claims 1-5, the method includes: Source code acquisition module: used to acquire the source code of the software under test and obtain the first source code file. Acquisition methods include GIT, SVN, TFS and Mercurial. Directed Acyclic Graph (DAG) Module: This module is used to extract configuration file metadata and source code file metadata from the first source code file and perform the first preprocessing to obtain the first dependent component DAG and the second dependent component DAG. Relationship island generation module: used to perform a second preprocessing on the first dependent component directed acyclic graph and the second dependent component directed acyclic graph to obtain relationship islands; Giant Island Generation Module: Used to fill in the gaps in relational isolated islands using a depth-first traversal algorithm to obtain giant islands; Open Source Code Component Identification Module: This module is used to acquire all open source code and build a knowledge base. It then inputs the giant island into the analysis engine of the corresponding computer language for reading and analysis, and inputs the reading and analysis results into the knowledge base for filtering of related data. Finally, it generates an identification result report in SBOM format.

7. A system for identifying components of open-source software code, characterized in that, The system includes a memory and a processor, the memory being used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method for identifying open-source software code components as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it implements a method for identifying open-source software code components as described in any one of claims 1-5.

Images