A method and system for security detection of artifacts in an artifact repository

Abstract

The application relates to a method for security detection of software products in a product library, comprising the following steps: encapsulating a business data standardization data interface development kit according to an interface document provided by different product management software; querying a security problem related data source according to a unique characteristic value of a product to obtain security data of the corresponding product; taking a product package required by a detection tool when performing problem diagnosis as a file input source of the detection tool, downloading or distributing the product package to a corresponding scanning position, and detecting security data results of the product by the detection tool; and calling an interface of the data interface development kit to backfill attribute values of the security data of the product in a key-value pair form. The application uses a product characteristic value matching query mode, so that a process of obtaining product security problems becomes very simple, and a detection tool does not need to be actually purchased or used, and implementation cost is extremely low for a user with basic security requirements.

Description

Technical Field

[0001] This invention relates to the field of software security technology, and in particular to a method and system for security testing of artifacts in a software artifact library. Background Technology

[0002] In software development and delivery, artifacts typically refer to the results generated by program code, configuration files, and other build tools, such as binary files, dependency packages, library files, and Docker images. These artifacts, as the fruits of developers' labor, become an important component of application deployment and production. Artifact repository software is a tool used to store, manage, and distribute software project artifacts (such as binary files, source code, and documentation). It is commonly used in automated software building and deployment processes to repeatedly generate the same artifacts and quickly distribute them to different environments as needed. Companies typically use commonly used artifact management software such as Artifactory or Nexus to build their own artifact management platform (referred to as a private server) on their intranet to uniformly manage third-party open-source artifact packages created during internal project iterations or pulled from the network. Most artifact management software provides robust management support for artifact packages from common package managers such as Maven, Npm, and PyPI.

[0003] Meanwhile, with software security issues becoming increasingly severe, numerous open-source software artifact packages are constantly being exposed for various security vulnerabilities; even artifact packages developed internally by companies have various security problems. If artifact packages with security vulnerabilities are blindly introduced and used in the project development process without diagnosis, it will inevitably lead to significant security risks in the developed applications. Many current artifact package network mirror repositories, such as Maven's mirror repository, label the corresponding versions of artifacts with their corresponding security vulnerabilities to help users choose more secure artifact versions. However, artifact packages are now managed centrally by artifact management software. How to quickly and effectively diagnose the security of artifact packages has become an important requirement and function of artifact management today. For this reason, many artifact management software programs now provide artifact security testing tools adapted to their own artifact management, such as Xray (required by Artifactory), Nexus IQ (required by Nexus), and Dependency Check (required by Apache Archiva). Generally, using the security testing tools provided by the artifact management software itself can also help users perform basic diagnoses of security issues in the artifacts stored in the artifact repository. However, this approach of utilizing the artifact repository's own security testing tools has too many limitations in scenarios with stringent security requirements, mainly including the following aspects: 1. While most artifact management software is free or partially free, its associated diagnostic tools often require payment. For users, diagnosing security issues of corresponding components in the artifact repository requires additional costs; 2. The diagnostic tools of the artifact management software itself, based on product positioning and functional characteristics, may not necessarily meet the differentiated needs of users in diagnosing artifact security issues, such as whether they can diagnose security vulnerabilities, detect license compliance issues, or perform binary testing; 3. The diagnostic tools and artifact management software are too closely interdependent, making migration and adaptation difficult. If users need to change their artifact management software due to business requirements, it also means changing their testing tools. If the business scenario has too much coupled business logic based on the previous testing tools, it will lead to migration difficulties. Some companies also use various types of third-party security testing tools, such as Black Duck and Snyk. They can use security testing plugins developed by third-party independent testing tools for the corresponding product management software. By loading the plugin package into Artifactory or Nexus services, they can indirectly access the testing capabilities of independent testing tools.However, this plug-in-based detection capability integration still faces many practical problems. For example, extension plugins are developed by various security detection tools to adapt to artifact management software based on their own detection capabilities. If a user's security detection tool does not have an officially supported extension plugin, and the user wants to apply the capabilities of this detection tool to artifacts in the artifact management software, then a custom plugin must be developed, which undoubtedly incurs huge development and maintenance costs. Moreover, these plugins are not portable; different detection plugins need to be developed and maintained for different artifact management software. Furthermore, due to the limitations of plugin integration, plug-in-based detection tools have limited access to their own detection capabilities, failing to fully utilize the powerful detection capabilities of the detection tool itself.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for security testing of artifacts in a software artifact library, thereby solving the problems existing in the prior art.

[0006] The objective of this invention is achieved through the following technical solution: a method for security testing of artifacts in a software artifact library, the method comprising:

[0007] S1. A data interface development toolkit that encapsulates standardized business data for different product management software interfaces;

[0008] S2. Based on the unique characteristic value of the product, query the relevant data source for safety issues to obtain the safety data of the corresponding product;

[0009] S3. Using the data interface development toolkit, the product package required by the testing tool for problem diagnosis is used as the file input source of the testing tool. It is downloaded or distributed to the corresponding scanning location, and the safety data results of the product are detected by the testing tool.

[0010] S4. Call the data interface development toolkit interface to backfill the attribute values ​​of the product's safety data in the form of key-value pairs.

[0011] The data interface development toolkit enables the following functions: parsing and obtaining relevant data of products in the product library; uploading and downloading product files; adding additional attributes or modifying attribute values ​​for products with specified characteristics or all products under a specified product library by calling the interface; and filtering and querying products based on business conditions.

[0012] The methods for obtaining data sources related to security issues in step S2 include: using the network mirror repository of each type of manager to query the security data of relevant products by using the feature values ​​of the products through interface calls; or querying the security vulnerability data of the corresponding products based on the feature values ​​related to the product name; or using the data query interface provided by the security issue knowledge base maintained by the detection tool itself to query the security issue-related data of the corresponding products by using the parsed feature names of the products.

[0013] The S3 step specifically includes the following:

[0014] For the connected detection tools, a command-line driver script is set up to control the input and output of commands. The driver script realizes the logical connection of each execution command step of the specific detection tool. It determines whether to execute the next command based on the success or failure of the previous scan command, thus realizing the process control of tool scanning.

[0015] The business platform controls the input of testing tool parameters and the scheduling and execution of scanning tasks. When the source file is needed in the internal process of the driver script, the corresponding product file is downloaded through the standard API interface of the product management data interface development toolkit. The testing command of the subsequent steps is called in sequence according to the testing process to scan and analyze the product file. After the analysis is completed, the driver script judges whether the preceding process has been executed successfully. After all the processes have been executed successfully, the result data of the safety analysis of the testing tool is written in a fixed format to a temporary file named after the product feature.

[0016] Once the driver script is executed and the preceding processes are deemed successful, the detection results are read from a temporary file named after the product characteristics and written to an intermediate temporary file for storage. After the business platform successfully calls the driver script, it reads the detection result file to obtain the detection result data.

[0017] A system for security inspection of artifacts in a software artifact library includes: a data interface development kit encapsulation module, an artifact feature value matching module, an inspection scanning module, and an attribute value backfilling module;

[0018] The data interface development toolkit encapsulation module is used to encapsulate standardized business data into a data interface development toolkit for different product management software interface documents.

[0019] The product feature value matching module is used to query relevant data sources based on the unique feature value of the product to obtain the safety data of the corresponding product.

[0020] The detection and scanning module is used to download or distribute the product package required by the detection tool for problem diagnosis as the file input source of the detection tool through the data interface development toolkit, and detect the safety data results of the product through the detection tool.

[0021] The attribute value backfilling module is used to call the data interface development toolkit interface to backfill the attribute values ​​of the product's safety data in the form of key-value pairs.

[0022] The detection and scanning module includes: a driver script setting unit, a detection tool scheduling and execution unit, and a detection result acquisition unit;

[0023] The driver script setting unit is used to set up a command-line driver script for the connected detection tool to control the input and output of commands. The driver script realizes the logical connection of each execution command step of the specific detection tool, and decides whether to execute the next command based on the success or failure of the previous scan command, thereby realizing the process control of tool scanning.

[0024] The detection tool scheduling and execution unit is used by the business platform to control the input of detection tool parameters and the scheduling and execution of scanning tasks. When the source file is needed in the internal process of the driver script, the corresponding product file is downloaded through the standard API interface of the product management data interface development toolkit. The detection command of the subsequent steps is called in sequence according to the detection process to scan and analyze the product file. After the analysis is completed, the driver script judges whether the preceding process has been executed successfully. After all the processes have been executed successfully, the result data of the safety analysis of the detection tool is written in a fixed format into a temporary file named after the product feature.

[0025] The detection result acquisition unit is used to read the detection results in a temporary file named after the product characteristics after the execution of the driver script is completed and the preceding processes are judged to be executed successfully, and write the detection results into an intermediate temporary file for storage. After the business platform successfully calls the driver script, it reads the detection result file to obtain the detection result data.

[0026] The data interface development toolkit enables the following functions: parsing and obtaining relevant data of products in the product library; uploading and downloading product files; adding additional attributes or modifying attribute values ​​for products with specified characteristics or all products under a specified product library by calling the interface; and filtering and querying products based on business conditions.

[0027] This invention offers the following advantages: A method and system for security testing of artifacts in a software artifact library. By standardizing and encapsulating the open interfaces of various artifact management software, the interface implementation of different artifact management software is unified. Upper-layer business processes are decoupled from specific artifact management software, and data interaction is achieved indirectly through a toolkit. Business changes require modifications to the artifact management software or the integration of new artifact management software data, while the upper-layer business processes require minimal changes, thus enabling flexible switching of artifact management software. Utilizing artifact feature value matching queries simplifies the process of identifying artifact security issues, eliminating the need to purchase or use actual testing tools. For users with basic security needs, the implementation cost is extremely low. By customizing and backfilling some artifact attribute data, the usage scenarios of artifacts by business users can be flexibly controlled. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the method of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided below with reference to the accompanying drawings is not intended to limit the scope of protection of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The present invention will be further described below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, this invention completely breaks free from the testing constraints of the related testing tools that the product management software itself relies on, completely decoupling the product management software from the product safety testing, allowing the management software to focus on product management itself, while the product safety testing is handled by one or more more specialized testing tools of different types; specifically including the following:

[0031] Step 1: Interface Encapsulation: For different artifact management software (Artifactory, Nexus, etc.), encapsulate standardized data interface development kits (SDKs) for business data based on the interface documentation provided. This SDK should have at least the following basic functionalities:

[0032] 1. Able to parse and obtain product-related data in the product library, including but not limited to: the name of the repository to which the product belongs, storage time, storage path, package manager type, product name, product file type, product file size, unique feature value - feature value extracted according to the characteristics of the package manager or product file hash value, etc.

[0033] For example, if an artifact stored in the Maven repository named Test-M in Artifactory has the following coordinates, its feature value can be obtained by directly taking its file hash value (e.g., MD5 value 599b8ba07d1d04f0ea34414e861d7ad1 or SHA1 value 03b254c872b95141751f414e353a25c2ac261b51), or by constructing the artifact feature value as log4j:log4j:1.2.14 using the coordinates. This feature value serves as a unique identifier for the artifact across all network distribution channels and can be used for subsequent data lookups. Artifacts for other package managers are handled similarly.

[0034] For example, in a PyPI repository named Test-P within Nexus, if an artifact with the storage path `packages / aiohttp / 3.8.2 / aiohttp-3.8.2-cp36-cp36m-win_amd64.whl` is stored, its feature value can be directly obtained from its file hash value, such as the MD5 value `8cf16dfd9967942da565529602f9f282` or the SHA1 value `6a35d4f45b297b4f39a9983d5b3e0f15c24c05901`. Alternatively, its artifact and version name can be parsed to obtain the artifact's feature value as `aiohttp:3.8.2`. Artifacts under other artifact management software and their corresponding package managers can be parsed using the package manager's feature parsing method.

[0035] 2. Capable of uploading and downloading product files;

[0036] 3. It can perform operations such as adding, modifying or deleting attributes of products. That is, by calling the SDK interface, it can add additional attributes or modify attribute values ​​for products with specified characteristics or all products under a specified product warehouse.

[0037] 4. Able to filter and query products based on business conditions, including but not limited to product warehouse name, product name, product version, product release time, product file size, product attribute value, etc.

[0038] Step 2: Basic Scan Based on Product Feature Value Matching: Based on the unique feature value of the product (such as the product's package manager feature representation, the product file's hash value, etc.), query existing security issue-related data sources to obtain the corresponding product's security data. This method requires the support of product security-related data sources.

[0039] The methods for obtaining security-related data sources include:

[0040] 1. Utilize the network mirror repositories of various package managers, since they generally have security-related data for the corresponding products. Security data for the relevant products can be obtained by querying the characteristic values ​​of the products through interface calls.

[0041] 2. By utilizing the security data query interface provided by a professional online security website, you can query data such as security vulnerabilities of the corresponding product based on the characteristic values ​​related to the product name;

[0042] 3. Utilize the security issue database maintained by the detection tool itself. For example, users of Black Duck can access a massive knowledge base of security issues maintained by the detection tool. By using the data query interface provided by the knowledge base and the parsed product's characteristic name, they can also query relevant data on the security issues of the corresponding product.

[0043] This method of using product feature matching can easily obtain product security data without relying on the detection capabilities of the detection tools themselves, and can basically meet the diagnostic needs of general business scenarios for product security issues. However, for some products developed by the business itself, because there is no product-related security data source support, simple feature matching alone is unlikely to find relevant security issues, and actual detection of the products by detection tools is necessary.

[0044] Step 3: Deep Scanning Based on the Testing Tool: Utilizing the standardized product management software data interface, the product packages required for problem diagnosis by the testing tool are used as the file input source for the testing tool. These packages are downloaded or distributed to the corresponding scanning locations as needed according to the testing process. In this invention, to facilitate the universality and convenience of testing tool access, the following process is adopted to achieve testing tool access and scan result acquisition:

[0045] A1. For the connected testing tools, develop command-line driver scripts for the testing tools using scripting languages ​​such as shell or Python. For example, if the business platform connects to the product testing tool A, the developed command-line driver script is Tool-A.sh.

[0046] Almost all detection tools provide corresponding scanning command lines. Through custom driver scripts, it's possible to control the input and output of these command lines, encapsulate differences and details in scan execution, and achieve various effects such as business decoupling. Internally, the driver script logically connects the various execution command steps of the specific detection tool (general detection tools typically include stages such as source acquisition, source file parsing, security analysis, and result processing). The driver script determines whether to proceed to the next command line execution based on the success or failure of the previous scan command (the success / failure indicators are provided by the diagnostic tool's native commands; for example, many commands output 0 to indicate success, and other non-zero numbers indicate failure or anomalies). This allows for very simple control of the tool's scanning process.

[0047] A2. The business platform controls the input of detection tool parameters and the scheduling and execution of scanning tasks. For example, if the business platform needs to access detection tool A to perform product detection on a certain product B, then when the business platform calls the detection tool driver script Tool-A.sh, it passes in the main command-line parameters such as the detection tool's authentication token, the feature value of product B, and other auxiliary scanning tool parameters, etc., and then directly calls the server-side driver script through the business platform service to start the scanning process for product B. Different programming languages ​​support calling the driver script at the language level. For example, in a business platform system developed in Java, the service runtime can execute the driver script program passed to the exec method via Runtime.exec(); in Go, the os / exec package can be used to execute shell commands or scripts.

[0048] Within the driver script's internal workflow, when source files are needed (i.e., the source file scanning and acquisition step), the corresponding artifact file is downloaded by calling the standard API interface of the artifact management SDK. Subsequently, the driver script sequentially calls the detection commands of subsequent steps to scan and analyze the artifact file. After analysis, the driver script internally determines that all preceding processes have executed successfully. Then, it writes the security analysis results from the detection tool into a temporary file named after the artifact's characteristics, using a fixed format (the format depends on the data characteristics of the specific detection tool being accessed). The business platform then retrieves the artifact's detection results by reading this temporary file.

[0049] A3. Product Inspection Result Acquisition: The business platform service completes this by executing the driver script. Upon successful output, the system reads the inspection results from a temporary file named after the product's characteristics and writes the scan results to an intermediate temporary file for storage. After successfully calling the driver script, the business platform reads the inspection result file to obtain the inspection result data. The business platform then performs further processing on the corresponding product's result data, such as backfilling attribute values ​​or performing other business operations.

[0050] Step 4: Attribute Value Backfilling: Regardless of whether the detection is done through basic scanning or by detection tools, after identifying the safety-related data corresponding to the artifact, the corresponding artifact is assigned safety result data using the SDK interface encapsulated in Step 1. Different artifact management tools provide support for the development and extension of artifact attributes. For example, Artifactory has a Properties attribute column, and Nexus has an Artifacts attribute column. These attribute columns annotate some basic attributes of the artifact in the form of key-value pairs, such as the MD5 value of the artifact file, artifact name, artifact path, artifact entry time, artifact last download time, etc. Different artifact management software have slight differences in basic attributes, but the business meaning is the same. The API interfaces they provide can also meet the user's needs to extend the business attributes of the artifact (basic attributes are the attributes of the artifact itself and cannot be modified). The standard SDK package provided by this invention encapsulates the implementation details of adding and modifying specific attributes of different artifacts, providing users with standardized operation interfaces. Business users only need to call the SDK interface to add, modify, or delete artifact attributes in the form of key-value pairs.

[0051] In this invention, the attribute backfilling step is not limited to adding attribute values ​​to the security data of the scanned product in the manner described above. It can also include user-defined business data, such as assigning a blacklist attribute to a product, whether or not to allow downloading, etc. Attributes can also be changed or removed according to business changes, such as removing the blacklist attribute of a product, or modifying its blacklist / whitelist attribute to whitelist, etc.

[0052] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method of security testing artifacts in a repository of software artifacts, characterized by: The method includes: S1. A data interface development toolkit that encapsulates standardized business data for different product management software interfaces; S2. Based on the unique characteristic value of the product, query the relevant data source for safety issues to obtain the safety data of the corresponding product; S3. Using the data interface development toolkit, the product package required by the testing tool for problem diagnosis is used as the file input source of the testing tool. It is downloaded or distributed to the corresponding scanning location, and the safety data results of the product are detected by the testing tool. S4. Call the data interface development toolkit interface to backfill the attribute values ​​of the product's safety data in the form of key-value pairs; The S3 step specifically includes the following: For the connected detection tools, a command-line driver script is set up to control the input and output of commands. The driver script implements the logical chaining of each execution command step of the detection tool, and determines whether to execute the next command based on the success or failure of the previous scan command, thereby realizing the flow control of the tool scan. Each execution command step includes source acquisition, source file parsing, security analysis, and result processing. The business platform controls the input of testing tool parameters and the scheduling and execution of scanning tasks. When the source file is needed in the internal process of the driver script, the corresponding product file is downloaded through the standard API interface of the product management data interface development toolkit. The testing command of source file parsing and security analysis is called in sequence according to the testing process to scan and analyze the product file. After the analysis is completed, the driver script internally judges whether the source file parsing and security analysis have been executed successfully. After both are executed successfully, the result data of the security analysis of the testing tool is written in a fixed format to a temporary file named with the unique feature value of the product. Once the driver script is executed and the source file parsing and security analysis are both successful, the detection results are read from a temporary file named after the unique feature value of the product. The detection results are then written to an intermediate temporary file for storage. After the business platform successfully calls the driver script, it reads the detection result file to obtain the detection result data.

2. The method for security testing of artifacts in a software artifact library according to claim 1, characterized in that: The data interface development toolkit implements the following functions: parsing and obtaining relevant data of products in the product library; uploading and downloading product files; adding additional attributes or modifying attribute values ​​for products with specified unique feature values ​​or all products under a specified product warehouse by calling the interface; and filtering and querying products based on business conditions.

3. The method for security testing of artifacts in a software artifact library according to claim 1, characterized in that: The methods for obtaining data sources related to security issues in step S2 include: using the network mirror repository of each type of manager to query the security data of relevant products by using the feature values ​​of the products through interface calls; or querying the security vulnerability data of the corresponding products based on the feature values ​​related to the product name; or using the data query interface provided by the security issue knowledge base maintained by the detection tool itself to query the security issue-related data of the corresponding products by using the parsed feature names of the products.

4. A system for security testing of artifacts in a software artifact library, characterized in that: It includes: The data interface development toolkit includes a packaging module, a product feature value matching module, a detection and scanning module, and an attribute value backfilling module. The data interface development toolkit encapsulation module is used to encapsulate standardized business data into a data interface development toolkit for different product management software interface documents. The product feature value matching module is used to query relevant data sources based on the unique feature value of the product to obtain the safety data of the corresponding product. The detection and scanning module is used to download or distribute the product package required by the detection tool for problem diagnosis as the file input source of the detection tool through the data interface development toolkit, and detect the safety data results of the product through the detection tool. The attribute value backfilling module is used to call the data interface development toolkit interface to backfill the attribute values ​​of the product's safety data in the form of key-value pairs. The detection and scanning module includes: a driver script setting unit, a detection tool scheduling and execution unit, and a detection result acquisition unit; The driver script setting unit is used to set up a command-line driver script for the connected detection tool to control the input and output of commands. The driver script realizes the logical connection of each execution command step of the detection tool, and determines whether to execute the next command based on the success or failure of the previous scan command, thereby realizing the flow control of the tool scan. Each execution command step includes scanning source acquisition, source file parsing, security analysis, and result processing. The detection tool scheduling and execution unit is used by the business platform to control the input of detection tool parameters and the scheduling and execution of scanning tasks. When the source file is needed in the internal process of the driver script, the corresponding product file is downloaded through the standard API interface of the product management data interface development toolkit. The detection commands for source file parsing and security analysis are called sequentially according to the detection process to scan and analyze the product file. After the analysis is completed, the driver script internally judges whether the source file parsing and security analysis have been executed successfully. After both are executed successfully, the result data of the security analysis of the detection tool is written in a fixed format into a temporary file named with the unique feature value of the product. The detection result acquisition unit is used to read the detection result from a temporary file named after the unique feature value of the product after the execution of the driver script is completed and the source file parsing and security analysis are both successfully executed. The detection result is then written into an intermediate temporary file for storage. After the business platform successfully calls the driver script, it reads the detection result file to obtain the detection result data.

5. The system for security detection of artifacts in a software artifact library according to claim 4, characterized in that: The data interface development toolkit implements the following functions: parsing and obtaining relevant data of products in the product library; uploading and downloading product files; adding additional attributes or modifying attribute values ​​for products with specified unique feature values ​​or all products under a specified product warehouse by calling the interface; and filtering and querying products based on business conditions.

Images