A multi-type scanning tool source code index statistics method, a storage medium and a terminal

Abstract

The present application relates to a kind of multi-type scanning tool source code index statistics method, storage medium and terminal, comprising: integrated development design general index statistics module and diagnostic tool drive module, the general index item required by index statistics module on server to tool integration application platform is carried out statistics;When user triggers diagnostic task, create task, diagnostic task is bound with diagnostic tool drive module and general index statistics module;Diagnostic tool drive module executes diagnostic tool command, and after completing diagnostic tool command, the general index item of statistics to code is executed, finally data is assembled into the data format required by tool and is sent to integration application platform.The present application embodiment is simple, only needs to utilize existing professional tool in the server of source code, can complete the acquisition of general index data, does not need complex interface call;And index acquisition function can be plugged, can be flexibly controlled according to the general index acquisition required by platform business demand.

Description

Technical Field

[0001] This invention relates to the field of software security technology, and in particular to a method for statistical analysis of source code indicators of multiple types of scanning tools, a storage medium, and a terminal. Background Technology

[0002] In the field of software security, there are various code security-related diagnostic tools, such as Component Analysis (SCA) tools that focus on detecting software components, Static Analysis and Testing (SAST) tools that focus on source code problem analysis, and Dynamic Analysis and Testing (DAST) tools that focus on detecting program vulnerabilities. This diversity of tool types has led to the development of corresponding tool integration application platforms, which integrate various types of tools to meet users' diagnostic needs for various code problems. The diagnostic data from these platforms can be divided into tool-related diagnostic indicators and some tool-independent general indicators. General indicators typically include source file size, number of lines of source code, number of lines of comments, and cyclomatic complexity of the source code. Some diagnostic tools can provide these indicators, while others cannot or do not provide comprehensive data. Therefore, a method for statistically analyzing general source code indicators is needed to meet the requirements of tool integration application platforms for displaying general indicators and for secondary business development.

[0003] However, existing implementation methods have the following shortcomings: 1. The tool integration application platform obtains general indicators through diagnostic tools, but the data from different diagnostic tools may not meet some key indicator items required by the tool integration application platform, and the platform's indicators are highly dependent on diagnostic tools, making business expansion inflexible; 2. Since the general indicators of the tool integration application platform come from different diagnostic tools, the platform needs to connect to data interfaces for different tools. However, the large number of tools and the different interface data formats lead to high maintenance costs; 3. The general indicator data of the tool integration application platform comes from tools. If platform users do not use certain diagnostic tools, general data items will be missing; 4. Data conflicts occur because the general data items come from diagnostic tools. Different diagnostic tools may have different statistical results for some indicators of the same source code, posing a challenge to the tool integration application platform in integrating conflicting data.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for statistical analysis of source code indicators of multi-type scanning tools, a storage medium, and a terminal, thereby solving the deficiencies of existing methods.

[0006] The objective of this invention is achieved through the following technical solution: a method for statistical analysis of source code metrics in multi-type scanning tools, the method comprising:

[0007] Step 1: Integrate and develop a general indicator statistics module and a diagnostic tool driver module. Execute the indicator statistics module on the server to perform statistics on the general indicator items required by the tool integration application platform. The indicator statistics driver module encapsulates command-line parameters for various general indicators required for statistical business. The diagnostic tool driver module encapsulates scan command-line parameters related to various diagnostic tools.

[0008] Step 2: When a user triggers a diagnostic task, a task is created. In the task creation project, creation parameters are passed to bind the diagnostic task with the diagnostic tool driver module and the general indicator statistics module.

[0009] Step 3: The diagnostic tool driver module executes the diagnostic tool command, and after completing the diagnostic tool command, it executes the general indicator statistics module to complete the statistics of the general indicator items of the code. After the general indicator statistics module completes its execution, it assembles the data into a data format that meets the requirements and sends it to the tool integration application platform.

[0010] When a user triggers a diagnostic task, a task is created. Creation parameters are passed within the task creation process, and the diagnostic task is bound to the diagnostic tool driver module and the general indicator statistics module. Specifically, this includes:

[0011] After a user selects diagnostic tool A through the tool integration application platform to diagnose code problems, a diagnostic task is triggered. The tool integration application platform creates a task named Job-A in the task orchestration tool through an internal service program. During the task creation process, when the tool integration application platform passes creation parameters to Job-A, it also transmits various business parameters required for command execution as input parameters for the diagnostic tool driver module and / or general indicator statistics module.

[0012] On the execution node of the created task Job-A, bind the diagnostic tool driver module and the general indicator statistics module in sequence according to the business order.

[0013] The simultaneous transmission of various business parameters required for command execution as input parameters for the diagnostic tool driver module and / or general indicator statistics module includes: for the diagnostic tool driver module, the source code address to be scanned by diagnostic tool A or the source code retrieval address, as well as the necessary tool connection information of diagnostic tool A; for the general indicator statistics module, the code path, platform callback address, and access token information.

[0014] The diagnostic tool driver module executes diagnostic tool commands including:

[0015] The diagnostic tool driver module logically connects the command lines that execute the source code scanning steps of the diagnostic tool's internal implementation program according to the type of the diagnostic tool, and encapsulates the diagnostic command steps that need to be called multiple times within the diagnostic tool driver module.

[0016] Of course, when the initial parameters of the external input command line are sent to the diagnostic tool driver module, the diagnostic tool driver module judges the flag of the previous step command. If the flag of the previous step command is a success flag, the next step command is executed; otherwise, the diagnosis ends.

[0017] The general indicator statistics module performs statistics on the general indicator items of the code, including:

[0018] The general indicator statistics module encapsulates and links different indicator statistics commands. It calls the script program and passes in initial parameters such as the source code path and platform callback address. The script program processes the data logic of each statistical indicator, and the execution results of commands for different indicator data items do not affect each other.

[0019] The successfully collected data is appended to a standard format and output to the agreed server path. After all indicator commands are executed, the script sends the statistical data to the tool integration application platform via a callback address, thus completing the collection and statistics of general indicator items.

[0020] During the execution of Job-A after it is triggered, the Job-A orchestration task is executed on the server side. When this task is executed, the diagnostic tool driver module and the general indicator statistics module, which are driven on the same service host, are executed in sequence.

[0021] A computer-readable storage medium having a computer program stored thereon, wherein the steps of the source code index statistics method are executed by a processor.

[0022] A terminal device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the source code indicator statistics method.

[0023] This invention offers the following advantages: a method, storage medium, and terminal for statistical analysis of source code metrics in multi-type scanning tools; decoupling the acquisition of general metrics from specific diagnostic tools; regardless of whether the diagnostic tool itself supports the acquisition of certain general metric data items, the tool integration application platform can accurately and standardly statistically analyze the general metrics required for the platform's business; furthermore, it can integrate other specialized metric statistical tools according to the platform's own expansion needs, providing platform users with more dimensions of general code metric data; the implementation is simple, requiring only the use of existing professional tools on the server where the source code resides to complete the acquisition of general metric data, without the need for complex interface calls; the metric acquisition function is pluggable, allowing flexible control over the acquisition of required general metrics according to the platform's business needs; the tool integration application platform uses professional general metric detection tools as standard data, eliminating data differences, conflicts, and missing data caused by different diagnostic tools. Attached Figure Description

[0024] Figure 1 This is a schematic flowchart of the method of the present invention;

[0025] Figure 2 This is a logical diagram of the internal workings of the utility script Cloc.py;

[0026] Figure 3 This is a schematic diagram of the diagnostic steps for diagnostic tool A. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided below with reference to the accompanying drawings is not intended to limit the scope of protection of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The present invention will be further described below with reference to the accompanying drawings.

[0028] like Figure 1 As shown, one embodiment of the present invention specifically relates to a method for statistical analysis of source code metrics compatible with different types of scanning tools, which specifically includes the following:

[0029] S1. Based on the common metrics required by the tool integration application platform, independently develop and design a scripted program for metric statistics that is easy to execute directly on the server, such as a Python script or a Shell script. For simplicity, it is named Cloc.py. This script encapsulates various command lines that can calculate common metrics required for business operations. The command lines can be system commands or commands executed by installed professional statistical tools.

[0030] like Figure 2 As shown, for example, a business requirement might be to calculate file size, lines of code, and code comments in source code. If file size is obtained using the Linux command `du -h sourceDir`, and lines of code and comments are obtained using a third-party statistics tool called `Cloc` (executed with the command `cloc sourceDir`), then the script `Cloc.py` encapsulates and concatenates these different metric calculation commands. Calling `Cloc.py` with necessary initial parameters such as the source code path and platform callback address allows `Cloc.py` to internally process the data logic for each metric. The execution results of commands for different metric data items are independent of each other; that is, the success or failure of previous metric calculations does not affect subsequent metric calculations, thus achieving loose coupling in the architecture. Successfully calculated data is appended and assembled into a standard JSON format and output to the agreed-upon server path. After all metric commands have been executed, `Cloc.py` internally sends the statistical data to the tool's integrated application platform via the callback address, thereby completing the collection of common metrics.

[0031] When a business requires adding new statistical metrics, simply add the corresponding metric statistics tool to the server-side environment and add the command-line call step for the new metric statistics to the Cloc.py driver script. If the current business requirement is to calculate the cyclomatic complexity of the source code, a new command-line tool called lizard, specifically for calculating the cyclomatic complexity of the source code, has been added. Its script execution command is lizard sourceDir. Simply add the command related to adding cyclomatic complexity to the sequential flow within the Cloc.py script; other steps remain unchanged, thus quickly achieving the ability to dynamically expand general metric statistics. Similarly, to remove a metric, simply modify the Cloc.py script and remove the unnecessary metric statistics command-line steps. Alternatively, control parameters can be passed externally, allowing the Cloc.py internal logic to dynamically select the required metric, achieving flexible control over the acquisition of general metrics.

[0032] S2, such as Figure 3As shown, the tool integration application platform pre-develops and designs diagnostic tool driver scripts, named Tool-A.py, based on the different integrated diagnostic tools. This script encapsulates the scan command-line parameters related to the diagnostic tools. For example, if tool A's scan of a source code program involves sequential steps such as source code retrieval, compilation, scanning, problem analysis, and result uploading (the tool command's own behavior), then the driver script Tool-A.py logically links these execution steps together. It completely encapsulates the diagnostic command-line steps that need to be called multiple times within Tool-A.py. External callers only need to pass in the necessary initial command-line parameters, and Tool-A.py integrates the various execution steps, thus greatly simplifying the command-line invocation of diagnostic tools.

[0033] Because Tool-A.py internally connects the various execution steps logically, it can decide whether to proceed to the next command line execution based on the success or failure of the previous command (the success or failure indicators of the command are provided by the native commands of the diagnostic tool; for example, many commands use 0 to indicate success and other non-zero numbers to indicate failure or exception. Therefore, after calling the tool to execute the command in the Tool-A.py script program, it is only necessary to check its output result). This makes the flow control more convenient.

[0034] Because the diagnostic tool scripts are self-developed and designed, external input commands can be standardized. Regardless of the type of diagnostic tool, the tool integration application platform only needs to design one set of processes and one set of standardized parameter input command lines. The diagnostic driver script internally parses the standardized parameters into the diagnostic parameters of the diagnostic tool and executes them. This completely shields the upper-layer application from the differences in command lines of different diagnostic tools, thereby greatly reducing the repetitive coding and maintenance work when the tool integration application platform connects to different diagnostic tools.

[0035] The success or failure of a diagnostic tool script is determined by the commands of the diagnostic tool itself. For example, if the connection address entered into the diagnostic tool is incorrect or the source code file path is wrong, the diagnostic tool command will fail to execute. After the failure, it is meaningless to execute the indicator statistics tool again, and the process is interrupted. The success or failure of the indicator item commands within the general indicator script is also determined by the indicator script itself based on the input items. The success or failure of the entire indicator script depends on whether a statistical indicator file is generated.

[0036] S3. After a user selects a diagnostic tool (e.g., Tool A) to diagnose code issues using the tool integration application platform, a diagnostic task is triggered. The tool integration application platform will create a task named Job-A in the task orchestration tool (e.g., Jenkins) through its internal service program. On the script execution node of the created task Job-A (e.g., the ExecuteShell node in Jenkins), the diagnostic tool script Tool-A.py and the general metrics script Cloc.py are bound sequentially according to the business order.

[0037] S4. During the task creation process in step 3, when creating Job-A, the tool integration application platform transmits various business parameters required for command execution as input parameters to the execution script. For example, for Tool-A.py, it transmits the source code address or source code pull address (such as git_url) to be scanned by diagnostic tool A, as well as necessary tool connection information for diagnostic tool A. Similarly, for the script Cloc.py, in order to collect general code metrics, it also transmits necessary information such as the code path source_path, the platform callback address callback_Url, and the access token. This means that the execution script needs to transmit information such as the callback address, access token, and other necessary information required for the tool to perform diagnostics.

[0038] During the execution of S5 and Job-A after being triggered, the Job-A orchestration task will be executed on the server side. When this task is executed, the script execution configuration node configured in the task orchestration tool for orchestrating script execution (such as the Execute Shell script command configuration node that can be selected in the build step on the commonly used Jenkins orchestration tool) will be executed, that is, the Tool-A.py and Cloc.py on the same service host will be executed in sequence.

[0039] S6. Tool-A.py executes normal diagnostic tool commands. After the diagnostic tool commands are completed, the Cloc.py script is executed to complete the statistics of common metrics of the code. After the Cloc.py script finishes execution, it assembles the data into a standard, extensible JSON format agreed upon by the tool integration application platform and sends it to the tool integration application platform via an HTTP request. The platform then completes the subsequent business data processing.

[0040] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the source code index statistics method. The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0041] Another embodiment of the present invention relates to a terminal device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the source code indicator statistics method.

[0042] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for statistical analysis of source code metrics for multi-type scanning tools, characterized in that: The source code metric statistics method includes: Step 1: Integrate and develop a general indicator statistics module and a diagnostic tool driver module. Execute the indicator statistics module on the server to perform statistics on the general indicator items required by the tool integration application platform. The indicator statistics driver module encapsulates command-line parameters for various general indicators required for statistical business. The diagnostic tool driver module encapsulates scan command-line parameters related to various diagnostic tools. Step 2: When a user triggers a diagnostic task, a task is created. During the task creation process, creation parameters are passed, and the diagnostic task is bound to the diagnostic tool driver module and the general indicator statistics module. Step 3: The diagnostic tool driver module executes the diagnostic tool command, and after completing the diagnostic tool command, it executes the general indicator statistics module to complete the statistics of the general indicator items of the code. After the general indicator statistics module completes the execution, it assembles the data into a data format that meets the requirements and sends it to the tool integration application platform. When a user triggers a diagnostic task, a task is created. During the task creation process, creation parameters are passed, and the diagnostic task is bound to the diagnostic tool driver module and the general indicator statistics module. Specifically, this includes: After a user selects diagnostic tool A through the tool integration application platform to diagnose code problems, a diagnostic task is triggered. The tool integration application platform creates a task named Job-A in the task orchestration tool through an internal service program. During the task creation process, when the tool integration application platform passes creation parameters to Job-A, it also transmits various business parameters required for command execution as input parameters for the diagnostic tool driver module and the general indicator statistics module. On the execution node of the created task Job-A, bind the diagnostic tool driver module and the general indicator statistics module in the order of business operations; The general indicator statistics module performs statistics on the general indicator items of the code, including: The general indicator statistics module encapsulates and links different indicator statistics commands. It calls the script program and passes in initial parameters such as the source code path and platform callback address. The script program processes the data logic of each statistical indicator, and the execution results of commands for different indicator data items do not affect each other. The successfully collected data is appended to a standard format and output to the agreed server path. After all indicator commands are executed, the script sends the statistical data to the tool integration application platform via a callback address, thus completing the collection and statistics of general indicator items.

2. The method for statistical analysis of source code metrics for multi-type scanning tools according to claim 1, characterized in that: The simultaneous transmission of various business parameters required for command execution as input parameters for the diagnostic tool driver module and the general indicator statistics module includes: for the diagnostic tool driver module, the source code address or source code retrieval address to be scanned by diagnostic tool A, and the tool connection information of diagnostic tool A; for the general indicator statistics module, the code path, platform callback address, and access token information.

3. The method for statistical analysis of source code metrics for multi-type scanning tools according to claim 1, characterized in that: The diagnostic tool driver module executes diagnostic tool commands including: The diagnostic tool driver module logically connects the command lines that execute the source code scanning steps of the diagnostic tool's internal implementation program according to the type of the diagnostic tool, and encapsulates the diagnostic command steps that need to be called multiple times within the diagnostic tool driver module. Of course, when the initial parameters of the external input command line are sent to the diagnostic tool driver module, the diagnostic tool driver module judges the flag of the previous step command. If the flag of the previous step command is a success flag, the next step command is executed; otherwise, the diagnosis ends.

4. The method for statistical analysis of source code metrics for multi-type scanning tools according to claim 1, characterized in that: During the execution of Job-A after it is triggered, the Job-A orchestration task is executed on the server side. When this task is executed, the diagnostic tool driver module and the general indicator statistics module, which are driven on the same service host, are executed in sequence.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the source code indicator statistics method according to any one of claims 1-4.

6. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the source code indicator statistics method according to any one of claims 1-4.

Images