Static program safety test tool evaluation method, device and equipment

Abstract

The embodiment of the specification discloses a static program security testing tool evaluation method, device and equipment, the method comprises the following steps: receiving the evaluation request of the static program security testing tool for the preset programming language; based on the evaluation request, the syntax characteristic information of the preset programming language and the information of different dimensions of the sensitivity analysis related to the program analysis ability accuracy of the static program security testing tool are obtained; based on the syntax characteristic information of the preset programming language, a plurality of different first evaluation indexes for the evaluation completeness are constructed, and based on the different dimensions of the sensitivity analysis related to the program analysis ability accuracy, a plurality of different second evaluation indexes for the evaluation accuracy are constructed, based on each evaluation index in the plurality of first evaluation indexes and the plurality of second evaluation indexes, one or more different positive and negative sample pairs are obtained, and the static program security testing tool is respectively evaluated based on the obtained positive and negative sample pairs.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to an evaluation method, device, and equipment for a static program security testing tool. Background Art

[0002] Software is now ubiquitous. However, with its widespread adoption, software security issues are becoming increasingly serious. System crashes and security vulnerabilities, for example, can lead to significant financial losses and even threaten lives. To ensure software security and reliability, and to meet people's need to protect their private data, application security testing technologies are widely used, including Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), and Interactive Application Security Testing (IAST). These security testing technologies are used to uncover security vulnerabilities in proprietary and open source code and conduct compliance checks for intellectual property and industry standards.

[0003] Currently, there are numerous commercial and open-source static application security testing tools available, and some companies are also developing their own. However, the primary challenge facing developers and users is how to objectively measure the technical performance of these tools. Evaluating these products can provide valuable reference information for developers and users, helping them make more informed technology selection and product planning decisions. To this end, a more granular evaluation method for static application security testing tools is needed to comprehensively assess their performance. Summary of the Invention

[0004] The purpose of the embodiments of this specification is to provide a more fine-grained evaluation method for static application security testing tools, so as to comprehensively evaluate the performance of static application security testing tools.

[0005] In order to implement the above technical solution, the embodiments of this specification are implemented as follows:

[0006] An embodiment of the present specification provides an evaluation method for a static program security testing tool, the method comprising: receiving an evaluation request for a static program security testing tool for a preset programming language. Based on the evaluation request, the syntax characteristic information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool are obtained. Based on the syntax characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, a plurality of different first evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0007] The embodiment of the present specification provides an evaluation device of a static program security testing tool, the device comprises: a request receiving module, receiving an evaluation request for a static program security testing tool of a preset programming language. An information acquisition module acquires information of different dimensions of sensitivity analysis related to the program analysis ability and precision of the static program security testing tool based on the syntax characteristics information of the preset programming language. An evaluation index construction module constructs a plurality of different first evaluation indexes for the evaluation completeness based on the syntax characteristics information of the preset programming language, and constructs a plurality of different second evaluation indexes for the evaluation accuracy based on the different dimensions of the sensitivity analysis related to the program analysis ability and precision of the static program security testing tool. The plurality of different first evaluation indexes include a first evaluation index for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation on the data structure, and / or a first evaluation index for analyzing a logic feature in the program. The plurality of different second evaluation indexes include a second evaluation index for sensitivity analysis of the static program security testing tool. A tool evaluation module acquires one or more different positive and negative sample pairs matched with each evaluation index in the plurality of different first evaluation indexes and the plurality of different second evaluation indexes based on each evaluation index, and respectively evaluates the static program security testing tool based on the acquired positive and negative sample pairs to determine the evaluation result of the static program security testing tool.

[0008] An embodiment of the present specification provides an evaluation device for a static program security testing tool. The evaluation device includes: a processor; and a memory configured to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to: receive an evaluation request for a static program security testing tool in a preset programming language; obtain, based on the evaluation request, syntax characteristic information of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool; construct multiple different first evaluation indicators for evaluation completeness based on the syntax characteristic information of the preset programming language; and construct multiple different second evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The multiple different first evaluation indicators include a first evaluation indicator for analyzing and detecting data structures and / or preset operations of the data structures contained in a program written in the preset programming language, and / or a first evaluation indicator for analyzing logical features in the program; and the multiple different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0009] The embodiments of this specification also provide a storage medium for storing computer-executable instructions, which implement the following process when executed by a processor: receiving an evaluation request for a static program security testing tool for a preset programming language. Based on the evaluation request, obtaining the syntax characteristic information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. Based on the syntax characteristic information of the preset programming language, constructing a plurality of different first evaluation indicators for evaluation completeness, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, constructing a plurality of different second evaluation indicators for evaluation accuracy, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting the data structure contained in the program written in the preset programming language and / or the preset operation of the data structure, and / or a first evaluation indicator for analyzing the logical features in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0010] The embodiments of this specification also provide a computer program product, including a computer program, which implements the following process when executed by a processor: receiving an evaluation request for a static program security testing tool for a preset programming language. Based on the evaluation request, the syntax characteristic information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool are obtained. Based on the syntax characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, a plurality of different first evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting the data structure contained in the program written in the preset programming language and / or the preset operation of the data structure, and / or a first evaluation indicator for analyzing the logical features in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. Those skilled in the art can also derive other drawings based on these drawings without inventive work.

[0012] Figure 1 This is an embodiment of an evaluation method for a static program security testing tool in this specification;

[0013] Figure 2 This is a schematic diagram of the evaluation page of a static program security testing tool in this manual;

[0014] Figure 3 This is an embodiment of an evaluation device for a static program security testing tool in this specification;

[0015] Figure 4 This is an embodiment of an evaluation device for a static program security testing tool in this specification. DETAILED DESCRIPTION

[0016] The embodiments of this specification provide a method, apparatus, and device for evaluating a static program security testing tool.

[0017] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0018] The embodiments of this specification provide a federated learning mechanism. Currently, software is ubiquitous. However, with the popularization of software, software security issues are becoming increasingly serious. For example, system crashes and security vulnerabilities may cause huge economic losses or even threaten life. In order to ensure the security and reliability of software and meet people's needs for protecting their own privacy data, application security testing technologies are widely used, including static application security testing (SAST), dynamic application security testing (DAST), interactive application security testing (IAST), etc. The above security testing technologies are used to discover security vulnerabilities in private and open source codes, conduct intellectual property and industry standard compliance checks, etc.

[0019] At present, there are many commercial or open source products available for various types of static application security testing tools, and some companies are also developing related products on their own. However, the main challenge facing developers and users is how to objectively measure the technical level of various types of static application security testing tools. Evaluating the above products can provide valuable reference information for developers and users, helping them make more informed technology selection and product planning decisions. To this end, it is necessary to provide a more fine-grained evaluation method for static application security testing tools, so that the performance of static application security testing tools can be comprehensively evaluated. The embodiments of this specification can clarify the test function points corresponding to each test sample through multi-dimensional evaluation indicators and fine-grained test samples, thereby not only being able to more accurately reflect the detection capabilities of static application security testing tools and avoid ambiguous test results, but also providing detailed explanations of the test results, thereby comprehensively evaluating the performance of static application security testing tools. For specific processing, please refer to the specific content in the following embodiments.

[0020] like Figure 1As shown, an embodiment of this specification provides an evaluation method for a static program security testing tool, and the execution subject of the method can be a terminal device or a server, etc., wherein the terminal device can be a mobile terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically such as a smart watch, a car-mounted device, etc.), etc., wherein the server can be an independent server, or a server cluster composed of multiple servers, etc. The server can be a background server in the financial field or the online shopping field, etc., or a background server of an application, etc. In this embodiment, the execution subject is described in detail as an example of a server. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method can specifically include the following steps:

[0021] In step S102 , an evaluation request for a static program security testing tool for a preset programming language is received.

[0022] The preset programming language may include multiple languages. For example, the preset programming language may be Java programming language, C language, or C++ language, etc. The specific setting may be based on actual conditions and is not limited in the embodiments of this specification. Static program security testing tool, namely SAST, is a commonly used application security (AppSec) tool that can scan the source code, binary code, or byte code of an application. The static program security testing tool can identify the root cause of the vulnerability and help fix potential security vulnerabilities. The solution of the static program security testing tool SAST analyzes the application from the "inside out" without running the operating system to perform the scanning operation.

[0023] In practice, there are currently many types of static program security testing tools, such as OWASP Benchmark or Webgoat, Juliet Test Suite, SecuriBench Micro, Openrasp, etc. However, due to the lack of a systematic evaluation system guidance, the test results of the above test sample sets are relatively vague. They are basically a simple stacking of various samples and cannot provide fine-grained evaluation results. The current industry evaluation method mainly achieves comprehensive evaluation through upper-level standard evaluation or non-technical indicators, but lacks subjective stratification of the capabilities of static program security testing tools. For example, OWASP Benchmark covers a limited number of vulnerability types and evaluation results, and most of the evaluation results are recall rate and false alarm rate, which makes it difficult to fully and finely characterize the performance of the tool; Webgoat has a limited number of samples and coverage, and is mainly used for beginners to learn about security vulnerabilities, and is difficult to use to comprehensively evaluate the performance of professional tools; Juliet Test Suite has a large number of samples, but due to the single sample type, it may not be able to fully reflect the performance of the tool in a variety of scenarios; SecuriBench Micro mainly targets specific security issues, with a narrow coverage range, making it difficult to comprehensively evaluate the versatility of the tool; Openrasp has a small number of samples, a limited number of vulnerability types, and incomplete evaluation results. Based on the above content, we can design evaluation indicators from dimensions such as completeness and accuracy according to the preset programming language characteristics and SAST tool principles, and evaluate static program security testing tools using the above evaluation indicators. Specifically, Figure 2 As shown in the following example, when it is necessary to evaluate the static program security testing tool of a programming language, Figure 2 Upload the ID of the static program security testing tool in the "Information of the tool to be evaluated" information input box, and generate a corresponding evaluation request based on the ID. Subsequently, you can obtain the data of the static program security testing tool based on the ID of the static program security testing tool, or you can use Figure 2 In the "Information of the tool to be evaluated" information input box, upload the data of the static program security testing tool, and generate a corresponding evaluation request based on the data of the static program security testing tool. The specific settings can also be made according to actual conditions.

[0024] In step S104, based on the evaluation request, information on syntax characteristics of a preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of a static program security testing tool are obtained.

[0025] The syntax characteristic information of the preset programming language can include multiple types. For example, the syntax characteristic information of the preset programming language can include basic language syntax characteristic information and complex language syntax characteristic information. For example, the basic language syntax characteristic information of the Java programming language can include syntax characteristic information such as set, array, object, and the like, and the complex language syntax characteristic information can include syntax characteristic information such as control flow, exception, and the like. For another example, the basic language syntax characteristic information of the C language can include syntax characteristic information such as array, pointer, keyword, and the like, and the complex language syntax characteristic information can include flow control, and the like. The different dimensions of the sensitivity analysis related to the program analysis capability and accuracy of the static program security testing tool can include multiple types. For example, the different dimensions of the sensitivity analysis related to the program analysis capability and accuracy of the static program security testing tool can include flow sensitivity dimension, object sensitivity dimension, path sensitivity dimension, and context sensitivity dimension, and the like. The dimensions of the sensitivity analysis corresponding to different static program security testing tools can be different. In addition, in actual application, the different dimensions of the sensitivity analysis related to the program analysis capability and accuracy of the static program security testing tool can also be set according to the type of the tool. For example, the dimensions of the sensitivity analysis can be set according to the type of the OWASP Benchmark or Webgoat, or the type of the Juliet Test Suite, or the type of the Openrasp, and the like. The dimensions of the sensitivity analysis can also be set according to actual conditions.

[0026] In implementation, after receiving the evaluation request in the above manner, the data of the static program security testing tool can be obtained. In addition, in order to improve the processing efficiency, the identifier of the preset programming language can be uploaded, such as the Java programming language or the C language, and the static program security testing tool and the preset programming language can be analyzed respectively, so that the syntax characteristic information of the preset programming language and the information of the different dimensions of the sensitivity analysis related to the program analysis capability and accuracy of the static program security testing tool can be determined. Alternatively, the static program security testing tool can be analyzed to determine the information of the preset programming language corresponding to the static program security testing tool, and the information of the different dimensions of the sensitivity analysis related to the program analysis capability and accuracy of the static program security testing tool. In addition, the preset programming language can be analyzed to determine the syntax characteristic information of the preset programming language, and the like. The specific implementation can be set according to actual conditions, which is not limited in the embodiments of the present disclosure.

[0027] In step S106, a plurality of different first evaluation indicators for evaluation completeness are constructed based on the grammatical characteristic information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy are constructed based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing the logical features in the above program. The plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool.

[0028] Among them, the data structure may include multiple types, and different programming languages ​​may contain different data structures. For example, the data structures contained in the Java programming language may include arrays, collections, objects, etc., and the data structures contained in the C language may include pointers, keywords, etc., which can be set according to actual conditions. The preset operations may include multiple types, such as input operations, access operations, data protection operations, etc., which can be set according to actual conditions. Static program security testing tools use different sensitivities (such as sensitivity to flow sensitivity (i.e., flow sensitivity dimension), sensitivity to object sensitivity (i.e., object sensitivity dimension), sensitivity to path sensitivity (i.e., path sensitivity dimension), and sensitivity to context sensitivity (i.e., context sensitivity dimension)) during the analysis process, which has always been one of the important strategies to achieve accuracy-performance trade-offs. Therefore, evaluation indicators can be set for the sensitivity of static program security testing tools.

[0029] In practice, usually, vulnerability-based test sample sets are generally based on vulnerability types as the evaluation perspective, and test samples of different vulnerability types constitute the test sample set. Then, different types of static program security testing tools are evaluated on the same set of test sample sets. However, the principles of different types of static program security testing tools are different, making it difficult for a set of test sample sets to be used universally among different types of static program security testing tools, otherwise it will affect the rationality of the test results. Therefore, in response to the above situation, it is necessary to change the evaluation perspective and determine different evaluation items (i.e., evaluation indicators) based on different dimensions of sensitivity analysis related to specific programming languages ​​and the accuracy of program analysis capabilities of static program security testing tools (or the principles of static program security testing tools or the tool types of static program security testing tools, etc.), so that the design of evaluation items and test sample sets is more reasonable. Ultimately, an evaluation system for static program security testing tools for preset programming languages ​​can be designed from multiple dimensions such as completeness, accuracy, and performance.

[0030] In the completeness dimension, the syntax feature information of the preset programming language and the principle of the static program security testing tool can be preset, and one or more aspects such as a regular taint link (i.e., a basic language feature) and a special taint link (i.e., a complex language feature) are selected to set the evaluation indicators. On the one hand, by modeling the basic language features (which can include syntax features), a plurality of different first evaluation indicators for evaluating completeness are constructed, such as initialization, access, and specific method operations of sets, arrays, objects, and the like. The analysis and detection capabilities of the above data structures and operations contained in the program written in the preset programming language can be evaluated, and potential security problems such as input verification, access control, and data protection can be revealed. On the other hand, modeling of complex syntax features such as control flow and exceptions is used to construct first evaluation indicators for evaluating completeness, so that the analysis capabilities of the logical features in the above program can be evaluated, which is also crucial to the correctness and security of the written program. Through the above two aspects, more different first evaluation indicators for evaluating completeness can be constructed.

[0031] In addition, in the accuracy dimension, based on the kernel-sensitivity analysis of the static analysis capability of the static program security testing tool, the precision analysis capability of the static program security testing tool can be further measured, the evaluation precision can be improved, and based on this, a plurality of different second evaluation indicators for evaluating accuracy, including second evaluation indicators for sensitivity analysis of the static program security testing tool, can be constructed based on different dimensions of the sensitivity analysis of the program analysis capability precision of the static program security testing tool.

[0032] In step S108, based on each of the plurality of different first evaluation indicators and the plurality of different second evaluation indicators, one or more different pairs of positive and negative samples matched with each evaluation indicator are obtained, and the static program security testing tool is respectively evaluated based on the obtained pairs of positive and negative samples to determine the evaluation results of the static program security testing tool.

[0033] In implementation, after obtaining multiple different first evaluation indicators and multiple different second evaluation indicators in the above manner, for any one of the above multiple evaluation indicators (including multiple different first evaluation indicators and multiple different second evaluation indicators) (which can be a first evaluation indicator, or, it can also be a second evaluation indicator), one or more different positive and negative sample pairs (i.e., a positive sample and a negative sample) matching the evaluation indicator can be obtained from a specified database, or, corresponding positive and negative sample pairs can be generated for the evaluation indicator (specifically, corresponding positive and negative sample pairs can be generated through a specified algorithm, or corresponding positive and negative sample pairs can be generated through a pre-trained model or a large model, etc., which can be set specifically according to actual conditions). Through the above method, positive and negative sample pairs corresponding to other evaluation indicators can be obtained or generated. Finally, one or more different positive and negative sample pairs matching each evaluation indicator can be obtained. The obtained positive and negative sample pairs can be used as test samples, and the static program security testing tool can be evaluated for different evaluation indicators respectively through the test samples. For example, the test indicators include test indicator 1, test indicator 2 and test indicator 3. In addition, one or more different positive and negative sample pairs matching test indicator 1, one or more different positive and negative sample pairs matching test indicator 2 and one or more different positive and negative sample pairs matching test indicator 3 can be obtained. One or more different positive and negative sample pairs that match test indicator 1 are used to evaluate the static program security testing tool to determine whether the static program security testing tool supports test indicator 1. One or more different positive and negative sample pairs that match test indicator 2 are used to evaluate the static program security testing tool to determine whether the static program security testing tool supports test indicator 2. One or more different positive and negative sample pairs that match test indicator 3 are used to evaluate the static program security testing tool to determine whether the static program security testing tool supports test indicator 3. Finally, the evaluation results of the static program security testing tool are obtained by combining the results of the above support conditions.

[0034] It should be noted that the positive and negative samples obtained in the above manner can not only comprehensively evaluate the static program security testing tool based on different evaluation indicators, but also more effectively and prominently evaluate the core functions (or core capabilities) of the static program security testing tool.

[0035] An embodiment of the present specification provides an evaluation method for a static program security testing tool. When an evaluation request for a static program security testing tool for a preset programming language is received, the syntax feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool are obtained based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the syntax feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, the plurality of different second evaluation indicators can be constructed based on the plurality of different For each evaluation indicator of the same first evaluation indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and the static program security testing tool is evaluated for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (that is, positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which not only can more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also can provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0036] In practical applications, one or more different positive and negative sample pairs that match the evaluation indicators can be obtained from a specified database. To this end, the above-mentioned database can be constructed by constructing a test sample set, wherein the test sample set can be constructed in the following manner. For details, please refer to the processing of steps A2 to A8 below.

[0037] In step A2, a target static program security testing tool for a target programming language is obtained.

[0038] The target programming language can be any programming language, such as C or Java, and can be set based on actual circumstances. The target static program security testing tool can be any type of static program security testing tool. In actual applications, the target static program security testing tool can also be the same as the aforementioned static program security testing tool and can be set based on actual circumstances.

[0039] In implementation, the target static program security testing tool for the target programming language can be obtained in a variety of different ways. For example, the target static program security testing tool for the target programming language can be crawled from the Internet through a web crawler, or one or more target static program security testing tools for the target programming language can be selected from a specified database, or the target static program security testing tool for the target programming language can be obtained from different users (such as through exchange or from users who are willing to provide data free of charge, etc.). The specific setting can be based on actual conditions, and the embodiments of this specification do not limit this.

[0040] In step A4, a language characteristic analysis is performed on the target programming language to obtain the grammatical characteristic information of the target programming language, and the target static program security testing tool is analyzed to determine different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool.

[0041] The grammatical feature information of the target programming language can vary depending on the target programming language, specifically based on the above-mentioned related content, and will not be repeated here. The different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool can vary depending on the target static program security testing tool, specifically based on the above-mentioned related content, and will not be repeated here.

[0042] In step A6, multiple different third evaluation indicators for evaluation completeness are constructed based on the grammatical feature information of the target programming language, and multiple different fourth evaluation indicators for evaluation accuracy are constructed based on different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool. The multiple different third evaluation indicators include third evaluation indicators for analyzing and detecting target data structures and / or target operations on target data structures contained in a target program written in the target programming language, and / or third evaluation indicators for analyzing logical features in the target program. The multiple different fourth evaluation indicators include fourth evaluation indicators for performing sensitivity analysis on the target static program security testing tool.

[0043] The specific processing process of the above step A6 can be found in the above related content and will not be repeated here.

[0044] In step A8, one or more different positive and negative sample pairs are generated based on each third evaluation indicator, and one or more different positive and negative sample pairs are generated based on each fourth evaluation indicator, and the generated positive and negative sample pairs are stored in the test sample set.

[0045] In implementation, for any third evaluation indicator, one or more different positive and negative sample pairs can be generated based on the third evaluation indicator, and the same method can be used to generate one or more different positive and negative sample pairs for each third evaluation indicator. For any fourth evaluation indicator, one or more different positive and negative sample pairs can be generated based on the fourth evaluation indicator, and the same method can be used to generate one or more different positive and negative sample pairs for each fourth evaluation indicator. The positive and negative sample pairs generated above can be stored in the test sample set.

[0046] After generating the test sample set in the above manner, the specific processing methods for obtaining one or more different positive and negative sample pairs matching each evaluation indicator based on multiple different first evaluation indicators and multiple different second evaluation indicators in the above step S108 can be varied. An optional processing method is provided below, which may specifically include the following: based on multiple different first evaluation indicators and each evaluation indicator in multiple different second evaluation indicators, obtaining one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set.

[0047] In practical applications, after obtaining the test sample set in the above manner, the positive and negative sample pairs in the test sample set can also be evaluated. For details, please refer to the following processing: Based on the positive and negative sample pairs in the test sample set, use the preset static code analysis tool for the target programming language to evaluate the test sample set and obtain the corresponding evaluation results.

[0048] Among them, static code analysis tools can include multiple types, for example, SonarQube, FindBugs, TscanCode, etc., which can be set according to actual conditions. In addition, static code analysis tools can also vary according to the target programming language. For a specified target programming language, a specified static code analysis tool can be included. In actual applications, the static code analysis tool for the target programming language can be one tool or multiple different tools, which can be set according to actual conditions. The embodiments of this specification do not limit this.

[0049] In implementation, one or more different positive and negative sample pairs corresponding to a certain evaluation metric can be obtained, and the positive and negative samples can be used to test the above-mentioned static code analysis tool to test the support of the above-mentioned evaluation metric and obtain corresponding results. In this way, the support of each evaluation metric can be tested and the corresponding results can be obtained, thereby completing the evaluation of the test sample set and obtaining the corresponding evaluation results.

[0050] Based on the above processing, the specific processing methods for obtaining one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set based on multiple different first evaluation indicators and multiple different second evaluation indicators can be varied. An optional processing method is provided below, which may specifically include the following: If the above evaluation result is that the evaluation passes, then based on multiple different first evaluation indicators and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained from the test sample set.

[0051] In addition, if the above evaluation result is that the evaluation fails, the above steps A2 to A8 can be used to continue generating new positive and negative sample pairs, and the new positive and negative sample pairs can be evaluated until the evaluation result is that the evaluation passes.

[0052] In practical applications, in order to highlight the evaluation advantages of the test sample set and reflect the "completeness" of the test sample set in code structure analysis, the above-mentioned positive and negative sample pairs based on the test sample set use the preset static code analysis tool for the target programming language to evaluate the test sample set, and obtain the corresponding evaluation results. The test sample set can also be evaluated by cross-validation. For details, please refer to the following content: Based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set, the preset static code analysis tool for the target programming language is used respectively to evaluate the test sample set in a cross-validation manner to obtain the corresponding evaluation results.

[0053] Among them, the benchmark sample set may include multiple types. For example, the benchmark sample set may be a commonly used sample set, such as a Benchmark sample set, a Geekbench sample set, a PassMark sample set, etc., which can be set according to actual conditions.

[0054] In implementation, one sample (which can be a positive sample or a negative sample) in a corresponding positive and negative sample pair in the evaluation designated sample set can be selected from the test sample set, in addition, one sample can be selected from the benchmark sample set, and then a preset static code analysis tool for the target programming language can be used to analyze the two selected samples respectively to obtain corresponding analysis results. In this way, the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set can be analyzed using the static code analysis tool to obtain corresponding analysis results. The analysis results can be collected, and the final evaluation can be made based on the collected analysis results to obtain corresponding evaluation results.

[0055] In practical application, cross-validation with an existing benchmark sample set can be performed to ensure the completeness and rationality of the evaluation system. To evaluate the relative completeness of the designed evaluation system and test sample set under a specific standard, the constructed test sample set can be cross-validated with an existing test set in the same field for evaluation capability. Based on this, the target programming language is Java, the benchmark sample set is the Benchmark sample set, and the static code analysis tool is JavaParser. Based on the above, the test sample set can be evaluated in a cross-validation manner using the preset static code analysis tool for the target programming language based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set to obtain corresponding evaluation results. The specific processing method can be various, and an optional processing method is provided below, which can include the following steps B2 to B6.

[0056] In step B2, JavaParser is used to perform AST node type detection on the positive and negative sample pairs in the test sample set to obtain the coverage of the AST node types corresponding to the test sample set.

[0057] In step B4, JavaParser is used to perform AST node type detection on the samples in the benchmark sample set to obtain the coverage of the AST node types corresponding to the benchmark sample set.

[0058] In step B6, based on the coverage of the AST node types corresponding to the test sample set and the coverage of the AST node types corresponding to the benchmark sample set, the evaluation result of evaluating the test sample set is determined.

[0059] In implementation, the degree of support for the analysis of common code structures can be reflected by comparing the coverage of AST node types by different sample sets (i.e., test sample set and benchmark sample set), reflecting the "completeness" of the test sample set in code structure analysis. Specifically, the coverage of the AST node types corresponding to the test sample set and the coverage of the AST node types corresponding to the benchmark sample set can be compared. If the coverage of the AST node types corresponding to the test sample set is greater than the coverage of the AST node types corresponding to the benchmark sample set, then the evaluation result of the test sample set is determined to be passed. If the coverage of the AST node types corresponding to the test sample set is greater than the coverage of the AST node types corresponding to the benchmark sample set, then the evaluation result of the test sample set is determined to be failed. In the above manner, combined with the core technology of static analysis, corresponding evaluation items (i.e., evaluation indicators) are set from the core technical perspectives of static analysis such as AST node types, characteristics of programming languages, sensitivity analysis, etc., rather than designing categories such as vulnerability types, so that the core performance of static application security testing tools can be more scientifically characterized.

[0060] In actual applications, the preset programming language in the above steps S102 to S108 can be the Java programming language, and the data structure can include one or more of a collection, an array, and an object. For the Java programming language, the specific data structures included above are only some exemplary specific data structures. In actual applications, the data structure does not only include the one or more of the above, but may also include other specific data structures in addition to the above specific data structures, which can be set specifically according to actual conditions. The logical features in the above program can include one or more of control flow, exceptions, and reflection. For the Java programming language, the specific logical features included in the above program are only some exemplary specific logical features. In actual applications, the above program does not only include the one or more of the above, but may also include other specific logical features in addition to the above specific logical features, which can be set specifically according to actual conditions.

[0061] In actual applications, multiple different second evaluation indicators include second evaluation indicators for performing sensitivity analysis on static program security testing tools through taint objects and / or taint links. Taint objects are objects that track data flow to discover potential risks, and taint links are links that track data flow to discover potential risks.

[0062] In actual applications, sensitivity analysis of static program security testing tools includes one or more of sensitivity analysis of static program security testing tools for flow sensitivity, sensitivity analysis of static program security testing tools for object sensitivity, sensitivity analysis of static program security testing tools for domain sensitivity, sensitivity analysis of static program security testing tools for path sensitivity, and sensitivity analysis of static program security testing tools for context sensitivity. In actual applications, the sensitivity analysis of static program security testing tools for specific sensitive types included in the above sensitivity analysis are only some exemplary sensitivity analyses, and in actual applications, the sensitivity analysis of static program security testing tools does not only include one or more of the above, but may also include sensitivity analysis of other sensitive types in addition to the sensitivity analysis of the above specific sensitive types, which can be set specifically according to actual conditions.

[0063] An embodiment of the present specification provides an evaluation method for a static program security testing tool. When an evaluation request for a static program security testing tool for a preset programming language is received, grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool are obtained based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the grammatical feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, a plurality of different first evaluation indicators can be constructed based on the plurality of different first evaluation indicators. For each evaluation indicator in the evaluation indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and the static program security testing tool is evaluated for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (that is, positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which not only can more accurately reflect the detection capability of the static application security testing tool, reduce the evaluation complexity, and avoid ambiguous test results, but also provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0064] Furthermore, when developing the evaluation system, we fully considered the grammatical characteristics of the Java language and the principles of static program security testing tools. We set evaluation metrics based on comprehensiveness, accuracy, and performance, ensuring a systematic and scientific evaluation system and avoiding subjectivity and unsystematicity during the evaluation process. Furthermore, we cross-validated the test sample set using a specific benchmark sample set to ensure the completeness and rationality of the evaluation system. This verification method not only enhances the credibility of the evaluation system but also makes the evaluation results more universal and applicable.

[0065] The above is an evaluation method for a static program security testing tool provided in the embodiment of this specification. Based on the same idea, the embodiment of this specification also provides an evaluation device for a static program security testing tool, such as Figure 3 shown.

[0066] The evaluation device of the static program security testing tool includes: a request receiving module 301, an information acquisition module 302, an evaluation index construction module 303 and a tool evaluation module 304, wherein:

[0067] A request receiving module 301 receives an evaluation request for a static program security testing tool for a preset programming language;

[0068] An information acquisition module 302 acquires, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool;

[0069] An evaluation indicator construction module 303 constructs a plurality of different first evaluation indicators for evaluation completeness based on the grammatical characteristic information of the preset programming language, and constructs a plurality of different second evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting data structures contained in a program written in the preset programming language and / or preset operations on the data structures, and / or a first evaluation indicator for analyzing logical features in the program. The plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool.

[0070] The tool evaluation module 304 obtains one or more different positive and negative sample pairs that match each evaluation indicator based on the multiple different first evaluation indicators and each of the multiple different second evaluation indicators, and evaluates the static program security testing tool based on different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool.

[0071] In the embodiment of this specification, the device further includes:

[0072] A tool acquisition module, which acquires a target static program security testing tool for a target programming language;

[0073] an analysis module that performs language characteristic analysis on the target programming language to obtain grammatical characteristic information of the target programming language, and analyzes the target static program security testing tool to determine information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool;

[0074] an indicator construction module, which constructs a plurality of different third evaluation indicators for evaluation completeness based on the grammatical characteristic information of the target programming language, and constructs a plurality of different fourth evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the target static program security testing tool, wherein the plurality of different third evaluation indicators include a third evaluation indicator for analyzing and detecting a target data structure contained in a target program written in the target programming language and / or a target operation on the target data structure, and / or a third evaluation indicator for analyzing logical features in the target program, and the plurality of different fourth evaluation indicators include a fourth evaluation indicator for performing sensitivity analysis on the target static program security testing tool;

[0075] A test set generation module generates one or more different positive and negative sample pairs based on each of the third evaluation indicators, and generates one or more different positive and negative sample pairs based on each of the fourth evaluation indicators, and stores the generated positive and negative sample pairs in a test sample set;

[0076] The tool evaluation module 304 obtains one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators.

[0077] In the embodiment of this specification, the device further includes:

[0078] An evaluation module, which evaluates the test sample set based on the positive and negative sample pairs in the test sample set using a preset static code analysis tool for the target programming language to obtain corresponding evaluation results;

[0079] The tool evaluation module 304 obtains one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set based on the multiple different first evaluation indicators and each evaluation indicator of the multiple different second evaluation indicators if the evaluation result is passed.

[0080] In an embodiment of the present specification, the evaluation module uses a preset static code analysis tool for the target programming language based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set to evaluate the test sample set in a cross-validation manner to obtain corresponding evaluation results.

[0081] In the embodiment of this specification, the target programming language is the Java programming language, the benchmark sample set is the Benchmark sample set, and the static code analysis tool is JavaParser;

[0082] The evaluation module includes:

[0083] A first coverage determination unit, using the JavaParser, performs AST node type detection on the positive and negative sample pairs in the test sample set to obtain coverage of the AST node type corresponding to the test sample set;

[0084] A second coverage determination unit, using the JavaParser to perform AST node type detection on samples in the benchmark sample set to obtain coverage of the AST node type corresponding to the benchmark sample set;

[0085] The evaluation unit determines an evaluation result of evaluating the test sample set based on the coverage of the AST node type corresponding to the test sample set and the coverage of the AST node type corresponding to the benchmark sample set.

[0086] In the embodiment of this specification, the preset programming language is the Java programming language, the data structure includes but is not limited to one or more of a collection, an array, and an object, and the logical features in the program include but are not limited to one or more of control flow, exceptions, and reflection.

[0087] In an embodiment of the present specification, the multiple different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool through taint objects and / or taint links, wherein the taint objects are objects that track data flow to discover potential risks, and the taint links are links that track data flow to discover potential risks.

[0088] In an embodiment of the present specification, the sensitivity analysis performed on the static program security testing tool includes one or more of a sensitivity analysis performed on the static program security testing tool for flow sensitivity, a sensitivity analysis performed on the static program security testing tool for object sensitivity, a sensitivity analysis performed on the static program security testing tool for domain sensitivity, a sensitivity analysis performed on the static program security testing tool for path sensitivity, and a sensitivity analysis performed on the static program security testing tool for context sensitivity.

[0089] An embodiment of the present specification provides an evaluation device for a static program security testing tool. When an evaluation request for a static program security testing tool for a preset programming language is received, the device obtains grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the grammatical feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, the plurality of different second evaluation indicators can be constructed based on the plurality of different For each evaluation indicator of the same first evaluation indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and the static program security testing tool is evaluated for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (that is, positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which not only can more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also can provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0090] Furthermore, when developing the evaluation system, we fully considered the characteristics of the Java language and the principles of static program security testing tools. We set evaluation metrics based on comprehensiveness, accuracy, and performance, ensuring a systematic and scientific evaluation system and avoiding subjectivity and unsystematicity during the evaluation process. Furthermore, we cross-validated the test sample set using a specific benchmark sample set to ensure the completeness and rationality of the evaluation system. This verification method not only enhances the credibility of the evaluation system but also makes the evaluation results more versatile and applicable.

[0091] The above is an evaluation device for a static program security testing tool provided in the embodiment of this specification. Based on the same idea, the embodiment of this specification also provides an evaluation device for a static program security testing tool, such as Figure 4 shown.

[0092] The evaluation device of the static program security testing tool may provide a terminal device or a server, etc. for the above embodiment.

[0093] The evaluation device of the static program security testing tool may have relatively large differences due to different configurations or performances, and may include one or more processors 401 and memory 402, and the memory 402 may store one or more storage applications or data. Among them, the memory 402 can be a temporary storage or a permanent storage. The application stored in the memory 402 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the evaluation device of the static program security testing tool. Furthermore, the processor 401 can be configured to communicate with the memory 402 to execute a series of computer executable instructions in the memory 402 on the evaluation device of the static program security testing tool. The evaluation device of the static program security testing tool may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input and output interfaces 405, and one or more keyboards 406.

[0094] Specifically in this embodiment, the evaluation device of the static program security testing tool includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the evaluation device of the static program security testing tool, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0095] Receive evaluation requests for static program security testing tools for pre-defined programming languages;

[0096] Acquire, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool;

[0097] A plurality of different first evaluation indicators for evaluation completeness are constructed based on the grammatical characteristic information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy are constructed based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool, wherein the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting data structures contained in a program written in the preset programming language and / or preset operations on the data structures, and / or a first evaluation indicator for analyzing logical features in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool;

[0098] Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0099] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the evaluation device embodiment of the static program security testing tool is described briefly because it is fundamentally similar to the method embodiment. For relevant parts, refer to the description of the method embodiment.

[0100] The embodiment of the present specification provides an evaluation device of a static program security test tool. When an evaluation request of a static program security test tool for a preset programming language is received, the evaluation device obtains syntax characteristic information of the preset programming language and information of different dimensions of sensitivity analysis related to program analysis ability accuracy of the static program security test tool based on the evaluation request. Then, a plurality of different first evaluation indexes for evaluation completeness can be constructed based on the syntax characteristic information of the preset programming language, and a plurality of different second evaluation indexes for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the program analysis ability accuracy of the static program security test tool. The plurality of different first evaluation indexes include a first evaluation index for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation on the data structure, and / or a first evaluation index for analyzing a logic feature in the program. The plurality of different second evaluation indexes include a second evaluation index for sensitivity analysis of the static program security test tool. Finally, one or more different positive and negative sample pairs matched with each evaluation index can be obtained based on each evaluation index in the plurality of different first evaluation indexes and the plurality of different second evaluation indexes, and the static program security test tool is evaluated based on the obtained positive and negative sample pairs to determine an evaluation result of the static program security test tool. In this way, through the multi-dimensional evaluation indexes and the fine-grained test samples (i.e., the positive and negative sample pairs corresponding to the plurality of different evaluation indexes), the test function points corresponding to each test sample can be determined, so that not only the detection ability of the static application security test tool can be more accurately reflected, but also the fuzzy test result can be avoided, and detailed test result explanation can be provided, and thus the performance of the static application security test tool can be comprehensively evaluated. In addition, one test function point is described by a plurality of positive and negative sample combinations, and detailed test result explanation is provided, so that the result is more targeted and operable. Moreover, through scientific design of the evaluation indexes, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and the development and application of SAST technology are promoted.

[0101] Further, based on the above Figures 1 to 2 The one or more embodiments of the present specification also provide a storage medium for storing computer executable instruction information. In a specific embodiment, the storage medium can be a U disk, an optical disk, a hard disk, etc. The computer executable instruction information stored in the storage medium can implement the following processes when executed by a processor.

[0102] receiving an evaluation request of a static program security test tool for a preset programming language;

[0103] Based on the evaluation request, obtain the syntax characteristic information of the preset programming language and the information of different dimensions of sensitivity analysis related to the program analysis ability accuracy of the static program security testing tool;

[0104] Based on the syntax characteristic information of the preset programming language, construct a plurality of different first evaluation indexes for evaluation completeness, and based on the different dimensions of sensitivity analysis related to the program analysis ability accuracy of the static program security testing tool, construct a plurality of different second evaluation indexes for evaluation accuracy. The plurality of different first evaluation indexes include a first evaluation index for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation on the data structure, and / or a first evaluation index for analyzing a logic feature in the program. The plurality of different second evaluation indexes include a second evaluation index for sensitivity analysis of the static program security testing tool.

[0105] Based on each evaluation index in the plurality of different first evaluation indexes and the plurality of different second evaluation indexes, obtain one or more different positive and negative sample pairs matched with the each evaluation index, and based on the obtained positive and negative sample pairs, respectively evaluate the static program security testing tool according to different evaluation indexes to determine the evaluation result of the static program security testing tool.

[0106] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the above-mentioned storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0107] An embodiment of the present specification provides a storage medium, which, upon receiving an evaluation request for a static program security testing tool for a preset programming language, obtains grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the grammatical feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, a plurality of different first evaluation indicators can be constructed based on the plurality of different first evaluation indicators. For each evaluation indicator in the measurement indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and the static program security testing tool is evaluated for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (that is, positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which not only can more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also can provide detailed test result explanations, and thus can comprehensively evaluate the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0108] Furthermore, based on the above Figures 1 to 2 In addition to the method shown in FIG. 1 , one or more embodiments of this specification further provide a computer program product, including a computer program. When the computer program in the computer program product is executed by a processor, the computer program can implement the following process:

[0109] Receive evaluation requests for static program security testing tools for pre-defined programming languages;

[0110] Acquire, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool;

[0111] A plurality of different first evaluation indicators for evaluation completeness are constructed based on the grammatical characteristic information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy are constructed based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool, wherein the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting data structures contained in a program written in the preset programming language and / or preset operations on the data structures, and / or a first evaluation indicator for analyzing logical features in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool;

[0112] Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0113] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the aforementioned computer program product embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant portions, reference can be made to the description of the method embodiment.

[0114] The embodiment of the specification provides a computer program product, when a test request for a static program security test tool of a preset programming language is received, information of a syntax characteristic of the preset programming language and different dimensions of sensitivity analysis related to program analysis ability accuracy of the static program security test tool are obtained based on the test request, then, a plurality of different first test indexes for test completeness can be constructed based on the syntax characteristic of the preset programming language, and a plurality of different second test indexes for test accuracy can be constructed based on the different dimensions of sensitivity analysis related to program analysis ability accuracy of the static program security test tool, finally, one or more different positive and negative sample pairs matched with each test index can be obtained based on each test index in the plurality of different first test indexes and the plurality of different second test indexes, and the static program security test tool is tested based on the obtained positive and negative sample pairs to determine the test result of the static program security test tool, in this way, through the multi-dimensional test indexes and the fine-grained test samples (i.e. the positive and negative sample pairs corresponding to the plurality of different test indexes), the test function points corresponding to each test sample can be determined, so that not only the detection ability of the static application security test tool can be more accurately reflected, the ambiguous test result is avoided, but also detailed test result explanation can be provided, and then the performance of the static application security test tool can be comprehensively evaluated, in addition, one test function point is described by a plurality of positive and negative sample combinations, and detailed test result explanation is provided, so that the result is more targeted and operable, and through scientific design of the test indexes, the shortcomings of unclear and unsystematic test standards are overcome, the systematization level of the security test is improved, and the development and application of the SAST technology are promoted.

[0115] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or possible.

[0116] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0117] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0118] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0120] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] The embodiments of this specification are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable fraud case serial and parallel device to produce a machine, so that the instructions executed by the processor of the computer or other programmable fraud case serial and parallel device generate instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable fraud case serial and parallel device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, the instruction device being implemented in the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0126] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0128] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, one or more embodiments of this specification may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0130] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0131] The foregoing description is merely an example of the present invention and is not intended to limit this document. Various modifications and variations are possible within the scope of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A method for evaluating a static program security testing tool, the method comprising: Receive evaluation requests for static program security testing tools for pre-defined programming languages; Acquire, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool; Based on the grammatical characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting data structures contained in a program written in the preset programming language and / or preset operations on the data structures, and / or a first evaluation indicator for analyzing logical features in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool; Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

2. The method according to claim 1, further comprising: Obtain a target static program security testing tool for the target programming language; Performing a language characteristic analysis on the target programming language to obtain grammatical characteristic information of the target programming language, and analyzing the target static program security testing tool to determine information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool; A plurality of different third evaluation indicators for evaluation completeness are constructed based on the grammatical characteristic information of the target programming language, and a plurality of different fourth evaluation indicators for evaluation accuracy are constructed based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the target static program security testing tool, the plurality of different third evaluation indicators including a third evaluation indicator for analyzing and detecting a target data structure contained in a target program written in the target programming language and / or a target operation of the target data structure, and / or a third evaluation indicator for analyzing logical features in the target program, and the plurality of different fourth evaluation indicators including a fourth evaluation indicator for performing sensitivity analysis on the target static program security testing tool; Generating one or more different positive and negative sample pairs based on each of the third evaluation indicators, and generating one or more different positive and negative sample pairs based on each of the fourth evaluation indicators, and storing the generated positive and negative sample pairs in a test sample set; The acquiring, based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator comprises: Based on each evaluation indicator of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained from the test sample set.

3. The method according to claim 2, further comprising: Based on the positive and negative sample pairs in the test sample set, the test sample set is evaluated using a preset static code analysis tool for the target programming language to obtain corresponding evaluation results; The acquiring, based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set comprises: If the evaluation result is passed, based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained from the test sample set.

4. The method according to claim 3, wherein the evaluating the test sample set based on the positive and negative sample pairs in the test sample set using a preset static code analysis tool for the target programming language to obtain corresponding evaluation results comprises: Based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set, the test sample set is evaluated in a cross-validation manner using a preset static code analysis tool for the target programming language to obtain corresponding evaluation results.

5. The method according to claim 4, wherein the target programming language is Java programming language, the benchmark sample set is Benchmark sample set, and the static code analysis tool is JavaParser; Based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set, a preset static code analysis tool for the target programming language is used to evaluate the test sample set in a cross-validation manner to obtain corresponding evaluation results, including: Using the JavaParser to perform AST node type detection on the positive and negative sample pairs in the test sample set, and obtaining the coverage of the AST node type corresponding to the test sample set; Using the JavaParser to perform AST node type detection on samples in the benchmark sample set to obtain coverage of AST node types corresponding to the benchmark sample set; An evaluation result of evaluating the test sample set is determined based on the coverage of the AST node type corresponding to the test sample set and the coverage of the AST node type corresponding to the benchmark sample set.

6. The method according to claim 1, wherein the preset programming language is Java programming language, the data structure includes but is not limited to one or more of a collection, an array, and an object, and the logical features in the program include but are not limited to one or more of control flow, exceptions, and reflection.

7. According to the method according to claim 1, the multiple different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool through taint objects and / or taint links, the taint objects are objects that track data flow to discover potential risks, and the taint links are links that track data flow to discover potential risks.

8. According to the method according to claim 1, the sensitivity analysis of the static program security testing tool includes one or more of a sensitivity analysis of the static program security testing tool for flow sensitivity, a sensitivity analysis of the static program security testing tool for object sensitivity, a sensitivity analysis of the static program security testing tool for domain sensitivity, a sensitivity analysis of the static program security testing tool for path sensitivity, and a sensitivity analysis of the static program security testing tool for context sensitivity.

9. An evaluation device for a static program security testing tool, the device comprising: A request receiving module receives an evaluation request for a static program security testing tool for a preset programming language; An information acquisition module, which acquires, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool; An evaluation indicator construction module is configured to construct a plurality of different first evaluation indicators for evaluation completeness based on grammatical characteristic information of the preset programming language, and to construct a plurality of different second evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool, wherein the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting data structures contained in programs written in the preset programming language and / or preset operations on the data structures, and / or a first evaluation indicator for analyzing logical features in the programs, and the plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool; The tool evaluation module obtains one or more different positive and negative sample pairs that match each evaluation indicator based on the multiple different first evaluation indicators and each of the multiple different second evaluation indicators, and evaluates the static program security testing tool based on different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool.

10. An evaluation device for a static program security testing tool, comprising: processor; as well as a memory arranged to store computer-executable instructions which, when executed, cause the processor to: Receive evaluation requests for static program security testing tools for pre-defined programming languages; Acquire, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool; Based on the grammatical characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on the test types corresponding to different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators including a first evaluation indicator for analyzing and detecting data structures contained in a program written in the preset programming language and / or preset operations on the data structures, and / or a first evaluation indicator for analyzing logical features in the program, and the plurality of different second evaluation indicators including a second evaluation indicator for performing sensitivity analysis on the static program security testing tool; Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

Images