Evaluation method, device and equipment for static program security test tool

Abstract

The embodiment of the invention discloses an evaluation method, device and equipment for a static program security test tool. The method comprises the steps that an evaluation request for the static program security test tool in a preset programming language is received; obtaining grammar characteristic information of a preset programming language and information of different dimensions of sensitivity analysis related to program analysis capability precision of the static program security test tool based on the evaluation request; a plurality of different first evaluation indexes for evaluating integrity are constructed on the basis of grammatical characteristic information of a preset programming language, and a plurality of different second evaluation indexes for evaluating accuracy are constructed on the basis of different dimensions of sensitivity analysis related to program analysis capability precision; and based on each evaluation index in the plurality of first evaluation indexes and the plurality of second evaluation indexes, obtaining one or more corresponding different positive and negative sample pairs, and based on the obtained positive and negative samples, carrying out evaluation of different evaluation indexes on the static program security test tool.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to an evaluation method, device and equipment for a static program security testing tool. Background Art

[0002] Currently, software is everywhere. However, with the popularity of software, software security issues are becoming increasingly serious. For example, system crashes and security vulnerabilities may lead to huge economic losses or even threaten lives. In order to ensure the security and reliability of software and meet people's needs for protecting their privacy data, application security testing technologies are widely used, including static application security testing (SAST), dynamic application security testing (DAST), interactive application security testing (IAST), etc. The above security testing technologies are used to discover security vulnerabilities in private and open source codes, conduct intellectual property and industry standard compliance checks, etc.

[0003] At present, there are many commercial or open source products available for various types of static application security testing tools, and some companies are also developing related products in-house. However, the main challenge facing developers and users is how to objectively measure the technical level of various types of static application security testing tools. Evaluating the above products can provide developers and users with valuable reference information to help them make more informed technology selection and product planning decisions. To this end, it is necessary to provide a more fine-grained evaluation method for static application security testing tools, so that the performance of static application security testing tools can be comprehensively evaluated. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a more fine-grained evaluation method for static application security testing tools, so that the performance of static application security testing tools can be comprehensively evaluated.

[0005] In order to implement the above technical solution, the embodiments of this specification are implemented as follows:

[0006] An evaluation method for a static program security testing tool provided in an embodiment of the present specification includes: receiving an evaluation request for a static program security testing tool for a preset programming language. Based on the evaluation request, the syntax characteristic information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool are obtained. Based on the syntax characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0007] An evaluation device for a static program security testing tool provided in an embodiment of the present specification includes: a request receiving module, receiving an evaluation request for a static program security testing tool for a preset programming language. An information acquisition module, based on the evaluation request, acquires the syntax characteristic information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. An evaluation indicator construction module, based on the syntax characteristic information of the preset programming language, constructs a plurality of different first evaluation indicators for evaluation completeness, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, constructs a plurality of different second evaluation indicators for evaluation accuracy, wherein the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. The tool evaluation module obtains one or more different positive and negative sample pairs matching each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, and evaluates the static program security testing tool for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool.

[0008] An evaluation device for a static program security testing tool provided by an embodiment of the present specification includes: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor receives an evaluation request for a static program security testing tool for a preset programming language. Based on the evaluation request, the syntax characteristic information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool are obtained. Based on the syntax characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, wherein the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0009] The embodiment of the present specification also provides a storage medium, the storage medium is used to store computer executable instructions, and the executable instructions implement the following process when executed by the processor: receive an evaluation request for a static program security testing tool for a preset programming language. Based on the evaluation request, obtain the syntax characteristic information of the preset programming language and the information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. Based on the syntax characteristic information of the preset programming language, construct a plurality of different first evaluation indicators for evaluation completeness, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, construct a plurality of different second evaluation indicators for evaluation accuracy, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0010] The embodiment of the present specification also provides a computer program product, including a computer program, which implements the following process when executed by a processor: receiving an evaluation request for a static program security testing tool for a preset programming language. Based on the evaluation request, the syntax characteristic information of the preset programming language and the information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool are obtained. Based on the syntax characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0012] Figure 1 This is an embodiment of an evaluation method of a static program security testing tool in this specification;

[0013] Figure 2 This is a schematic diagram of an evaluation page of a static program security testing tool in this specification;

[0014] Figure 3 This is an embodiment of an evaluation device of a static program security testing tool in this specification;

[0015] Figure 4 This is an example of an evaluation device for a static program security testing tool in this specification. DETAILED DESCRIPTION

[0016] The embodiments of this specification provide a method, device and equipment for evaluating a static program security testing tool.

[0017] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0018] The embodiments of this specification provide a federated learning mechanism. Currently, software is ubiquitous. However, with the popularization of software, software security issues are becoming increasingly serious. For example, system crashes and security vulnerabilities may cause huge economic losses or even threaten life. In order to ensure the security and reliability of software and meet people's needs for protecting their privacy data, application security testing technologies are widely used, including static application security testing (SAST), dynamic application security testing (DAST), interactive application security testing (IAST), etc. The above security testing technologies are used to discover security vulnerabilities in private and open source codes, conduct intellectual property and industry standard compliance checks, etc.

[0019] At present, there are many commercial or open source products available for various types of static application security testing tools, and some companies are also developing related products on their own. However, the main challenge facing developers and users is how to objectively measure the technical level of various types of static application security testing tools. Evaluating the above products can provide valuable reference information for developers and users, helping them make more informed technology selection and product planning decisions. To this end, it is necessary to provide a more fine-grained evaluation method for static application security testing tools, so that the performance of static application security testing tools can be comprehensively evaluated. The embodiments of this specification can clarify the test function points corresponding to each test sample through multi-dimensional evaluation indicators and fine-grained test samples, so as to not only more accurately reflect the detection capabilities of static application security testing tools and avoid ambiguous test results, but also provide detailed test result explanations, thereby comprehensively evaluating the performance of static application security testing tools. For specific processing, please refer to the specific content in the following embodiments.

[0020] like Figure 1As shown, an embodiment of this specification provides an evaluation method for a static program security testing tool, wherein the execution subject of the method may be a terminal device or a server, etc., wherein the terminal device may be a mobile terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically a smart watch, a vehicle-mounted device, etc.), etc., wherein the server may be an independent server, or a server cluster composed of multiple servers, etc., and the server may be a background server in the financial field or the online shopping field, etc., or a background server of an application, etc. In this embodiment, a detailed description is given by taking the execution subject as a server as an example. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method may specifically include the following steps:

[0021] In step S102, an evaluation request for a static program security testing tool for a preset programming language is received.

[0022] Among them, the preset programming language may include multiple languages, for example, the preset programming language may be Java programming language, C language or C++ language, etc., which may be set specifically according to actual conditions, and the embodiments of this specification do not limit this. Static program security testing tool is SAST, which is a commonly used application security (AppSec) tool that can scan the source code, binary code or byte code of an application. The static program security testing tool can find the root cause of the vulnerability and help fix potential security vulnerabilities. The solution of the static program security testing tool SAST analyzes the application from the "inside out" without running the operating system to perform the scanning operation.

[0023] In practice, there are currently many types of static program security testing tools, such as OWASP Benchmark or Webgoat, Juliet Test Suite, SecuriBench Micro, Openrasp, etc. However, at present, due to the lack of systematic evaluation system guidance, the test results of the above test sample sets are relatively vague, basically a simple stacking of various samples, and cannot provide fine-grained evaluation results. The current industry evaluation method mainly achieves comprehensive evaluation through upper-level standard evaluation or non-technical indicators, but lacks subjective stratification of the capabilities of static program security testing tools. For example, OWASP Benchmark covers a limited number of vulnerability types and numbers, and the evaluation results are mostly recall rate and false alarm rate, which makes it difficult to fully and finely characterize the performance of the tool; Webgoat has a limited number of samples and coverage, and is mainly used for beginners to learn about security vulnerabilities, making it difficult to fully evaluate the performance of professional tools; Juliet Test Suite has a large number of samples, but due to the single sample type, it may not be able to fully reflect the performance of the tool in a variety of scenarios; SecuriBench Micro mainly targets specific security issues, with a narrow coverage range, making it difficult to fully evaluate the versatility of the tool; Openrasp has a small number of samples, covers a limited number of vulnerability types, and the evaluation results are not comprehensive enough. Based on the above content, we can choose to design evaluation indicators from dimensions such as completeness and accuracy according to the preset programming language characteristics and SAST tool principles, and use the above evaluation indicators to evaluate static program security testing tools. Specifically, Figure 2 As shown in the figure, when you need to evaluate the static program security testing tool of a programming language, you can use Figure 2 In the "Information of the tool to be evaluated" information input box, upload the identifier of the static program security testing tool, and generate a corresponding evaluation request based on the identifier. Subsequently, the data of the static program security testing tool can be obtained based on the identifier of the static program security testing tool, or the static program security testing tool can be obtained through Figure 2 In the "Information of the tool to be evaluated" information input box, upload the data of the static program security testing tool, and generate a corresponding evaluation request based on the data of the static program security testing tool. The specific settings can also be made according to actual conditions.

[0024] In step S104, based on the above evaluation request, information on syntax characteristics of a preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of a static program security testing tool are obtained.

[0025] Among them, the grammatical feature information of the preset programming language may include multiple types. For example, the grammatical feature information of the preset programming language may include basic language grammatical feature information and complex language grammatical feature information. For example, the basic language grammatical feature information for the Java programming language may include grammatical feature information such as collections, arrays, objects, etc., and the complex language grammatical feature information may include grammatical feature information such as control flow, exceptions, etc. For another example, the basic language grammatical feature information for the C language may include grammatical feature information such as arrays, pointers, keywords, etc., and the complex language grammatical feature information may include process control, etc., which may be set specifically according to actual conditions. The different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool may include multiple types, such as flow sensitivity dimension, object sensitivity dimension, path sensitivity dimension, context sensitivity dimension, etc., which may be set specifically according to actual conditions. In addition, the dimensions of sensitivity analysis corresponding to different static program security testing tools may be different. In addition, in actual applications, different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of static program security testing tools can also be set according to the tool type, such as setting the corresponding sensitivity analysis dimensions similar to the type of OWASP Benchmark or Webgoat, or the type similar to Juliet Test Suite, or the type similar to Openrasp, etc., which can also be set according to actual conditions.

[0026] In implementation, after receiving the evaluation request in the above manner, the data of the static program security testing tool can be obtained. In addition, in order to improve processing efficiency, the identifier of the preset programming language, such as Java programming language or C language, etc., can be uploaded, and the static program security testing tool and the preset programming language can be analyzed separately, so as to determine the grammatical characteristic information of the preset programming language and the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. Alternatively, the static program security testing tool can also be analyzed to determine the relevant information of the preset programming language corresponding to the static program security testing tool, as well as the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. In addition, the preset programming language can also be analyzed to determine the grammatical characteristic information of the preset programming language, etc. The specific setting can be based on actual conditions, and the embodiments of this specification are not limited to this.

[0027] In step S106, a plurality of different first evaluation indicators for evaluation completeness are constructed based on the grammatical characteristic information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy are constructed based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program safety testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing logical features in the above program. The plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program safety testing tool.

[0028] Among them, the data structure may include multiple types, and different programming languages ​​may contain different data structures. For example, the data structures contained in the Java programming language may include arrays, collections, objects, etc., and the data structures contained in the C language may include pointers, keywords, etc., which can be set according to actual conditions. The preset operations may include multiple types, such as input operations, access operations, data protection operations, etc., which can be set according to actual conditions. Static program security testing tools use different sensitivities (such as sensitivity to flow sensitivity (i.e., flow sensitivity dimension) sensitivity, sensitivity to object sensitivity (i.e., object sensitivity dimension) sensitivity, sensitivity to path sensitivity (i.e., path sensitivity dimension) sensitivity, and sensitivity to context sensitivity (i.e., context sensitivity dimension)) during the analysis process, which has always been one of the important strategies to achieve accuracy-performance trade-offs. Therefore, evaluation indicators can be set for the sensitivity of static program security testing tools.

[0029] In implementation, usually, the vulnerability-based test sample set is generally based on the vulnerability type as the evaluation perspective, and test samples of different vulnerability types constitute the test sample set. Then, different types of static program security testing tools are evaluated on the same set of test sample sets. However, the principles of different types of static program security testing tools are different, making it difficult for a set of test sample sets to be used universally between different types of static program security testing tools, otherwise it will affect the rationality of the test results. Therefore, in view of the above situation, it is necessary to change the evaluation perspective, and determine different evaluation items (i.e., evaluation indicators) based on different dimensions of sensitivity analysis related to the program analysis capability accuracy of specific programming languages ​​and static program security testing tools (or the principles of static program security testing tools or the tool types of static program security testing tools, etc.), so that the design of evaluation items and test sample sets is more reasonable. Finally, the evaluation system of static program security testing tools for preset programming languages ​​can be designed from multiple dimensions such as completeness, accuracy and performance.

[0030] In the dimension of completeness, the syntax characteristic information of the programming language and the principle of the static program security testing tool can be preset, and one or more aspects such as conventional taint links (i.e., basic language characteristics) and special taint links (i.e., complex language characteristics) can be selected to set the evaluation indicators. On the one hand, by modeling the basic language characteristics (which may include syntax characteristics) to construct multiple different first evaluation indicators for evaluation completeness, such as initialization, access, specific method operations, etc. of collections, arrays, objects, etc., the analysis and detection capabilities of the above data structures and operations contained in the program written in the preset programming language can be evaluated, and potential security issues such as input verification, access control, data protection, etc. can be revealed. On the other hand, modeling of complex syntax characteristic information such as control flow and exceptions can be used to construct the first evaluation indicator for evaluation completeness. In this way, the analysis capabilities of the logical features in the above program can be evaluated, which is also crucial to the correctness and security of the written program. Through the processing of the above two aspects, more different first evaluation indicators for evaluation completeness can be constructed.

[0031] In addition, in terms of accuracy, based on the core of the static analysis capability of the static program security testing tool - sensitivity analysis, the precise analysis capability of the static program security testing tool can be further measured to improve the evaluation accuracy. Based on this, multiple different second evaluation indicators for evaluation accuracy can be constructed based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool, including a second evaluation indicator for sensitivity analysis of the static program security testing tool.

[0032] In step S108, based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0033] In implementation, after obtaining multiple different first evaluation indicators and multiple different second evaluation indicators in the above manner, for any one of the above multiple evaluation indicators (including multiple different first evaluation indicators and multiple different second evaluation indicators) (which can be a first evaluation indicator, or, can also be a second evaluation indicator), one or more different positive and negative sample pairs (i.e., a positive sample and a negative sample) matching the evaluation indicator can be obtained from a specified database, or, corresponding positive and negative sample pairs can be generated for the evaluation indicator (specifically, the corresponding positive and negative sample pairs can be generated by a specified algorithm, or, the corresponding positive and negative sample pairs can be generated by a pre-trained model or a large model, etc., which can be set specifically according to actual conditions). Through the above method, positive and negative sample pairs corresponding to other evaluation indicators can be obtained or generated. Finally, one or more different positive and negative sample pairs matching each evaluation indicator can be obtained. The obtained positive and negative sample pairs can be used as test samples, and the static program security testing tool can be evaluated for different evaluation indicators through the test samples. For example, the test indicators include test indicator 1, test indicator 2 and test indicator 3. In addition, one or more different positive and negative sample pairs matching test indicator 1, one or more different positive and negative sample pairs matching test indicator 2 and one or more different positive and negative sample pairs matching test indicator 3 can be obtained. One or more different positive and negative sample pairs matching test indicator 1 are used to evaluate the static program security testing tool to determine its support for test indicator 1. One or more different positive and negative sample pairs matching test indicator 2 are used to evaluate the static program security testing tool to determine its support for test indicator 2. One or more different positive and negative sample pairs matching test indicator 3 are used to evaluate the static program security testing tool to determine its support for test indicator 3. Finally, the evaluation result of the static program security testing tool is obtained by combining the above support results.

[0034] It should be noted that the positive and negative samples obtained in the above manner can not only conduct a comprehensive evaluation of the static program security testing tool based on different evaluation indicators, but also can more effectively and prominently evaluate the core functions (or core capabilities) of the static program security testing tool.

[0035] An embodiment of the present specification provides an evaluation method for a static program security testing tool. When an evaluation request for a static program security testing tool for a preset programming language is received, the syntax feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool are obtained based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the syntax feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, the plurality of different second evaluation indicators can be constructed based on the different For each of the same first evaluation indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (that is, positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which can not only more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0036] In practical applications, one or more different positive and negative sample pairs matching the evaluation index can be obtained from a specified database. To this end, the above database can be constructed by constructing a test sample set, wherein the test sample set can be constructed in the following manner. For details, please refer to the processing of the following steps A2 to A8.

[0037] In step A2, a target static program security testing tool for a target programming language is obtained.

[0038] The target programming language may be any programming language, such as C language or Java programming language, etc., and may be set according to actual conditions. The target static program security testing tool may be any type of static program security testing tool. In actual applications, the target static program security testing tool may also be the same as the above-mentioned static program security testing tool, and may be set according to actual conditions.

[0039] In implementation, the target static program security testing tool for the target programming language can be obtained in a variety of different ways. For example, the target static program security testing tool for the target programming language can be crawled from the Internet by a web crawler, or one or more target static program security testing tools for the target programming language can be selected from a specified database, or the target static program security testing tool for the target programming language can be obtained from different users (such as through exchange or from users who are willing to provide data free of charge, etc.). The specific setting can be based on actual conditions, and the embodiments of this specification are not limited to this.

[0040] In step A4, a language characteristic analysis is performed on the target programming language to obtain the grammatical characteristic information of the target programming language, and the target static program security testing tool is analyzed to determine different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the target static program security testing tool.

[0041] Among them, the grammatical characteristic information of the target programming language can be different according to the target programming language, which can be specifically based on the above-mentioned related content and will not be repeated here. The different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool can be different according to the target static program security testing tool, which can be specifically based on the above-mentioned related content and will not be repeated here.

[0042] In step A6, multiple different third evaluation indicators for evaluation completeness are constructed based on the grammatical characteristic information of the target programming language, and multiple different fourth evaluation indicators for evaluation accuracy are constructed based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the target static program security testing tool. The multiple different third evaluation indicators include a third evaluation indicator for analyzing and detecting a target data structure contained in a target program written in the target programming language and / or a target operation of the target data structure, and / or a third evaluation indicator for analyzing logical features in the target program. The multiple different fourth evaluation indicators include a fourth evaluation indicator for performing sensitivity analysis on the target static program security testing tool.

[0043] The specific processing process of the above step A6 can refer to the above related content and will not be repeated here.

[0044] In step A8, one or more different positive and negative sample pairs are generated based on each third evaluation indicator, and one or more different positive and negative sample pairs are generated based on each fourth evaluation indicator, and the generated positive and negative sample pairs are stored in the test sample set.

[0045] In implementation, for any third evaluation indicator, one or more different positive and negative sample pairs can be generated based on the third evaluation indicator, and the same method can be used to generate one or more different positive and negative sample pairs for each third evaluation indicator. For any fourth evaluation indicator, one or more different positive and negative sample pairs can be generated based on the fourth evaluation indicator, and the same method can be used to generate one or more different positive and negative sample pairs for each fourth evaluation indicator. The positive and negative sample pairs generated above can be stored in the test sample set.

[0046] After the test sample set is generated in the above manner, the specific processing method of obtaining one or more different positive and negative sample pairs matching each evaluation indicator based on multiple different first evaluation indicators and multiple different second evaluation indicators in the above step S108 can be varied. An optional processing method is provided below, which may specifically include the following: based on multiple different first evaluation indicators and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained from the test sample set.

[0047] In practical applications, after obtaining the test sample set in the above manner, the positive and negative sample pairs in the test sample set can also be evaluated. For details, please refer to the following processing: Based on the positive and negative sample pairs in the test sample set, use the preset static code analysis tool for the target programming language to evaluate the test sample set and obtain the corresponding evaluation results.

[0048] Among them, static code analysis tools can include multiple types, for example, SonarQube, FindBugs, TscanCode, etc., which can be set according to actual conditions. In addition, static code analysis tools can also vary according to the target programming language. For a specified target programming language, a specified static code analysis tool can be included. In actual applications, the static code analysis tool for the target programming language can be one tool or multiple different tools, which can be set according to actual conditions. The embodiments of this specification do not limit this.

[0049] In implementation, one or more different positive and negative sample pairs corresponding to a certain evaluation indicator can be obtained, and the positive and negative samples can be used to test the above static code analysis tool to test the support of the above evaluation indicators and obtain corresponding results. In the above manner, the support of each evaluation indicator can be tested and the corresponding results can be obtained, thereby completing the evaluation of the test sample set and obtaining the corresponding evaluation results.

[0050] Based on the above processing, the specific processing method of obtaining one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set based on each evaluation indicator in the multiple different first evaluation indicators and the multiple different second evaluation indicators can be varied. An optional processing method is provided below, which may specifically include the following contents: If the above evaluation result is that the evaluation passes, then based on each evaluation indicator in the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained from the test sample set.

[0051] In addition, if the above evaluation result is that the evaluation fails, the above steps A2 to A8 can be used to continue generating new positive and negative sample pairs, and the new positive and negative sample pairs can be evaluated until the evaluation result is that the evaluation passes.

[0052] In practical applications, in order to highlight the evaluation advantages of the test sample set and reflect the "completeness" of the test sample set in code structure analysis, the above-mentioned positive and negative sample pairs in the test sample set are based on the use of a preset static code analysis tool for the target programming language to evaluate the test sample set, and the corresponding evaluation results are obtained. The test sample set can also be evaluated by cross-validation. For details, please refer to the following content: Based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set, the preset static code analysis tool for the target programming language is used to evaluate the test sample set in a cross-validation manner to obtain the corresponding evaluation results.

[0053] Among them, the benchmark sample set may include multiple types. For example, the benchmark sample set may be a commonly used sample set, such as a Benchmark sample set, a Geekbench sample set, a PassMark sample set, etc., which can be set according to actual conditions.

[0054] In implementation, a sample (either a positive sample or a negative sample) in a positive and negative sample pair corresponding to a specified evaluation may be selected from a test sample set. In addition, a sample may be selected from a benchmark sample set. Then, a preset static code analysis tool for the target programming language may be used to analyze the two selected samples respectively to obtain corresponding analysis results. In the above manner, the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set may be analyzed using the above static code analysis tool to obtain corresponding analysis results. The above analysis results may be collected, and a final evaluation may be made based on the collected analysis results to obtain corresponding evaluation results.

[0055] In practical applications, the completeness and rationality of the evaluation system can be ensured by cross-validation with existing benchmark sample sets. In order to evaluate the relative completeness of the designed evaluation system and its test sample set under specific standards, the constructed test sample set can be cross-validated with the existing test sets in the same field for evaluation capabilities. Based on this, the target programming language is the Java programming language, the benchmark sample set is the Benchmark sample set, and the static code analysis tool is JavaParser. Based on the above situation, the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set are evaluated by using the preset static code analysis tool for the target programming language in a cross-validation manner. The specific processing methods for obtaining the corresponding evaluation results can be varied. An optional processing method is provided below, which can specifically include the processing of the following steps B2 to B6.

[0056] In step B2, JavaParser is used to perform AST node type detection on the positive and negative sample pairs in the test sample set to obtain the coverage of the AST node type corresponding to the test sample set.

[0057] In step B4, JavaParser is used to perform AST node type detection on the samples in the benchmark sample set to obtain the coverage of the AST node types corresponding to the benchmark sample set.

[0058] In step B6, based on the coverage of the AST node type corresponding to the test sample set and the coverage of the AST node type corresponding to the benchmark sample set, an evaluation result of evaluating the test sample set is determined.

[0059] In implementation, the degree of analysis support for common code structures can be reflected by comparing the coverage of AST node types by different sample sets (i.e., test sample set and benchmark sample set) to reflect the "completeness" of the test sample set in code structure analysis. Specifically, the coverage of AST node types corresponding to the test sample set and the coverage of AST node types corresponding to the benchmark sample set can be compared. If the coverage of AST node types corresponding to the test sample set is greater than the coverage of AST node types corresponding to the benchmark sample set, the evaluation result of the test sample set is determined to be passed. If the coverage of AST node types corresponding to the test sample set is greater than the coverage of AST node types corresponding to the benchmark sample set, the evaluation result of the test sample set is determined to be failed. In the above manner, combined with the core technology of static analysis, the corresponding evaluation items (i.e., evaluation indicators) are set from the core technology perspectives of static analysis such as AST node types, characteristics of programming languages, sensitivity analysis, etc., rather than designing classifications such as vulnerability types, so that the core performance of static application security testing tools can be more scientifically characterized.

[0060] In practical applications, the preset programming language in the above steps S102 to S108 can be the Java programming language, and the data structure can include one or more of a collection, an array, and an object. For the Java programming language, the specific data structures included above are only some exemplary specific data structures, and in practical applications, the data structure does not only include the above one or more, but also includes other specific data structures in addition to the above specific data structures, which can be set specifically according to actual conditions. The logical features in the above program can include one or more of control flow, exceptions, and reflection. For the Java programming language, the specific logical features included in the above program are only some exemplary specific logical features, and in practical applications, the above program does not only include the above one or more specific logical features, but also includes other specific logical features in addition to the above specific logical features, which can be set specifically according to actual conditions.

[0061] In actual applications, multiple different second evaluation indicators include second evaluation indicators for performing sensitivity analysis on static program security testing tools through taint objects and / or taint links. Taint objects are objects that track data flow to discover potential risks, and taint links are links that track data flow to discover potential risks.

[0062] In practical applications, sensitivity analysis of static program security testing tools includes one or more of sensitivity analysis of flow sensitivity, sensitivity analysis of object sensitivity, sensitivity analysis of domain sensitivity, sensitivity analysis of path sensitivity, and sensitivity analysis of context sensitivity. In practical applications, the sensitivity analysis of specific sensitive types included in the sensitivity analysis of static program security testing tools are only some exemplary sensitivity analyses, and in practical applications, the sensitivity analysis of static program security testing tools does not only include one or more of the above, but may also include sensitivity analysis of other sensitive types in addition to the sensitivity analysis of the above specific sensitive types, which can be set specifically according to actual conditions.

[0063] An embodiment of the present specification provides an evaluation method for a static program security testing tool. When an evaluation request for a static program security testing tool for a preset programming language is received, the grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool are obtained based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the grammatical feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, the plurality of different first evaluation indicators can be constructed based on the plurality of different first evaluation indicators. For each evaluation indicator in the evaluation indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and the static program security testing tool is evaluated for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (i.e., positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which can not only more accurately reflect the detection capability of the static application security testing tool, reduce the evaluation complexity, and avoid ambiguous test results, but also provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0064] In addition, when setting up the evaluation system, the Java language syntax characteristics and the principles of static program security testing tools were fully considered, and the evaluation indicators were set by integrating multiple dimensions such as completeness, accuracy and performance. This setting ensures the systematization and scientificity of the evaluation system and avoids subjectivity and unsystematicity in the evaluation process. In addition, the test sample set is cross-validated through a specific Benchmark sample set to ensure the completeness and rationality of the evaluation system. This verification method not only improves the credibility of the evaluation system, but also makes the evaluation results more universal and applicable.

[0065] The above is an evaluation method for a static program security testing tool provided in an embodiment of this specification. Based on the same idea, an evaluation device for a static program security testing tool is also provided in an embodiment of this specification. Figure 3 shown.

[0066] The evaluation device of the static program security testing tool includes: a request receiving module 301, an information acquisition module 302, an evaluation index construction module 303 and a tool evaluation module 304, wherein:

[0067] A request receiving module 301 receives an evaluation request for a static program security testing tool for a preset programming language;

[0068] An information acquisition module 302 is configured to acquire, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool;

[0069] An evaluation indicator construction module 303 constructs a plurality of different first evaluation indicators for evaluation completeness based on the grammatical characteristic information of the preset programming language, and constructs a plurality of different second evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the accuracy of program analysis capability of the static program safety testing tool, wherein the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program safety testing tool;

[0070] The tool evaluation module 304 obtains one or more different positive and negative sample pairs matching each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, and evaluates the static program security testing tool for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool.

[0071] In the embodiment of this specification, the device further includes:

[0072] A tool acquisition module, which acquires a target static program security testing tool for a target programming language;

[0073] An analysis module performs language characteristic analysis on the target programming language to obtain syntax characteristic information of the target programming language, and analyzes the target static program security testing tool to determine information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool;

[0074] An indicator construction module constructs a plurality of different third evaluation indicators for evaluation completeness based on the grammatical characteristic information of the target programming language, and constructs a plurality of different fourth evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program security testing tool, wherein the plurality of different third evaluation indicators include a third evaluation indicator for analyzing and detecting a target data structure contained in a target program written in the target programming language and / or a target operation of the target data structure, and / or a third evaluation indicator for analyzing logical features in the target program, and the plurality of different fourth evaluation indicators include a fourth evaluation indicator for performing sensitivity analysis on the target static program security testing tool;

[0075] A test set generation module, which generates one or more different positive and negative sample pairs based on each of the third evaluation indicators, and generates one or more different positive and negative sample pairs based on each of the fourth evaluation indicators, and stores the generated positive and negative sample pairs in a test sample set;

[0076] The tool evaluation module 304 obtains one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set based on each evaluation indicator of the multiple different first evaluation indicators and the multiple different second evaluation indicators.

[0077] In the embodiment of this specification, the device further includes:

[0078] An evaluation module, based on the positive and negative sample pairs in the test sample set, uses a preset static code analysis tool for the target programming language to evaluate the test sample set and obtain corresponding evaluation results;

[0079] The tool evaluation module 304, if the evaluation result is passed, obtains one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set based on each evaluation indicator of the multiple different first evaluation indicators and the multiple different second evaluation indicators.

[0080] In an embodiment of the present specification, the evaluation module, based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set, respectively uses a preset static code analysis tool for the target programming language to evaluate the test sample set in a cross-validation manner to obtain corresponding evaluation results.

[0081] In the embodiment of this specification, the target programming language is the Java programming language, the benchmark sample set is the Benchmark sample set, and the static code analysis tool is JavaParser;

[0082] The evaluation module comprises:

[0083] A first coverage determination unit, using the JavaParser to perform AST node type detection on the positive and negative sample pairs in the test sample set, to obtain the coverage of the AST node type corresponding to the test sample set;

[0084] A second coverage determination unit, using the JavaParser to perform AST node type detection on samples in the benchmark sample set to obtain coverage of AST node types corresponding to the benchmark sample set;

[0085] The evaluation unit determines an evaluation result of evaluating the test sample set based on the coverage of the AST node type corresponding to the test sample set and the coverage of the AST node type corresponding to the benchmark sample set.

[0086] In the embodiment of this specification, the preset programming language is the Java programming language, the data structure includes but is not limited to one or more of a collection, an array, and an object, and the logical features in the program include but are not limited to one or more of control flow, exceptions, and reflection.

[0087] In an embodiment of the present specification, the multiple different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool by means of taint objects and / or taint links, wherein the taint objects are objects that track data flow to discover potential risks, and the taint links are links that track data flow to discover potential risks.

[0088] In an embodiment of the present specification, the sensitivity analysis performed on the static program security testing tool includes one or more of a sensitivity analysis performed on the static program security testing tool for flow sensitivity, a sensitivity analysis performed on the static program security testing tool for object sensitivity, a sensitivity analysis performed on the static program security testing tool for domain sensitivity, a sensitivity analysis performed on the static program security testing tool for path sensitivity, and a sensitivity analysis performed on the static program security testing tool for context sensitivity.

[0089] An embodiment of the present specification provides an evaluation device for a static program security testing tool. When an evaluation request for a static program security testing tool for a preset programming language is received, the evaluation device obtains grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the grammatical feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, the plurality of different second evaluation indicators can be constructed based on the plurality of different For each of the same first evaluation indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (that is, positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which can not only more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0090] In addition, when setting up the evaluation system, we fully considered the characteristics of the Java language and the principles of static program security testing tools, and set evaluation indicators by integrating multiple dimensions such as completeness, accuracy, and performance. This setting ensures the systematization and scientificity of the evaluation system and avoids subjectivity and unsystematicity in the evaluation process. In addition, the test sample set is cross-validated through a specific Benchmark sample set to ensure the completeness and rationality of the evaluation system. This verification method not only improves the credibility of the evaluation system, but also makes the evaluation results more universal and applicable.

[0091] The above is an evaluation device for a static program security testing tool provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide an evaluation device for a static program security testing tool, such as Figure 4 shown.

[0092] The evaluation device of the static program security testing tool may provide a terminal device or a server, etc. for the above-mentioned embodiment.

[0093] The evaluation device of the static program security testing tool may have relatively large differences due to different configurations or performances, and may include one or more processors 401 and memory 402, and one or more storage applications or data may be stored in the memory 402. Among them, the memory 402 may be a short-term storage or a persistent storage. The application stored in the memory 402 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the evaluation device of the static program security testing tool. Furthermore, the processor 401 may be configured to communicate with the memory 402, and execute a series of computer executable instructions in the memory 402 on the evaluation device of the static program security testing tool. The evaluation device of the static program security testing tool may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input and output interfaces 405, and one or more keyboards 406.

[0094] Specifically, in this embodiment, the evaluation device of the static program security testing tool includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the evaluation device of the static program security testing tool, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions:

[0095] Receive evaluation requests for static program security testing tools for a preset programming language;

[0096] Based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool are obtained;

[0097] Based on the grammatical characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program safety testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing a sensitivity analysis on the static program safety testing tool;

[0098] Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0099] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the evaluation device embodiment of the static program security testing tool, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0100] An embodiment of the present specification provides an evaluation device for a static program security testing tool. When an evaluation request for a static program security testing tool for a preset programming language is received, the device obtains grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the grammatical feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, the plurality of different second evaluation indicators can be constructed based on the plurality of different For each of the same first evaluation indicator and multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (that is, positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which can not only more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0101] Furthermore, based on the above Figure 1 to Figure 2 In one embodiment, the present specification further provides a storage medium for storing computer executable instruction information. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instruction information stored in the storage medium is executed by the processor, the following process can be implemented:

[0102] Receive evaluation requests for static program security testing tools for a preset programming language;

[0103] Based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool are obtained;

[0104] Based on the grammatical characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program safety testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing a sensitivity analysis on the static program safety testing tool;

[0105] Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0107] An embodiment of the present specification provides a storage medium, which, upon receiving an evaluation request for a static program security testing tool for a preset programming language, obtains grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool based on the evaluation request. Then, a plurality of different first evaluation indicators for evaluation completeness can be constructed based on the grammatical feature information of the preset programming language, and a plurality of different second evaluation indicators for evaluation accuracy can be constructed based on the different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. The plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program. The plurality of different second evaluation indicators include a second evaluation indicator for sensitivity analysis of the static program security testing tool. Finally, the plurality of different first evaluation indicators can be constructed based on the plurality of different first evaluation indicators. The method comprises the following steps: first, evaluating the first evaluation indicator and each of the multiple different second evaluation indicators, obtaining one or more different positive and negative sample pairs matching each evaluation indicator, and evaluating the static program security testing tool for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (i.e., positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which can not only more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also provide a detailed explanation of the test results, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed explanation of the test results is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0108] Furthermore, based on the above Figure 1 to Figure 2 In one or more embodiments of the present specification, a computer program product is provided, including a computer program. When the computer program in the computer program product is executed by a processor, the following process can be implemented:

[0109] Receive evaluation requests for static program security testing tools for a preset programming language;

[0110] Based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool are obtained;

[0111] Based on the grammatical characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program safety testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing a sensitivity analysis on the static program safety testing tool;

[0112] Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

[0113] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned computer program product embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0114] An embodiment of the present specification provides a computer program product, which, upon receiving an evaluation request for a static program security testing tool for a preset programming language, obtains grammatical feature information of the preset programming language and information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool based on the evaluation request, and then, can construct a plurality of different first evaluation indicators for evaluation completeness based on the grammatical feature information of the preset programming language, and construct a plurality of different second evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the program analysis capability accuracy of the static program security testing tool. Finally, based on each of the plurality of different first evaluation indicators and the plurality of different second evaluation indicators, can obtain one or more different positive and negative sample pairs matching each evaluation indicator, and based on the obtained positive and negative sample pairs, static program security testing can be tested. The security testing tool evaluates different evaluation indicators respectively to determine the evaluation results of the static program security testing tool. In this way, through multi-dimensional evaluation indicators and fine-grained test samples (i.e., positive and negative sample pairs corresponding to multiple different evaluation indicators), the test function point corresponding to each test sample can be clarified, which can not only more accurately reflect the detection capability of the static application security testing tool and avoid ambiguous test results, but also provide detailed test result explanations, thereby comprehensively evaluating the performance of the static application security testing tool. In addition, a test function point is characterized by a combination of multiple positive and negative samples, and a detailed test result explanation is provided, making the results more targeted and operational. Moreover, through the scientific design of evaluation indicators, the shortcomings of unclear and unsystematic evaluation standards are overcome, the systematization level of security testing is improved, and it is helpful to promote the development and application of SAST technology.

[0115] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0117] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0118] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0120] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0121] The embodiments of this specification are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable fraud case serial and parallel device to produce a machine, so that the instructions executed by the processor of the computer or other programmable fraud case serial and parallel device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable fraud case serial and parallel device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0126] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0128] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0129] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0130] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0131] The above description is only an embodiment of this specification and is not intended to limit this document. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this specification.

Claims

1. A static program security testing tool evaluation method, the method comprising: Receive evaluation requests for static program security testing tools for a preset programming language; Based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool are obtained; Based on the grammatical characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program safety testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing a sensitivity analysis on the static program safety testing tool; Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

2. The method according to claim 1, further comprising: Obtain a target static program security testing tool for a target programming language; Performing language characteristic analysis on the target programming language to obtain syntax characteristic information of the target programming language, and analyzing the target static program safety testing tool to determine information of different dimensions of sensitivity analysis related to the program analysis capability accuracy of the target static program safety testing tool; Based on the syntax characteristic information of the target programming language, a plurality of different third evaluation indicators for evaluation completeness are constructed, and based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the target static program security testing tool, a plurality of different fourth evaluation indicators for evaluation accuracy are constructed, the plurality of different third evaluation indicators include a third evaluation indicator for analyzing and detecting a target data structure contained in a target program written in the target programming language and / or a target operation of the target data structure, and / or a third evaluation indicator for analyzing logical features in the target program, and the plurality of different fourth evaluation indicators include a fourth evaluation indicator for performing sensitivity analysis on the target static program security testing tool; Generate one or more different positive and negative sample pairs based on each of the third evaluation indicators, and generate one or more different positive and negative sample pairs based on each of the fourth evaluation indicators, and store the generated positive and negative sample pairs in a test sample set; The acquiring, based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator comprises: Based on each evaluation indicator of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained from the test sample set.

3. The method according to claim 2, further comprising: Based on the positive and negative sample pairs in the test sample set, the test sample set is evaluated using a preset static code analysis tool for the target programming language to obtain corresponding evaluation results; The acquiring, based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator from the test sample set comprises: If the evaluation result is passed, based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained from the test sample set.

4. The method according to claim 3, wherein based on the positive and negative sample pairs in the test sample set, the test sample set is evaluated using a preset static code analysis tool for a target programming language to obtain a corresponding evaluation result, comprising: Based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set, the test sample set is evaluated in a cross-validation manner using a preset static code analysis tool for the target programming language to obtain corresponding evaluation results.

5. The method according to claim 4, wherein the target programming language is the Java programming language, the benchmark sample set is the Benchmark sample set, and the static code analysis tool is JavaParser; Based on the positive and negative sample pairs in the test sample set and the samples in the benchmark sample set, the preset static code analysis tool for the target programming language is used to evaluate the test sample set in a cross-validation manner to obtain corresponding evaluation results, including: Using the JavaParser to perform AST node type detection on the positive and negative sample pairs in the test sample set, and obtaining the coverage of the AST node type corresponding to the test sample set; Using the JavaParser to perform AST node type detection on samples in the benchmark sample set to obtain coverage of AST node types corresponding to the benchmark sample set; Based on the coverage of the AST node type corresponding to the test sample set and the coverage of the AST node type corresponding to the benchmark sample set, an evaluation result of evaluating the test sample set is determined.

6. According to the method of claim 1, the preset programming language is Java programming language, the data structure includes but is not limited to one or more of collection, array, and object, and the logical features in the program include but are not limited to one or more of control flow, exception, and reflection.

7. According to the method described in claim 1, the multiple different second evaluation indicators include a second evaluation indicator for performing sensitivity analysis on the static program security testing tool by means of taint objects and / or taint links, wherein the taint object is an object that tracks data flow to discover potential risks, and the taint link is a link that tracks data flow to discover potential risks.

8. According to the method according to claim 1, the sensitivity analysis of the static program security testing tool includes one or more of a sensitivity analysis of the static program security testing tool for flow sensitivity, a sensitivity analysis of the static program security testing tool for object sensitivity, a sensitivity analysis of the static program security testing tool for domain sensitivity, a sensitivity analysis of the static program security testing tool for path sensitivity, and a sensitivity analysis of the static program security testing tool for context sensitivity.

9. An evaluation device for a static program security testing tool, the device comprising: A request receiving module, receiving an evaluation request for a static program security testing tool for a preset programming language; An information acquisition module, which acquires, based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool; An evaluation indicator construction module is configured to construct a plurality of different first evaluation indicators for evaluation completeness based on the grammatical characteristic information of the preset programming language, and to construct a plurality of different second evaluation indicators for evaluation accuracy based on different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program safety testing tool, wherein the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing a sensitivity analysis on the static program safety testing tool; The tool evaluation module obtains one or more different positive and negative sample pairs matching each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, and evaluates the static program security testing tool for different evaluation indicators based on the obtained positive and negative samples to determine the evaluation results of the static program security testing tool.

10. An evaluation device for a static program security testing tool, the evaluation device for a static program security testing tool comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Receive evaluation requests for static program security testing tools for a preset programming language; Based on the evaluation request, information on syntax characteristics of the preset programming language and information on different dimensions of sensitivity analysis related to the accuracy of program analysis capabilities of the static program security testing tool are obtained; Based on the grammatical characteristic information of the preset programming language, a plurality of different first evaluation indicators for evaluation completeness are constructed, and based on the test types corresponding to different dimensions of sensitivity analysis related to the accuracy of the program analysis capability of the static program safety testing tool, a plurality of different second evaluation indicators for evaluation accuracy are constructed, the plurality of different first evaluation indicators include a first evaluation indicator for analyzing and detecting a data structure contained in a program written in the preset programming language and / or a preset operation of the data structure, and / or a first evaluation indicator for analyzing a logical feature in the program, and the plurality of different second evaluation indicators include a second evaluation indicator for performing a sensitivity analysis on the static program safety testing tool; Based on each of the multiple different first evaluation indicators and the multiple different second evaluation indicators, one or more different positive and negative sample pairs matching each evaluation indicator are obtained, and based on the obtained positive and negative samples, the static program security testing tool is evaluated for different evaluation indicators respectively to determine the evaluation results of the static program security testing tool.

Images