A testing method and device for multiple vulnerability scanners

Abstract

The present application relates to the field of network security technology, and in particular to a testing method and device for multiple vulnerability scanners. For each application supported by each vulnerability scanner, the number of samples corresponding to an application is determined according to the number of CVEs of an application and the total number of CVEs; for each application supported by each vulnerability scanner, a preset number of version information is randomly sampled from the obtained version information set according to the number of samples corresponding to each application, and the application samples corresponding to the preset number of version information are determined; each vulnerability scanner is called respectively to scan the container environment generated by each application sample to obtain each CVE scanned by each vulnerability scanner; for each vulnerability scanner, the accuracy detection result of a vulnerability scanner is determined according to each CVE of a vulnerability scanner to be tested and the obtained benchmark data set. In this way, the detection accuracy of multiple types of vulnerability scanners can be evaluated at the same time.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a testing method and device for a multi-vulnerability scanner. Background Art

[0002] As computer system security issues continue to emerge, different industries are paying more and more attention to computer system security issues. Therefore, various types of vulnerability scanners are gradually increasing. However, different vulnerability scanners may have different types of Common Vulnerabilities & Exposures (CVE) that they can scan and their detection accuracy. Therefore, how to determine the detection accuracy of each vulnerability scanner and provide important indicators for users to choose vulnerability scanners has become an urgent problem to be solved.

[0003] In the related art, when evaluating the detection accuracy of vulnerability scanners, it is generally only possible to evaluate a single type of vulnerability scanner. If it is necessary to evaluate the detection accuracy of multiple types of vulnerability scanners, this method in the related art obviously cannot meet the needs of multiple vulnerability scanners due to the lack of evaluation standards. Therefore, how to evaluate the detection accuracy of multiple types of vulnerability scanners at the same time has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method and device for testing multiple vulnerability scanners to simultaneously evaluate the detection accuracy of multiple types of vulnerability scanners.

[0005] The specific technical solutions provided by the embodiments of this application are as follows:

[0006] A method for testing a multi-vulnerability scanner, comprising:

[0007] For each application supported by each vulnerability scanner, determine the number of samples corresponding to the application according to the number of CVEs of the application and the total number of CVEs;

[0008] According to the number of samples corresponding to each application, a preset number of version information is randomly sampled from the acquired version information set, and application samples corresponding to the preset number of version information are determined, wherein the version information set at least includes each version information corresponding to each application supported by each vulnerability scanner for scanning;

[0009] Invoke the vulnerability scanners respectively to scan the container environment generated by the application samples, and obtain the CVEs scanned by the vulnerability scanners;

[0010] For each vulnerability scanner, the accuracy detection result of the vulnerability scanner is determined according to the CVEs of the vulnerability scanner to be tested and the obtained benchmark data set, wherein the benchmark data set includes all CVEs corresponding to the applications supported by the vulnerability scanners.

[0011] Optionally, determining the number of samples corresponding to an application according to the number of CVEs of the application and the total number of CVEs includes:

[0012] Determine the number of CVEs corresponding to each application supported by each vulnerability scanner, and the total number of CVEs, wherein the total number of CVEs represents the sum of the numbers of each CVE;

[0013] Calculate the ratio between the number of CVEs for an application and the total number of CVEs;

[0014] The number of samples corresponding to the application is determined according to the ratio and a preset total number of samples.

[0015] Optionally, determine the number of CVEs corresponding to each application that each vulnerability scanner supports scanning, including:

[0016] Determine respectively each application supported by each vulnerability scanner for scanning, as well as the type information of the corresponding CVE and the number of application CVEs, wherein the number of application CVEs is the number of CVEs included in an application of a vulnerability scanner;

[0017] For each application, the sum of the number of application CVEs of any one application in the vulnerability scanners is calculated, and the sum is used as the number of CVEs of the one application.

[0018] Optionally, the benchmark data set is obtained in the following manner:

[0019] Query the applications and version information corresponding to the applications from a standard vulnerability database;

[0020] For each of the applications, respectively, according to each version information of an application, determine the CVE included in the application under the each version information;

[0021] Generate a benchmark data set containing the CVEs included in each application under each version information.

[0022] Optionally, for each vulnerability scanner, according to each CVE of a vulnerability scanner to be tested and the acquired benchmark data set, determining the accuracy detection result of the vulnerability scanner specifically includes:

[0023] For each of the CVEs, if it is determined that a CVE is included in the benchmark data set, then the detection result of the CVE is determined to be correct; otherwise, the detection result of the CVE is determined to be a false positive;

[0024] If it is determined that the CVE included in the benchmark data set is not included in the CVEs, then determining that the detection result of the CVE is a missed report;

[0025] According to the detection results of each CVE, the accuracy detection result of the vulnerability scanner is determined.

[0026] Optionally, determining the accuracy detection result of the vulnerability scanner according to the detection results of each CVE includes:

[0027] Determine, for each of the applications, the number of CVEs for which detection results are correct, the number of CVEs for which detection results are false positives, and the number of CVEs for which detection results are missed negatives for one application, and determine the total number of CVEs for the one application included in the benchmark data set;

[0028] For each of the applications, respectively, determine the accuracy of the application according to the number of CVEs whose detection results are correct and the total number, determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number, and determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number;

[0029] The accuracy detection result of the vulnerability scanner is determined based on the accuracy rate, false positive rate and missed negative rate corresponding to each application.

[0030] A testing device for a multi-vulnerability scanner, comprising:

[0031] A first determination module is used to determine, for each application supported by each vulnerability scanner, the number of samples corresponding to the application according to the number of CVEs of the application and the total number of CVEs;

[0032] A sampling module, configured to randomly sample a preset number of version information from the acquired version information set according to the number of samples corresponding to each application, and determine application samples corresponding to the preset number of version information, wherein the version information set at least includes each version information corresponding to each application supported by each vulnerability scanner;

[0033] A scanning module, used to call each vulnerability scanner respectively, scan the container environment generated by each application sample, and obtain each CVE scanned by each vulnerability scanner;

[0034] A detection module is used to determine the accuracy detection result of a vulnerability scanner for each vulnerability scanner respectively according to each CVE of the vulnerability scanner to be tested and an acquired benchmark data set, wherein the benchmark data set includes all CVEs corresponding to each application supported by the vulnerability scanner.

[0035] Optionally, when determining the number of samples corresponding to an application according to the number of CVEs of the application and the total number of CVEs, the sampling module is specifically configured to:

[0036] Determine the number of CVEs corresponding to each application supported by each vulnerability scanner, and the total number of CVEs, wherein the total number of CVEs represents the sum of the numbers of each CVE;

[0037] Calculate the ratio between the number of CVEs for an application and the total number of CVEs;

[0038] The number of samples corresponding to the application is determined according to the ratio and a preset total number of samples.

[0039] Optionally, when determining the number of CVEs corresponding to each application supported by each vulnerability scanner, the scanning module is specifically used for:

[0040] Determine respectively each application supported by each vulnerability scanner for scanning, as well as the type information of the corresponding CVE and the number of application CVEs, wherein the number of application CVEs is the number of CVEs included in an application of a vulnerability scanner;

[0041] For each application, the sum of the number of application CVEs of any one application in the vulnerability scanners is calculated, and the sum is used as the number of CVEs of the one application.

[0042] Optionally, when obtaining the benchmark data set, the step further includes:

[0043] A query module, used to query the applications and version information corresponding to the applications from a standard vulnerability database;

[0044] A second determination module is used to determine, for each of the applications, the CVEs included in the application under each version information according to each version information of the application;

[0045] The generation module is used to generate a benchmark data set including the CVE included in each application under each version information.

[0046] Optionally, the detection module is specifically used for:

[0047] For each of the CVEs, if it is determined that a CVE is included in the benchmark data set, then the detection result of the CVE is determined to be correct; otherwise, the detection result of the CVE is determined to be a false positive;

[0048] If it is determined that the CVE included in the benchmark data set is not included in the CVEs, then determining that the detection result of the CVE is a missed report;

[0049] According to the detection results of each CVE, the accuracy detection result of the vulnerability scanner is determined.

[0050] Optionally, when determining the accuracy detection result of the vulnerability scanner according to the detection results of each CVE, the detection module is specifically used to:

[0051] Determine, for each of the applications, the number of CVEs for which detection results are correct, the number of CVEs for which detection results are false positives, and the number of CVEs for which detection results are missed negatives for one application, and determine the total number of CVEs for the one application included in the benchmark data set;

[0052] For each of the applications, respectively, determine the accuracy of the application according to the number of CVEs whose detection results are correct and the total number, determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number, and determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number;

[0053] The accuracy detection result of the vulnerability scanner is determined based on the accuracy rate, false positive rate and missed negative rate corresponding to each application.

[0054] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for testing a multi-vulnerability scanner when executing the program.

[0055] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned multi-vulnerability scanner testing method when executed by a processor.

[0056] In the embodiment of the present application, for each application supported by each vulnerability scanner, the number of samples corresponding to an application is determined according to the number of CVEs of an application and the total number of CVEs, and a preset number of version information is randomly sampled from the obtained version information set according to the number of samples corresponding to each application, and the application samples corresponding to the preset number of version information are determined, and each vulnerability scanner is called to scan the container environment generated by each application sample to obtain each CVE scanned by each vulnerability scanner, and for each vulnerability scanner, the accuracy detection result of a vulnerability scanner is determined according to each CVE of a vulnerability scanner to be tested and the obtained benchmark data set. In this way, by taking the CVEs of each application supported by each vulnerability scanner as scanning samples and obtaining a benchmark data set, the scanned CVEs are compared with the benchmark data set to detect the accuracy of each vulnerability scanner, which can provide a unified standard for the detection of multiple types of vulnerability scanners, and realize the detection of multiple types of vulnerability scanners at the same time, thereby improving the accuracy of the detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a method for testing a multi-vulnerability scanner in an embodiment of the present application;

[0058] Figure 2 This is a schematic diagram of the version impact interval division in the embodiment of this application;

[0059] Figure 3 This is a schematic diagram of dividing the number of CVEs based on versions in an embodiment of the present application;

[0060] Figure 4 Another flow chart of a method for testing a multi-vulnerability scanner in an embodiment of the present application;

[0061] Figure 5 This is a schematic diagram of the structure of a testing device for a multi-vulnerability scanner in an embodiment of the present application;

[0062] Figure 6 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] As computer system security issues continue to emerge, different industries are paying more and more attention to computer system security issues. Therefore, various types of vulnerability scanners are gradually increasing. The number of vulnerabilities contained in each product and the accuracy of the vulnerabilities are important indicators for users to evaluate the quality of the product. However, different vulnerability scanners may have different CVEs that can be scanned and the accuracy of detection. Therefore, how to determine the detection accuracy of each vulnerability scanner and provide important indicators for users to choose vulnerability scanners has become an urgent problem to be solved.

[0065] In the related art, when evaluating the detection accuracy of vulnerability scanners, for a single type of scanner, only the accuracy of a certain vulnerability scanner can be evaluated, and this evaluation is based on the vulnerabilities added by the vulnerability scanner. If there are multiple types of scanners that need to be evaluated, due to the lack of standards, what is used as the standard for comparing multiple types of vulnerability scanners to obtain accuracy data, therefore, using this test method in the related art obviously cannot meet the needs of the scenario. If one of the vulnerability scanners is selected as the standard, it will inevitably have adverse factors on other vulnerability scanners. Therefore, when multiple vulnerability scanners are compared, how to more objectively evaluate the accuracy of the vulnerability scanner has become an urgent problem to be solved. In addition, how to verify the reporting of vulnerabilities in different vulnerability scanners, without understanding the rules, it is necessary to build a test environment to scan. In the related art, when building a test environment, it is basically based on a physical machine or a virtual machine. When the number of applications and systems increases, building, copying and maintaining the container environment also becomes time-consuming and laborious.

[0066] In order to solve the above problems, a test method for multiple vulnerability scanners is provided in an embodiment of the present application. For each application supported by each vulnerability scanner, the number of samples corresponding to an application is determined according to the number of CVEs of an application and the total number of CVEs. According to the number of samples corresponding to each application, a preset number of version information is randomly sampled from the obtained version information set, and the application samples corresponding to the preset number of version information are determined. Each vulnerability scanner is called to scan the container environment generated by each application sample to obtain each CVE scanned by each vulnerability scanner. According to each CVE of a vulnerability scanner to be tested and the obtained benchmark data set, the accuracy detection result of a vulnerability scanner is determined for each vulnerability scanner. In this way, using the benchmark data as the comparison standard, different vulnerability scanners use the same standard for accuracy calculation, which can make the output accuracy detection result more objective and reliable.

[0067] Based on the above embodiments, see Figure 1 FIG. 1 is a flow chart of a method for testing a multi-vulnerability scanner in an embodiment of the present application, which specifically includes:

[0068] Step 100: for each application supported by each vulnerability scanner, determine the number of samples corresponding to one application according to the number of CVEs of one application and the total number of CVEs.

[0069] In an embodiment of the present application, when performing stratified sampling, the number of samples corresponding to each application supported by each vulnerability scanner is determined.

[0070] It should be noted that, since each vulnerability scanner supports a large number of applications and CVEs corresponding to the applications, it is necessary to sample from each application supported by each vulnerability scanner according to the preset stratified sampling rules and the number of CVEs of each application, and use the sampled applications as sample data of each vulnerability scanner, thereby building a container environment containing sample data. In other words, the samples in the embodiments of the present application are sampled from each application supported by each vulnerability scanner.

[0071] In addition, it should be noted that in the standard vulnerability database, the number of CVEs corresponding to each application is large, which does not mean that the number of CVEs that can be scanned by the actual vulnerability scanner is large. If for one of the applications, the total number of CVEs contained in the standard vulnerability database is large, but the total number of CVEs that can be scanned by the actual vulnerability scanner is small. Then, if the total number of CVEs contained in the standard database is used as the basis for stratified sampling, the proportion of samples obtained after stratified sampling will be relatively small. The accuracy of the vulnerability scanner finally determined will also have deviations, so the total number of CVEs that have been added to the applications of each vulnerability scanner for comparison needs to be used as the basis for stratified sampling. Therefore, in the embodiment of the present application, the number of CVEs corresponding to each application of each vulnerability scanner and the total number of CVEs that can be scanned by each vulnerability scanner are used as the basis for stratified sampling.

[0072] The embodiment of the present application provides a possible implementation method for determining the number of samples corresponding to an application. The following is a detailed description of the step of determining the number of samples corresponding to an application according to the number of CVEs and the total number of CVEs in the embodiment of the present application, specifically including:

[0073] S1: Determine the number of CVEs corresponding to each application supported by each vulnerability scanner, as well as the total number of CVEs.

[0074] The total number of CVEs represents the sum of the numbers of each CVE.

[0075] Specifically, when executing step S1, it specifically includes:

[0076] A1: Determine the applications that each vulnerability scanner supports scanning, as well as the type information of each corresponding CVE and the number of application CVEs.

[0077] The number of application CVEs is the number of CVEs included in a vulnerability scanner of the application.

[0078] In the embodiment of the present application, different vulnerability scanners may support different applications and application version information, and the types of CVEs that can be scanned by each vulnerability scanner may also be different.

[0079] Therefore, in an embodiment of the present application, first, for each vulnerability scanner, each application supported by a vulnerability scanner and the number of applications supported by the vulnerability scanner are obtained, and an application set containing each application is generated for the vulnerability scanner.

[0080] For example, suppose you need to test the accuracy of three vulnerability scanners, namely P a , P b and P c . Vulnerability Scanner P a A total of 5 applications are supported for scanning, namely app1, app2, app3, app4 and app5. The vulnerability scanner P a The application set can be expressed as: a ={app1,app2,app 3,app4,app5}, vulnerability scanner P a The number of applications supported for scanning is 5. Vulnerability Scanner P b A total of 4 applications are supported for scanning, namely app1, app3, app4 and app7. The vulnerability scanner P b The application set can be expressed as: b ={app1,app3,app4,app7}, vulnerability scanner P b The number of applications supported for scanning is 4. Vulnerability Scanner P c A total of 4 applications are supported for scanning, namely app2, app3, app4 and app8. c The application set can be expressed as: c ={app2,app3,app4,app8}, scanner to be tested P c The number of applications that support scanning is 4.

[0081] The application may be, for example, MySQL, which is not limited in the embodiments of the present application.

[0082] Furthermore, in an embodiment of the present application, after obtaining the applications supported by each vulnerability scanner, the number of applications, and the application set, the application sets corresponding to all vulnerability scanners are unioned to obtain an application set containing all applications.

[0083] For example, suppose three vulnerability scanners are P a , P b and P c , then the application set P can be expressed as: P = P a∪ P b∪ P c .

[0084] Then, the type information of CVE corresponding to each application supported by each vulnerability scanner and the number of application CVEs are determined, where the number of application CVEs is the number of CVEs included in an application in a vulnerability scanner.

[0085] A2: For each application, calculate the sum of the number of application CVEs of any application in each vulnerability scanner, and use the sum as the number of CVEs of the application.

[0086] In an embodiment of the present application, for each application, after obtaining the number of application CVEs of any application in each vulnerability scanner, the sum of the number of application CVEs in each vulnerability scanner is calculated, and the obtained sum is used as the number of CVEs of the application in each vulnerability scanner, thereby obtaining the number of CVEs of each application.

[0087] The CVE number represents the total number of CVEs applied in all vulnerability scanners, which is the sum of the number of CVEs applied in each vulnerability scanner.

[0088] For example, suppose there are three vulnerability scanners, namely vulnerability scanner P a , Vulnerability Scanner P b and vulnerability scanner P c . Vulnerability Scanner P a A total of 4 applications are supported for scanning, namely app1, app2, app3, and app4. The number of application CVEs corresponding to app1 is cve a (app1), the number of application CVEs corresponding to app2 is cve a (app2), the number of application CVEs corresponding to app3 is cve a (app3), the number of application CVEs corresponding to app4 is cve a (app4). Vulnerability Scanner P b A total of 4 applications are supported for scanning, namely app1, app3, app4, and app5. The number of application CVEs corresponding to app1 is cve b (app1), the number of application CVEs corresponding to app3 is cve b (app3), the number of application CVEs corresponding to app4 is cveb (app4), the number of application CVEs corresponding to app5 is cve b (app5). Vulnerability Scanner P c A total of 4 applications are supported for scanning, namely app2, app3, app4, and app6. The number of application CVEs corresponding to app2 is cve c (app2), the number of application CVEs corresponding to app3 is cve c (app3), the number of application CVEs corresponding to app4 is cve c (app4), the number of application CVEs corresponding to app6 is cve c (app6).

[0089] Therefore, the number of application CVEs corresponding to each application in different vulnerability scanners can be expressed as:

[0090] cve(P a )={cve a (app1),cve a (app2),cve a (app3),cve a (app4)};

[0091] cve(P b )={cve b (app1),cve b (app3),cve b (app4),cve b (app5)};

[0092] cve(P c )={cve c (app2),cve c (app3),cve c (app4),cve c (app6)}.

[0093] Then, after determining the number of application CVEs corresponding to each application in each vulnerability scanner, calculate P a The number of CVEs in app1 a (app1) and P b The number of CVEs in app1 b (app1), the number of CVEs corresponding to app1 is SUM(cve a (app1),cve b (app1)). Calculate P aNumber of CVEs in app2 a (app2) and P c Number of CVEs in app2 c (app2), the number of CVEs corresponding to app2 is SUM(cve a (app2),cve c (app2)). Calculate P a Number of CVEs in app3 a (app3) and P b Number of CVEs in app3 b (app3), and P c Number of CVEs in app3 c (app3) SUM(cve a (app3),cve b (app3), cve c (app3)). Calculate P a Number of CVEs in app4 a (app4) and P b Number of CVEs in app4 b (app4), and P c Number of CVEs in app4 c (app4) SUM(cve a (app4),cve b (app4), cve c (app4)). Calculate the number of CVEs corresponding to app5 as cve b (app5). Calculate the number of CVEs corresponding to app6 as cve c (app6).

[0094] Finally, the total number of CVEs corresponding to the CVEs that can be scanned by each vulnerability scanner to be tested is determined.

[0095] For example, the total number of CVEs of all apps included in each vulnerability scanner is cve(app), cve(app) = {cve(app1), cve(app2), cve(app3), cve(app4), cve(app5), cve(app6)}.

[0096] S2: Calculate the ratio between the number of CVEs of an application and the total number of CVEs.

[0097] In the embodiment of the present application, the ratio between the number of CVEs of the application program and the number of CVEs of all vulnerability scanners including the application and the total number of CVEs is calculated.

[0098] S3: Determine the number of samples corresponding to an application according to the ratio and a preset total number of samples.

[0099] In the embodiment of the present application, the calculated ratio is used as the proportion of the CVE of the application in the CVE of all vulnerability scanners. Then, according to the calculated ratio and the preset total number of samples, the number of samples corresponding to the samples that the application should extract is determined.

[0100] For example, assuming that the application is MySQL, the preset total number of samples is 20, and the calculated number of CVEs for MySQL accounts for 10% of the total number of CVEs corresponding to the CVEs contained in all vulnerability scanners. Therefore, the product between the preset total number of samples and the ratio is calculated to determine that the number of MySQL samples extracted is 2.

[0101] It should be noted that in the embodiments of the present application, the number of samples directly affects the error of the final accuracy test result. When the number of samples is larger, although the accuracy of stratified sampling can be improved, the sampling cost is also greater. On the contrary, the smaller the number of samples, although the accuracy of stratified sampling will be reduced, the cost is also smaller. Therefore, the number of samples for each application can be determined according to actual needs.

[0102] In addition, it should be noted that since the embodiment of the present application performs stratified sampling based on all applications to be scanned for vulnerabilities, the number of samples will affect all vulnerability scanners. Under the same influence, the influence of this part can be ignored.

[0103] Step 110: randomly sampling a preset number of version information from the acquired version information set according to the number of samples corresponding to each application, and determining application samples corresponding to the preset number of version information.

[0104] The version information set includes at least the version information corresponding to each application supported by each vulnerability scanner for scanning.

[0105] In an embodiment of the present application, after determining the number of samples of each application, version information corresponding to the number of samples is extracted from the acquired version information set through random sampling, and the application sample corresponding to each extracted version information, that is, the sample data, is determined.

[0106] It should be noted that in the embodiment of the present application, the process of generating a version information set is as follows: for each application supported by each vulnerability scanner, the version information corresponding to the version of the application containing a certain CVE is obtained from the standard vulnerability database, thereby generating a version information set containing each version information.

[0107] For example, see Figure 2 As shown, it is a schematic diagram of the version impact interval division in the embodiment of the present application, the applications are app1, app2, app3, app4 and app5, the version information of app1 is banner1 and banner2, the version information of app2 is banner1, the version information of app3 is banner1 and banner2, the affected version of app4 is banner1, and the version information of app5 is banner1 and banner2.

[0108] After determining the sample space, you can build a container environment based on each application sample.

[0109] It should be noted that in the embodiment of the present application, the Docker container technology is used to build the container environment of each application sample, and multiple container environments are built in a virtual machine, and the environment port is mapped out. In this way, the container environment can be obtained by calling the environment port.

[0110] Furthermore, accuracy comparison is a cyclical process. Saving the existing container environment and using it directly later can greatly improve the test efficiency. Choosing the Harbor warehouse to save the sample environment is more convenient for maintenance on the one hand. On the other hand, when the sample data is used later, it can be directly pulled from the Harbor warehouse for easy migration and copying.

[0111] Among them, the development and operation of Docker container applications cannot be separated from reliable image management. Harbor is an enterprise-level Docker Registry management project that provides a role-based access control mechanism and organizes images and controls access rights through projects. Container technology is an emerging and hot new technology after virtualization, cloud computing, and big data. Container technology improves the utilization of hardware resources, facilitates the rapid horizontal expansion of enterprise business, and realizes the self-healing function of business downtime. Container technology greatly simplifies the creation and maintenance of containers from file systems, network interconnection to process isolation, making Docker technology lighter and faster than virtual machine technology.

[0112] Step 120: Call each vulnerability scanner respectively to scan the container environment generated by each application sample, and obtain each CVE scanned by each vulnerability scanner.

[0113] In an embodiment of the present application, each vulnerability scanner is called separately, and the determined container environment is scanned by each vulnerability scanner respectively, so as to determine the applications with CVEs scanned by each vulnerability scanner, the CVEs existing in each application, and the number of CVEs that each vulnerability scanner can scan.

[0114] Step 130: for each vulnerability scanner, determine the accuracy test result of a vulnerability scanner according to each CVE of the vulnerability scanner to be tested and the obtained benchmark data set.

[0115] The benchmark data set includes all CVEs corresponding to each application supported by each vulnerability scanner.

[0116] First, the method for obtaining the benchmark data set in the embodiment of the present application is described in detail, including:

[0117] S1: Query each application and the version information corresponding to each application from the standard vulnerability database.

[0118] In the embodiment of the present application, each application included in each vulnerability scanner is queried from a standard vulnerability database, and version information corresponding to the version affected by each application is obtained from the standard vulnerability database.

[0119] It should be noted that, for the same application, some versions of the application may be affected by CVE, but some versions of the application may not be affected. Therefore, it is necessary to determine the version information corresponding to the affected application version.

[0120] The version information represents the version information corresponding to the version of the application affected by the CVE, for example, the version information of the application mysql is 8.0.0, which is not limited in the embodiments of the present application.

[0121] In addition, it should be noted that the standard vulnerability database contains CVE information of all applications. The standard vulnerability database can be, for example, a security vulnerability sharing platform, which is not limited in the embodiments of the present application.

[0122] S2: for each application, respectively, according to each version information of an application, determine the CVE included in the application under each version information.

[0123] In the embodiment of the present application, for each version information of each application, the CVE included in the application under each version information is determined.

[0124] Taking one of the applications as an example, step S2 in the embodiment of the present application is described in detail below.

[0125] First, determine the version information of the application.

[0126] Then, for each application under each version information, the CVEs and the number of application version CVEs contained in the application under any version information are determined.

[0127] It should be noted that the number of CVEs in an application version represents the total number of CVEs contained in an application under a certain version. For example, if the application is mysql and its version information is 8.0.0, the number of CVEs contained in mysql under version 8.0.0 is num(cve(8.0.0)).

[0128] Finally, through the above steps, the CVEs contained in the applications under each version information and the number of CVEs of the application versions can be obtained.

[0129] For example, the following uses a specific example to describe step S2 in detail in the embodiment of the present application, taking MySQL as an example, see Figure 3As shown, it is a schematic diagram of dividing the number of CVEs based on versions in an embodiment of the present application. First, the total number of CVEs of all applications in the standard vulnerability database is obtained as CVE(app). Then, according to the type information of the application, the applications are classified, and the number of application version CVEs of each application is obtained, which are CVE(app1), CVE(app2), CVE(app3), CVE(app4) and CVE(app5). Then, the version information corresponding to the version affected by each application is determined. The version information of app1 is banner1 and banner2, the version information of app2 is banner1, the version information of app3 is banner1 and banner2, the version information of app4 is banner1, and the version information of app5 is banner1 and banner2. The number of application version CVEs num(cve(banner1)) of application app1 under version information banner1 and the number of application version CVEs num(cve(banner2)) of application app1 under version information banner2 are obtained. The number of application version CVEs num(cve(banner2)) of application app2 under version information banner1 is determined. The number of application version CVEs num(cve(banner1)) is used to determine the number of application version CVEs num(cve(banner1)) of app3 under version information banner1 and the number of application version CVEs num(cve(banner2)) of app3 under version information banner2. The number of application version CVEs num(cve(banner1)) of app4 under version information banner1 is used to determine the number of application version CVEs num(cve(banner1)) of app5 under version information banner1 and the number of application version CVEs num(cve(banner2)) of app5 under version information banner2.

[0130] S3: Generate a benchmark data set including the CVE included in each application under each version information.

[0131] In an embodiment of the present application, after determining the CVEs contained in each application under each version information and the number of application version CVEs, a benchmark data set is generated based on each CVE, and the generated benchmark data set also corresponds to the number of application version CVEs of each application under each version information.

[0132] In this way, by obtaining a benchmark data set, it is possible to determine the accuracy detection result of each vulnerability scanner for each vulnerability scanner based on each CVE of any vulnerability scanner and the obtained benchmark data set.

[0133] In the embodiment of the present application, when determining the accuracy detection result of the vulnerability scanner to be tested, it specifically includes:

[0134] S1: For each CVE, if it is determined that a CVE is included in the benchmark data set, then the detection result of the CVE is determined to be correct; otherwise, the detection result of the CVE is determined to be a false positive.

[0135] S2: If it is determined that the CVE included in the benchmark data set is not included in each CVE, then it is determined that the detection result of the CVE is a missed report.

[0136] In the embodiment of the present application, for each vulnerability scanner, each CVE obtained after any vulnerability scanner scans the container environment and the detection result of each CVE are determined.

[0137] Among them, the detection results of CVE can be specifically divided into the following three types, but are not limited to the following three types.

[0138] The first test result: correct.

[0139] If it is determined that the scanned CVE is included in the benchmark data set, then the detection result corresponding to the CVE is determined to be correct. In other words, after scanning the container environment, the vulnerability scanner to be tested determines that the vulnerability CVE exists in the container environment and is correctly reported, then the detection result of the CVE is determined to be correct.

[0140] The second detection result: false positive.

[0141] If it is determined that the scanned CVE is not included in the benchmark data set, then the detection result corresponding to the CVE is determined to be a false positive. In other words, the CVE does not exist in the container environment. After the vulnerability scanner scans the container environment, the CVE is scanned and obtained, but in fact the CVE does not exist in the application corresponding to the CVE. Therefore, the detection result of the CVE is determined to be a false positive.

[0142] Furthermore, in an embodiment of the present application, if each CVE in the benchmark data set corresponds to a CVE identifier, it is possible to determine whether the scanned CVE is included in the benchmark data set by determining whether the CVE identifier corresponding to the scanned CVE is included in the benchmark data set.

[0143] The third test result: false negative.

[0144] If it is determined that a CVE existing in the benchmark data set is not scanned by the vulnerability scanner, that is, a CVE that actually exists in the container environment is not scanned by the vulnerability scanner, it is determined that the detection result of the CVE is a missed report.

[0145] For example, using vulnerability scanner P a For example, if P aIf the CVE of app1 is found and exists in the benchmark data set, the detection result of the CVE is determined to be correct. a If a CVE of app1 is found but it does not exist in the benchmark data set, the detection result of the CVE is determined to be a false positive. a After scanning, it is determined that the detection result of the CVE is a false negative.

[0146] S3: Determine the accuracy of the vulnerability scanner detection result based on the detection results of each CVE.

[0147] Specifically, when executing step S3, it specifically includes:

[0148] A1: For each application, determine the number of CVEs with correct detection results, the number of CVEs with false positive detection results, and the number of CVEs with missed detection results for an application, and determine the total number of CVEs included in the benchmark data set for an application.

[0149] In the embodiment of the present application, for each application supported by the vulnerability scanner, first, the CVE corresponding to any scanned application is determined, and the detection results corresponding to these CVEs are obtained, then the number of CVEs with correct detection results, the number of CVEs with false positive detection results, and the number of CVEs with missed detection results are determined. Finally, the total number of CVEs that should be included in the benchmark data set for the application is determined.

[0150] For example, suppose the application is app1 and the vulnerability scanner P a The scanned CVEs are a1, a2, a3, and a4. Assuming that the detection result of a1 is correct, the detection result of a2 is a false alarm, the detection result of a3 is a missed alarm, and the detection result of a4 is correct, then the number of CVEs with correct detection results is determined to be 2, the number of CVEs with false alarms is 1, and the number of CVEs with missed alarms is 1. The total number of CVEs that should be included in the benchmark data set for the application is determined to be 4.

[0151] It should be noted that in step A1, since the benchmark data set contains the CVEs that each application should contain, the total number of CVEs contained in each application can be determined by counting the CVEs contained in each application. In actual application, the total number of CVEs that the application should include in the benchmark data set is the sum of the number of CVEs with correct detection results, the number of CVEs with false positive detection results, and the number of CVEs with missed detection results.

[0152] A2: For each application, the accuracy of an application is determined according to the number and total number of CVEs with correct detection results for an application, the false alarm rate of an application is determined according to the number and total number of CVEs with false alarm results, and the missed alarm rate of an application is determined according to the number and total number of CVEs with missed detection results.

[0153] In an embodiment of the present application, for each application, a first ratio between the number of CVEs whose detection results are correct for any one application and the total number is calculated, and the calculated first ratio is used as the accuracy of the application, a second ratio between the number of CVEs whose detection results are false alarms for the application and the total number is calculated, and the calculated second ratio is used as the false alarm rate of the application, and a third ratio between the number of CVEs whose detection results are missed alarms for the application and the total number is calculated, and the calculated third ratio is used as the missed alarm rate of the application.

[0154] For example, suppose the vulnerability scanner P a For example, if the application app1 is detected correctly, the number of CVEs detected correctly by app1 is TP(app1), and the total number is cve(app1), then the accuracy of app1 can be expressed as:

[0155]

[0156] Among them, accuracy(app1) is the accuracy of app1, TP(app1) is the number of CVEs with correct detection results, and cve(app1) is the total number.

[0157] The false alarm rate of app1 is FP(app1), the number of CVEs with false alarm results is FP(app1), and the total number is cve(app1). The false alarm rate of app1 can be expressed as:

[0158] FPR(app1)=(FP(app1)) / (cve(app1))

[0159] Among them, FPR(app1) is the false alarm rate of app1, FP(app1) is the number of CVEs with false alarms, and cve(app1) is the total number.

[0160] The false negative rate of app1 is FNR(app1), the number of CVEs with false negative detection results is FNR(app1), and the total number is cve(app1). The false negative rate of app1 can be expressed as:

[0161]

[0162] Among them, FPR(app1) is the false alarm rate of app1, FP(app1) is the number of CVEs with missed detection results, and cve(app1) is the total number.

[0163] A3: Determine the accuracy detection result of the vulnerability scanner based on the accuracy, false positive rate, and missed negative rate of each application.

[0164] In the embodiment of the present application, according to the accuracy, false alarm rate and missed alarm rate corresponding to each application, it is more advantageous when the accuracy of the vulnerability scanner is high and the false alarm rate and the vulnerability rate are relatively low.

[0165] In the embodiment of the present application, the CVE of each application in the standard vulnerability database is used as the comparison standard, and different vulnerability scanners use the same standard for accuracy calculation, which can make the output data more objective and reliable. In addition, for the selection of samples of the container environment, stratified sampling is performed according to the number of CVEs of applications supported by each vulnerability scanner, which can diversify the sample space coverage. In addition, the accuracy of the vulnerability scanner is measured from three dimensions: accuracy, false negative rate, and false positive rate, which increases the comprehensiveness of the comparison. The container environment is built using docker technology, which reduces the resource consumption of building the container environment. At the same time, the Harbor warehouse is used to maintain the container environment image, which is convenient for maintenance and migration.

[0166] Based on the above embodiments, see Figure 4 FIG. 1 is another flow chart of a method for testing a multi-vulnerability scanner in an embodiment of the present application, which specifically includes:

[0167] Step 400: Start.

[0168] Step 401: Obtain each application supported by vulnerability scanner A and the corresponding application CVE quantity.

[0169] It should be noted that the application in the embodiment of the present application is used to search for a corresponding benchmark data set from a standard vulnerability database, and on the other hand, is used as an input for stratified sampling of a container environment.

[0170] Step 402: Obtain each application supported by vulnerability scanner B and the corresponding application CVE quantity.

[0171] Step 403: Obtain each application supported by vulnerability scanner C and the corresponding application CVE quantity.

[0172] In the embodiment of the present application, the applications supported by the vulnerability scanner C for scanning are:

[0173] P c ={app2,app3,app4,app8....}

[0174] The number of application CVEs corresponding to each application supported by vulnerability scanner C is:

[0175] cve(P b )={cve b (app1),cve b (app3),cve b (app4),cve b (app5)…}

[0176] Step 404: Store the application set and the number of CVEs corresponding to each application in a test database.

[0177] In an embodiment of the present application, an application set is generated that includes applications supported by vulnerability scanners A, B, and C, and for each application, the number of applications that can be scanned by A, B, and C is calculated, that is, the sum of the number of CVEs corresponding to the application in A, B, and C.

[0178] Specifically, it can be expressed as:

[0179] cve(app1)=SUM(cve a (app1),cve b (app1),..)

[0180] cve(app2)=SUM(cve a (app2),cve c (app2),..)

[0181] cve(app)={cve(app1),cve(app2),…}

[0182] Finally, the application set and the number of CVEs corresponding to each application are stored in the test database.

[0183] Step 405: Obtain a benchmark database from a standard vulnerability database.

[0184] The benchmark database includes at least a benchmark data set, a total number of CVEs, version information affected by CVEs, and a total number of CVEs included in applications under each version information.

[0185] It should be noted that the benchmark database in the embodiment of the present application is used as a comparison standard for determining the detection results of CVE.

[0186] In addition, it should be noted that, based on the application set obtained in step 404, the CVE corresponding to each application set, the version information of the affected CVE version, and the number of CVEs of different affected versions of applications are obtained from the standard vulnerability database, and these data are used as the standard for subsequent accuracy judgment.

[0187] In addition, it should be noted that three data are obtained in this step, namely, the total number of CVEs cve(app) corresponding to each application, the version information cve(banner) corresponding to the affected version of each CVE, and the number of CVEs contained in the application under the version information.

[0188] Step 406: randomly sample a preset number of version information from the acquired version information set according to the number of samples corresponding to each application, and determine the application samples corresponding to the preset number of version information.

[0189] In the embodiment of the present application, based on the determined number of samples corresponding to each application, application samples corresponding to the version information corresponding to the number of samples are randomly sampled from the version information set by a random sampling method.

[0190] Step 407: Determine whether the sample image exists in the Harbor repository. If so, execute step 408; if not, execute step 409.

[0191] Step 408: Obtain the image from the warehouse and build a container environment based on container technology.

[0192] Step 409: Use Docker to obtain the image from the external network warehouse and build a container environment.

[0193] Step 410: The container environment is set up.

[0194] Step 411: Call A, B, and C respectively to scan the container environment.

[0195] Step 412: Obtain each CVE after A, B, and C scan.

[0196] Step 413: Determine the accuracy of each application corresponding to A, B, C and the accuracy detection results of A, B, C according to each CVE of A, B, C respectively.

[0197] Step 414: End.

[0198] Based on the above embodiments, see Figure 5 FIG. 1 is a schematic diagram of a structure of a test device for a multi-vulnerability scanner in an embodiment of the present application, which specifically includes:

[0199] A first determination module 500 is used to determine the number of samples corresponding to each application supported by each vulnerability scanner according to the number of CVEs of the application and the total number of CVEs;

[0200] The sampling module 510 is used to randomly sample a preset number of version information from the acquired version information set according to the number of samples corresponding to each application, and determine the application samples corresponding to the preset number of version information, wherein the version information set at least includes each version information corresponding to each application supported by each vulnerability scanner;

[0201] The scanning module 520 is used to call each vulnerability scanner respectively to scan the container environment generated by each application sample to obtain each CVE scanned by each vulnerability scanner;

[0202] The detection module 530 is used to determine the accuracy detection result of a vulnerability scanner for each vulnerability scanner respectively according to each CVE of the vulnerability scanner to be tested and the obtained benchmark data set, wherein the benchmark data set includes all CVEs corresponding to each application supported by the vulnerability scanner.

[0203] Optionally, when determining the number of samples corresponding to an application according to the number of CVEs of the application and the total number of CVEs, the sampling module 510 is specifically configured to:

[0204] Determine the number of CVEs corresponding to each application supported by each vulnerability scanner, and the total number of CVEs, wherein the total number of CVEs represents the sum of the numbers of each CVE;

[0205] Calculate the ratio between the number of CVEs for an application and the total number of CVEs;

[0206] The number of samples corresponding to the application is determined according to the ratio and a preset total number of samples.

[0207] Optionally, when determining the number of CVEs corresponding to each application supported by each vulnerability scanner, the scanning module 520 is specifically configured to:

[0208] Determine respectively each application supported by each vulnerability scanner for scanning, as well as the type information of the corresponding CVE and the number of application CVEs, wherein the number of application CVEs is the number of CVEs included in an application of a vulnerability scanner;

[0209] For each application, the sum of the number of application CVEs of any one application in the vulnerability scanners is calculated, and the sum is used as the number of CVEs of the one application.

[0210] Optionally, when obtaining the benchmark data set, the step further includes:

[0211] A query module 540 is used to query the applications and the version information corresponding to the applications from the standard vulnerability database;

[0212] The second determination module 550 is used to determine, for each of the applications, the CVEs included in the application under each version information according to each version information of the application;

[0213] The generating module 560 is used to generate a benchmark data set including the CVE included in each application under each version information.

[0214] Optionally, the detection module 530 is specifically used for:

[0215] For each of the CVEs, if it is determined that a CVE is included in the benchmark data set, then the detection result of the CVE is determined to be correct; otherwise, the detection result of the CVE is determined to be a false positive;

[0216] If it is determined that the CVE included in the benchmark data set is not included in the CVEs, then determining that the detection result of the CVE is a missed report;

[0217] According to the detection results of each CVE, the accuracy detection result of the vulnerability scanner is determined.

[0218] Optionally, when determining the accuracy detection result of the vulnerability scanner according to the detection results of each CVE, the detection module 530 is specifically used to:

[0219] Determine, for each of the applications, the number of CVEs for which detection results are correct, the number of CVEs for which detection results are false positives, and the number of CVEs for which detection results are missed negatives for one application, and determine the total number of CVEs for the one application included in the benchmark data set;

[0220] For each of the applications, respectively, determine the accuracy of the application according to the number of CVEs whose detection results are correct and the total number, determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number, and determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number;

[0221] The accuracy detection result of the vulnerability scanner is determined based on the accuracy rate, false positive rate and missed negative rate corresponding to each application.

[0222] Based on the above embodiments, see Figure 6 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0223] An embodiment of the present application provides an electronic device, which may include a processor 610 (Center Processing Unit, CPU), a memory 620, an input device 630 and an output device 640, etc. The input device 630 may include a keyboard, a mouse, a touch screen, etc., and the output device 640 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0224] The memory 620 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 610 with program instructions and data stored in the memory 620. In the embodiment of the present application, the memory 620 may be used to store the program of any of the multi-vulnerability scanner testing methods in the embodiment of the present application.

[0225] The processor 610 calls the program instructions stored in the memory 620, and the processor 610 is used to execute any one of the multi-vulnerability scanner testing methods in the embodiments of the present application according to the obtained program instructions.

[0226] Based on the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the testing method of the multiple vulnerability scanner in any of the above method embodiments is implemented.

[0227] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0228] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0229] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0230] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0231] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for testing a multi-vulnerability scanner, characterized in that: include: For each application supported by each vulnerability scanner, determine the number of samples corresponding to the application according to the number of common vulnerability disclosures (CVEs) and the total number of CVEs of the application, wherein the number of CVEs of the application represents the total number of CVEs of the application in all vulnerability scanners, which is the sum of the number of CVEs of the application in each vulnerability scanner, the total number of CVEs represents the sum of the number of CVEs of all applications in each vulnerability scanner, and the samples are obtained by sampling from each application supported by each vulnerability scanner; According to the number of samples corresponding to each application, randomly sample the version information of the corresponding number of samples from the acquired version information set, and determine the application program samples corresponding to the version information, wherein the version information set at least includes the version information corresponding to each application supported by each vulnerability scanner; Invoke the vulnerability scanners respectively to scan the container environment generated by the application samples, and obtain the CVEs existing in each application scanned by the vulnerability scanners; For each vulnerability scanner, the accuracy detection result of the vulnerability scanner is determined according to the CVEs of the vulnerability scanner to be tested and the obtained benchmark data set, wherein the benchmark data set includes all CVEs corresponding to the applications supported by the vulnerability scanners.

2. The method according to claim 1, characterized in that According to the number of common vulnerability disclosures (CVEs) of an application and the total number of CVEs, the number of samples corresponding to the application is determined, specifically including: Determine the number of CVEs corresponding to each application supported by each vulnerability scanner, and the total number of CVEs, wherein the total number of CVEs represents the sum of the numbers of each CVE; Calculate the ratio between the number of CVEs for an application and the total number of CVEs; The number of samples corresponding to the application is determined according to the ratio and a preset total number of samples.

3. The method according to claim 2, characterized in that Determine the number of CVEs corresponding to each application that each vulnerability scanner supports scanning, including: Determine respectively each application supported by each vulnerability scanner for scanning, as well as the type information of the corresponding CVE and the number of application CVEs, wherein the number of application CVEs is the number of CVEs included in an application of a vulnerability scanner; For each application, the sum of the number of application CVEs of any one application in the vulnerability scanners is calculated, and the sum is used as the number of CVEs of the one application.

4. The method according to claim 1, characterized in that The benchmark data set is obtained in the following manner: Query the applications and version information corresponding to the applications from a standard vulnerability database; For each of the applications, respectively, according to each version information of an application, determine the CVE included in the application under the each version information; Generate a benchmark data set containing the CVEs included in each application under each version information.

5. The method according to claim 4, characterized in that For each vulnerability scanner, respectively, according to each CVE of a vulnerability scanner to be tested and the acquired benchmark data set, determining the accuracy detection result of the vulnerability scanner, specifically including: For each of the CVEs, if it is determined that a CVE is included in the benchmark data set, then the detection result of the CVE is determined to be correct; otherwise, the detection result of the CVE is determined to be a false positive; If it is determined that the CVE included in the benchmark data set is not included in the CVEs, then determining that the detection result of the CVE is a missed report; According to the detection results of each CVE, the accuracy detection result of the vulnerability scanner is determined.

6. The method according to claim 5, characterized in that According to the detection results of each CVE, the accuracy detection result of the vulnerability scanner is determined, specifically including: Determine, for each of the applications, the number of CVEs for which detection results are correct, the number of CVEs for which detection results are false positives, and the number of CVEs for which detection results are missed negatives for one application, and determine the total number of CVEs for the one application included in the benchmark data set; For each of the applications, respectively, determine the accuracy of the application according to the number of CVEs whose detection results are correct and the total number, determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number, and determine the false alarm rate of the application according to the number of CVEs whose detection results are false alarms and the total number; The accuracy detection result of the vulnerability scanner is determined based on the accuracy rate, false positive rate and missed negative rate corresponding to each application.

7. A testing device for a multi-vulnerability scanner, characterized in that: include: A first determination module is used to determine the number of samples corresponding to an application, for each application supported by each vulnerability scanner, according to the number of common vulnerability disclosures (CVEs) of the application and the total number of CVEs, wherein the number of CVEs of the application represents the total number of CVEs of the application in all vulnerability scanners, which is the sum of the number of CVEs of the application in each vulnerability scanner, the total number of CVEs represents the sum of the number of CVEs of all applications in each vulnerability scanner, and the samples are obtained by sampling from each application supported by each vulnerability scanner; A sampling module, used to randomly sample the version information of the corresponding number of samples from the acquired version information set according to the number of samples corresponding to the applications, and determine the application program samples corresponding to the version information, wherein the version information set at least includes the version information corresponding to the applications supported by the vulnerability scanners; A scanning module, used to call each vulnerability scanner respectively, scan the container environment generated by each application sample, and obtain each CVE existing in each application scanned by each vulnerability scanner; A detection module is used to determine the accuracy detection result of a vulnerability scanner for each vulnerability scanner respectively according to each CVE of the vulnerability scanner to be tested and an acquired benchmark data set, wherein the benchmark data set includes all CVEs corresponding to each application supported by the vulnerability scanner.

8. The device according to claim 7, characterized in that When determining the number of samples corresponding to an application according to the number of common vulnerability disclosures (CVEs) and the total number of CVEs of the application, the sampling module is specifically used to: Determine the number of CVEs corresponding to each application supported by each vulnerability scanner, and the total number of CVEs, wherein the total number of CVEs represents the sum of the numbers of each CVE; Calculate the ratio between the number of CVEs for an application and the total number of CVEs; The number of samples corresponding to the application is determined according to the ratio and a preset total number of samples.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images