An industrial control protocol fuzzing test benchmark platform

Abstract

The application discloses an industrial control protocol fuzzy test benchmark platform, and relates to an industrial control network platform, which comprises a virtualized communication module, a model loading module, a fuzzy test module, a database module, a performance evaluation module, a monitoring log module and an entity industrial control network and equipment; wherein the communication module is responsible for scanning and connecting the industrial control equipment; the model loading module loads the fuzzy test model through an API interface; the fuzzy test module is connected with the communication module, and sends corresponding test cases according to the scanned industrial control equipment; the database module calls a fuzzy tool to be tested, and stores the test cases and manages abnormal flow; the application is compatible with multiple models, provides more comprehensive and unified index evaluation, solves the problem that evaluation scales of fuzzy tools are not unified in the field of industrial control safety, modular design guarantees flexibility and high expansibility of the platform, and helps development and test work of efficient and high-performance industrial control protocol fuzzy tools.

Description

TECHNICAL FIELD

[0001] The present application relates to an industrial control network platform, in particular to an industrial control protocol fuzzing benchmark platform. BACKGROUND

[0002] Industrial control systems, as the basic core infrastructure in industrial production, have been widely used in the fields of petrochemical industry, power transmission, traffic control, etc. With the advancement of industrial informatization, industrial control systems are gradually connected to the Internet, the Internet of Things and other open networks, forming an open structure system. However, this open structure makes industrial systems face more security threats, and they are often subjected to external intrusion and attacks, seriously affecting the stable operation of industrial control systems. As the medium for exchanging data between devices in industrial control systems, industrial control protocols are easy to be used by attackers to attack industrial control systems through protocol vulnerabilities due to their simple structure and imperfect encryption measures.

[0003] Fuzzing is an effective technical means for industrial control protocol vulnerability mining at present. Its principle is to send a large number of unexpected test cases to the target to be tested, monitor the response of the system to the test cases, and determine whether the system has protocol vulnerabilities. Although fuzzing has achieved certain results in the field of industrial control protocol vulnerability mining, the widespread use of private industrial control protocols seriously limits the effectiveness and scalability of fuzzing tools. In addition, most fuzzing tools cannot prove that they have conducted appropriate and sufficient experimental evaluation in the field of industrial control security, and it is difficult to repeat the experiment, resulting in strong randomness and unreliability of the test results. The existing indicators are difficult to apply to the fuzzing task of industrial control protocols, and the evaluation scale cannot comprehensively measure the advantages of improved fuzzing tools.

[0004] In summary, the lack of standard industrial control vulnerability benchmark tools, comprehensive and unified performance indicator systems and representative data sets seriously hinders the evaluation and development of industrial control protocol fuzzing tools. Therefore, there is an urgent need for a public and shared industrial control protocol fuzzing benchmark platform to provide comprehensive and unified performance indicators to solve the problem that different models cannot be comprehensively measured in performance, and to promote the development of high-performance and efficient industrial control protocol fuzzing tools. SUMMARY

[0005] The purpose of the present application is to provide an industrial control protocol fuzzing benchmark platform, which solves the problem of the lack of shared fuzzing benchmark platforms and comprehensive and unified performance indicator evaluation systems in the current industrial control protocol fuzzing field, thereby effectively measuring the test breadth and depth capabilities of fuzzing tools, obtaining stable and repeatable quantitative evaluation, and assisting the development and testing of efficient and high-performance industrial control protocol fuzzing frameworks.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] The industrial control protocol fuzzing test benchmark platform comprises a communication module, a model loading module, a fuzzing test module, a database module, a performance evaluation module and a monitoring log module; wherein the communication module is connected with the model loading module, the fuzzing test module, the database module, the monitoring log module and the performance evaluation module;

[0008] The communication module is electrically connected with the entity industrial control network and the equipment, scans and obtains the basic information of the industrial control equipment, such as the IP address, the port number and the equipment type, and establishes a basis for subsequent communication;

[0009] The model loading module is connected with the communication module, and the user uploads the model to be tested to the fuzzing test benchmark platform through the connected published standardized API interface, and the platform analyzes and stores the model configuration information;

[0010] The fuzzing test module is connected with the communication module, generates and stores test cases with the loaded model, matches the corresponding industrial control equipment, and sends the test cases for fuzzing test;

[0011] The database module mainly comprises abnormal test cases, abnormal test cases, normal test cases and general fuzzing test tools generated by the abnormal test cases, and is used for matching and similarity comparison of abnormal test cases in the fuzzing test process;

[0012] The monitoring log module is connected with the communication module, and the monitoring analysis system responds;

[0013] The performance evaluation module is connected with the fuzzing test module, the database module and the monitoring log module, the abnormal test cases in the test process are matched with the database content, and a plurality of performance indexes are calculated and stored in the monitoring log module.

[0014] The communication module of the industrial control protocol fuzzing test benchmark platform comprises a device scanning submodule, a communication establishing submodule, a protocol adaptation submodule and a data transmission submodule; wherein,

[0015] The device scanning submodule is electrically connected with the entity industrial control network and the equipment, scans the IP address, the port number and the equipment type of the industrial control equipment connected with the platform;

[0016] The communication establishing submodule is used for establishing the communication connection with the industrial control equipment according to the equipment type and the protocol, and ensuring that the platform and the industrial control equipment normally interact with data;

[0017] The protocol adaptation submodule is connected with the fuzzing test module, and is used for selecting the corresponding industrial control equipment for data interaction according to the test case type;

[0018] The data transmission submodule is used for sending a test case and receiving a response of a device under test.

[0019] The model loading module comprises an API interface management submodule, a model storage management submodule, a model analysis and verification submodule, and a model loading submodule.

[0020] The API interface management submodule is developed based on a FastAPI framework and a python language, and the platform is deployed to a cloud server.

[0021] The model storage management submodule is used for persistent storage of a test model to meet the needs of a large number of users.

[0022] The model analysis and verification submodule analyzes a test model uploaded by a user, extracts configuration information therein, and verifies structural integrity and consistency of the test model.

[0023] The model loading submodule deploys a model to a correct running environment to generate a required test case.

[0024] The fuzzy test module comprises a test case generation submodule, a test case management submodule, an industrial control device matching submodule, and a test case sending submodule.

[0025] The test case generation submodule controls a model under test to generate a test case of a required type of protocol.

[0026] The test case management submodule stores and manages a generated test case and dispatches the test case according to user needs.

[0027] The industrial control device matching submodule is electrically connected with the communication module and cooperates with a protocol adaptation submodule to accurately locate a required industrial control device.

[0028] The test case sending submodule is connected with the communication module, and in combination with a communication submodule, establishes a communication connection with an industrial control device and sends a test case to realize data interaction.

[0029] The database module comprises a reproduction abnormal test case submodule, a deformed test case submodule, a normal test case submodule, and AFL, AFLfast, Peach, Honggfuzz, MOPT, and T-Fuzz universal fuzzy test tool submodules.

[0030] The abnormal test case reproduction module reproduces industrial control protocol vulnerabilities including but not limited to CVE, CNVD, CNNVD and the like, captures abnormal pcap traffic and stores the same in a classified manner;

[0031] The abnormal test case reproduction module reproduces industrial control protocol vulnerabilities including but not limited to CVE, CNVD, CNNVD and the like, captures abnormal pcap traffic and stores the same in a classified manner;

[0032] The normal test case reproduction module collects normal data traffic of mainstream industrial control protocols including Modbus / TCP, Modbus / RTU, S7Comm, DNP3, Ethernet / IP and Profibus, and stores the same in a classified manner;

[0033] The general fuzzing tool sub-module generates test cases of various protocols using a general fuzzing tool under the same environment, and stores the same in a classified manner.

[0034] The performance evaluation module includes a vulnerability mining speed sub-module, a vulnerability mining probability sub-module, an abnormal case matching degree sub-module, an abnormal case similarity sub-module, a test case receiving rate sub-module and a test case diversity sub-module; wherein:

[0035] The vulnerability mining speed sub-module includes two performance indicators of cumulative curve (CC) and time to exposure (TTE), and evaluates the number of vulnerabilities discovered within a given time;

[0036] The vulnerability mining probability sub-module calculates a p value as the probability of discovering a vulnerability within a given time;

[0037] The abnormal case matching sub-module calculates a value to evaluate the number of matching abnormal test cases and abnormal cases in the database;

[0038] The abnormal case similarity sub-module calculates a Euclidean distance D (X, Y) to evaluate the similarity of abnormal test cases in the database;

[0039] The test case receiving rate sub-module evaluates the proportion of test cases received by the industrial control device through a fuzzy test acceptance rate (TCAR);

[0040] The test case diversity sub-module calculates a state coverage rate C state to evaluate the richness of the types of generated test cases.

[0041] The monitoring log module includes a response data receiving sub-module, a response analysis sub-module, a log management sub-module and a performance indicator display sub-module; wherein:​

[0042] The response data receiving submodule is connected with the communication module and is used for receiving the response information of the industrial control device.

[0043] The response analysis submodule is used for analyzing the response information of the industrial control device to determine whether it is timeout, non-response, error response or device crash.

[0044] The log management submodule is connected with the response analysis submodule and is used for recording and storing the response information.

[0045] The performance index display submodule is connected with the performance evaluation module and is used for recording and displaying the performance indexes of the measured model, combining the Nemenyi test and the critical difference diagram (CDD) to evaluate the performance difference of the fuzzy test tool.

[0046] The advantages and effects of the present application are as follows:

[0047] The present application provides a virtual-real combined industrial control protocol fuzzy test benchmark platform scheme, the communication module is electrically connected with the entity industrial control network and device, and the IP address, port number and device model of the industrial control device are automatically scanned and obtained under the control of the communication module; the model loading module is connected with the communication module, a general API interface is developed based on the FastAPI framework combined with the python language, the user can upload the test model through the API interface, and the model loading module will automatically perform model analysis and verification; the fuzzy test module is connected with the communication module, the measured model is deployed in a suitable environment, test cases are generated according to the user's demand and are stored in categories, the industrial control device matching submodule cooperates with the protocol adaptation submodule to find the industrial control device corresponding to the test case, and the fuzzy test is executed; the database module is connected with the performance evaluation module and the monitoring log module, the monitoring log module automatically captures the device response information, under the synergistic action of the abnormal response information and the database module, the performance indexes defined in each submodule of the performance evaluation module are calculated and stored in the log management submodule in the monitoring log module, and the experimenters can comprehensively evaluate and compare the performance of each fuzzy tester in a unified environment, so as to develop high-performance and high-efficiency industrial control protocol fuzzy test tools. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application and the prior art, the drawings needed in the embodiments and the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 It is a virtual-real combined industrial control protocol fuzzy test benchmark platform structure schematic diagram provided by the embodiment of the present application.

[0050] Figure 2 Figure 1 is a structural schematic diagram of a communication module provided by an embodiment of the present application;

[0051] Figure 3 Figure 2 is a structural schematic diagram of a model loading module provided by an embodiment of the present application;

[0052] Figure 4 Figure 3 is a structural schematic diagram of a fuzzy test module provided by an embodiment of the present application;

[0053] Figure 5 Figure 4 is a structural schematic diagram of a performance index module provided by an embodiment of the present application;

[0054] Figure 6 Figure 5 is a critical difference diagram of a fuzzy test result of a Modbus TCP protocol provided by an embodiment of the present application.

[0055] Figure 1 is a structural schematic diagram of a communication module provided by an embodiment of the present application; DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0057] The present application discloses an industrial control protocol fuzzy test benchmark platform, comprising:

[0058] The communication module, the model loading module, the fuzz testing module, the database module, the performance evaluation module, the monitoring log module and the industrial control network and equipment module are connected, and communication and data transmission are performed through a local area network to scan and obtain basic information of the industrial control equipment, such as an IP address, a port number and a device type, to establish a basis for subsequent communication and testing;

[0059] The communication module is used as a communication medium between the modules, is responsible for data exchange and signal transmission with the model loading module, the fuzz testing module, the database module, the monitoring log module and the performance evaluation module, and is deployed in a server to realize interconnection and intercommunication through an internal bus and a network;

[0060] Further, the communication module is connected with the industrial control network and equipment module, and communication and data transmission are performed through a local area network to scan and obtain basic information of the industrial control equipment, such as an IP address, a port number and a device type, to establish a basis for subsequent communication and testing;

[0061] The model loading module is connected with the communication module, a user uploads a model to be tested to the fuzz testing benchmark platform through a standardized API interface connected by the user, and the platform stores configuration information of the model after analysis;

[0062] The fuzz testing module is connected with the communication module, and test cases are generated and stored by using the loaded model, the test cases are matched with corresponding industrial control equipment, and the test cases are sent for fuzz testing;

[0063] The database module is mainly responsible for storing abnormal test cases of various industrial control protocol reproductions, abnormal test cases of construction, normal test cases and test cases generated by a general fuzz testing tool, and is used for matching and similarity comparison of abnormal test cases occurring in a fuzz testing process;

[0064] The monitoring log module is connected with the communication module and is used for monitoring and analyzing system responses;

[0065] The performance evaluation module is connected with the fuzz testing module, the database module and the monitoring log module, abnormal test cases in a test process are matched with database content, a plurality of performance indexes are calculated and stored in the monitoring log module;

[0066] Preferably, the industrial control protocol fuzz testing benchmark platform, the communication module comprises a device scanning submodule, a communication establishing submodule, a protocol adapting submodule and a data transmission submodule;

[0067] The device scanning submodule is connected with the industrial control network and is used for scanning an IP address, a port number and a device type of an industrial control equipment connected with the platform;

[0068] The communication establishing submodule is used for establishing a communication connection with the industrial control equipment according to a device type and a protocol, and ensuring that the platform and the industrial control equipment normally interact with each other;

[0069] The protocol adaptation submodule is connected with the fuzz testing module, and is used for selecting a corresponding industrial control device for data interaction according to a test case type;

[0070] The data transmission submodule is used for sending a test case and receiving a response of a device under test;

[0071] Preferably, the industrial protocol fuzz testing benchmark platform, the model loading module comprises an API interface management submodule, a model storage management submodule, a model analysis and verification submodule and a model loading submodule; wherein,

[0072] The API interface management submodule is based on the FastAPI framework developed by the python language, and the platform is deployed to a cloud server. A user can upload a test model through an API interface to perform comparative testing and performance evaluation;

[0073] The model storage management submodule is used for persistent storage of a test model and a comparative test fuzzy tool to meet the needs of multi-dimensional performance index comparative testing;

[0074] The model analysis and verification submodule is used for analyzing a test model uploaded by a user, extracting configuration information therein, verifying structural integrity and consistency of the test model, and facilitating rapid execution of deployment and test evaluation tasks of the test model;

[0075] The model loading submodule is used for deploying a model to a correct running environment to generate a test case meeting specific task requirements;

[0076] Preferably, the industrial protocol fuzz testing benchmark platform, the fuzz testing module comprises:

[0077] A test case generation submodule, a test case management submodule, an industrial control device matching submodule and a test case sending submodule; wherein,

[0078] The test case generation submodule is used for controlling a test model to generate a test case of a required type protocol;

[0079] The test case management submodule is used for storing and managing a generated test case and scheduling the test case according to user requirements;

[0080] The industrial control device matching submodule is electrically connected with the communication module, and cooperates with the protocol adaptation submodule to accurately position a target industrial control device. The test case sending submodule is connected with the communication module, and combines the communication submodule and the industrial control device to establish a communication connection and send a test case, thereby realizing data interaction;

[0081] Preferably, the industrial protocol fuzz testing benchmark platform, the database module comprises:

[0082] The abnormality reproducing test case sub-module, the abnormal test case sub-module, the normal test case sub-module and the general fuzzing test tool sub-module; wherein,

[0083] The abnormality reproducing test case sub-module is used for reproducing industrial control protocol vulnerabilities published by professional vulnerability databases including but not limited to CVE, CNVD and CNNVD, capturing pcap data leading to abnormality and storing in categories.

[0084] The abnormal test case sub-module is used for constructing test cases with unexpected functions, formats and contents and storing in categories.

[0085] The normal test case sub-module is used for capturing normal data flow of mainstream industrial control protocols such as Modbus / TCP, S7Comm, DNP3 and Ethernet / IP, covering various function flow messages of each protocol and storing in categories.

[0086] The fuzzing test tool sub-module is used for performing evaluation test comparison tasks in the same environment by using mainstream general fuzzing test tools including AFL, AFLfast, Peach, Sully, Honggfuzz, MOPT, Boofuzz, T-Fuzz, Onefuzz and LibAFL, constructing corresponding protocol test cases, marking and storing in categories.

[0087] Preferably, the performance evaluation module of the industrial control protocol fuzzing test benchmark platform includes a vulnerability mining speed sub-module, a vulnerability mining probability sub-module, an abnormal case matching degree sub-module, an abnormal case similarity sub-module, a test case receiving rate sub-module and a test case diversity sub-module; wherein,

[0088] The vulnerability mining speed sub-module includes two performance indexes of cumulative curve (CC) and time to exposure (TTE) and is used for describing the efficiency of vulnerability detection.

[0089] Preferably, CC assumes that the number of all unique vulnerabilities detected within time t is B(t), and the cumulative curve can be expressed as:

[0090]

[0091] B(t) is the cumulative number of all unique vulnerabilities detected within time t, b i (t) is an indication function of whether the i-th vulnerability is detected within time t (if the i-th vulnerability is detected within time t, b i (t) = 1; otherwise b i (t) = 0), N bug is the total number of unique vulnerabilities.

[0092] The CC qualitatively analyzes the vulnerability detection speed of the fuzzing tool by plotting the curve of B(t) versus time t. The greater the slope of the curve, the faster the detection speed.

[0093] Preferably, the TTE assumes that the first exposure time of the jth vulnerability is T j , the exposure time can be expressed as:

[0094] TTE j = T j - T0,

[0095] where TTE j is the exposure time of the jth vulnerability, T j is the time when the jth vulnerability is first detected, and T0 is the time when the test starts.

[0096] The TTE quantitatively analyzes the first detection time of a specific vulnerability by recording the TTE j of each vulnerability. A smaller TTE j value indicates faster detection of a specific vulnerability.

[0097] The vulnerability mining probability submodule calculates the P value as an evaluation of the probability of discovering a vulnerability within a given time;

[0098] Preferably, the P value assumes that in M independent repeated tests, the number of unique vulnerabilities discovered within time t is recorded B m (t), where m is the index of the test, and the average probability of discovering a vulnerability within time t for each test is calculated

[0099]

[0100] The obeys a normal distribution, and the p value is calculated using the sample mean and standard deviation:

[0101]

[0102] where Φ is the cumulative distribution function of the standard normal distribution, σ is the standard deviation of , and P0 is the vulnerability discovery probability of the first test;

[0103] The abnormal use case matching submodule evaluates the number of matching abnormal test cases generated by the abnormal use case matching submodule and the database by calculating the value;

[0104] Preferably, it is assumed that the generated abnormal test cases X1, X2, …, X m and the known abnormal test cases Y1, Y2, …, Y n; the Value is the number of matching estimates between the two groups of data, the formula is as follows:

[0105]

[0106] Where # (X i > Y j ) represents the number of times the event X i > Y j occurs in the sample, # (X i = Y j ) represents the number of times the event X i = Y j occurs in the sample, m, n are the number of generated abnormal test cases and the number of known abnormal cases in the database respectively;

[0107] The abnormal case similarity submodule calculates the similarity with the abnormal test cases in the database through the Euclidean distance D (X, Y) ;

[0108] Preferably, assuming that the generated test case vector X i = (x i1 , x i2 ,…, x id ) and the abnormal test case vector Y j = (y j1 , y j2 ,…, y jd ) in the database, in the d-dimensional feature space, the Euclidean distance D (X i , Y j ) formula is as follows:

[0109]

[0110] The Euclidean distance sets a distance threshold ε, if D (X i , Y j ), then the test case X i matches the abnormal test case Y j in the database, and the number N match of all test cases matching the abnormal test cases in the database is counted:

[0111]

[0112] The test case acceptance rate submodule evaluates the proportion of test cases accepted by the industrial control device through the fuzzy test acceptance rate (TCAR), and the formula is as follows:

[0113]

[0114] Where N testN represents the total number of test cases that can pass in a one-time fuzz test acp N represents the number of test cases that are legally accepted in this test; if the test system responds normally to the test case, it means that the test case is accepted successfully, and if the test system returns an abnormal code, it means that the test case does not meet the standard of the protocol to be tested.

[0115] The test case diversity submodule evaluates the richness of the generated test case types by calculating the state coverage C state of the test feedback.

[0116] Preferably, the M test cases X i triggered within a specified time are i1 {S i2 ,…,S ik}, the number of unique states covered by all test cases is counted and the coverage rate is calculated as follows:

[0117]

[0118] where ∪ represents the union of all test case triggered protocol states, |·| represents the cardinality of the set, S represents the number of states of the protocol specification; C state quantifies the coverage of the generated test cases in the protocol state space, measures the diversity and effectiveness of the test case generation tool.

[0119] Preferably, the industrial control protocol fuzz testing benchmark platform comprises a monitoring log module.

[0120] The response data receiving submodule, the response analysis submodule, the log management submodule and the performance index display submodule; wherein,

[0121] The response data receiving submodule is connected with the communication module and is used for receiving the industrial control device response information;

[0122] The response analysis submodule is used for analyzing the industrial control device response information and judging whether it is timeout, no response, error response or device crash;

[0123] The log management submodule is connected with the response analysis submodule and is used for recording and storing the response information;

[0124] The performance index display submodule is connected with the performance evaluation module and is used for recording and displaying various performance indexes of the tested model, combining the Nemenyi test and the critical difference (CD) value to evaluate the performance difference of the fuzz testing tool.

[0125] Preferably, the Nemenyi test is a non-parametric test used for post-hoc comparison after the Friedman test, aiming to determine the significant difference between different test groups; the Friedman test is used to determine whether there is a significant difference in the ranking of multiple fuzzing tools on different benchmark tests. The statistical quantity Q of the Friedman test is calculated as follows:

[0126]

[0127] where k is the number of tools compared in the fuzzing test, N is the number of protocols of the benchmark test, R j is the total ranking of the jth fuzzing tool in all benchmark tests;

[0128] Preferably, if the Friedman test indicates that there is a significant difference between the various fuzzing tools, the Nemenyi post-hoc test is performed. The critical difference (CD) value of the Nemenyi test is calculated as follows:

[0129]

[0130] where q α is the critical value obtained from the standardized range distribution table according to the significance level a. In the performance index display submodule, the Nemenyi test and the critical difference (CD) value can be used to generate a "critical difference diagram" (CDD) to visualize the average ranking of multiple fuzzing tools on different benchmark tests and their significant differences.

[0131] Embodiments of the present application provide a specific technical solution:

[0132] The interconnection of industrial control networks and open networks attracts more intrusion attacks on industrial control networks through industrial control protocol vulnerabilities, which poses a serious threat to the security of industrial control networks. Therefore, it is urgent to timely exploit potential vulnerabilities of industrial control protocols. With the emergence of various fuzzing tools, authors have shown that their models or methods are more excellent in some aspects, but there is a lack of an open platform for comprehensive performance evaluation of various fuzzing tools. Under the same experimental environment and evaluation indicators, the performance evaluation data comparison is more credible, and a scalable and flexible shared fuzzing benchmark platform is provided for researchers in the same field.

[0133] To achieve the above object, please refer to Figure 1 , Figure 1 is a structure diagram of a virtual-real combined industrial control protocol fuzzing benchmark platform provided by the embodiment of the present application. As Figure 1 shown, the industrial control protocol fuzzing benchmark platform comprises:

[0134] The communication module 01, the model loading module 02, the fuzz testing module 03, the database module 04, the performance evaluation module 05 and the monitoring log module 06; wherein,

[0135] The fuzz testing benchmark platform is constructed in a front-end and back-end form. A user entering the platform can register a personal account and manage relevant information. The front-end uses a Vue.js framework to develop a graphical interactive interface. The functions of various modules are displayed to the user in the form of icons. Real-time data transmission is achieved through RESTAPI and the back-end. The back-end develops relevant functions based on the open-source Fuzz bench platform. Modular design makes the entire benchmark platform more flexible and expandable. Finally, the entire platform is deployed in a Docker environment and packaged into an independent container. The platform can run in any Docker environment to provide users with testing;

[0136] The communication module 01 is connected with the model loading module 02, the fuzz testing module 03, the database module 04, the performance evaluation module 05 and the monitoring log module 06. The platform initializes the interactive interface upon startup. The functions of various modules are displayed on the interactive interface. The communication module 01 is electrically connected with the entity industrial control network and the device 33. It is used to scan the industrial control devices connected with the platform, obtain the IP address, port number and other basic information of the industrial control devices, identify the protocol types supported by each device, establish a communication connection and perform data interaction. The fuzz testing module 03 cooperates with the communication module to complete the sending of test cases. The monitoring log module receives the device responses returned by the communication module and visually displays performance index information to the user.

[0137] The model loading module 02 is connected with the fuzz testing module 03. It supports users to upload model files and perform integrity and consistency verification through API interfaces published to users. It ensures the correctness of the model running environment and allows users to call the model at any time. It also supports loading multiple general fuzz testing tools, including AFL, AFLfast, Peach, Sully, Honggfuzz, MOPT, Boofuzz, T-Fuzz, Onefuzz, LibAFL, etc., for performance comparison.

[0138] The fuzz testing module 03 is connected with the communication module 01, the model loading module 02 and the monitoring log module 06. It is used to generate and store test cases. It determines the target industrial control device through device matching and the communication module and sends test cases for fuzz testing.

[0139] The database module 04 is connected with the performance index module 05 and the monitoring log module 06. It is used to store normal test cases, abnormal test cases and test cases for reproducing public vulnerabilities of various mainstream industrial control protocols. It matches the abnormal responses received during testing with the database content and calculates performance indicators such as similarity.

[0140] The performance evaluation module 05 is used for defining a series of new fuzzy test performance evaluation indexes, including but not limited to vulnerability mining speed, vulnerability mining probability, abnormal test case matching degree, abnormal test case similarity, test case receiving rate and test case diversity, and the like, to provide a more comprehensive performance evaluation system for users, and meanwhile, users can feed back other meaningful index definitions, and developers will supplement the performance evaluation module after evaluation, so as to continuously improve the module composition.

[0141] The monitoring log module 06 is connected with the communication module 01 and the performance evaluation module 05, and is used for monitoring the response information of the measured device, transmitting the result to the performance evaluation module for detailed settlement, returning to the module for storage and finally displaying the performance indexes.

[0142] In summary, the industrial control protocol fuzzy test benchmark platform scheme provided by the application has the following advantages: the communication module 01 is electrically connected with the entity industrial control network and the device 33, and automatically scans and obtains the IP address, port number and device model of the industrial control device under the control of the communication module 01; the model loading module 02 is connected with the communication module 01, and uses a general API interface developed based on a FastAPI framework and a python language, so that users can upload test models through the API interface, and the model loading module will automatically perform model analysis and verification; the fuzzy test module 03 is connected with the communication module 01, and deploys the measured model to a suitable environment, generates test cases according to user requirements and stores the test cases in a classified manner, the industrial control device matching sub-module cooperates with the protocol adaptation sub-module to find the industrial control device corresponding to the test case, and performs fuzzy test; the database module 04 is connected with the performance evaluation module 05 and the monitoring log module 06, the monitoring log module 06 automatically captures device response information, calculates the performance indexes defined in each sub-module of the performance evaluation module under the cooperative action of the abnormal response information and the database module, and stores the performance indexes in the log management sub-module of the monitoring log module, so that experimenters can comprehensively evaluate and compare the performance of each fuzzy tester in a unified environment, so as to develop an industrial control protocol fuzzy test tool with high performance and high efficiency.

[0143] In addition, as a preferred embodiment, as shown in Figure 2 The communication module in the industrial control protocol fuzzy test benchmark platform includes:

[0144] The device scanning sub-module 07, the communication establishing sub-module 08, the protocol adaptation sub-module 09 and the data transmission sub-module 10; wherein,

[0145] The device scanning sub-module 07 is electrically connected with the industrial control network and the device 33, and is used for scanning and obtaining the IP address, port number, device model and the like of the industrial control device connected to the platform.

[0146] The communication establishing submodule 08 is connected with the test case sending submodule 18, and is used for executing a command of sending a test case.

[0147] The protocol adapting submodule 09 is connected with the industrial control device matching submodule 17, and is used for assisting in determining a target industrial control device of the fuzz testing.

[0148] The data transmission submodule 10 is connected with the communication establishing submodule 08, and starts to execute a data transmission instruction after the communication is established.

[0149] In the technical scheme provided by the embodiment of the application, the communication module 01 is connected with the fuzz testing module 03 and the monitoring log module 06, and the main task is to establish a connection with a communication device before sending a test case, and return device response information to the monitoring log module, so as to analyze a test result.

[0150] In addition, as a preferred embodiment, as shown in Figure 3 The model loading module in the industrial control protocol fuzz testing benchmark platform includes:

[0151] The API interface management submodule 11 is connected with the model parsing and verifying submodule 13 and the model storage management submodule 12, and is used for providing a standard API interface for a user to upload a model file to be tested.

[0152] The API interface management submodule 11 is connected with the model parsing and verifying submodule 13 and the model storage management submodule 12, and is used for providing a standard API interface for a user to upload a model file to be tested.

[0153] The model loading submodule 14 is used for calling a specified model through a unique ID, loading a suitable running environment, and preparing for subsequent test case generation.

[0154] In addition, as a preferred embodiment, as shown in Figure 4 The fuzz testing module in the industrial control protocol fuzz testing benchmark platform includes:

[0155] The test case generation submodule 15, the test case management submodule 16, the industrial control device matching submodule 17 and the test case sending submodule 18 are connected with each other.

[0156] The test case generation submodule 15 is connected with the test case management submodule 16, and is used for generating a test case according to a requirement of an experimental personnel, and performing classified storage management on the test case.

[0157] The industrial control device matching sub-module 17 is connected with the protocol adaptation sub-module 09, and is used for matching the corresponding industrial control device by combining the IP address, the port number and the like before sending the test case.

[0158] The test case sending sub-module 18 is connected with the communication establishing sub-module 08, and is used for encapsulating the generated test case, adding the CRC verification, the Ethernet header and the like necessary information, supplementing the complete data frame, and starting to send the encapsulated test case to the target device after the communication is completed. The basic logic is as follows: the stored test case is called through the index, the necessary data segment is added for encapsulation, the encapsulated test case is temporarily stored in the to-be-sent list, the target device is determined and the communication connection is established, the sending button is clicked, and the test case is started to be sent to the communication module 01 at a speed of 1 piece per second, and the data interaction is performed between the communication module 01 and the target industrial control device in the industrial control network 33.

[0159] In addition, as a preferred embodiment, the industrial control protocol fuzzing test benchmark platform database module comprises:

[0160] The abnormal test case reproduction storage sub-module 19, the abnormal test case storage sub-module 20, the normal test case storage sub-module 21 and the general fuzzy test tool test case storage sub-module 22; wherein,

[0161] The abnormal test case reproduction storage sub-module 19 is used for reproducing the industrial control protocol vulnerabilities published by the professional vulnerability library including but not limited to CVE, CNVD and CNNVD, and capturing the test case causing the abnormality as the benchmark abnormal case.

[0162] The abnormal test case storage sub-module 20 is used for collecting the test cases constructed by violating the normal protocol format and adding various illegal characters as the benchmark abnormal case.

[0163] The normal test case storage sub-module 21 is used for collecting and storing the traffic data of most mainstream industrial control protocols, including but not limited to the various executable operation captured data traffic of common industrial control protocols such as Modbus / TCP, S7Comm, DNP3 and Ethernet / IP, so as to support the index evaluation of the test case coverage and diversity.

[0164] The general fuzzy test tool test case storage sub-module 22 is used for storing various test cases generated by the general fuzzy test tool and the specific model method which the user hopes to compare, so as to support the subsequent model performance comparison.

[0165] In addition, as a preferred embodiment, as shown in Figure 5 The performance evaluation module in the industrial control protocol fuzzing test benchmark platform comprises:

[0166] The vulnerability mining speed submodule 23, the vulnerability mining probability submodule 24, the abnormal test case matching submodule 25, the abnormal test case similarity submodule 26, the test case receiving rate submodule 27, and the test case diversity submodule 28; wherein,

[0167] The test case management submodule 16 is connected with the database module 04;

[0168] The vulnerability mining speed submodule 23 is connected with the response data receiving submodule 29 and the response analysis submodule 30, and is used for calculating a cumulative curve (CC) of the number of vulnerabilities mined within a specified time and a time to exposure (TTE), so as to evaluate the efficiency of vulnerability detection.

[0169] The vulnerability mining probability submodule 24 is connected with the response data receiving submodule 29 and the response analysis submodule 30, and is used for testing and calculating a P value as an index for evaluating the probability of a fuzzy tool discovering a vulnerability within a specified time.

[0170] The abnormal test case matching submodule 25 is connected with the reproduced abnormal test case storage submodule 19 and the malformed test case storage submodule 20, and is used for matching abnormal test cases in the database, is defined as the number of abnormal test cases matched between the abnormal test cases discovered in the fuzzy testing process and the test cases stored in the database.

[0171] The abnormal test case similarity submodule 26 is connected with the reproduced abnormal test case storage submodule 19 and the malformed test case storage submodule 20, and is used for calculating the similarity between the discovered abnormal test cases and the abnormal test cases in the database, D(X, Y) is defined as the Euclidean distance between the test cases as the similarity, and the number N of matches is counted match as the final similarity evaluation index.

[0172] The test case receiving rate submodule 27 is connected with the response data receiving submodule 29 and the response analysis submodule 30, and is used for receiving response information of the device under test, and the fuzzy testing acceptance rate (TCAR) is defined as the proportion of the test cases normally received by the system to all the test cases sent.

[0173] The test case diversity submodule 28 is connected with the normal test case submodule 21 and the general fuzzy test tool case submodule 22, and defines the state coverage C state of the test feedback, and is used for calculating the ratio of the types of the test cases sent to the types of the normal test cases in the database, so as to evaluate the richness of the types of the test cases generated.

[0174] In addition, as a preferred embodiment, the log evaluation module in the industrial control protocol fuzzy testing benchmark platform comprises:

[0175] The response data receiving submodule 29, the response analysis submodule 30, the log management submodule 31 and the performance index display submodule 32; wherein,

[0176] The response data receiving submodule 29 is connected with the communication module 01, and is used to receive each piece of response information of the device under test. The abnormal response state is defined as that the device does not respond within a predetermined time range, that an undefined response is received, that the response returned by the device does not match the required response, that the device crashes, that the device stops, that the communication connection is interrupted and the like.

[0177] The log management submodule 31 is connected with the response analysis submodule 30 and the performance evaluation module 05, and is used to store the response record and the calculation result of the performance evaluation module.

[0178] The performance index display submodule 32 is connected with the log management submodule 31, and directly displays various performance indexes of the model under test and the test case triggering the exception in the fuzzy test process to the user. The Nemenyi test and the critical difference (CD) value can be used to generate a critical difference diagram (CDD) to visualize the average ranking of multiple fuzzy tools on different benchmark tests and the significant difference thereof. The specific steps are as follows:

[0179] S1: For each protocol benchmark test, the performance indexes of all fuzzy test tools are calculated, the fuzzy tools are ranked according to the parameters, and the average ranking of each fuzzy tool is calculated.

[0180] S2: The Friedman test is performed, the statistical quantity Q is calculated using the Friedman test formula, and it is determined whether there is a significant difference;

[0181] S3: If the Friedman test shows that there is a significant difference between the compared models, the Nemenyi test is performed, and the critical difference (CD) value is calculated.

[0182] S4: The average ranking of the fuzzy tools is drawn according to the CD value in S3, and the fuzzy tool groups without significant difference within the critical difference range are connected using thick lines.

[0183] In the embodiment, based on the benchmark test results of the Modbus / TCP protocol fuzzy test tool of the performance index display submodule, the critical difference diagram (CDD_Modbus / TCP) is as shown in Figure 6

[0184] ​It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0188] Although the principles and implementation methods of the present invention have been described, the above description of the embodiments is only intended to help understand the present invention and its core concept. Modifications and changes made by those skilled in the art without departing from the spirit and scope of the present invention should be within the scope of protection of the appended claims.

Claims

1. An industrial control protocol fuzzy testing benchmark platform, characterized by: The platform includes a communication module, a model loading module, a fuzzy testing module, a database module, a performance evaluation module and a monitoring log module; wherein the communication module is connected to the model loading module, the fuzzy testing module, the database module, the monitoring log module and the performance evaluation module; The communication module is electrically connected to the physical industrial control network and equipment, and scans and obtains basic information of the industrial control equipment, including IP address, port number, and device type, to establish a foundation for subsequent communication; The model loading module is connected to the communication module. The user uploads the model to be tested to the fuzzy test benchmark platform through the published standardized API interface, and the platform parses and stores the model configuration information; The fuzz testing module is connected to the communication module, generates and stores test cases using the loaded model, matches the corresponding industrial control equipment, and sends the test cases for fuzz testing; The database module is mainly composed of abnormal test cases reproduced by various industrial control protocols, constructed deformed test cases, normal test cases and test cases generated by general fuzz testing tools, which is used to match and compare the similarity of abnormal test cases that appear in the fuzz testing process; The monitoring log module is connected to the communication module to monitor and analyze system responses; The performance evaluation module is connected to the fuzz testing module, the database module and the monitoring log module, matches abnormal test cases in the test process with the database content, calculates multiple performance indicators and stores them in the monitoring log module; The database module includes: an abnormality test case module, a deformity test case module, a normal test case module and AFL, AFLfast, Peach, Honggfuzz, MOPT, T-Fuzz general fuzz testing tool submodules; wherein, The anomaly recurrence test case module includes a module that reproduces but is not limited to industrial control protocol vulnerabilities published by professional vulnerability libraries such as CVE, CNVD, and CNNVD, captures the pcap traffic that causes the anomaly, and stores it in a classified manner; The abnormal test case module constructs test cases with unexpected functions, formats and contents and stores them in categories; The normal test case module collects normal data traffic of mainstream industrial control protocols such as Modbus / TCP, Modbus / RTU, S7Comm, DNP3, Ethernet / IP, and Profibus. Each protocol covers multiple functional traffic messages and stores them in a classified manner. The general fuzz testing tool submodule uses the general fuzz testing tool to generate various protocol test cases in the same environment, mark and classify them for storage; The performance evaluation module includes: a vulnerability mining speed submodule, a vulnerability mining probability submodule, an abnormal case matching submodule, an abnormal case similarity submodule, a test case acceptance rate submodule and a test case diversity submodule; wherein: The vulnerability mining speed submodule includes two performance indicators: cumulative curve CC and exposure time TTE, which evaluate the number of vulnerabilities discovered in a given time; The vulnerability mining probability submodule calculates the p-value as the probability of finding a vulnerability within a given time period; The exception case matching submodule calculates The number of abnormal test cases generated by the value evaluation matches the abnormal test cases in the database; The value is an estimate of the number of matches between the two sets of data; The abnormal case similarity submodule calculates and evaluates the similarity with abnormal test cases in the database through the Euclidean distance D(X,Y); The test case acceptance rate submodule evaluates the proportion of test cases accepted by industrial control equipment through the fuzzy test acceptance rate TCAR; The test case diversity submodule calculates the state coverage C of the test feedback state Evaluate the richness of the generated test cases.

2. The industrial control protocol fuzzy test benchmark platform according to claim 1, characterized in that: The communication module includes: a device scanning submodule, a communication establishment submodule, a protocol adaptation submodule and a data transmission submodule; wherein, The device scanning submodule is electrically connected to the physical industrial control network and the device, and scans the IP address, port number, and device type of the industrial control device connected to the platform; The communication establishment submodule is used to establish a communication connection with the industrial control device according to the device type and protocol, ensuring normal data interaction between the platform and the industrial control device; The protocol adapter module is connected to the fuzzy test module and is used to select the corresponding industrial control device for data interaction according to the test case type; The data transmission submodule is used for sending test cases and receiving responses from the device under test.

3. The industrial control protocol fuzz testing benchmark platform according to claim 1, characterized in that: The model loading module includes: an API interface management submodule, a model storage management submodule, a model analysis and verification submodule, and a model loading submodule; wherein, The API interface management submodule is developed based on the FastAPI framework and Python language. The platform is deployed to the cloud server. Users can upload their own models through the API interface for overall testing and evaluation. The model storage management submodule is used for persistent storage of test models to meet the needs of a large number of users; The model parsing and verification submodule parses the test model uploaded by the user, extracts the configuration information therein, and verifies the structural integrity and consistency of the test model; The model loading submodule deploys the model to the correct operating environment to generate the required test cases.

4. The industrial control protocol fuzzy testing benchmark platform according to claim 1, characterized in that: The fuzzy test module includes: a test case generation submodule, a test case management submodule, an industrial control equipment matching submodule and a test case sending submodule; wherein: The test case generation submodule controls the model under test to generate test cases of the required type of protocol; The test case management submodule stores and manages the generated test cases and schedules the test cases according to user needs; The industrial control equipment matching submodule is electrically connected to the communication module and cooperates with the protocol adapter submodule to accurately locate the required industrial control equipment; The sending test case submodule is connected to the communication module, and combined with the establishing communication submodule, establishes a communication connection with the industrial control equipment and sends the test case to realize data interaction.

5. The industrial control protocol fuzzy test benchmark platform according to claim 1, characterized in that: The monitoring log module includes: a response data receiving submodule, a response analysis submodule, a log management submodule and a performance indicator display submodule; wherein: The response data receiving submodule is connected to the communication module and is used to receive response information from the industrial control equipment; The response analysis submodule is used to analyze the response information of the industrial control equipment to determine whether it is a timeout, no response, an error response or a device crash; The log management submodule is connected to the response analysis submodule and is used to record and store response information; The performance indicator display submodule is connected to the performance evaluation module and is used to record and display the performance indicators of the tested model, and evaluate the performance differences of the fuzz testing tools in combination with the Nemenyi test and the critical difference diagram CDD.

Images