A fuzzy testing system and method for unknown industrial control protocols

Through the HRLFuzz model based on hierarchical reinforcement learning, variant test samples are generated and different variant strategies are combined to conduct intelligent and automated fuzz testing on industrial control protocols, solving the problem of incomplete vulnerability detection in the existing technology and achieving more efficient testing Coverage and depth.

CN113886225BActive Publication Date: 2024-12-03STATE GRID HENAN ELECTRIC POWER ELECTRIC POWER SCI RES INST +2
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202111112940.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-18
Publication Date
2024-12-03
Estimated Expiration
2041-09-18

AI Technical Summary

Technical Problem

Existing fuzz testing methods cannot conduct comprehensive and complete vulnerability detection on industrial control protocols, especially security vulnerabilities that work together when multiple protocol states or functional type codes are difficult to detect.

Method used

Using the HRLFuzz model based on hierarchical reinforcement learning, a variant test sample is generated through the agent, and the message sending module, industrial control network system, industrial control network monitoring module and interpreter are used to mutate the message with different mutation strategies until the reward is maximized. Implement intelligent and automated fuzz testing of industrial control protocols.

Benefits of technology

It realizes comprehensive vulnerability detection of industrial control protocols, improves test coverage and depth, and can discover security vulnerabilities triggered by the combined action of multiple protocol states or functional type codes, providing better fuzzy testing samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113886225B_ABST
    Figure CN113886225B_ABST
Patent Text Reader

Abstract

The present application discloses a fuzz testing system and method for unknown industrial control protocols, including: an intelligent agent generates variant test samples by executing actions using a HRLFuzz model based on hierarchical reinforcement learning; receiving rewards and rewards, selecting the next action in combination with the new state derived from the test sample, and generating variant test samples through hierarchical reinforcement learning until the reward is maximized; a message sending module sends the test sample to an industrial control network system; the industrial control network system processes the test case to obtain rewards and message responses; an interpreter translates the rewards and sends them to the intelligent agent; the industrial control network monitoring module sends the rewards to the intelligent agent, and monitors the message responses for abnormalities. The present invention uses a reinforcement learning model to realize the variant generation of test cases, and proposes an industrial control protocol fuzz testing framework HRLFuzz, which solves the problem that the existing fuzz testing methods do not perform comprehensive and complete vulnerability detection on the tested object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of industrial control, and relates to a fuzzy testing system and method for unknown industrial control protocols. Background Art

[0002] Industrial control systems (ICS) are essential for the normal operation of critical infrastructure such as the oil and petrochemical industry, power systems, and nuclear power plants. Early industrial control systems were closed systems that were independent of external networks and were therefore less likely to be attacked. With the development of industry and manufacturing, industrial control systems are interconnected with the network, exposing them to external attacks such as worms and Trojans, posing a major threat to their normal operation. In fact, well-known attacks on industrial networks, such as Stuxnet and BlackEnergy, have caused serious damage worldwide. This shows the severity of the current industrial network security situation.

[0003] In recent years, fuzz testing (fuzzing) technology has been widely used in protocol vulnerability mining due to its high accuracy, no source code requirement, and high applicability. Initially, traditional fuzz testing methods were used to test ICPs. A large number of test cases were generated through mutation or generation methods and used as inputs to the protocol entity program. It was expected that these unusual inputs would cause protocol entity anomalies to improve the coverage of the target program and find the existing vulnerabilities in the protocol as comprehensively as possible. When using traditional fuzz testing methods to generate test cases, test cases are usually only sent to a certain state of the protocol or a single function type code. Vulnerabilities corresponding to a single protocol state or a single function type code can be found, but many security vulnerabilities are triggered by the combined action of multiple protocol states or multiple function type codes. Obviously, these existing fuzz testing methods do not perform comprehensive and complete vulnerability detection on the object under test. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present application provides a fuzzy testing system and method for unknown industrial control protocols.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A fuzzy testing system for unknown industrial control protocols, including an intelligent agent, a message sending module, an industrial control network system, an industrial control network monitoring module, and an interpreter;

[0007] The agent is used to generate variant test samples by executing actions using the HRLFuzz (HRL: Hierarchical Reinforcement Learning) model based on hierarchical reinforcement learning; it is used to receive the return and reward of the industrial control network system and the industrial control network monitoring module, and select the next action in combination with the new state derived from the test sample, and generate variant test samples by hierarchical reinforcement learning until the reward is maximized;

[0008] The message sending module is used to send the test sample to the industrial control network system;

[0009] The industrial control network system is used to process the test case through the industrial control network server to obtain feedback and message response;

[0010] The interpreter is used to translate the reward into a message entity and a message state sequence that can be accessed by the HRLFuzz framework and then send it to the agent;

[0011] The industrial control network monitoring module is used to send the reward obtained through the reward function to the intelligent agent, and monitor the message response for abnormalities, and issue an alarm when an abnormality is found.

[0012] The present invention further includes the following preferred embodiments:

[0013] Preferably, the intelligent agent includes a header mutation module, a content mutation module and a sequence mutation module, which respectively utilize different mutation strategies to mutate the message;

[0014] The header mutation module includes four types of mutation strategies: length, marker, version and command;

[0015] The mutation strategy of the content mutation module includes using a bit flipping strategy to discover new paths;

[0016] The mutation strategy of the sequence mutation module is to randomly insert the currently selected message into the position of the correct message sequence to change the direction of state transfer.

[0017] Preferably, the industrial control network system receives the test sample through the industrial control server and gives a corresponding response, and the response message is returned to the industrial control network monitoring module together with the trace of the target process for subsequent analysis of the reward function and update of the intelligent agent Q value.

[0018] Preferably, the industrial control network monitoring module performs three types of abnormal monitoring on message responses: timed connection, timed return and abnormal message.

[0019] The present invention also discloses a fuzzy testing method for unknown industrial control protocols, comprising the following steps:

[0020] Step 1: During the fuzz testing process, the agent uses the HRLFuzz model based on hierarchical reinforcement learning to generate mutation test samples by executing actions;

[0021] Step 2: The message sending module sends the test sample to the industrial control network system;

[0022] Step 3: The industrial control network server of the industrial control network system processes the test case and obtains feedback and message response;

[0023] Step 4: The message response is sent to the industrial control network monitoring module, and the report is fed back to the interpreter;

[0024] Step 5: The interpreter translates the reward into a sequence of message entities and message states that can be accessed by the HRLFuzz framework and sends it to the agent;

[0025] Step 6: The industrial control network monitoring module receives the reward through the reward function and sends it to the agent. It also monitors the message response for abnormalities and issues an alarm when an abnormality is found.

[0026] Step 7: The agent receives the return and reward, and combined with the new state derived from the test example, selects the next action and returns to step 1 until the reward is maximized.

[0027] Beneficial effects achieved by this application:

[0028] The present invention utilizes the reinforcement learning model to realize the variation generation of test cases, and proposes an intelligent and automatic industrial control protocol fuzz testing framework HRLFuzz. Experimental results show that HRLFuzz can achieve the expected effect on the evaluation indicators of industrial control protocol fuzz testing, and solves the problem that the existing fuzz testing methods do not perform comprehensive and complete vulnerability detection on the tested objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a structure and operation principle diagram of a fuzzy testing system for unknown industrial control protocols of the present invention;

[0030] Figure 2 is a header field mutation module mutation strategy table in an embodiment of the present invention;

[0031] Figure 3 is the interpreter input and output in the embodiment of the present invention;

[0032] Figure 4 is a schematic diagram of a reinforcement learning model in an embodiment of the present invention;

[0033] Figure 5 is a schematic diagram of a reinforcement learning framework in an embodiment of the present invention;

[0034] Figure 6It is the message format of Modbus-TCP in the embodiment of the present invention;

[0035] Figure 7 is the relationship between the feedback and the number of test fields in the embodiment of the present invention;

[0036] Figure 8 are the TIRR value and AoVD value of the test results in the embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0038] like Figure 1 As shown, a fuzzy testing system for unknown industrial control protocols of the present invention includes an intelligent agent, a message sending module, an industrial control network system, an industrial control network monitoring module, and an interpreter;

[0039] The intelligent agent is used to generate variant test samples by executing actions using the HRLFuzz model based on hierarchical reinforcement learning; it is used to receive the return and reward of the industrial control network system and the industrial control network monitoring module, and select the next action in combination with the new state derived from the test sample, and generate variant test samples by hierarchical reinforcement learning until the reward is maximized;

[0040] The agent is the core module of HRLFuzz and is responsible for optimizing the proposed test case mutation generation strategy. Test case mutation generation aims to trigger vulnerabilities in a concise manner, which involves interaction with the object under test, serialization decisions, and a test case that accumulates as test cases are continuously input.

[0041] For the serialized decision-making problem of test sample mutation generation, ordinary supervised learning algorithms and unsupervised learning algorithms do not take into account the mutual influence between different decision points in the sequence, and can only maximize the immediate reward of a single decision point. The reinforcement learning algorithm aims to maximize the final reward, takes into account the delay effect in the sequence, and is good at handling serialized decision-making problems. Considering the internal structure of the test message and the dependencies between messages, it is necessary to distinguish three levels when using the intelligent agent engine to mutate the test samples: header fields, content fields, and sequence fields. Among them, the header field mainly distinguishes four types: length, marker, version, and command (length, identifier, version, and command). The corresponding mutation strategy can be selected from the table.

[0042] The data in the content domain is usually random and messy, and strategies that can be used include discovering new paths using strategies such as bit flipping.

[0043] In addition, some header fields and content fields have dependencies on length or type fields. When updating the content field data, the header field needs to be updated accordingly.

[0044] The implementation of the sequence domain mutation strategy requires disrupting the order of the message sequence while maintaining the integrity of the message structure, thereby improving the sample coverage. The main method is to randomly insert the currently selected message into the position of the correct message sequence to change the direction of the state transfer.

[0045] By using different strategies to generate message mutations, fuzz testing can be performed from different dimensions, which helps to improve test sample coverage and test depth.

[0046] Specifically: the intelligent agent includes a header mutation module, a content mutation module and a sequence mutation module, which respectively utilize different mutation strategies to mutate the message;

[0047] The header mutation module includes four types of mutation strategies: length, marker, version, and command, see Figure 2 .

[0048] The mutation strategy of the content mutation module includes discovering new paths using strategies such as bit flipping;

[0049] The mutation strategy of the sequence mutation module is to randomly insert the currently selected message into the position of the correct message sequence to change the direction of state transfer.

[0050] The message sending module is used to send the test sample to the industrial control network system;

[0051] In specific implementation, this module is responsible for capturing test samples generated by reinforcement learning and sending them to the industrial control network system. This module needs to extract protocol-specific commands and store them as text files in the format of one message per line. In addition to packet capture, data can also be read from local pcap files and text files and sent to the industrial control network server.

[0052] The industrial control network system is used to process the test case through the industrial control network server to obtain feedback and message response;

[0053] After receiving the test sample, the industrial control server gives a corresponding response, and the response message and the trace of the target process are returned to the industrial control network monitoring module for subsequent reward function analysis and agent Q value update.

[0054] The interpreter is used to translate the reward into a message entity and a message state sequence that can be accessed by the HRLFuzz framework and then send it to the agent;

[0055] Network protocols need to comply with specification documents including the protocol message format and session message sequence. For industrial control networks, it is necessary to define the description language including protocol specifications and protocol state transitions based on actual conditions, and clarify the protocol message format and session message sequence in general. Figure 3 As shown, the interpreter module takes the above two description files as input, processes them and translates them into message entities and message state sequences that can be accessed by the HRLFuzz framework.

[0056] The industrial control network monitoring module is used to send the reward obtained through the reward function to the intelligent agent, and monitor the message response for abnormalities, and issue an alarm when an abnormality is found.

[0057] This module monitors the abnormalities of the industrial network during the fuzz testing process by monitoring and analyzing the interaction data, trying to find out whether there are any failures and errors. Specifically, there are three types of anomalies: timed-out connections, overtime returns, and exception messages. If any signs of abnormal behavior are found, an alarm will be issued and fed back to the agent through the reward function.

[0058] The present invention provides a fuzzy testing method for unknown industrial control protocols, which uses Markov decision process (MDP) to formalize the industrial control protocol fuzzy testing process into a reinforcement learning problem. Specifically, Figure 4 As shown. During the fuzz testing process, the agent M generates a test sample I by executing action a and sends it to the industrial control network server P, which processes the test sample. Then, the new state x derived directly from I is observed, as well as the feedback information after the test sample I is sent to the server P, and the reward r(x, a) is measured. The task of reinforcement learning is to continuously adjust the test sample input strategy according to the test situation during the fuzz testing process, and finally obtain the optimal strategy to achieve the goal of optimizing the test effect.

[0059] Reinforcement Learning (RL) aims to learn through interaction in a trial-and-error manner. It is the process of adjusting the behavior of an agent during interaction with a system. The goal is to obtain the maximum cumulative reward over the entire time of system interaction based on the actions performed and the system state transitions. Figure 5As shown in the figure, reinforcement learning regards learning as a trial-evaluation process. The agent selects action a_t to act on the environment based on the previous state S_t. The environment transfers to the next state S_(t+1) under the action of action a_t, and generates a reward r_t to feed back to the agent. The agent then selects the next action based on the reward r_t and the state S_(t+1). The selection principle is to maximize the reward.

[0060] A fuzzy testing method for unknown industrial control protocols of the present invention specifically includes the following steps:

[0061] Step 1: During the fuzz testing process, the agent uses the HRLFuzz model based on hierarchical reinforcement learning to generate mutation test samples by executing actions;

[0062] Step 2: The message sending module sends the test sample to the industrial control network system;

[0063] Step 3: The industrial control network server of the industrial control network system processes the test case and obtains feedback and message response;

[0064] Step 4: The message response is sent to the industrial control network monitoring module, and the report is fed back to the interpreter;

[0065] Step 5: The interpreter translates the reward into a sequence of message entities and message states that can be accessed by the HRLFuzz framework and sends it to the agent;

[0066] Step 6: The industrial control network monitoring module receives the reward through the reward function and sends it to the agent. It also monitors the message response for abnormalities and issues an alarm when an abnormality is found.

[0067] Step 7: The agent receives the return and reward, and combined with the new state derived from the test example, selects the next action and returns to step 1 until the reward is maximized.

[0068] 1. Training data: Modbus is an application layer protocol developed by Modicon in 1979. Today, it has become the de facto standard communication protocol for many industrial networks. ModbusTCP was invented to meet the application requirements of industrial Ethernet. It uses the TCP / IP protocol stack to equip the Modbus RTU protocol with a TCP interface so that it can run on Ethernet. Figure 6As shown, the Modbus-TCP message consists of a Modbus application protocol (MBAP) header and a protocol data unit (PDU). The MBAP header is 7 bytes long and consists of four data fields, including a transaction identifier field, a protocol identifier field, etc. The PDU consists of a function code field and a data field. The maximum length of a Modbus-TCP message is 260 bytes. The training data is generated using the famous python library Pymodbus. It is very convenient to perform Modbus-TCP requests and responses by calling the corresponding functions in Pymodbus. The present invention randomly calls functions in the program to generate 50,000 request strings.

[0069] 2. Training model: In order to train the parameters in the proposed hierarchical DQNs Q = {Q^((1)), Q^((2)), Q^((3))} and the state representation method, the present invention adopts the experience replay technology with a memory buffer M of a certain size. The main starting point for using experience replay is to reduce the deviation caused by the correlation between samples. The present invention simulates the selection process to generate training data and stores it in the memory buffer M. During training, a batch of experiences (s, a, s^') are uniformly randomly extracted from the stored memory buffer M, where a = {a^((1)), a^((2)), a^((3))}. The Q-learning loss function is defined as:

[0070] E_((s,a,s^')~M)[(r+γ(max)┬(a^')Q^(s^',a^'∣θ^-)-Q(s,a∣θ))^2]

[0071] Where Q^ represents the target action-value function.

[0072] In order to improve the stability of the algorithm, the present invention limits the error term to between -1 and 1.

[0073] The agent adopts an ∈-greedy strategy, that is, it chooses random actions with probability ∈.

[0074] In the proposed model, the present invention uses two layers of multilayer perceptrons to implement all trainable parameters θ in the action-value function Q = {Q^((1)), Q^((2)), Q^((3))}. In fact, a more complex deep neural network can be used as a model, which is also a future research direction.

[0075] The test results of the present invention are as follows:

[0076] Learning performance: The evaluation of learning performance is mainly based on whether the agent can learn the expected strategy through interaction with the environment so that the reward converges to a stable state. As the testing process deepens, the relationship between the total reward obtained by HRLFuzz in each test and the number of test fields is as follows: Figure 7 As shown. Figure 7 It can be seen that during the testing process, HRLFuzz can learn from past fuzz tests, optimize the test sample input strategy, increase the occurrence rate of abnormal test cases, and gradually improve the return until the return converges to a stable state.

[0077] Test results: Using the HRLFuzz model based on hierarchical reinforcement learning to generate mutation test samples, the TIRR and AoVD values ​​of the alternating training process from 5 to 20 epochs are as follows: Figure 8 shown.

[0078] It can be seen that the more times the alternating training phase is iterated, the lower the TIRR that can be obtained, which confirms the claim of the present invention that the similarity of the generated messages can be adjusted by changing the number of trainings. As for the AoVD value, it can be noted that the AoVD value shows an upward trend as the number of alternating trainings increases.

[0079] Although HRLFuzz is still in a prototype version, it is able to reveal errors in industrial control network protocol implementations. The experimental TIRR and AoVD values ​​also demonstrate the potential and effectiveness of the HRLFuzz method.

[0080] In summary, the present invention aims to address the shortcomings of existing fuzz testing methods in terms of insufficient depth and low efficiency in vulnerability detection, and proposes an intelligent and automatic industrial control protocol fuzz testing system and method. The reinforcement learning model is used to realize the variation generation of test samples, and through continuous trial and error in the learning process, better fuzzy test samples are sought to be generated, attempting to provide a new solution to the problem of industrial control fuzz testing.

[0081] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A fuzzy testing system for unknown industrial control protocols, including an intelligent agent, a message sending module, an industrial control network system, an industrial control network monitoring module, and an interpreter, characterized in that: The intelligent agent is used to generate variant test samples by executing actions using the HRLFuzz model based on hierarchical reinforcement learning; it is used to receive the return and reward of the industrial control network system and the industrial control network monitoring module, and select the next action in combination with the new state derived from the test sample, and generate variant test samples by hierarchical reinforcement learning until the reward is maximized; The intelligent agent includes a header mutation module, a content mutation module and a sequence mutation module, which respectively utilize different mutation strategies to mutate the message; The header mutation module includes four types of mutation strategies: length, marker, version and command; The mutation strategy of the content mutation module includes using a bit flipping strategy to discover new paths; The mutation strategy of the sequence mutation module is to randomly insert the currently selected message into the position of the correct message sequence to change the direction of state transfer; The message sending module is used to send the test sample to the industrial control network system; The message sending module extracts the protocol-specific commands and stores them as a text file in the format of one message per line, reads data from the local pcap file and the text file and sends it to the industrial control network server; The industrial control network system is used to process the test case through the industrial control network server to obtain feedback and message response; The interpreter is used to translate the reward into a message entity and a message state sequence that can be accessed by the HRLFuzz framework and then send it to the agent; The industrial control network monitoring module is used to send the reward obtained through the reward function to the intelligent agent, and monitor the message response for abnormalities, and issue an alarm when an abnormality is found.

2. According to claim 1, a fuzzy testing system for unknown industrial control protocols is characterized in that: The industrial control network system receives the test sample through the industrial control server and gives a corresponding response. The response message and the trace of the target process are returned to the industrial control network monitoring module for subsequent reward function analysis and intelligent agent Q value update.

3. The fuzzy testing system for unknown industrial control protocols according to claim 1 is characterized in that: The industrial control network monitoring module performs three types of abnormal monitoring on message responses: timed connection, timed return and abnormal message.

4. A fuzzy testing method for an unknown industrial control protocol in a fuzzy testing system for an unknown industrial control protocol according to any one of claims 1 to 3, characterized in that: The method comprises the following steps: Step 1: During the fuzz testing process, the agent uses the HRLFuzz model based on hierarchical reinforcement learning to generate mutation test samples by executing actions; Step 2: The message sending module sends the test sample to the industrial control network system; Step 3: The industrial control network server of the industrial control network system processes the test case and obtains feedback and message response; Step 4: The message response is sent to the industrial control network monitoring module, and the report is fed back to the interpreter; Step 5: The interpreter translates the reward into a sequence of message entities and message states that can be accessed by the HRLFuzz framework and sends it to the agent; Step 6: The industrial control network monitoring module receives the reward through the reward function and sends it to the agent. It also monitors the message response for abnormalities and issues an alarm when an abnormality is found. Step 7: The agent receives the return and reward, and combined with the new state derived from the test example, selects the next action and returns to step 1 until the reward is maximized.

Citation Information

Patent Citations

  • Power-grid industrial control protocol vulnerability mining system and method based on fuzzy test

    CN105245403A

  • Industrial control protocol fuzzing test method based on protocol state

    CN105763392A

  • Penetration test framework suitable for industrial control system

    CN108809951A

  • Vulnerability detection method based on deep reinforcement learning and program path instrumentation

    CN110008710A

  • Fuzzy test case adaptive variation method and device based on reinforcement learning

    CN110175120A