Black box fuzzy testing method and device based on feedback enhancement

Abstract

The invention discloses a black box fuzzy testing method and device based on feedback enhancement. According to the method, field boundaries are divided by means of structural feature changes of messages, an automatic semantic inference method is constructed on the basis, abstract relation between field features and semantic categories is learned from a public protocol, and learned knowledge is applied to inference of semantics of all fields in a private protocol; and finally, equipment feedback messages are collected in the fuzzy testing process to establish a new feedback mechanism, and variation types and variation strategies are adjusted according to feedback information. According to the method, the field boundary and semantic information can be automatically identified based on the network communication data of the industrial site, manual participation is not needed, the field boundary and semantic information are used as the basis for establishing a feedback mechanism, and a good test effect is achieved for a complex and private industrial control protocol.

Description

Technical Field

[0001] The present invention belongs to the field of industrial control system security, and in particular relates to a black box fuzzy testing method and device based on feedback enhancement. Background Art

[0002] Industrial Control Systems (ICS) is a process control system that uses various automation equipment to collect and monitor the entire industrial production process in real time and make timely adjustments. It includes data monitoring and acquisition systems, distributed control systems, programmable logic controllers and other systems. Industrial control systems are widely used in key industries and fields such as chemical industry and electric power, and are the pillars of the basic industry on which the entire country and society depend for survival. With the rapid development of Internet technology in recent years and the key breakthroughs in hardware technology, the trend of the Internet of Everything has become increasingly unstoppable. The improvement in productivity and huge economic benefits it brings have set off a trend of informatization and networking in all walks of life, and the industrial control field is no exception. However, as more and more traditional industrial control devices are connected to the industrial Internet, many security issues have also arisen. Since most industrial control devices only consider intranet communication at the beginning of their design, they lack effective and strict identity authentication, and the operating systems used lack timely updates and vulnerability repairs. Once these industrial control devices are attacked, it will lead to data leakage and stagnation of industrial production at the least, and will further threaten the safety of the entire industrial system and staff at the worst. How to quickly and effectively discover vulnerabilities in industrial control systems is gradually becoming an important topic.

[0003] Protocol fuzz testing is a mainstream method for vulnerability mining. It generates a large amount of random, abnormal or boundary data, simulates various possible attack situations, sends data to the implementation of the target protocol and monitors the response. If the protocol is abnormal or crashes when processing this data, the test tool will save the test case and analyze the location and cause of the abnormality to determine the possible vulnerabilities in the device.

[0004] Traditional protocol fuzz testing methods rely on prior knowledge and need to construct seeds for fuzz testing based on known grammatical structures. However, in industrial scenarios, most devices have a set of private communication standards, and the execution information inside the device is often difficult to obtain. We cannot adjust the mutation strategy of fuzz testing based on feedback information. Therefore, fuzz testing methods for unknown protocols are needed for current research on the security of industrial control equipment. Summary of the invention

[0005] The purpose of the present invention is to provide a black box fuzzy testing method and device based on feedback enhancement in view of the deficiencies and shortcomings of the current fuzzy testing methods of private protocols.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a black box fuzz testing method based on feedback enhancement, the method comprising the following steps:

[0008] Collect the network communication traffic of the industrial control system, use the structural characteristics of the message itself, analyze the change distribution of the internal value of each message, and divide the field boundaries according to the inherent structural differences between the fields;

[0009] Build an automated semantic inference method to learn the abstract relationship between field features and semantic categories from public protocols, and apply the learned knowledge to infer the semantics of each field in private protocols;

[0010] Based on the results of field division and semantic extraction, a feedback-enhanced fuzz testing method is constructed. During the fuzz testing process, device feedback messages are collected to establish a new feedback mechanism, and the mutation strategy and the weight of the mutation type are adjusted according to the feedback information.

[0011] Furthermore, the structural feature of the message itself is the similarity of consecutive bytes in the message, that is, each bit of two adjacent bytes is identical or stale, and the sum is averaged to obtain the similarity of adjacent bytes; then the structural feature changes of adjacent bytes are calculated; finally, according to the structural feature change curve of the message itself, a Gaussian filter is used to smooth the curve, and the inflection point of the rising period of the curve is used as the field boundary.

[0012] Furthermore, the semantic inference method includes the following three parts: field feature extraction, fine-grained semantic aggregation, and similarity-based semantic inference.

[0013] Furthermore, the field characteristics integrate the characteristics of the field itself and the context characteristics of the field. The context characteristics of the field represent the timing relationship between the field and the fields at the same byte offset position in the message sequence and the relationship between the field and adjacent fields.

[0014] Furthermore, the field feature extraction is based on three types of gated recurrent network units. The first type of gated recurrent network unit G i Used to extract the features of the field itself, the second gated recurrent network unit G o The third gated recurrent network unit G is used to extract the temporal relationship between the field and the field at the same byte offset position in the message sequence. r It is used to extract the relationship between fields and adjacent fields, and finally integrates multiple gated recurrent network units into a large neural network model.

[0015] Furthermore, the field feature extraction is specifically as follows:

[0016] First, each byte of the sth field of the fth message is used as G i The input and output hidden states are That is, the extracted feature of the field itself; taking the longest length of each field as the benchmark, fill the fields that are less than the length;

[0017] Secondly, G i The extracted features of each field are input into G o In, G o The hidden state of the output is recorded as For a message sequence consisting of N fields, this part of the features of all fields constitutes a sequence

[0018] Finally, using the bidirectional GRU, G r , processing sequence use The corresponding output is used as the final representation of the field features, and the feature of each field is recorded as r s , the field feature R of the entire message sequence is expressed as (r 1 ,r 2 ,…,r N ).

[0019] Furthermore, the semantic aggregation is implemented based on text similarity. For known protocols, a natural language model is used to convert semantic descriptions into semantic vectors, and clustering is performed based on the similarity between the vectors.

[0020] Furthermore, the similarity-based semantic inference includes:

[0021] During the model training phase, for fields belonging to the same semantic category, the feature similarity between them needs to be higher, and vice versa. The loss function L established based on this feature is as follows:

[0022]

[0023] Among them, P represents the positive sample set, A represents the negative sample set, sim represents the inner product of two vectors, and τ represents the temperature parameter;

[0024] In the model utilization stage, after extracting the field features of the unknown protocol, the similarity between each field feature and the center of each semantic category is calculated, and the semantics of the field is inferred based on the similarity.

[0025] Furthermore, the collection device feedbacks messages to establish a new feedback mechanism, specifically:

[0026] Perform a complete deterministic mutation on the seeds in the current seed library that have not yet undergone deterministic mutation, and send them to the device under test, merging the feedback messages with high similarity into the same category; if a different response type is triggered, the test case is regarded as a new unique seed and added to the seed library; otherwise, the test case will not be saved;

[0027] When there is no seed in the seed library that has not undergone deterministic mutation, a seed is randomly selected for a havoc mutation. If the number of mutations reaches the preset number and no new response type is triggered, the fuzz test is terminated;

[0028] At the same time, in the process of deterministic mutation, the device status of each round of testing is collected, different weights are assigned to different statuses, and then the weight sum corresponding to each mutation type is calculated. The next round of testing will dynamically adjust the probability of their occurrence in the next round of testing based on the proportion of the weight sum of each mutation type.

[0029] In a second aspect, the present invention provides a black box fuzzy testing device based on feedback enhancement, comprising the following modules:

[0030] Field division module: Utilizes the structural characteristics of the message itself to analyze the distribution of changes in the internal values ​​of each message and divides the field boundaries according to the inherent structural differences between the fields;

[0031] Semantic inference module: Build an automated semantic inference method to learn the abstract connection between field features and semantic categories from public protocols, and apply the learned knowledge to infer the semantics of each field in private protocols;

[0032] Fuzz testing module: collects device feedback messages during the fuzz testing process to establish a new feedback mechanism, and adjusts the mutation strategy and the weight of the mutation type according to the feedback information.

[0033] The beneficial effects of the present invention are as follows: the black box fuzzy testing method based on feedback enhancement proposed by the present invention automatically identifies field boundaries and semantic information based on network communication data of industrial sites, and uses this as the basis for establishing a feedback mechanism, and adjusts the mutation type and mutation strategy of the fuzzy test according to real-time feedback, and also has a good test effect on complex and private industrial control protocols. The method of the present invention is a universal black box fuzzy testing method, which is not affected by the type of protocol, does not require too much prior knowledge, and is practical and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is an example of data preprocessing in the method of the present invention;

[0036] Figure 2 is a flow chart of the automated semantic inference method in the method of the present invention;

[0037] Figure 3 It is a flow chart of field feature extraction in the method of the present invention. DETAILED DESCRIPTION

[0038] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0039] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0040] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0041] This embodiment provides a black box fuzzy testing method based on feedback enhancement, which includes the following steps:

[0042] (1) Collect the network communication traffic of the industrial control system. Each traffic packet contains several message sequences.

[0043] Using the structural characteristics of the message itself, we analyze the distribution of changes in the internal values ​​of each message and divide the boundaries of the fields according to the inherent structural differences between the fields. The structural characteristics of the message itself are the similarity of consecutive bytes in the message, and the calculation method is as follows:

[0044]

[0045] Among them, P and Q represent two adjacent bytes, i represents the i-th bit in the byte, ⊙ represents the sameness or operation, and D represents the similarity. Then the structural feature changes of adjacent bytes can be expressed by the following formula:

[0046] △D k =(D(m k , m k+1 )-D(m k-1 , m k ))

[0047] Where k represents the kth position in message m, m k Represents the kth byte in message m. For a message of length n, k∈(0,n).

[0048] Therefore, according to the change curve of the structural characteristics of the message itself, after smoothing it with a Gaussian filter, the inflection point of the rising period of the curve is used as the boundary of the field.

[0049] (2) Data preprocessing is required before semantic inference. Figure 1 As shown, use open source protocol parsers such as wireshark or tshark to parse the traffic packet, output the result in json format, and then obtain the protocol field name and description information from the json file and store it in csv format.

[0050] (3) Build an automated semantic inference method to learn the abstract connection between field features and semantic categories from public protocols (i.e., known protocols), and apply the learned knowledge to infer the semantics of each field in private protocols (i.e., unknown protocols).

[0051] Specifically, Figure 2 As shown in the figure, the method consists of three parts: the first is field feature extraction, the second is fine-grained semantic aggregation, and the third is semantic inference based on similarity. The whole process can be divided into two stages. In the model training stage, for fields belonging to the same semantic category, the feature similarity between them needs to be higher, and vice versa. In the model utilization stage, after extracting the field features of the unknown protocol using the field extraction method, the similarity between each field feature and the center of each semantic category is calculated, and the semantics of the field is inferred based on the similarity.

[0052] The field feature extraction method is based on three types of gated recurrent network (GRU) units. The first type of gated recurrent network unit G i The second gated recurrent network unit G is used to extract the features of the field itself. o The third gated recurrent network unit G is used to extract the temporal relationship between the field and the field at the same byte offset position in the message sequence. r It is used to extract the relationship between fields and adjacent fields, and finally integrates multiple GRU units into a large neural network model.

[0053] Specifically, Figure 3As shown, each byte of the sth field of the fth message is used as G i The input and output hidden states are That is, the feature of the field itself is extracted; since not every message has the same length for this field, the longest length of the field is used as a benchmark, and 256 is used to fill the field that is less than the length, such as Figure 3 The "padding" in the , until all fields at that position reach the same length;

[0054] Secondly, G i The extracted features of each field are input into G o The more messages a message sequence contains, the greater the o The more input, the more G o The hidden state of the output is recorded as For a message sequence consisting of N fields, this part of the features of all fields constitutes a sequence

[0055] Finally, using the bidirectional GRU, G r , process the sequence use The corresponding output is used as the final representation of the field features, and the feature of each field is recorded as r s , the field feature R of the entire message sequence can be expressed as (r 1 ,r 2 ,…,r N ).

[0056] Fine-grained semantic aggregation is to aggregate multiple similar fine-grained semantics into coarse-grained semantic categories based on semantic similarity. This step aims to improve the scalability of the method. For some proprietary semantics appearing in private protocols, the meaning of the field can be roughly inferred by calculating its similarity with the existing public semantics, rather than matching one by one in a few pattern rules reflected in semantics such as length field, serial number field, function code field, etc. The specific implementation of semantic aggregation is based on text similarity. For known protocols, open source protocol decoders such as wireshark or tshark are used to obtain the semantic description of each field, and the natural language model is used to convert the semantic description into a semantic vector, and clustering is performed based on the similarity between the vectors.

[0057] During the model training phase, for fields belonging to the same semantic category, the feature similarity between them needs to be higher, and vice versa. The loss function L established based on this feature is as follows:

[0058]

[0059] Among them, P represents the set of all positive samples, A represents the set of negative samples, sim represents the inner product of two vectors, and τ represents the temperature parameter.

[0060] (4) Based on the results of field division and semantic extraction, a feedback-enhanced fuzz testing method is constructed. During the fuzz testing process, device feedback messages are collected to establish a new feedback mechanism, and the mutation strategy and mutation type weights are adjusted according to the feedback information. Specifically:

[0061] The change of feedback message type reflects the change of execution path of the device under test to a certain extent. The first step of fuzz testing is to perform a complete deterministic mutation on the seeds in the current seed library that have not yet undergone deterministic mutation, including but not limited to bit flipping, bit addition and subtraction, special value replacement, etc., and send it to the device under test. The Smith-Waterman algorithm is used to calculate the similarity between the feedback message and the message in the response library, and the hierarchical clustering method is used to merge the feedback messages with high similarity into the same category. If a different response type is triggered, the test case is regarded as a new unique seed and added to the seed library; otherwise, the test case will not be saved.

[0062] When there are no seeds in the seed library that have not undergone deterministic mutation, a seed is randomly selected for a havoc mutation. If the number of mutations reaches a certain number and no new response type is triggered, the fuzz test is terminated.

[0063] At the same time, in the process of deterministic mutation, the device status of each round of testing is collected, different weights are assigned to different states, and then the weight sum corresponding to each mutation type is calculated. The next round of testing will dynamically adjust the probability of each mutation type occurring in the next round of testing according to the proportion of the weight sum of each mutation type. In this embodiment, every 500 test cases is recorded as one round. In the first round of testing, the mutation probability of each mutation strategy is equal, and the dynamic adjustment strategy of each subsequent round is as follows:

[0064] ① Define the weight of each connection state, represented by the symbol S. The connection timeout is defined as 7, the successful response and triggering of a new path is defined as 4, and the successful response but no new path is defined as 1;

[0065] ②For the mutation type K, calculate the weight sum of the mutation type as Where h is the number of all states triggered by this mutation type, S represents the weight of the connection state, and count is the number of times the connection state appears;

[0066] ③ Based on the weight sum of the mutation types calculated in step ②, adjust the mutation probability of the mutation type to Where l represents the total number of mutation types.

[0067] On the other hand, the present invention also provides a black box fuzzy testing device based on feedback enhancement, the device comprising:

[0068] Field division module: Utilizes the structural characteristics of the message itself to analyze the distribution of changes in the internal values ​​of each message and divides the field boundaries according to the inherent structural differences between the fields;

[0069] Semantic inference module: Build an automated semantic inference method to learn the abstract connection between field features and semantic categories from public protocols, and apply the learned knowledge to infer the semantics of each field in private protocols;

[0070] Fuzz testing module: collects device feedback messages during the fuzz testing process to establish a new feedback mechanism, and adjusts the mutation strategy and the weight of the mutation type according to the feedback information.

[0071] The implementation of each module can refer to the steps in the above-mentioned black box fuzz testing method based on feedback enhancement.

[0072] Corresponding to the embodiment of the black-box fuzz testing method based on feedback enhancement, the present invention also provides an embodiment of a black-box fuzz testing device based on feedback enhancement. A black-box fuzz testing device based on feedback enhancement provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a black-box fuzz testing method based on feedback enhancement in the above embodiment.

[0073] The embodiment of a black box fuzz testing device based on feedback enhancement provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware.

[0074] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a black box fuzzy testing method based on feedback enhancement in the above embodiment is implemented.

[0075] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0076] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A black box fuzz testing method based on feedback enhancement, characterized in that: include: Collect the network communication traffic of the industrial control system, use the structural characteristics of the message itself, analyze the change distribution of the internal value of each message, and divide the field boundaries according to the inherent structural differences between the fields; Build an automated semantic inference method to learn the abstract relationship between field features and semantic categories from public protocols, and apply the learned knowledge to infer the semantics of each field in private protocols; Based on the results of field division and semantic extraction, a feedback-enhanced fuzz testing method is constructed. During the fuzz testing process, device feedback messages are collected to establish a new feedback mechanism, and the mutation strategy and the weight of the mutation type are adjusted according to the feedback information.

2. The black box fuzz testing method based on feedback enhancement according to claim 1 is characterized in that: The structural feature of the message itself is the similarity of consecutive bytes in the message, that is, the similarity of adjacent bytes is obtained by adding and averaging each bit of two adjacent bytes; then the structural feature changes of adjacent bytes are calculated; finally, according to the structural feature change curve of the message itself, a Gaussian filter is used to smooth the curve, and the inflection point of the rising period of the curve is used as the field boundary.

3. The black box fuzz testing method based on feedback enhancement according to claim 1 is characterized in that: The semantic inference method includes the following three parts: field feature extraction, fine-grained semantic aggregation, and similarity-based semantic inference.

4. The black box fuzz testing method based on feedback enhancement according to claim 1 is characterized in that: The field characteristics integrate the characteristics of the field itself and the context characteristics of the field. The context characteristics of the field represent the timing relationship between the field and the fields at the same byte offset position in the message sequence and the relationship between the field and adjacent fields.

5. The black box fuzz testing method based on feedback enhancement according to claim 3 is characterized in that: The field feature extraction is based on three types of gated recurrent network units. The first type of gated recurrent network unit G i Used to extract the features of the field itself, the second gated recurrent network unit G o The third gated recurrent network unit G is used to extract the temporal relationship between the field and the field at the same byte offset position in the message sequence. r It is used to extract the relationship between fields and adjacent fields, and finally integrates multiple gated recurrent network units into a large neural network model.

6. The feedback-enhanced black box fuzz testing method according to claim 5 is characterized in that: The field feature extraction is specifically as follows: First, each byte of the sth field of the fth message is used as G i The input and output hidden states are That is, the extracted features of the field itself; The longest length of each field is used as a benchmark to fill in the fields that are shorter than the length. Secondly, G i The extracted features of each field are input into G o In, G o The hidden state of the output is recorded as For a message sequence consisting of N fields, this part of the features of all fields constitutes a sequence Finally, using the bidirectional GRU, G r , processing sequence use The corresponding output is used as the final representation of the field features, and the feature of each field is recorded as r s , the field feature R of the entire message sequence is expressed as (r 1 ,r 2 ,…,r N ).

7. The feedback-enhanced black box fuzz testing method according to claim 3 is characterized in that: The semantic aggregation is implemented based on text similarity. For known protocols, a natural language model is used to convert semantic descriptions into semantic vectors, and clustering is performed based on the similarity between vectors.

8. The feedback-enhanced black box fuzz testing method according to claim 6, characterized in that: The similarity-based semantic inference includes: During the model training phase, for fields belonging to the same semantic category, the feature similarity between them needs to be higher, and vice versa. The loss function L established based on this feature is as follows: Among them, P represents the positive sample set, A represents the negative sample set, sim represents the inner product of two vectors, and τ represents the temperature parameter; In the model utilization stage, after extracting the field features of the unknown protocol, the similarity between each field feature and the center of each semantic category is calculated, and the semantics of the field is inferred based on the similarity.

9. The feedback-enhanced black box fuzz testing method according to claim 6, characterized in that: The collection device feedback message is used to establish a new feedback mechanism, specifically: Perform a complete deterministic mutation on the seeds in the current seed library that have not yet undergone deterministic mutation, and send them to the device under test, merging the feedback messages with high similarity into the same category; if a different response type is triggered, the test case is regarded as a new unique seed and added to the seed library; otherwise, the test case will not be saved; When there is no seed in the seed library that has not undergone deterministic mutation, a seed is randomly selected for a havoc mutation. If the number of mutations reaches the preset number and no new response type is triggered, the fuzz test is terminated; At the same time, in the process of deterministic mutation, the device status of each round of testing is collected, different weights are assigned to different statuses, and then the weight sum corresponding to each mutation type is calculated. The next round of testing will dynamically adjust the probability of their occurrence in the next round of testing based on the proportion of the weight sum of each mutation type.

10. A black box fuzzy testing device based on feedback enhancement, characterized in that: include: Field division module: Utilizes the structural characteristics of the message itself to analyze the distribution of changes in the internal values ​​of each message and divides the field boundaries according to the inherent structural differences between the fields; Semantic inference module: Build an automated semantic inference method to learn the abstract connection between field features and semantic categories from public protocols, and apply the learned knowledge to infer the semantics of each field in private protocols; Fuzz testing module: collects device feedback messages during the fuzz testing process to establish a new feedback mechanism, and adjusts the mutation strategy and the weight of the mutation type according to the feedback information.

Images