A network traffic-based industrial control protocol reverse analysis method
By performing dynamic field clustering operations and machine learning recognition on industrial control protocol traffic, the accuracy problem of industrial control protocol parsing was solved, and efficient reverse analysis and semantic recognition of complex protocols were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2022-10-18
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies struggle to effectively parse complex and proprietary industrial control protocols, resulting in low accuracy in network intrusion detection and vulnerability analysis.
By capturing protocol traffic from industrial control systems, message sequences are trimmed to the shortest possible length. Dynamic field clustering operations and machine learning are used to identify control fields. Hamming distance and random forest classifiers are combined to identify increments, addresses, checksums, lengths, and function codes, and the protocol format is output.
It enables efficient and automated reverse analysis of complex industrial control protocols, improves the accuracy of protocol format recognition and semantic understanding, and is applicable to nested protocols.
Smart Images
Figure CN115622926B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of protocol format and semantic parsing technology, specifically to a method for reverse analysis of industrial control protocols based on network traffic. Background Technology
[0002] With the proliferation of network applications, network protocols have grown rapidly. However, most protocols lack publicly available specifications or complete descriptions, especially proprietary protocols such as industrial control system (ICS) protocols and Internet of Things (IoT) protocols. Unlike most internet protocols, in the industrial control field, manufacturers generally do not choose to disclose the details of the formulation and use of such protocols due to considerations such as economy, security, and privacy. In today's increasingly important industrial control systems, these highly complex and proprietary protocols pose challenges to security measures such as network intrusion detection, fuzzing, and vulnerability analysis.
[0003] Current reverse engineering based on network traffic primarily targets Internet protocols with text-based data, employing analysis methods such as clustering and field segmentation using the Needleman-Wunsch sequence alignment algorithm to obtain the protocol format. However, Industrial Control System (ICS) protocols, compared to Internet protocols, have the following characteristics: strong privacy, making it impossible to obtain communication code; shorter fields, generally defined in bytes or even bits; no delimiters, and no printable characters, making string segmentation difficult compared to Internet protocols; and generally pure binary data, making it impossible to directly obtain semantic information. These characteristics of ICS protocols result in low accuracy for traditional reverse engineering methods that directly apply sequence alignment. Therefore, designing an efficient and automated protocol reverse engineering method for ICS protocols is currently needed in the field of ICS security. Summary of the Invention
[0004] The purpose of this invention is to propose a reverse analysis method for industrial control protocols based on network traffic, taking into account the format characteristics of industrial control protocols.
[0005] The objective of this invention is achieved through the following technical solution: a method for reverse analysis of industrial control protocols based on network traffic, the method comprising the following steps:
[0006] Step 1: Capture the protocol traffic of the actual industrial control system or collect the protocol traffic publicly available from the open source community, while also collecting the function click traffic according to specific rules;
[0007] Step 2: All message sequences are trimmed to the shortest sequence length for dynamic and static field identification. Dynamic fields are used as candidate fields for control fields. Cluster operations are performed on the message sequences. If the result of cluster operations using candidate fields meets the threshold condition, the candidate field is determined to be a control field. Control field semantic information is added to the field, and semantic identification is performed on the message sequences in each cluster. Otherwise, the candidate field is determined to be a non-control field, and semantic identification is performed directly on the original message sequence.
[0008] Step 3: Perform semantic recognition on the message sequences or original message sequences in each cluster, including incremental sequences, address fields, check fields, length fields, and function codes;
[0009] Step 4: Based on semantic recognition, perform industrial control protocol boundary division, merge adjacent static fields and fields with the same semantics, and output the protocol format.
[0010] Furthermore, in step 1, in order to locate the function code position of the industrial control protocol, function click traffic is collected according to specific rules, specifically referring to the traffic collected after performing a specified number of "run-pause" or "read-write" operation instructions with paired nature on the PLC.
[0011] Furthermore, step 2 specifically includes:
[0012] Iterate through all input message sequences, calculate the shortest sequence length, trim all message sequence lengths to the shortest sequence length, and then identify dynamic and static fields.
[0013] Dynamic fields are used as candidate fields for control fields. A random variable is introduced for each candidate field as the probability that the field becomes the control field. Based on each candidate field, cluster operations are performed on the message sequence to calculate the message similarity constraints of each cluster and select the candidate field with the highest probability.
[0014] If the result of clustering operations on the selected candidate fields meets the threshold condition, the candidate field is determined to be a control field; otherwise, the candidate field is determined to be a non-control field.
[0015] Furthermore, in step 2, the threshold condition is to simultaneously satisfy the following two conditions: ① the number of clusters after performing cluster operations with the inferred control field does not exceed n, and the length of the message sequence in each cluster is not completely consistent; ② the offset position of the inferred control field is in the first m fraction of the message sequence.
[0016] Further, in step 3, the incremental sequence is identified by analyzing the Hamming distance distribution after the message sequence is converted into bit-level data. Specifically, the Hamming distance distribution of each offset position is calculated in 8-bit units. If the offset position has at least 4 bits of increasing Hamming distance from one end to the other, the field corresponding to the offset position is considered to be an incremental sequence in a strict sense, and incremental sequence semantic information is added to the field. If the adjacent field of the identified strict incremental sequence has the characteristics of increasing Hamming distance distribution, the adjacent field is considered to be the high bit of the incremental sequence, and incremental sequence semantic information is added to the adjacent field.
[0017] Furthermore, in step 3, the address field is identified by determining whether it is consistent with the IP address or port number of the TCP / IP layer, using a single byte as the unit; 2-4 bytes are extracted from the end of the message sequence as candidate fields for the verification field, and the verification field is identified by calculating whether the CRC value of the message sequence after removing the candidate fields is equal to the value of the candidate fields.
[0018] Furthermore, in step 3, the length field is identified based on a random forest classifier, and the specific steps are as follows:
[0019] a. Cluster the message sequences by physical length, prune all message sequences in each cluster to the shortest sequence length and convert them into bit-level data, add length information labels to each cluster, and divide the training set and test set;
[0020] b. Input the data in the training set into the random forest classifier for training, and use the test set to test the trained classifier. If the classification accuracy of the test set is greater than the set threshold, output the feature importance ranking of the bit offset of the random forest classification.
[0021] c. Map the bit offsets of important features to bytes and remove duplicates, then output the length field to be selected;
[0022] d. Set the length field filtering rules, that is, the decimal value corresponding to the field should be less than or equal to the length of the entire message sequence. The filtered field is the estimated length field.
[0023] Furthermore, in step 3, the function code is located by analyzing the function click traffic collected according to specific rules. Specifically, after collecting the function click traffic, the message sequence is divided into dynamic and static fields. The unique value of each dynamic field remaining after the identified semantic information is counted. If the number of unique values of each field meets the aforementioned rules, the field is considered a candidate field for the function code, and the semantic information of the function code is added to it. If the identified unique values meet the aforementioned rules for multiple fields, the number of clicks is redefined and the function click traffic is collected again for multiple identifications.
[0024] Furthermore, in step 3, the fields filtered out by the filtering rules during the length field extraction process are used as candidate fields for the function code; if a candidate field meets the characteristics of a function code, it is considered to be part of the function code and semantic information of the function is added to it; the formula for calculating the function code score of the candidate field is as follows:
[0025] F(w i )=P(w i )*H(w i )
[0026]
[0027]
[0028] Where w i For the candidate field, P(w) i H(w) represents the function code uniqueness constraint. i ) represents the function code entropy value constraint, w i .num represents the field to be selected. i The number of unique values, m.len is the total number of message sequences, p(w i ) is the field to be selected w i The probability of occurrence; select F(w) i Fields that meet the threshold conditions are selected as function codes, and semantic information of the function codes is added to these fields.
[0029] Furthermore, the semantic recognition order of the incremental sequence, address field, check field, length field, and function code is not limited, but the semantic recognition of the function code must be ensured after the recognition of the length field.
[0030] Compared with the prior art, the present invention has the following advantages:
[0031] 1. An innovative method for locating function codes in industrial control protocols is proposed.
[0032] 2. Improving the inference of protocol format by identifying semantic information of special fields is actually a reverse thinking of the protocol parsing process.
[0033] 3. This invention also has a high accuracy rate in recognizing the syntax and semantics of complex and nested industrial control protocols. Attached Figure Description
[0034] Figure 1 This is an overall block diagram of the method of the present invention;
[0035] Figure 2 A schematic diagram of the industrial control protocol reverse analysis process provided for an exemplary embodiment of the present invention. Detailed Implementation
[0036] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0037] like Figure 1 As shown, this invention provides a method for reverse analysis of industrial control protocols based on network traffic, comprising the following steps:
[0038] Step 1: Capture the protocol traffic of the actual industrial control system or collect the protocol traffic publicly available from the open source community, while also collecting the function click traffic according to specific rules.
[0039] To enrich the protocol sample as much as possible, protocol traffic from actual industrial control systems is collected using traffic acquisition tools, or publicly available protocol traffic is collected from the open-source community. In addition to normal communication traffic, to locate the function code position of the industrial control protocol, this invention requires the additional collection of some traffic according to specific rules. Specifically, this refers to the traffic collected after performing a specified number of "run-pause" or "read-write" operation commands with paired nature on the PLC.
[0040] Step 2: All message sequences are trimmed to the shortest sequence length for dynamic and static field identification. Dynamic fields are used as candidate fields for control fields. Cluster operations are performed on the message sequences. If the result of cluster operations using candidate fields meets the threshold condition, the candidate field is determined to be a control field. Control field semantic information is added to the field, and semantic identification is performed on the message sequences in each cluster. If the threshold condition is not met, the candidate field is determined to be a non-control field, and semantic identification is performed directly on the original message sequence.
[0041] Many industrial control protocols are nested protocols and contain control fields. These fields indicate the type of the protocol message and are related to the structure of subsequent messages. Messages with the same control field have high similarity. This control field typically appears early in the message header. To determine whether a protocol has a control field and locate its position, all input message sequences are traversed, and the shortest sequence length l is calculated. min Trim all message sequence lengths to l minAfter identifying dynamic and static fields, dynamic fields are used as candidate fields for control fields. A random variable is introduced for each candidate field to represent the probability that the field will become a control field. Based on each candidate field, a cluster operation is performed on the message sequence, and the message similarity constraint of each cluster is calculated. The higher the message similarity, the greater the probability that the candidate field will become a control field. The candidate field with the highest probability is selected. If the result of the cluster operation using the selected candidate field meets the threshold condition, the candidate field is determined to be a control field, and control field semantic information is added to the field. Semantic recognition is then performed on the message sequences in each cluster. If the threshold condition is not met, the candidate field is determined to be a non-control field, and semantic recognition is performed directly on the original message sequence. The threshold conditions specifically include: ① The number of clusters after the cluster operation using the inferred control field does not exceed n, and the length of the message sequence in each cluster is not completely consistent; ② The offset position of the inferred control field is within the first m-th fraction of the message sequence. Conditions ① and ② must be met simultaneously. n and m are set based on experimental results. In this embodiment, n=4 and m=3.
[0042] Step 3: Perform semantic recognition on the message sequences or original message sequences in each cluster, including incremental sequences, address fields, check fields, length fields, and function codes.
[0043] The increment sequence is a field type characterizing the temporal features of a message sequence. It is an incrementing counter used for flow control and freshness, typically 1-2 bytes in size. This invention utilizes the distribution of Hamming distance for increment sequence identification. Since the increment sequence is an incrementing counter, it exhibits the characteristic of increasing Hamming distance. All message sequences are pruned to the shortest sequence length l. min The data is then converted to bit-level data, and the Hamming distance distribution at each offset position is calculated in 8-bit units. The calculation formula is shown in Equation (1), where i is the offset bit position, j is the message sequence number, ⊕ is the XOR operation, m.len is the total number of message sequences, and d is the Hamming distance. i A larger value indicates a higher frequency of bit reversal at the offset position i, and vice versa. If the offset position has at least a 4-bit increasing Hamming distance from one end to the other, the field corresponding to the offset position is considered a strictly defined increment sequence, and increment sequence semantic information is added to the field. If the adjacent field of the identified strictly defined increment sequence has characteristics that conform to the increasing Hamming distance distribution, the adjacent field is considered a high-order bit of the increment sequence, and increment sequence semantic information is added to the adjacent field.
[0044]
[0045] The address field represents the addresses of the sender and receiver in the protocol, including the source device address and the destination device address. The message sequence is divided into dynamic and static fields. The number of unique values for each dynamic field is counted. If the number of unique values is 2, it is considered a candidate address field. The message sequence is clustered by IP address or port number. It is checked whether the candidate address field becomes a static field. If so, the candidate field is determined to be an address field, and semantic information of the address field is added to it.
[0046] To ensure error-free data transmission, many industrial control protocols utilize verification methods such as CRC checksums or hash checksums. These checksums are highly random and typically appear at the end of the message sequence, usually consisting of 2-4 bytes. Two to four bytes are extracted from the end of the message sequence as candidate fields for the checksum. The CRC value of the message sequence excluding the candidate fields is calculated using common checksum parameters such as CRC16 and CRC32. If the calculated CRC value matches the candidate field value, the candidate field is determined to be the checksum field, and its semantic information is added.
[0047] The length field is a field that characterizes the length of a protocol message sequence. Its length is fixed, and its value is the length of the field it refers to. However, the length field value itself could be the total length of the message sequence, the length of subsequent message sequences starting from a certain position, or the length of one of multiple length fields representing a subsequent data block. Currently, there are three methods for identifying length fields based on network traffic: First, calculating the Pearson correlation coefficient between the candidate field value and the physical length of the message sequence; if this coefficient reaches a certain threshold, the candidate field is inferred to be a length field. Second, calculating whether the difference between candidate fields at the same offset position in the message sequence is equal to the difference in physical length between the two message sequences; if they are equal, the candidate field is inferred to be a length field. Third, clustering based on physical length; if the original candidate field changes from a dynamic field to a static field after clustering, the candidate field is inferred to be a length field. However, these three identification methods have shortcomings. The Pearson correlation coefficient-based method is sensitive to threshold parameters, and the latter two methods can basically only locate length fields whose length field value is the total message length, and cannot completely identify multiple length fields with different scopes. To accurately identify multiple length fields in a message sequence, this invention employs a machine learning-based length field identification method, the specific steps of which are as follows:
[0048] a. Cluster the message sequences by physical length, and trim all message sequences in each cluster to the shortest possible sequence length. min The data was then converted to bit-level data, and length information labels were added to each cluster. The training set and test set were then divided according to the ratio.
[0049] b. Input the data in the training set into the random forest classifier for training, and use the test set to test the trained classifier. If the classification accuracy on the test set is greater than 90%, output the feature importance ranking of the bit offset of the random forest classification.
[0050] c. Map the bit offsets of important features to bytes and remove duplicates, then output the length field of the candidate. candidate ;
[0051] d. Set the filtering rules for the length field, i.e., the decimal value of the field, int(length). candidate The length of the entire message sequence should be less than or equal to the length of the message sequence (message.size). The filtered field is the estimated length field, and the filtering conditions are as follows:
[0052] int(length candidate )≤message.size
[0053] By using a machine learning-based length field identification method, this invention can locate multiple length fields in a message sequence with high accuracy.
[0054] Function codes represent the functions performed by a message sequence in an industrial control protocol, such as PLC start / stop, read / write, upload / download, etc. They are fields with significant control implications and are crucial for industrial control protocol control. Existing methods mainly identify and locate function codes based on field offsets, value diversity, and entropy values, which require high-quality datasets and have low accuracy. This invention locates function codes by analyzing function click traffic collected according to specific rules. Specifically, the traffic collected according to specific rules refers to the traffic collected after performing a specified number of paired operation commands such as "run-pause" or "read-write" on the PLC. After collecting the traffic, the message sequence is divided into dynamic and static fields. The unique value of each dynamic field remaining after identifying semantic information is counted. If the number of unique values for a field conforms to the aforementioned rules, the field is considered a candidate field for a function code, and its semantic information is added. If multiple fields have unique values that conform to the aforementioned rules, the number of clicks is redefined, and function click traffic is collected again for multiple identifications.
[0055] However, complex nested protocols may contain more than one function code. For example, in the CIP PCCC protocol, the function code representing the start / stop function is a sub-function code, and there is a parent function code before the sub-function code that represents the change of running state. Therefore, although the function click traffic identification method based on the above specific rules can accurately locate the sub-function code, it cannot identify multiple function codes. For industrial control protocols, message sequences of different specific functions usually have different physical lengths. Based on this observation, we use the fields filtered out by the filtering rules during the length field extraction process as candidate fields for function codes. If a candidate field meets the characteristics of a function code, it is considered to be part of the function code and semantic information of the function is added to it. The function code calculation score of the candidate field is shown in Equation (2), where w i For the candidate field, P(w) i H(w) represents the function code uniqueness constraint. i ) represents the function code entropy value constraint, and the specific calculation method is shown in equations (3) and (4). In equation (3), w i .num represents the field to be selected. i The number of unique values, m.len is the total number of message sequences, in equation (4), p(w i ) is the field to be selected w i The probability of occurrence. Choose F(w) i Fields that meet the threshold criteria are selected as function codes, and semantic information of the function code is added to these fields. The threshold is set based on experimental results.
[0056] F(w i )=P(w i )*H(w i (2)
[0057]
[0058]
[0059] By using a function code recognition method based on a specific click strategy and a length-filtered field, this invention can accurately locate the offset position of the function code and extract the semantic information of multiple function codes.
[0060] The semantic recognition order of the incremental sequence, address field, check field, length field, and function code is not limited, but the semantic recognition of the function code must be guaranteed to be after the recognition of the length field.
[0061] In one embodiment, semantic recognition of the incremental sequence, address field, check field, length field, and function code is performed sequentially, such as... Figure 2 As shown.
[0062] Step 4: Based on semantic recognition, perform industrial control protocol boundary division, merge adjacent static fields and fields with the same semantics, and output the protocol format.
[0063] Adjacent static fields are treated as a protocol field. The present invention identifies adjacent fields with the same semantics as a protocol field, and finally outputs an industrial control protocol format with semantic information.
[0064] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.
Claims
1. A method for reverse analysis of industrial control protocol based on network traffic, characterized in that, include: Step 1: Capture the protocol traffic of the actual industrial control system or collect the protocol traffic publicly available from the open source community, while also collecting the function click traffic according to specific rules; Step 2: All message sequences are trimmed to the shortest sequence length for dynamic and static field identification. Dynamic fields are then used as candidate control fields for clustering operations on the message sequences. If the result of the clustering operation using the candidate field meets a threshold condition, the candidate field is determined to be a control field, and control field semantic information is added to it. Semantic identification is then performed on the message sequences in each cluster. Otherwise, the candidate field is determined not to be a control field, and semantic identification is directly performed on the original message sequence. Specifically, Step 2 involves: Iterate through all input message sequences, calculate the shortest sequence length, trim all message sequence lengths to the shortest sequence length, and then identify dynamic and static fields. Dynamic fields are used as candidate fields for control fields. A random variable is introduced for each candidate field as the probability that the field becomes the control field. Based on each candidate field, cluster operations are performed on the message sequence to calculate the message similarity constraints of each cluster and select the candidate field with the highest probability. If the result of clustering operations on the selected candidate fields meets the threshold condition, then the candidate field is determined to be a control field; otherwise, the candidate field is determined to be a non-control field. The threshold condition is that the following two conditions are met simultaneously: ① The number of clusters after performing cluster operations with the inferred control field does not exceed n, and the length of the message sequence in each cluster is not completely consistent; ② The offset position of the inferred control field is in the first m-th of the message sequence. Step 3: Perform semantic recognition on the message sequences or original message sequences in each cluster, including incremental sequences, address fields, check fields, length fields, and function codes; Step 4: Based on semantic recognition, perform industrial control protocol boundary delineation, merge adjacent static fields and fields with the same semantic meaning, and output the protocol format.
2. The network traffic based industrial control protocol reverse analysis method according to claim 1, characterized in that, In step 1, in order to locate the function code position of the industrial control protocol, function click traffic is collected according to specific rules. Specifically, this refers to the traffic collected after performing a specified number of "run-pause" or "read-write" operation instructions with paired nature on the PLC.
3. The network traffic based industrial control protocol reverse analysis method according to claim 1, characterized in that, In step 3, the incremental sequence is identified by analyzing the Hamming distance distribution after the message sequence is converted into bit-level data. Specifically, the Hamming distance distribution of each offset position is calculated in 8-bit units. If the offset position has at least 4 bits of increasing Hamming distance from one end to the other, the field corresponding to the offset position is considered to be an incremental sequence in a strict sense, and incremental sequence semantic information is added to the field. If the adjacent field of the identified strict incremental sequence has the characteristics of increasing Hamming distance distribution, the adjacent field is considered to be the high bit of the incremental sequence, and incremental sequence semantic information is added to the adjacent field.
4. The network traffic based industrial control protocol reverse analysis method according to claim 1, characterized in that, In step 3, the address field is identified by determining whether it is consistent with the IP address or port number of the TCP / IP layer, using a single byte as the unit; 2-4 bytes are extracted from the end of the message sequence as candidate fields for the verification field, and the verification field is identified by calculating whether the CRC value of the message sequence after removing the candidate fields is equal to the value of the candidate fields.
5. The network traffic based industrial control protocol reverse analysis method according to claim 1, characterized in that, In step 3, the length field is identified based on a random forest classifier. The specific steps are as follows: a. Cluster the message sequences by physical length, prune all message sequences in each cluster to the shortest sequence length, convert them into bit-level data, add length information labels to each cluster, and divide the training set and test set; b. Input the data in the training set into the random forest classifier for training, and use the test set to test the trained classifier. If the classification accuracy of the test set is greater than the set threshold, output the feature importance ranking of the bit offset of the random forest classification. c. Map the bit offsets of important features to bytes and remove duplicates, then output the length field to be selected; d. Set the length field filtering rules, that is, the decimal value corresponding to the field should be less than or equal to the length of the entire message sequence. The filtered field is the estimated length field.
6. The network traffic based industrial control protocol reverse analysis method according to claim 1, characterized in that, In step 3, the function code is located by analyzing the function click traffic collected according to specific rules. Specifically, after collecting the function click traffic, the message sequence is divided into dynamic and static fields. The unique value of each dynamic field remaining after the semantic information has been identified is counted. If the number of unique values of each field meets the aforementioned rules, the field is considered a candidate field for the function code, and the semantic information of the function code is added to it. If the identified unique values meet the aforementioned rules for multiple fields, the number of clicks is redefined and the function click traffic is collected again for multiple identifications.
7. The network traffic based industrial control protocol reverse analysis method according to claim 1, characterized in that, In step 3, the fields filtered out by the filtering rules during the length field extraction process are used as candidate fields for the function code. If a candidate field meets the characteristics of a function code, it is considered to be part of the function code, and semantic information of the function code is added to it. The formula for calculating the function code score of the candidate field is as follows: wherein is a candidate field, is a function code uniqueness constraint, is a function code entropy constraint, is a candidate field of unique values, is a total number of message sequences, is a candidate field of probability of occurrence; Selecting The selected field that meets the threshold condition is taken as the function code, and semantic information of the function code is added to the field.
8. The network traffic based industrial control protocol reverse analysis method according to claim 1, characterized in that, The semantic recognition order of the incremental sequence, address field, check field, length field, and function code is not limited, but the semantic recognition of the function code must be ensured after the recognition of the length field.