Unknown protocol reverse analysis and feature extraction method

Abstract

The invention discloses an unknown protocol reverse analysis and feature extraction method, which comprises the following steps of: extracting a message mixed with an unknown protocol data packet, and dividing the extracted message into tokens with additional attributes according to bytes by adopting an ABNF grammar core rule mapping table; carrying out similarity calculation on tokens in different messages to obtain token format distances of the tokens among the different messages; based on the token format distance, a text similarity measurement algorithm is adopted, and the message format distance between different messages is obtained; clustering different messages by adopting a DBSCAN (Density Based Spatial Clustering of Applications with Noise) algorithm based on the message format distance; and for each message type formed by clustering, extracting area content with area similarity value jump in the message, and marking the area content as a feature character of an unknown protocol. According to the method, a protocol reverse technology is utilized, and reverse analysis and feature extraction of an unknown protocol can be effectively realized.

Description

Technical Field

[0001] The present invention belongs to the field of cyberspace security technology, and in particular relates to a method for reverse analysis and feature extraction of unknown protocols. Background Art

[0002] Network protocols are a series of rules, standards and conventions established for network data exchange, and they play an important role in network communication. Protocol specifications are specific descriptions of network protocol syntax, semantics, synchronization and other information, and are an important basis for network security related research. For example, fuzz testing uses protocol specifications to guide the generation of test cases to improve effectiveness, and intrusion detection systems use protocol specifications to perform deep packet inspection to detect malicious traffic, etc. For standard protocols, their protocol descriptions can be obtained through public documents. However, in actual network environments, most software vendors and individuals are reluctant to disclose protocol details. For example, the network file sharing SMB (Server Message Block) protocol used by Microsoft, the TNS (Transparence Network Substrate) protocol for Oracle database access, and the protocols used by instant messaging software such as WeChat and QQ do not disclose protocol details, and a large number of malware use custom private protocols to achieve network communication. Technicians call network protocols that are not disclosed in various protocol specifications unknown protocols. The widespread use of various unknown protocols in the network has brought great obstacles to network security protection. Therefore, protocol reverse analysis technology with unknown protocols as the main analysis object came into being.

[0003] Protocol reverse analysis technology refers to the process of extracting protocol formats and state machines by monitoring and analyzing the network input and output, system behavior, and instruction execution process of protocol entities without relying on protocol specifications. Early protocol reverse work mainly relied on manual analysis, and its efficiency and accuracy depended on the professional knowledge level of the operator, and the cycle was long. For example, it took the Wine project 15 years to release the first stable version of the Windows application programming interface specification for Linux systems. As the scale of the network continues to expand and the network environment becomes increasingly complex, manual analysis can no longer meet actual needs. Automated protocol reverse analysis technology that combines efficiency and accuracy has gradually come into view and has been widely used. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a method for reverse analysis and feature extraction of unknown protocols.

[0005] The technical solution of the present invention is as follows: A method for reverse analysis and feature extraction of an unknown protocol, comprising: Extract the message of the mixed unknown protocol data packet, and then use the ABNF grammar core rule mapping table to divide the extracted message into tokens with additional attributes according to bytes; Use the Jaccard coefficient and the corresponding Jaccard distance to calculate the similarity of tokens in different messages, and obtain the token format distance between tokens in different messages; Based on the token format distance, a text similarity measurement algorithm is used to obtain the message format distance between different messages; Based on the message format distance, a DBSCAN algorithm based on silhouette coefficient and Dunn index is used to cluster different messages; For each message category formed by clustering, the NW algorithm is used to extract the content of the region where the region similarity value jumps in the message, and mark it as a characteristic character of the unknown protocol.

[0006] Furthermore, in the ABNF grammar core rule mapping table, For a mapping relationship with the attribute CR, its value set is 0x0D, and its description is carriage return; For a mapping relationship with the attribute LF, its value set is 0x0A, and its description is line feed; For a mapping relationship with the attribute HTAB, its value set is 0x09 and its description is horizontal label; For a mapping relationship whose attribute is SP, its value set is 0x20 and its description is a space; For a mapping relationship with the attribute DQUOTE, its value set is 0x22 and its description is double quotes; For the mapping relationship with attribute ALPHA, its value set is 0x41-0x5A and 0x61-0x7A, and its description is AZ and az; For a mapping relationship with attribute CHAR, its value set is 0x01-0x7F, and its description is any 7-bit US-ASCII character; For the mapping relationship CTL of the attribute, its value set is 0x00-0x1F, and its description is control character; For the mapping relationship DIGIT of the attribute, its value set is 0x30-0x39, and its description is 0-9; For the mapping relationship HEXDIG, the value set is 0x30-0x39 and 0x41-0x46, and the description is 0-9 and AF; For the mapping relationship OCTET of the attribute, its value set is 0x00-0xFF, and its description is 8-bit data; For the mapping relationship with attribute VCHAR, its value set is 0x21-0xFF, and its description is printable characters.

[0007] Furthermore, the specific steps of using the Jaccard coefficient and the Jaccard distance to calculate the similarity of tokens in different messages and obtain the token format distance of tokens between different messages include: for any two tokens formed by the division m and n , the attribute sets for constructing tokens are and In the formula, a 1, a 2 They are the first and second attributes in the attribute set of the token respectively; The Jaccard coefficient and the corresponding Jaccard distance are used to represent the similarity and difference of relative grammatical attributes. The calculation formula is as follows: ; In the formula, A 11 , A 10 , A 01 and A 00 are all subsets of the full set of attributes, the full set of attributes is a token m and n The union of the attribute sets of ; J ( m , n ) is the token m and n Jaccard distance; D J ( m , n ) is the token m and n The token format distance, TFD( m , n ).

[0008] Furthermore, the message format distance between two messages is calculated using the following formula: Set the initial message format distance between two messages to d ( i , 0)= i , d (0, j )= j ; Using the formula d ( i , j )=min{ d ( i , j -1)+1, d (i -1, j )+1, d ( i -1, j -1)+TFD( t i , t j )} to calculate the format distance matrix of two messages; where, t i and t j They are the first i and j tokens, TFD( t i , t j ) is the token format distance between two tokens; The final message format distance between the two messages is d ( x , y ),in x and y The token numbers of the two messages respectively.

[0009] Furthermore, the specific steps of clustering different messages based on the message format distance using the DBSCAN algorithm based on the silhouette coefficient and the Dunn index include: Based on the message format distance, the DBSCAN algorithm is used for clustering; the silhouette coefficient and Dunn index are used to evaluate the quality of the clustering results, where the silhouette coefficient a i The calculation formula is as follows: ; In the formula, C i For data samples in the dataset i A sample set of the message type. Representing a collection C i Except for the data samples i Data points other than those prepared for calculating the silhouette coefficient; d ( i , j ) is the format distance between two messages, and the silhouette coefficient a i represents the average intra-cluster distance; The Dunn Index b i The calculation formula is as follows: ; In the formula,C j For data samples in the dataset i A sample set of other message types that do not belong to the group. for C j The data points for calculating the Dunn Index are prepared in b i express i The average distance to the other clusters; Combining the silhouette coefficient and Dunn index, the following clustering quality evaluation formula is obtained: ; In the formula, s i is a clustering quality evaluation index combining silhouette coefficient and Dunn index. s i If the calculated result is greater than 0.5, the clustering result is judged to be correct.

[0010] Furthermore, for each message category formed by clustering, the specific steps of extracting the content of the region where the similarity value jumps in the message using the NW algorithm and marking it as a characteristic character of the unknown protocol include: The NW algorithm is used to calculate the regional similarity values ​​of messages of different message categories one by one, and the similarity values ​​of each region are counted. Assume s ( i , j ) indicates the first message i Bytes before the second message j The similarity value of bytes will be recorded once the following three situations occur: i and j The initial position of x 0, y 0): s ( i +1, j +1)> s ( i , j ) s ( i , j +1)> s ( i , j ) s ( i +1, j )> s ( i , j ); Then i andj Gradually increase the number of 1 until s ( i , j ) does not change, and record the result at this time i and j Location( x 1, y 1), and at the same time, the first message ( x 0, x 1) and the second message ( y 0, y 1) Recorded as the similarity value jump area; Then, the query starts for the next area where the similarity value changes, until all bytes of the two messages are analyzed; After the similarity value jump region is identified, the field set separated by each similarity value jump region is the division result of the unknown protocol, and the content of the similarity value jump region is the characteristic character of the unknown protocol.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a method for reverse analysis and feature extraction of unknown protocols. When processing data packets of unknown protocols, the method can complete reverse analysis and feature extraction of unknown protocols by calculating token format distance, message format distance, unsupervised message clustering, protocol feature extraction and other methods, and effectively solve the problem that current methods are difficult to process unknown protocols. The method of the present invention utilizes protocol reverse technology, and can effectively realize reverse analysis and feature extraction of unknown protocols, thereby realizing application classification and identification of unknown traffic in the network, and assisting in network management and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a method for reverse analysis and feature extraction of an unknown protocol provided in an embodiment. DETAILED DESCRIPTION

[0013] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0014] Example

[0015] The present invention provides a method for reverse analysis and feature extraction of an unknown protocol, comprising: Extract the message of the mixed unknown protocol data packet, and then use the ABNF (Augmented Backus-Naur Form) grammar core rule mapping table to divide the extracted message into tokens with additional attributes according to bytes; Use the Jaccard coefficient and the corresponding Jaccard distance to calculate the similarity of tokens in different messages, and obtain the token format distance (TFD) between tokens in different messages; Based on the token format distance, the Levenshtein Distance algorithm is used to obtain the message format distance (MFD) between different messages. Based on the message format distance, the DBSCAN algorithm based on the silhouette coefficient and Dunn index is used to cluster different messages; For each message category formed by clustering, the NW (Needleman-Wunsch) algorithm is used to extract the content of the area where the regional similarity value jumps in the message, and mark it as a characteristic character of the unknown protocol.

[0016] Furthermore, in the ABNF grammar core rule mapping table, For a mapping relationship with the attribute CR, its value set is 0x0D, and its description is carriage return; For a mapping relationship with the attribute LF, its value set is 0x0A, and its description is line feed; For a mapping relationship with the attribute HTAB, its value set is 0x09 and its description is horizontal label; For a mapping relationship whose attribute is SP, its value set is 0x20 and its description is a space; For a mapping relationship with the attribute DQUOTE, its value set is 0x22 and its description is double quotes; For the mapping relationship with attribute ALPHA, its value set is 0x41-0x5A and 0x61-0x7A, and its description is AZ and az; For a mapping relationship with attribute CHAR, its value set is 0x01-0x7F, and its description is any 7-bit US-ASCII character; For the mapping relationship CTL of the attribute, its value set is 0x00-0x1F, and its description is control character; For the mapping relationship DIGIT of the attribute, its value set is 0x30-0x39, and its description is 0-9; For the mapping relationship HEXDIG, the value set is 0x30-0x39 and 0x41-0x46, and the description is 0-9 and AF; For the mapping relationship OCTET of the attribute, its value set is 0x00-0xFF, and its description is 8-bit data; For the mapping relationship with attribute VCHAR, its value set is 0x21-0xFF, and its description is printable characters.

[0017] Furthermore, the specific steps of extracting the message of the mixed unknown protocol data packet and then using the ABNF grammar core rule mapping table to divide the extracted message into tokens with additional attributes by bytes include: Extract the message of the mixed unknown protocol data packet, divide the extracted message into tokens by bytes, and define the core rules of the protocol as basic units; the core rules of these message format basic units (minimum units) form a series of value sets when constituting combined grammatical units, representing different value types.

[0018] Furthermore, the Jaccard coefficient and the Jaccard distance are used to calculate the similarity of the tokens in different messages, and the specific steps of obtaining the token format distance of the tokens between different messages include: for any two tokens formed by the division m and n , the attribute sets for constructing tokens are and In the formula, a 1, a 2 They are the first and second attributes of the attribute set of the token respectively; then the ABNF grammar core rule mapping table is used to mark the elements in the attribute set; For example, the character "P" is marked as { ALPHA, CHAR, OCTET, VCHAR}, and the character " / " is marked as { CHAR,OCTET, VCHAR}. These marks are used to calculate the similarity of their relative grammatical properties.

[0019] In order to measure the similarity of message attribute sets, the present invention uses the Jaccard coefficient and the corresponding Jaccard distance to represent the similarity and difference of relative grammatical attributes. The calculation formula is as follows: ; In the formula, A 11 , A 10 , A 01 and A 00 are all subsets of the full set of attributes, among which,A 11 , A 10 , A 01 and A 00 They are the first subset, the second subset, the third subset and the fourth subset respectively; the full set of attributes is a token m and n The union of the attribute sets of ; J ( m , n ) is the token m and n Jaccard distance; D J ( m , n ) is the token m and n The token format distance, TFD( m , n ).

[0020] The token can be calculated by Jaccard coefficient and Jaccard distance m and n TokenFormat Distance (TFD), that is, D J ( m , n ), so as to measure their grammatical differences and use them to make up for the corresponding message format distance.

[0021] Furthermore, the present invention refers to the text similarity measurement LD (Levenshtein Distance) algorithm widely used in natural language processing to calculate the message format distance between two messages. The calculation formula is as follows: Set the initial message format distance between two messages to d ( i , 0)= i , d (0, j )= j ; Using the formula d ( i , j )=min{ d ( i , j -1)+1, d ( i -1, j )+1, d ( i -1,j -1)+TFD( t i , t j )} to calculate the format distance matrix of two messages; where, t i and t j They are the first i and j tokens, TFD( t i , t j ) is the token format distance between two tokens; The final message format distance between the two messages is d ( x , y ),in x and y The token numbers of the two messages respectively.

[0022] Furthermore, based on the message format distance, the specific steps of clustering different messages using the DBSCAN algorithm based on the silhouette coefficient and the Dunn index include: In view of the distribution characteristics of unknown message format distances, the density-based DBSCAN algorithm is used for clustering based on the message format distance. When dealing with the clustering problem of unknown protocol messages, since the shape of the clusters is uncertain and the distances of the elements are calculated in a high-dimensional space rather than the Euclidean space widely used in most clustering algorithms, two unsupervised clustering quality metrics, namely the silhouette coefficient and the Dunn index, are used to evaluate the quality of the clustering results. Among them, the silhouette coefficient a i The calculation formula is as follows: ; In the formula, C i For data samples in the dataset i A sample set of the message type. Representing a collection C i Except for the data samples i Data points other than those prepared for calculating the silhouette coefficient; d ( i , j ) is the format distance between two messages, and the silhouette coefficient a i represents the average intra-cluster distance, i The distribution of the clusters to which they belong. a i The smaller,i The better the distribution; Dunn Index b i The calculation formula is as follows: ; In the formula, C j For data samples in the dataset i A sample set of other message types that do not belong to the group. for C j The data points for calculating the Dunn Index are prepared in b i express i The average distance to the other clusters; b i The larger the i The better the separation from other clusters; Combining the silhouette coefficient and Dunn index, the following clustering quality evaluation formula is obtained: ; In the formula, s i is a clustering quality evaluation index combining silhouette coefficient and Dunn index. s i If the calculated result is greater than 0.5, the clustering result is judged to be correct.

[0023] Furthermore, for each message category formed by clustering, the NW (Needleman-Wunsch) algorithm is used to extract the content of the region where the similarity value jumps in the message, and the specific steps of marking it as a characteristic character of the unknown protocol include: The NW algorithm is used to calculate the regional similarity values ​​of messages of different message categories one by one. If messages belonging to the same message category after clustering are used to construct a category set, the messages in different sets are processed; And count the similarity values ​​of each region, assuming s ( i , j ) indicates the first message i Bytes before the second message j The similarity value of bytes will be recorded once the following three situations occur: i and j The initial position of x 0, y 0): s ( i +1, j +1)> s ( i ,j ) s ( i , j +1)> s ( i , j ) s ( i +1, j )> s ( i , j );

[0024] Then i and j Gradually increase the number of 1 until s ( i , j ) does not change, and record the result at this time i and j Location( x 1, y 1), and at the same time, the first message ( x 0, x 1) and the second message ( y 0, y 1) Recorded as the similarity value jump area; Then, the next similarity value jump area is queried until all bytes of the two messages are analyzed.

[0025] After the similarity value transition region is identified, the field set separated by each similarity value transition region is the format division result of the unknown protocol, and the content of the similarity value transition region is the characteristic character of the unknown protocol.

[0026] This example simulates an unknown protocol reverse analysis and feature extraction method of the present invention. The simulated operating environment is an Intel-Windows architecture PC, a PC with a Core eight-core CPU with a hardware main frequency of 2.5GHz or above, a memory ≥8GB, a hard disk of 500GB, and runs the Windows 10 operating system.

[0027] like Figure 1 As shown, it starts from step S101, reads the messages in the unknown protocol data packet one by one, and then goes to step S102; In step S102, the message is divided into tokens by bytes using the ABNF grammar core rule mapping table, and then the process goes to step S103; In step S103, the Jaccard coefficient and Jaccard distance are used to calculate the similarity and difference between the messages, and the Jaccard distance is recorded as the token format distance, and then the process goes to step S104; In step S104, the message format distance between the messages is calculated by using the LD algorithm in combination with the calculation result of the token format distance, and then the process goes to step S105; In step S105, the messages are clustered using the DBSCAN algorithm according to the silhouette coefficient and Dunn index between the messages, and then the process goes to step S106; In step S106, for each message category in the clustering result, the sampling NW algorithm calculates the similarity value between the messages, and mines the field range where the similarity jump occurs, and then goes to step S107; In step S107, the content of the region where the similarity value jumps is extracted and marked as a characteristic character of the unknown protocol, and then the process goes to step S108; In step S108, the analysis is completed and the program is exited.

[0028] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for reverse analysis and feature extraction of unknown protocols, characterized in that: include: Extract the message of the mixed unknown protocol data packet, and then use the ABNF grammar core rule mapping table to divide the extracted message into tokens with additional attributes according to bytes; Use the Jaccard coefficient and the corresponding Jaccard distance to calculate the similarity of tokens in different messages, and obtain the token format distance between tokens in different messages; Based on the token format distance, a text similarity measurement algorithm is used to obtain the message format distance between different messages; Based on the message format distance, a DBSCAN algorithm based on silhouette coefficient and Dunn index is used to cluster different messages; For each message category formed by clustering, the NW algorithm is used to extract the content of the region where the region similarity value jumps in the message, and mark it as a characteristic character of the unknown protocol.

2. The method for reverse analysis and feature extraction of unknown protocols according to claim 1, characterized in that: In the ABNF grammar core rule mapping table, For a mapping relationship with the attribute CR, its value set is 0x0D, and its description is carriage return; For a mapping relationship with the attribute LF, its value set is 0x0A, and its description is line feed; For a mapping relationship with the attribute HTAB, its value set is 0x09 and its description is horizontal label; For a mapping relationship whose attribute is SP, its value set is 0x20 and its description is a space; For a mapping relationship with the attribute DQUOTE, its value set is 0x22 and its description is double quotes; For the mapping relationship with attribute ALPHA, its value set is 0x41-0x5A and 0x61-0x7A, and its description is AZ and az; For a mapping relationship with the attribute CHAR, its value set is 0x01-0x7F, and its description is any 7-bit US-ASCII character; For a mapping relationship with the attribute CTL, its value set is 0x00-0x1F, and its description is a control character; For a mapping relationship with the attribute DIGIT, its value set is 0x30-0x39, and its description is 0-9; For the mapping relationship with the attribute HEXDIG, its value set is 0x30-0x39 and 0x41-0x46, and its description is 0-9 and AF; For a mapping relationship with the attribute OCTET, its value set is 0x00-0xFF, and its description is 8-bit data; For a mapping relationship with the attribute VCHAR, its value set is 0x21-0xFF, and its description is printable characters.

3. The method for reverse analysis and feature extraction of unknown protocols according to claim 1, characterized in that: The specific steps of using the Jaccard coefficient and the Jaccard distance to calculate the similarity of tokens in different messages and obtaining the token format distance of tokens between different messages include: For any two tokens formed by the partition m and n , the attribute sets for constructing tokens are and In the formula, a 1, a 2 They are the first and second attributes in the attribute set of the token respectively; The Jaccard coefficient and the corresponding Jaccard distance are used to represent the similarity and difference of relative grammatical attributes. The calculation formula is as follows: ； In the formula, A 11 , A 10 , A 01 and A 00 are all subsets of the full set of attributes, the full set of attributes is a token m and n The union of the attribute sets of ; J ( m , n ) is the token m and n Jaccard distance; D J ( m , n ) is the token m and n The token format distance, TFD( m , n ).

4. The method for reverse analysis and feature extraction of unknown protocols according to claim 3 is characterized in that: Calculate the message format distance between two messages. The calculation formula is as follows: Set the initial message format distance between two messages to d ( i , 0)= i , d (0, j )= j ; Using the formula d ( i , j )=min{ d ( i , j -1)+1, d ( i -1, j )+1, d ( i -1, j -1)+TFD( t i , t j )} to calculate the format distance matrix of two messages; where, t i and t j They are the first i and j tokens, TFD( t i , t j ) is the token format distance between two tokens; The final message format distance between the two messages is d ( x , y ),in x and y The token numbers of the two messages respectively.

5. The method for reverse analysis and feature extraction of unknown protocols according to claim 4, characterized in that: The specific steps of clustering different messages based on the message format distance using the DBSCAN algorithm based on the silhouette coefficient and the Dunn index include: Based on the message format distance, the DBSCAN algorithm is used for clustering; the silhouette coefficient and Dunn index are used to evaluate the quality of the clustering results, where the silhouette coefficient a i The calculation formula is as follows: ； In the formula, C i For data samples in the dataset i A sample set of the message type. Representing a collection C i Except for the data samples i Data points other than those prepared to calculate the silhouette coefficient; d ( i , j ) is the format distance between two messages, and the silhouette coefficient a i represents the average intra-cluster distance; The Dunn Index b i The calculation formula is as follows: ； In the formula, C j For data samples in the dataset i A sample set of other message types that do not belong to the group. for C j The data points for calculating the Dunn Index are prepared in b i express i The average distance to the other clusters; Combining the silhouette coefficient and Dunn index, the following clustering quality evaluation formula is obtained: ; In the formula, s i is a clustering quality evaluation index combining silhouette coefficient and Dunn index. s i If the calculated result is greater than 0.5, the clustering result is judged to be correct.

6. The method for reverse analysis and feature extraction of unknown protocols according to claim 5, characterized in that: The specific steps of extracting the content of the region where the similarity value jump occurs in the message by using the NW algorithm for each message category formed by clustering, and marking it as a characteristic character of the unknown protocol include: The NW algorithm is used to calculate the regional similarity values ​​of messages of different message categories one by one, and the similarity values ​​of each region are counted. Assume s ( i , j ) indicates the first message i Bytes before the second message j The similarity value of bytes will be recorded once the following three situations occur: i and j The initial position of x 0, y 0): s ( i +1, j +1)> s ( i , j ) s ( i , j +1)> s ( i , j ) s ( i +1, j )> s ( i , j )； Then i and j Gradually increase the number of 1 until s ( i , j ) does not change, and records the i and j Location( x 1, y 1), and at the same time, the first message ( x 0, x 1) and the second message ( y 0, y 1) Recorded as the similarity value jump area; Then, the query starts for the next area where the similarity value changes, until all bytes of the two messages are analyzed; After the similarity value jump region is identified, the field set separated by each similarity value jump region is the division result of the unknown protocol, and the content of the similarity value jump region is the characteristic character of the unknown protocol.

Images