Data protocol parsing method and device based on hierarchical sequence structure reasoning

Abstract

The present invention discloses a data protocol parsing method based on hierarchical sequence structure reasoning, comprising: importing the traffic of a network port into a parsing device through traffic mirroring, performing session diversion on the traffic, and identifying unknown protocol traffic; performing message assembly on the unknown protocol traffic to form a message unit; performing field-level sequence structure reasoning operations based on the message unit, extracting all possible values of a fixed-value fixed-length field, and processing the possible values of the fixed-value fixed-length field to obtain a fixed-value fixed-length field and an enumerated indefinite-length field; performing message-level sequence structure reasoning operations on the message unit according to the extracted fixed-value fixed-length field and the enumerated indefinite-length field to obtain a message format; performing semantic reasoning based on the obtained message format to obtain a semantic type; performing session-level sequence structure reasoning operations based on the obtained message format to obtain a protocol state machine and a session stream format for each session. The method can perform comprehensive parsing of data.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a data protocol parsing method and device based on hierarchical sequence structure reasoning. Background Art

[0002] With the construction of electric power data communication networks, the access of massive heterogeneous terminals and the explosive growth of data pose significant challenges to terminal access security and information security within these networks. The existence of various proprietary communication protocols between a large number of heterogeneous devices in current electric power data communication networks, coupled with the proliferation of traffic applications, has led to a diversity of private network protocols, making the regulation of grid traffic increasingly difficult. Criminals exploit the hidden nature of unknown traffic to disrupt electric power data communication networks, causing irreparable damage. To ensure the stable operation and security of the power system, accurate and comprehensive analysis and monitoring of the power data communication network is necessary. Reverse protocol analysis, as an advanced power system analysis method, can help achieve intelligent monitoring and management of the power system.

[0003] Although reverse protocol analysis methods for various protocols have emerged, there is currently no fully standardized method for extracting protocol specifications, and each method has certain limitations. Existing methods are mostly manual, time-consuming, and error-prone, making them difficult to apply in complex power data communication network environments. Although many automated protocol reverse engineering methods have been applied to address this issue, the message formats they extract are overly specific or general, meaning they cannot extract the precise protocol specifications by extracting too many message formats, but can only extract a portion of the protocol specifications. For example, patent document CN112039196A provides a method for parsing private protocols in power monitoring systems based on protocol reverse engineering. The method first imports the private protocol traffic into the test system for analysis through bypass traffic mirroring. The protocol reverse module uses a local sequence alignment algorithm and an unweighted pairwise average algorithm to perform initial clustering on the protocol messages of the private protocol. The variable and invariant fields of the protocol message are separated through a multiple sequence alignment algorithm to obtain the structural information of the protocol message. The semantic fields in the protocol message are searched through a search algorithm to obtain the protocol format information. The structural and semantic information of the protocol message are used to obtain the state transition sequence in each session, construct a state prefix tree, merge redundant states, and then optimize the state prefix tree to obtain the final minimum deterministic protocol state machine. This method also extracts protocol specifications through message format, which has certain limitations. Summary of the Invention

[0004] The present invention provides a data protocol parsing method and device based on hierarchical sequence structure reasoning, which can realize comprehensive parsing of the syntax, semantics and behavior of unknown protocols.

[0005] A data protocol parsing method based on hierarchical sequence structure reasoning, comprising:

[0006] Importing the traffic of the network port into the analysis device through traffic mirroring, performing session diversion on the traffic, and identifying unknown protocol traffic;

[0007] Performing message assembly on the unknown protocol traffic to form a message unit;

[0008] Performing a field-level sequence structure inference operation based on the message unit, extracting all possible values of the fixed-value fixed-length field, and processing the possible values of the fixed-value fixed-length field to obtain the fixed-value fixed-length field and the enumerated variable-length field;

[0009] Performing a message-level sequence structure inference operation on the message unit according to the extracted fixed-value fixed-length field and the enumerated variable-length field to obtain a message format;

[0010] Perform semantic reasoning based on the obtained message format to obtain the semantic type;

[0011] Based on the obtained message format, a session-level sequence structure reasoning operation is performed to obtain the protocol state machine and session flow format of each session.

[0012] Furthermore, the unknown protocol traffic is subjected to message assembly to form a message unit, including:

[0013] Determining whether the location protocol traffic type is UDP traffic or TCP traffic;

[0014] For the UDP traffic, each data packet is considered as a message unit; for the TCP traffic, a set of consecutive data packets with the same direction is considered as a message unit.

[0015] Furthermore, the field formats in the message unit are defined as fixed-value fixed-length fields, enumerated variable-length fields, dynamic fixed-length fields, and fully dynamic fields;

[0016] Performing field-level sequence structure inference operations based on the message unit to extract all possible values of the fixed-value and fixed-length fields, including:

[0017] S31, scanning the data sequence of the message unit in units of two bytes to generate a first sequence;

[0018] S32, counting the frequency of occurrence of each first sequence to obtain the first sequence support, deleting the first sequence whose first sequence support is lower than the minimum preset support threshold, and obtaining a first frequent sequence;

[0019] S33. Perform pairwise merging based on the first frequent sequences to generate a first candidate sequence;

[0020] S34. Count the frequency of occurrence of each first candidate sequence to obtain the support of the first candidate sequence, delete the first candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new first sequence;

[0021] S35. Loop through steps S32 to S34 until a longer first sequence cannot be generated. Perform frequency calculation on the first sequence finally obtained, and output the first sequence whose support is higher than a minimum preset support threshold as the final target sequence. The final target sequence is a possible value of a fixed-value, fixed-length field.

[0022] Furthermore, possible values of the fixed-value fixed-length field are processed to obtain a fixed-value fixed-length field and an enumerated variable-length field, including:

[0023] Recording a position offset of a possible value of a current fixed-value fixed-length field relative to a message unit header, and calculating a position variance of the possible value of the fixed-value fixed-length field based on the position offset;

[0024] Calculating the support of the possible values of the fixed-value fixed-length field, and determining the fields corresponding to the possible values of the fixed-value fixed-length field whose support is less than a preset support threshold and whose position variance is lower than a preset variance threshold as convertible enumerated indefinite-length fields;

[0025] Determine the remaining fields after excluding the convertible enumerated indefinite-length fields from the fields corresponding to the possible values of the fixed-value fixed-length fields as fixed-value fixed-length fields;

[0026] The message unit is truncated according to the minimum offset and maximum depth of the convertible enumeration indefinite length field in the message unit, and the support of the truncated sequence is calculated. The sequence with the highest support is stored in an array. This step is repeated until the message unit no longer generates a new sequence, and the sequence in the array is used as the enumeration indefinite length field.

[0027] Furthermore, based on the extracted fixed-value fixed-length fields and enumerated variable-length fields, a message-level sequence structure inference operation is performed on the message unit to obtain a message format, including:

[0028] S41, scanning the fixed-value fixed-length field and enumerating the values of the variable-length field with one field as a unit to obtain a second sequence;

[0029] S42, counting the frequency of occurrence of each second sequence to obtain the second sequence support, deleting the second sequence whose second sequence support is lower than the minimum preset support threshold, and obtaining a second frequent sequence;

[0030] S43. Perform pairwise combinations based on the second frequent sequences to generate a second candidate sequence;

[0031] S44. Count the frequency of occurrence of each second candidate sequence to obtain the support of the second candidate sequence, delete the second candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new second sequence;

[0032] S45. Loop through steps S42 to S44 until a longer second sequence cannot be generated. Perform frequency calculation on the second sequence finally obtained, and output a second sequence whose second sequence support is higher than a minimum preset support threshold as an initial message format. The initial message format includes a fixed-length field and an enumerated variable-length field.

[0033] S46. Extract the additional field between the fixed-value fixed-length field and the enumerated indefinite-length field and determine its field format, and use the fixed-value fixed-length field, the enumerated indefinite-length field, and the additional field between them as the message format;

[0034] The field format of the additional field includes an enumerated indefinite length field, a dynamic fixed length field, and a completely dynamic field.

[0035] Furthermore, extracting the additional field between the fixed-value fixed-length field and the enumerated variable-length field in the initial message format and determining its field format includes:

[0036] Using the length of the fixed-length field and the enumerated variable-length field as the screening condition, the message sequence corresponding to the initial message format is collected from the original message unit, and the corresponding data set is found according to the position and length of the additional field in the initial message format;

[0037] Calculate the maximum value and variance of the data set length. If the variance of the data set is greater than a preset variance threshold, determine the data set as a fully dynamic field. If the variance of the data set is less than or equal to the preset variance threshold, determine the data set as a non-fully dynamic field.

[0038] If the variance of the non-completely dynamic field is less than the preset length threshold, the non-completely dynamic field is determined to be an enumerated indefinite length field; if the variance of the non-completely dynamic field is greater than or equal to the preset length threshold, the non-completely dynamic field is determined to be a dynamic fixed-length field.

[0039] Furthermore, semantic reasoning is performed based on the obtained message format to obtain the semantic type, including:

[0040] Retrieving data corresponding to the message format from the original message unit according to the position and length of the fields in the message format;

[0041] If the data is a dynamic field, the information theory index is calculated based on the data, and the field whose information theory index is greater than the preset index value is determined as S Mtype Candidate fields for

[0042] If the data is a dynamic field, the Pearson correlation coefficient is calculated based on the data. When the Pearson correlation coefficient is greater than the preset coefficient value, the corresponding field is determined to be S Mlen Candidate fields for

[0043] If the data is a dynamic field and the data is IP data, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the first preset correlation threshold is determined as S Host Candidate fields for

[0044] If the data is a dynamic field and the data is a session, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the second preset correlation threshold is determined as S Sess Candidate fields for

[0045] Calculate whether the field corresponding to the data is the same as the entropy value of the user's IP address. If they are the same, determine that the field corresponding to the data is S Trans Candidate fields for

[0046] Calculate the difference between the field values in two consecutive message units, and determine that the two consecutive message units are S when the difference is less than the preset maximum difference value. Accum Candidate fields.

[0047] Furthermore, based on the obtained message format, a session-level sequence structure inference operation is performed to obtain the protocol state machine of each session, including:

[0048] Matching the message format with the message unit in terms of field position and content, and if the message and state in the message unit correspond to the message format, determining that the message is in the corresponding state;

[0049] The transition relationship between states is identified based on the order of messages in the message unit and the changes in state. If two messages that appear in sequence correspond to two different states, it is determined that there is a transition path between the two states that appear in sequence. The number of occurrences of each transition path is recorded, and the transition probability of each state is calculated based on the statistical number of transitions to generate a protocol state machine.

[0050] Furthermore, based on the obtained message format, a session-level sequence structure inference operation is performed to obtain the session flow format of each session, including:

[0051] S61, scanning the message units with one message format as one unit to generate a third sequence;

[0052] S62: Count the frequency of occurrence of each third sequence to obtain the third sequence support, delete the third sequence whose third sequence support is lower than the minimum preset support threshold, and obtain the third frequent sequence;

[0053] S63: performing pairwise merging based on the third frequent sequences to generate a third candidate sequence;

[0054] S64: Count the frequency of occurrence of each third candidate sequence to obtain the support of the third candidate sequence, delete the third candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new third sequence;

[0055] S65 , looping through steps S62 to S64 until a longer third sequence cannot be generated, performing frequency calculation on the third sequence finally obtained, and outputting a third sequence whose support is higher than a minimum preset support threshold as a conversation stream format.

[0056] A data protocol parsing device based on hierarchical sequence structure reasoning, comprising:

[0057] A traffic diversion module, configured to import traffic from a network port through traffic mirroring, perform session diversion on the traffic, and identify traffic with unknown protocols;

[0058] An assembling module, configured to assemble messages of the unknown protocol traffic into message units;

[0059] A field reasoning module performs field-level sequence structure reasoning operations based on the message unit, extracts all possible values of the fixed-value and fixed-length fields, and processes the possible values of the fixed-value and fixed-length fields to obtain fixed-value and fixed-length fields and enumerate variable-length fields;

[0060] A message inference module, configured to perform a message-level sequence structure inference operation on the message unit based on the extracted fixed-value fixed-length fields and the enumerated variable-length fields to obtain a message format;

[0061] The semantic reasoning module performs semantic reasoning based on the obtained message format to obtain the semantic type;

[0062] The session reasoning module performs session-level sequence structure reasoning operations based on the obtained message format to obtain the protocol state machine and session flow format of each session.

[0063] Furthermore, the assembling module assembles messages of the unknown protocol traffic to form message units, including:

[0064] Determining whether the location protocol traffic type is UDP traffic or TCP traffic;

[0065] For the UDP traffic, each data packet is considered as a message unit; for the TCP traffic, a set of consecutive data packets with the same direction is considered as a message unit.

[0066] Furthermore, the field formats in the message unit are defined as fixed-value fixed-length fields, enumerated variable-length fields, dynamic fixed-length fields, and fully dynamic fields;

[0067] The field reasoning module performs field-level sequence structure reasoning operations based on the message unit to extract all possible values of the fixed-value and fixed-length fields, including:

[0068] S31, scanning the data sequence of the message unit in units of two bytes to generate a first sequence;

[0069] S32, counting the frequency of occurrence of each first sequence to obtain the first sequence support, deleting the first sequence whose first sequence support is lower than the minimum preset support threshold, and obtaining a first frequent sequence;

[0070] S33. Perform pairwise merging based on the first frequent sequences to generate a first candidate sequence;

[0071] S34. Count the frequency of occurrence of each first candidate sequence to obtain the support of the first candidate sequence, delete the first candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new first sequence;

[0072] S35. Loop through steps S32 to S34 until a longer first sequence cannot be generated. Perform frequency calculation on the first sequence finally obtained, and output the first sequence whose support is higher than a minimum preset support threshold as the final target sequence. The final target sequence is a possible value of a fixed-value, fixed-length field.

[0073] Furthermore, the field inference module processes possible values of the fixed-value fixed-length field to obtain the fixed-value fixed-length field and the enumerated variable-length field, including:

[0074] Recording a position offset of a possible value of a current fixed-value fixed-length field relative to a message unit header, and calculating a position variance of the possible value of the fixed-value fixed-length field based on the position offset;

[0075] Calculating the support of the possible values of the fixed-value fixed-length field, and determining the fields corresponding to the possible values of the fixed-value fixed-length field whose support is less than a preset support threshold and whose position variance is lower than a preset variance threshold as convertible enumerated indefinite-length fields;

[0076] Determine the remaining fields after excluding the convertible enumerated indefinite-length fields from the fields corresponding to the possible values of the fixed-value fixed-length fields as fixed-value fixed-length fields;

[0077] The message unit is truncated according to the minimum offset and maximum depth of the convertible enumeration indefinite length field in the message unit, and the support of the truncated sequence is calculated. The sequence with the highest support is stored in an array. This step is repeated until the message unit no longer generates a new sequence, and the sequence in the array is used as the enumeration indefinite length field.

[0078] Furthermore, the message inference module performs a message-level sequence structure inference operation on the message unit based on the extracted fixed-value fixed-length fields and the enumerated variable-length fields to obtain a message format, including:

[0079] S41, scanning the fixed-value fixed-length field and enumerating the values of the variable-length field with one field as a unit to obtain a second sequence;

[0080] S42, counting the frequency of occurrence of each second sequence to obtain the second sequence support, deleting the second sequence whose second sequence support is lower than the minimum preset support threshold, and obtaining a second frequent sequence;

[0081] S43. Perform pairwise combinations based on the second frequent sequences to generate a second candidate sequence;

[0082] S44. Count the frequency of occurrence of each second candidate sequence to obtain the support of the second candidate sequence, delete the second candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new second sequence;

[0083] S45. Loop through steps S42 to S44 until a longer second sequence cannot be generated. Perform frequency calculation on the second sequence finally obtained, and output a second sequence whose second sequence support is higher than a minimum preset support threshold as an initial message format. The initial message format includes a fixed-length field and an enumerated variable-length field.

[0084] S46. Extract the additional field between the fixed-value fixed-length field and the enumerated indefinite-length field and determine its field format, and use the fixed-value fixed-length field, the enumerated indefinite-length field, and the additional field between them as the message format;

[0085] The field format of the additional field includes an enumerated indefinite length field, a dynamic fixed length field, and a completely dynamic field.

[0086] Furthermore, the message inference module extracts the additional fields between the fixed-value fixed-length fields and the enumerated variable-length fields in the initial message format and determines the field formats thereof, including:

[0087] Using the length of the fixed-length field and the enumerated variable-length field as the screening condition, the message sequence corresponding to the initial message format is collected from the original message unit, and the corresponding data set is found according to the position and length of the additional field in the initial message format;

[0088] Calculate the maximum value and variance of the data set length. If the variance of the data set is greater than a preset variance threshold, determine the data set as a fully dynamic field. If the variance of the data set is less than or equal to the preset variance threshold, determine the data set as a non-fully dynamic field.

[0089] If the variance of the non-completely dynamic field is less than the preset length threshold, the non-completely dynamic field is determined to be an enumerated indefinite length field; if the variance of the non-completely dynamic field is greater than or equal to the preset length threshold, the non-completely dynamic field is determined to be a dynamic fixed-length field.

[0090] Furthermore, the semantic reasoning module performs semantic reasoning based on the obtained message format to obtain a semantic type, including:

[0091] Retrieving data corresponding to the message format from the original message unit according to the position and length of the fields in the message format;

[0092] If the data is a dynamic field, the information theory index is calculated based on the data, and the field whose information theory index is greater than the preset index value is determined as S Mtype Candidate fields for

[0093] If the data is a dynamic field, the Pearson correlation coefficient is calculated based on the data. When the Pearson correlation coefficient is greater than the preset coefficient value, the corresponding field is determined to be S Mlen Candidate fields for

[0094] If the data is a dynamic field and the data is IP data, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the first preset correlation threshold is determined as S Host Candidate fields for

[0095] If the data is a dynamic field and the data is a session, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the second preset correlation threshold is determined as S Sess Candidate fields for

[0096] Calculate whether the field corresponding to the data is the same as the entropy value of the user's IP address. If they are the same, determine that the field corresponding to the data is S Trans Candidate fields for

[0097] Calculate the difference between the field values in two consecutive message units, and determine that the two consecutive message units are S when the difference is less than the preset maximum difference value. Accum Candidate fields.

[0098] Furthermore, the session reasoning module performs session-level sequence structure reasoning operations based on the obtained message format to obtain a protocol state machine for each session, including:

[0099] Matching the message format with the message unit in terms of field position and content, and if the message and state in the message unit correspond to the message format, determining that the message is in the corresponding state;

[0100] The transition relationship between states is identified based on the order of messages in the message unit and the changes in state. If two messages that appear in sequence correspond to two different states, it is determined that there is a transition path between the two states that appear in sequence. The number of occurrences of each transition path is recorded, and the transition probability of each state is calculated based on the statistical number of transitions to generate a protocol state machine.

[0101] Furthermore, the conversation reasoning module performs conversation-level sequence structure reasoning operations based on the obtained message format to obtain the conversation flow format of each conversation, including:

[0102] S61, scanning the message units with one message format as one unit to generate a third sequence;

[0103] S62: Count the frequency of occurrence of each third sequence to obtain the third sequence support, delete the third sequence whose third sequence support is lower than the minimum preset support threshold, and obtain the third frequent sequence;

[0104] S63: performing pairwise merging based on the third frequent sequences to generate a third candidate sequence;

[0105] S64: Count the frequency of occurrence of each third candidate sequence to obtain the support of the third candidate sequence, delete the third candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new third sequence;

[0106] S65 , looping through steps S62 to S64 until a longer third sequence cannot be generated, performing frequency calculation on the third sequence finally obtained, and outputting a third sequence whose support is higher than a minimum preset support threshold as a conversation stream format.

[0107] The data protocol parsing method and device based on hierarchical sequence structure reasoning provided by the present invention have at least the following beneficial effects:

[0108] The field format, message format and session format of the unknown protocol are extracted in layers, and on this basis, the unknown protocol is parsed from the syntax, semantics and behavior of the unknown protocol. This method can achieve a comprehensive analysis of the syntax, semantics and behavior of the unknown protocol. This method has good applicability and can be applied to various power data communication flow supervision systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 This is a flow chart of an embodiment of the data protocol parsing method based on hierarchical sequence structure reasoning provided by the present invention.

[0110] Figure 2 A schematic diagram of an embodiment of the data protocol parsing method based on hierarchical sequence structure reasoning provided by the present invention.

[0111] Figure 3 This is a flow chart of an embodiment of field-level sequence structure reasoning in the data protocol parsing method based on hierarchical sequence structure reasoning provided by the present invention.

[0112] Figure 4 This is a flow chart of an embodiment of message-level sequence structure reasoning in the data protocol parsing method based on hierarchical sequence structure reasoning provided by the present invention.

[0113] Figure 5 A flow chart of an embodiment of extracting additional fields in the data protocol parsing method based on hierarchical sequence structure reasoning provided by the present invention.

[0114] Figure 6 This is a flow chart of an embodiment of semantic reasoning in the data protocol parsing method based on hierarchical sequence structure reasoning provided by the present invention.

[0115] Figure 7 This is a flow chart of an embodiment of session-level sequence structure reasoning in the data protocol parsing method based on hierarchical sequence structure reasoning provided by the present invention.

[0116] Figure 8 A structural diagram of a data protocol parsing device based on hierarchical sequence structure reasoning provided by the present invention. DETAILED DESCRIPTION

[0117] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0118] refer to Figure 1 and Figure 2 In some embodiments, a data protocol parsing method based on hierarchical sequence structure reasoning is provided, comprising:

[0119] S1. Importing the traffic of the network port into the parsing device through traffic mirroring, performing session diversion on the traffic, and identifying unknown protocol traffic;

[0120] S2. assembling the unknown protocol traffic into a message unit;

[0121] S3. Performing a field-level sequence structure inference operation based on the message unit to extract all possible values of the fixed-value and fixed-length fields, and processing the possible values of the fixed-value and fixed-length fields to obtain fixed-value and fixed-length fields and enumerated variable-length fields;

[0122] S4. Performing a message-level sequence structure inference operation on the message unit based on the extracted fixed-value fixed-length fields and the enumerated variable-length fields to obtain a message format;

[0123] S5. Perform semantic reasoning based on the obtained message format to obtain a semantic type;

[0124] S6. Perform session-level sequence structure reasoning based on the obtained message format to obtain the protocol state machine and session flow format of each session.

[0125] Specifically, in step S1, for a specific power data communication scenario, the traffic at the network entrance and exit is copied to a separate server through traffic mirroring, and then all the traffic is session-divided and protocol-divided to filter out traffic of unknown protocol types.

[0126] Traffic splitting is to divide the data packets with the same source IP, destination IP, source port, destination port and application layer protocol in the original traffic into one flow.

[0127] Furthermore, in step S2, the unknown protocol traffic is subjected to message assembly to form a message unit, including:

[0128] S21. Determine whether the unknown protocol traffic type is UDP traffic or TCP traffic;

[0129] S22. For the UDP traffic, each data packet is regarded as a message unit; for the TCP traffic, a set of consecutive data packets with the same direction is regarded as a message unit.

[0130] Specifically, in step S2, the field formats in the message unit are defined as fixed-value fixed-length fields, enumerated variable-length fields, dynamic fixed-length fields, and fully dynamic fields.

[0131] Among them, the fixed-value fixed-length field (FVF(v) field) represents a field with a static value and a fixed length; the enumerated variable-length field (FNVF(v) field) represents that its value exists in a finite set and can be obtained by enumeration in the set, and its length is not fixed; the values of the dynamic fixed-length field DVF and the fully dynamic field FDV are both dynamic, among which the length of the fully dynamic field FDV field varies too much and is unpredictable, while the length of the dynamic fixed-length field DVF is fixed. The fixed-value fixed-length field is a common substring in the message unit that meets a specific frequency; it is a continuous character, hexadecimal value, or a combination thereof in the message unit.

[0132] Further, refer to Figure 3 In step S3, a field-level sequence structure inference operation is performed based on the message unit to extract all possible values of the fixed-value and fixed-length fields, including:

[0133] S31, scanning the data sequence of the message unit in units of two bytes to generate a first sequence;

[0134] S32, counting the frequency of occurrence of each first sequence to obtain the first sequence support, deleting the first sequence whose first sequence support is lower than the minimum preset support threshold, and obtaining a first frequent sequence;

[0135] S33. Perform pairwise merging based on the first frequent sequences to generate a first candidate sequence;

[0136] S34. Count the frequency of occurrence of each first candidate sequence to obtain the support of the first candidate sequence, delete the first candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new first sequence;

[0137] S35. Loop through steps S32 to S34 until a longer first sequence cannot be generated. Perform frequency calculation on the first sequence finally obtained, and output the first sequence whose support is higher than a minimum preset support threshold as the final target sequence. The final target sequence is a possible value of a fixed-value, fixed-length field.

[0138] Furthermore, in step S3, possible values of the fixed-value fixed-length field are processed to obtain a fixed-value fixed-length field and an enumerated variable-length field, including:

[0139] S3A, recording a position offset of a possible value of the current fixed-value fixed-length field relative to the message unit header, and calculating a position variance of the possible value of the fixed-value fixed-length field based on the position offset;

[0140] S3B. Calculate the support of the possible values of the fixed-value and fixed-length fields, and determine the fields corresponding to the possible values of the fixed-value and fixed-length fields whose support is less than a preset support threshold and whose position variance is lower than a preset variance threshold as convertible enumerated indefinite-length fields;

[0141] S3C. Determine the remaining fields after excluding the convertible enumerated indefinite-length fields from the fields corresponding to the possible values of the fixed-value fixed-length fields as fixed-value fixed-length fields;

[0142] S3D. Truncate the message unit according to the minimum offset and maximum depth of the convertible enumeration indefinite length field in the message unit, calculate the support of the truncated sequence, store the sequence with the highest support in an array, repeat this step until the message unit no longer generates a new sequence, and use the sequence in the array as the enumeration indefinite length field.

[0143] In step S3A, the position offset of the possible value of the current fixed-value fixed-length field relative to the message unit header, that is, the byte difference between the possible value of the fixed-value fixed-length field in the message unit and the message unit header, is calculated based on all the position offsets, and then the position variance of each possible value is further calculated.

[0144] In step S3D, the minimum offset is the first position of the convertible enumerated indefinite-length field in the message unit, and the maximum depth is the last position that the convertible enumerated indefinite-length field can be greater than in the message unit. Assume that the minimum offset of the GET field in the two message sequences "GETPOSTPUT" and "POSTGETPUT" is 0, and the maximum depth is 7.

[0145] Further, refer to Figure 4 In step S4, based on the extracted fixed-value fixed-length fields and the enumerated variable-length fields, a message-level sequence structure inference operation is performed on the message unit to obtain a message format, including:

[0146] S41, scanning the fixed-value fixed-length field and enumerating the values of the variable-length field with one field as a unit to obtain a second sequence;

[0147] S42, counting the frequency of occurrence of each second sequence to obtain the second sequence support, deleting the second sequence whose second sequence support is lower than the minimum preset support threshold, and obtaining a second frequent sequence;

[0148] S43. Perform pairwise combinations based on the second frequent sequences to generate a second candidate sequence;

[0149] S44. Count the frequency of occurrence of each second candidate sequence to obtain the support of the second candidate sequence, delete the second candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new second sequence;

[0150] S45. Loop through steps S42 to S44 until a longer second sequence cannot be generated. Perform frequency calculation on the second sequence finally obtained, and output a second sequence whose second sequence support is higher than a minimum preset support threshold as an initial message format. The initial message format includes a fixed-length field and an enumerated variable-length field.

[0151] S46. Extract the additional field between the fixed-value fixed-length field and the enumerated variable-length field in the initial message format and determine its field format, and use the fixed-value fixed-length field, the enumerated variable-length field, and the additional field between them as the message format;

[0152] The field format of the additional field includes an enumerated indefinite length field, a dynamic fixed length field, and a completely dynamic field.

[0153] Specifically, refer to Figure 5 In step S46, the additional fields between the fixed-value fixed-length field and the enumerated variable-length field in the initial message format are extracted and their field formats are determined, including:

[0154] S46a, using the lengths of the fixed-length fields and the enumerated variable-length fields as screening conditions, collecting message sequences corresponding to the initial message format from the original message units, and searching for the corresponding data set according to the position and length of the additional fields in the initial message format;

[0155] S46b. Calculate the maximum value and variance of the data set length. If the variance of the data set is greater than a preset variance threshold, determine the data set as a fully dynamic field. If the variance of the data set is less than or equal to the preset variance threshold, determine the data set as a non-fully dynamic field.

[0156] S46c. If the variance of the non-completely dynamic field is less than the preset length threshold, the non-completely dynamic field is determined to be an enumerated indefinite length field; if the variance of the non-completely dynamic field is greater than or equal to the preset length threshold, the non-completely dynamic field is determined to be a dynamic fixed-length field.

[0157] Specifically, the message format is extracted using a message-level sequence structure inference algorithm, which is performed under the condition that the FVF(v) field and the FNVF(v) field contain only one value. The final message format is, for example: {FVF(v), FDV, DVF, FNVF(v), FDV,…FNVF(v)}, where FVF(v) and FNVF(v) have only one fixed value in the corresponding position.

[0158] Further, refer to Figure 6 In step S5, semantic reasoning is performed based on the obtained message format to obtain the semantic type, including:

[0159] S51. Retrieve data corresponding to the message format from the original message unit according to the position and length of the fields in the message format;

[0160] S52: If the data is a dynamic field, calculate the information theory index based on the data, and determine the field whose information theory index is greater than the preset index value as S Mtype Candidate fields for

[0161] Specifically, S Mtype The field is MSG-Type, whose value is neither random nor constant, and the field has an opposite field that can be matched. Specifically, the message sent from the client (local machine) to the server is defined as a request message, and the message sent from the server to the client is defined as a response message. If a field is in the request message, then its opposite field appears in the response message, and vice versa. Calculate the information theory index of the data, and take the preset number of fields whose information theory index is greater than the preset value as S Mtype The candidate fields of ; the information theory index is calculated according to the following formula:

[0162] I(q;r)=H(q)+H(r)-H(q,r); (1)

[0163] H(x)=-∑ i p i log2p i ; (2)

[0164] Where q represents the value of the field to be determined, r represents the value of the opposite field corresponding to q, I(q;r) represents the mutual information between field q and field r, H(q) represents the entropy of field q, H(r) represents the entropy of field r, H(q,r) represents the joint entropy of field q and field r, H(x) represents the entropy of field x, and p i It represents the probability of field x appearing, and the information theory indicator is I(q;r) / H(q).

[0165] S53: If the data is a dynamic field, calculate the Pearson correlation coefficient based on the data. When the Pearson correlation coefficient is greater than the preset coefficient value, determine that the corresponding field is S54. Mlen Candidate fields for

[0166] Specifically, S Mlen The MSG-Len field, whose value indicates the message length, is used to determine whether the message length field is linearly correlated with the actual physical message size. This can be calculated using the Pearson correlation coefficient. When the Pearson correlation coefficient is greater than 0.6, the corresponding field is considered a candidate field.

[0167] S54: If the data is a dynamic field and the data is IP data, calculate the correlation between the field format and the user IP address classification, and determine the field whose classification correlation is greater than the first preset correlation threshold as S. Host Candidate fields for

[0168] Specifically, S Host The Host-ID field has a value specific to the source address, such as an email address, user ID, or host IP address. Based on the heuristic assumption, all messages sent by the same host carry the same S Host , that is, for a given source IP, its information is likely to have the same S Host , then S Host It should be closely related to the user's IP address. Based on this assumption, the classification correlation R(x,y) between n field formats x and user IP address y is calculated by the following formula:

[0169] R(x,y)=I(x,y) / H(x,y); (2)

[0170] H(x,y)=-∑ i ∑ j p(q i , r j )log2p(q i ,r j ); (3)

[0171] Among them, I(x,y) represents the conditional mutual information between the field format x to be confirmed and the user IP address y, H(x,y) represents the joint entropy of the field format x to be confirmed and the user IP address y, and p(q i ,r j ) represents the joint probability distribution of field format x and user IP address y. When the above ratio is greater than 0.9, the corresponding field is S Host Candidate fields.

[0172] S55: If the data is a dynamic field and the data is a session, calculate the correlation between the field format and the user IP address classification, and determine the field whose classification correlation is greater than the second preset correlation threshold as S. Sess Candidate fields for

[0173] Specifically, S Sess The Session-ID field is a dynamic field whose value is specific to the session and is calculated in the same way as the Session-ID field. Host If a field in S Host Calculation and S Sess When the calculation is greater than the threshold, compare S Host The calculation results and S Sess The calculation results of , the larger result is given priority.

[0174] S56, calculate whether the field corresponding to the data is the same as the entropy value of the user's IP address, and if they are the same, determine that the field corresponding to the data is S Trans Candidate fields for

[0175] Specifically, S Trans The field, Trans-ID, is a dynamic field whose value is specific to a transaction. Fields with high entropy are possible candidates for the transaction identification field. On this basis, all consecutive request / response information is paired and measured using H(x). The entropy value of the field corresponding to the data and the user's IP address is calculated according to formula (2), and their identity is verified to determine the field.

[0176] S57: Calculate the difference between the field values in two consecutive message units, and determine that the two consecutive message units are S when the difference is less than a preset maximum difference value. Accum Candidate fields.

[0177] Specifically, S Accum The fields in the accumulators are dynamic fields whose values increase over time. These fields typically represent message sequence numbers, acknowledgment numbers, or timestamps. To calculate the difference between the field values in two consecutive message units, simply subtract the original hexadecimal values of the corresponding fields from each other, expressed as Δ. This difference is typically expected to be a stable positive integer. However, it's important to note that this difference doesn't need to be completely constant; any value less than 1% of the maximum value can be considered a candidate field.

[0178] Furthermore, in step S6, a session-level sequence structure inference operation is performed based on the obtained message format to obtain the protocol state machine of each session, including:

[0179] S6A, matching the message format with the message unit in terms of field position and content, and if the message and state in the message unit correspond to the message format, determining that the message is in the corresponding state;

[0180] S6B. Identify the transition relationship between states based on the order of messages in the message unit and the change of states. If two messages that appear in sequence correspond to two different states, determine that there is a transition path between the two states that appear in sequence, record the number of occurrences of each transition path, and calculate the transition probability of each state based on the statistical number of transitions to generate a protocol state machine.

[0181] A state in the protocol state machine is a single extracted message format, that is, a group of messages of the same type. The extracted message format is matched with the original conversation stream in terms of field position and content. If a message matches the message format corresponding to a state, it is considered to be in the corresponding state. The transition relationship between states is then identified based on the order of messages and state changes in the conversation stream. For example, if two messages appear in the input traffic, one after the other, and they correspond to two different states, then a transition path is determined between the two states. During this process, the number of occurrences of each transition is recorded, and based on this count, the transition probability of each state is calculated to ultimately generate the state machine.

[0182] Furthermore, the conversation flow format in step S6 meets the specific frequency of appearing in the same conversation. Under the condition that the length of the extracted message format is 1 (only a single message unit is considered, and the message value completes the current operation or request, and is not combined with other message sequences), the conversation-level sequence structure reasoning operation is used to extract the flow format.

[0183] Specifically, refer to Figure 7 In step S6, a conversation-level sequence structure inference operation is performed based on the obtained message format to obtain the conversation flow format of each conversation, including:

[0184] S61, scanning the message units with one message format as one unit to generate a third sequence;

[0185] S62: Count the frequency of occurrence of each third sequence to obtain the third sequence support, delete the third sequence whose third sequence support is lower than the minimum preset support threshold, and obtain the third frequent sequence;

[0186] S63: performing pairwise merging based on the third frequent sequences to generate a third candidate sequence;

[0187] S64: Count the frequency of occurrence of each third candidate sequence to obtain the support of the third candidate sequence, delete the third candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new third sequence;

[0188] S65 , looping through steps S62 to S64 until a longer third sequence cannot be generated, performing frequency calculation on the third sequence finally obtained, and outputting a third sequence whose support is higher than a minimum preset support threshold as a conversation stream format.

[0189] refer to Figure 8 In some embodiments, a data protocol parsing device based on hierarchical sequence structure reasoning is provided, comprising:

[0190] A traffic diversion module 201 is used to import traffic from a network port through traffic mirroring, perform session diversion on the traffic, and identify unknown protocol traffic;

[0191] An assembling module 202 is configured to assemble the unknown protocol traffic into a message unit;

[0192] A field reasoning module 203 performs field-level sequence structure reasoning operations based on the message unit, extracts all possible values of the fixed-value and fixed-length fields, and processes the possible values of the fixed-value and fixed-length fields to obtain fixed-value and fixed-length fields and enumerated variable-length fields;

[0193] A message inference module 204 is configured to perform a message-level sequence structure inference operation on the message unit based on the extracted fixed-value fixed-length fields and the enumerated variable-length fields to obtain a message format;

[0194] Semantic reasoning module 205 performs semantic reasoning based on the obtained message format to obtain a semantic type;

[0195] The session reasoning module 206 performs session-level sequence structure reasoning operations based on the obtained message format to obtain the protocol state machine and session flow format of each session.

[0196] Furthermore, the assembling module 202 assembles the unknown protocol traffic into a message unit, including:

[0197] Determining whether the location protocol traffic type is UDP traffic or TCP traffic;

[0198] For the UDP traffic, each data packet is considered as a message unit; for the TCP traffic, a set of consecutive data packets with the same direction is considered as a message unit.

[0199] Furthermore, the field formats in the message unit are defined as fixed-value fixed-length fields, enumerated variable-length fields, dynamic fixed-length fields, and fully dynamic fields;

[0200] The field reasoning module 203 performs field-level sequence structure reasoning operations based on the message unit to extract all possible values of the fixed-value and fixed-length fields, including:

[0201] S31, scanning the data sequence of the message unit in units of two bytes to generate a first sequence;

[0202] S32, counting the frequency of occurrence of each first sequence to obtain the first sequence support, deleting the first sequence whose first sequence support is lower than the minimum preset support threshold, and obtaining a first frequent sequence;

[0203] S33. Merge two by two of the first frequent sequences to generate a first candidate sequence;

[0204] S34. Count the frequency of occurrence of each first candidate sequence to obtain the support of the first candidate sequence, delete the first candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new first sequence;

[0205] S35. Loop through steps S32 to S34 until a longer first sequence cannot be generated. Perform frequency calculation on the first sequence finally obtained, and output the first sequence whose support is higher than a minimum preset support threshold as the final target sequence. The final target sequence is a possible value of a fixed-value, fixed-length field.

[0206] Furthermore, the field reasoning module 203 processes the possible values of the fixed-value fixed-length field to obtain the fixed-value fixed-length field and the enumerated variable-length field, including:

[0207] Recording a position offset of a possible value of a current fixed-value fixed-length field relative to a message unit header, and calculating a position variance of the possible value of the fixed-value fixed-length field based on the position offset;

[0208] Calculating the support of the possible values of the fixed-value fixed-length field, and determining the fields corresponding to the possible values of the fixed-value fixed-length field whose support is less than a preset support threshold and whose position variance is lower than a preset variance threshold as convertible enumerated indefinite-length fields;

[0209] Determine the remaining fields after excluding the convertible enumerated indefinite-length fields from the fields corresponding to the possible values of the fixed-value fixed-length fields as fixed-value fixed-length fields;

[0210] The message unit is truncated according to the minimum offset and maximum depth of the convertible enumeration indefinite length field in the message unit, and the support of the truncated sequence is calculated. The sequence with the highest support is stored in an array. This step is repeated until the message unit no longer generates a new sequence, and the sequence in the array is used as the enumeration indefinite length field.

[0211] Furthermore, the message inference module 204 performs a message-level sequence structure inference operation on the message unit based on the extracted fixed-value fixed-length fields and the enumerated variable-length fields to obtain a message format, including:

[0212] S41, scanning the fixed-value fixed-length field and enumerating the values of the variable-length field with one field as a unit to obtain a second sequence;

[0213] S42, counting the frequency of occurrence of each second sequence to obtain the second sequence support, deleting the second sequence whose second sequence support is lower than the minimum preset support threshold, and obtaining a second frequent sequence;

[0214] S43. Perform pairwise combinations based on the second frequent sequences to generate a second candidate sequence;

[0215] S44. Count the frequency of occurrence of each second candidate sequence to obtain the support of the second candidate sequence, delete the second candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new second sequence;

[0216] S45. Loop through steps S42 to S44 until a longer second sequence cannot be generated. Perform frequency calculation on the second sequence finally obtained, and output a second sequence whose second sequence support is higher than a minimum preset support threshold as an initial message format. The initial message format includes a fixed-length field and an enumerated variable-length field.

[0217] S46. Extract the additional field between the fixed-value fixed-length field and the enumerated indefinite-length field and determine its field format, and use the fixed-value fixed-length field, the enumerated indefinite-length field, and the additional field between them as the message format;

[0218] The field format of the additional field includes an enumerated indefinite length field, a dynamic fixed length field, and a completely dynamic field.

[0219] Furthermore, the message inference module 204 extracts the additional fields between the fixed-value fixed-length fields and the enumerated variable-length fields in the initial message format and determines the field formats thereof, including:

[0220] Using the length of the fixed-length field and the enumerated variable-length field as the screening condition, the message sequence corresponding to the initial message format is collected from the original message unit, and the corresponding data set is found according to the position and length of the additional field in the initial message format;

[0221] Calculate the maximum value and variance of the data set length. If the variance of the data set is greater than a preset variance threshold, determine the data set as a fully dynamic field. If the variance of the data set is less than or equal to the preset variance threshold, determine the data set as a non-fully dynamic field.

[0222] If the variance of the non-completely dynamic field is less than the preset length threshold, the non-completely dynamic field is determined to be an enumerated indefinite length field; if the variance of the non-completely dynamic field is greater than or equal to the preset length threshold, the non-completely dynamic field is determined to be a dynamic fixed-length field.

[0223] Furthermore, the semantic reasoning module 205 performs semantic reasoning based on the obtained message format to obtain a semantic type, including:

[0224] If the data is a dynamic field, the information theory index is calculated based on the data, and the field whose information theory index is greater than the preset index value is determined as S Mtype Candidate fields for

[0225] If the data is a dynamic field, the Pearson correlation coefficient is calculated based on the data. When the Pearson correlation coefficient is greater than the preset coefficient value, the corresponding field is determined to be S Mlen Candidate fields for

[0226] If the data is a dynamic field and the data is IP data, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the first preset correlation threshold is determined as S Host Candidate fields for

[0227] If the data is a dynamic field and the data is a session, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the second preset correlation threshold is determined as S Sess candidate fields;

[0228] Calculate whether the field corresponding to the data is the same as the entropy value of the user's IP address. If they are the same, determine that the field corresponding to the data is S Trans Field;

[0229] Calculate the difference between the field values in two consecutive message units, and determine that the two consecutive message units are S when the difference is less than the preset maximum difference value. Accum Candidate fields.

[0230] Furthermore, the session reasoning module 206 performs session-level sequence structure reasoning operations based on the obtained message format to obtain the protocol state machine for each session, including:

[0231] Matching the message format with the message unit in terms of field position and content, and if the message and state in the message unit correspond to the message format, determining that the message is in the corresponding state;

[0232] The transition relationship between states is identified based on the order of messages in the message unit and the changes in state. If two messages that appear in sequence correspond to two different states, it is determined that there is a transition path between the two states that appear in sequence. The number of occurrences of each transition path is recorded, and the transition probability of each state is calculated based on the statistical number of transitions to generate a protocol state machine.

[0233] Furthermore, the conversation reasoning module 206 performs conversation-level sequence structure reasoning based on the obtained message format to obtain the conversation flow format of each conversation, including:

[0234] S61, scanning the message units with one message format as one unit to generate a third sequence;

[0235] S62: Count the frequency of occurrence of each third sequence to obtain the third sequence support, delete the third sequence whose third sequence support is lower than the minimum preset support threshold, and obtain the third frequent sequence;

[0236] S63: performing pairwise merging based on the third frequent sequences to generate a third candidate sequence;

[0237] S64: Count the frequency of occurrence of each third candidate sequence to obtain the support of the third candidate sequence, delete the third candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new third sequence;

[0238] S65 , looping through steps S62 to S64 until a longer third sequence cannot be generated, performing frequency calculation on the third sequence finally obtained, and outputting a third sequence whose support is higher than a minimum preset support threshold as a conversation stream format.

[0239] The data protocol parsing method and device based on hierarchical sequence structure reasoning provided in the above embodiment have at least the following beneficial effects:

[0240] The field format, message format, and session format of the unknown protocol are extracted layer by layer, and the unknown protocol is parsed based on its syntax, semantics, and behavior. This method has good applicability and can be applied to various power data communication flow monitoring systems.

[0241] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A data protocol parsing method based on hierarchical sequence structure reasoning, characterized in that: include: Importing the traffic of the network port into the analysis device through traffic mirroring, performing session diversion on the traffic, and identifying unknown protocol traffic; Performing message assembly on the unknown protocol traffic to form a message unit; Performing a field-level sequence structure inference operation based on the message unit, extracting all possible values of the fixed-value fixed-length field, and processing the possible values of the fixed-value fixed-length field to obtain the fixed-value fixed-length field and the enumerated variable-length field; Performing a message-level sequence structure inference operation on the message unit according to the extracted fixed-value fixed-length field and the enumerated variable-length field to obtain a message format; Perform semantic reasoning based on the obtained message format to obtain the semantic type; Perform session-level sequence structure inference based on the obtained message format to obtain the protocol state machine and session flow format of each session; The field formats in the message unit are defined as fixed-value fixed-length fields, enumerated variable-length fields, dynamic fixed-length fields, and fully dynamic fields; Performing field-level sequence structure inference operations based on the message unit to extract all possible values of the fixed-value and fixed-length fields, including: S31, scanning the data sequence of the message unit in units of two bytes to generate a first sequence; S32, counting the frequency of occurrence of each first sequence to obtain the first sequence support, deleting the first sequence whose first sequence support is lower than the minimum preset support threshold, and obtaining a first frequent sequence; S33. Perform pairwise merging based on the first frequent sequences to generate a first candidate sequence; S34. Count the frequency of occurrence of each first candidate sequence to obtain the support of the first candidate sequence, delete the first candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new first sequence; S35. Loop through steps S32 to S34 until a longer first sequence cannot be generated. Perform frequency calculation on the first sequence finally obtained, and output the first sequence whose support is higher than a minimum preset support threshold as the final target sequence. The final target sequence is a possible value of a fixed-value, fixed-length field.

2. The method according to claim 1, characterized in that Assembling the unknown protocol traffic into a message unit includes: Determining whether the unknown protocol traffic type is UDP traffic or TCP traffic; For the UDP traffic, each data packet is considered as a message unit; for the TCP traffic, a set of consecutive data packets with the same direction is considered as a message unit.

3. The method according to claim 1, characterized in that The possible values of the fixed-value fixed-length field are processed to obtain the fixed-value fixed-length field and the enumerated variable-length field, including: Recording a position offset of a possible value of a current fixed-value fixed-length field relative to a message unit header, and calculating a position variance of the possible value of the fixed-value fixed-length field based on the position offset; Calculating the support of the possible values of the fixed-value fixed-length field, and determining the fields corresponding to the possible values of the fixed-value fixed-length field whose support is less than a preset support threshold and whose position variance is lower than a preset variance threshold as convertible enumerated indefinite-length fields; Determine the remaining fields after excluding the convertible enumerated indefinite length fields from the fields corresponding to the possible values of the fixed-value fixed-length fields as fixed-value fixed-length fields; The message unit is truncated according to the minimum offset and maximum depth of the convertible enumeration indefinite length field in the message unit, and the support of the truncated sequence is calculated. The sequence with the highest support is stored in an array. This step is repeated until the message unit no longer generates a new sequence, and the sequence in the array is used as the enumeration indefinite length field.

4. The method according to claim 1, wherein According to the extracted fixed-value fixed-length fields and the enumerated variable-length fields, a message-level sequence structure inference operation is performed on the message unit to obtain a message format, including: S41, scanning the fixed-value fixed-length field and enumerating the values of the variable-length field with one field as a unit to obtain a second sequence; S42, counting the frequency of occurrence of each second sequence to obtain the second sequence support, deleting the second sequence whose second sequence support is lower than the minimum preset support threshold, and obtaining a second frequent sequence; S43. Perform pairwise combinations based on the second frequent sequences to generate a second candidate sequence; S44. Count the frequency of occurrence of each second candidate sequence to obtain the support of the second candidate sequence, delete the second candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new second sequence; S45. Loop through steps S42 to S44 until a longer second sequence cannot be generated. Perform frequency calculation on the second sequence finally obtained, and output a second sequence whose second sequence support is higher than a minimum preset support threshold as an initial message format. The initial message format includes a fixed-length field and an enumerated variable-length field. S46. Extract the additional field between the fixed-value fixed-length field and the enumerated variable-length field in the initial message format and determine its field format, and use the fixed-value fixed-length field, the enumerated variable-length field, and the additional field between them as the message format; The field format of the additional field includes an enumerated indefinite length field, a dynamic fixed length field, and a completely dynamic field.

5. The method according to claim 4, characterized in that Extracting an additional field between the fixed-value fixed-length field and the enumerated variable-length field in the initial message format and determining its field format includes: Using the length of the fixed-length field and the enumerated variable-length field as the screening condition, the message sequence corresponding to the initial message format is collected from the original message unit, and the corresponding data set is found according to the position and length of the additional field in the initial message format; Calculate the maximum value and variance of the data set length. If the variance of the data set is greater than a preset variance threshold, determine the data set as a fully dynamic field. If the variance of the data set is less than or equal to the preset variance threshold, determine the data set as a non-fully dynamic field. If the variance of the non-completely dynamic field is less than the preset length threshold, the non-completely dynamic field is determined to be an enumerated indefinite length field; if the variance of the non-completely dynamic field is greater than or equal to the preset length threshold, the non-completely dynamic field is determined to be a dynamic fixed-length field.

6. The method according to claim 1, characterized in that Perform semantic reasoning based on the obtained message format to obtain the semantic type, including: Retrieving data corresponding to the message format from the original message unit according to the position and length of the fields in the message format; If the data is a dynamic field, the information theory index is calculated based on the data, and the field whose information theory index is greater than the preset index value is determined as S Mtype Candidate fields for If the data is a dynamic field, the Pearson correlation coefficient is calculated based on the data. When the Pearson correlation coefficient is greater than the preset coefficient value, the corresponding field is determined to be S Mlen Candidate fields for If the data is a dynamic field and the data is IP data, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the first preset correlation threshold is determined as S Host Candidate fields for If the data is a dynamic field and the data is a session, the correlation between the field format and the user IP address classification is calculated, and the field whose classification correlation is greater than the second preset correlation threshold is determined as S Sess Candidate fields for Calculate whether the field corresponding to the data is the same as the entropy value of the user's IP address. If they are the same, determine that the field corresponding to the data is S Trans Candidate fields for Calculate the difference between the field values in two consecutive message units, and determine that the two consecutive message units are S when the difference is less than the preset maximum difference value. Accum Candidate fields.

7. The method according to claim 1, characterized in that Based on the obtained message format, the session-level sequence structure inference operation is performed to obtain the protocol state machine of each session, including: Matching the message format with the message unit in terms of field position and content, and if the message and state in the message unit correspond to the message format, determining that the message is in the corresponding state; The transition relationship between states is identified based on the order of messages in the message unit and the changes in state. If two messages that appear in sequence correspond to two different states, it is determined that there is a transition path between the two states that appear in sequence. The number of occurrences of each transition path is recorded, and the transition probability of each state is calculated based on the statistical number of transitions to generate a protocol state machine.

8. The method according to claim 1, characterized in that Based on the obtained message format, a session-level sequence structure inference operation is performed to obtain the session flow format of each session, including: S61, scanning the message units with one message format as one unit to generate a third sequence; S62: Count the frequency of occurrence of each third sequence to obtain the third sequence support, delete the third sequence whose third sequence support is lower than the minimum preset support threshold, and obtain the third frequent sequence; S63: performing pairwise merging based on the third frequent sequences to generate a third candidate sequence; S64: Count the frequency of occurrence of each third candidate sequence to obtain the support of the third candidate sequence, delete the third candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new third sequence; S65 , looping through steps S62 to S64 until a longer third sequence cannot be generated, performing frequency calculation on the third sequence finally obtained, and outputting a third sequence whose support is higher than a minimum preset support threshold as a conversation stream format.

9. A data protocol parsing device based on hierarchical sequence structure reasoning, characterized in that: include: A traffic diversion module, configured to import traffic from a network port through traffic mirroring, perform session diversion on the traffic, and identify traffic with unknown protocols; An assembling module, configured to assemble messages of the unknown protocol traffic into message units; A field reasoning module performs field-level sequence structure reasoning operations based on the message unit, extracts all possible values of the fixed-value and fixed-length fields, and processes the possible values of the fixed-value and fixed-length fields to obtain fixed-value and fixed-length fields and enumerate variable-length fields; A message inference module, configured to perform a message-level sequence structure inference operation on the message unit based on the extracted fixed-value fixed-length fields and the enumerated variable-length fields to obtain a message format; The semantic reasoning module performs semantic reasoning based on the obtained message format to obtain the semantic type; The session reasoning module performs session-level sequence structure reasoning based on the obtained message format to obtain the protocol state machine and session flow format of each session; The field formats in the message unit are defined as fixed-value fixed-length fields, enumerated variable-length fields, dynamic fixed-length fields, and fully dynamic fields; The field reasoning module performs field-level sequence structure reasoning operations based on the message unit to extract all possible values of the fixed-value and fixed-length fields, including: S31, scanning the data sequence of the message unit in units of two bytes to generate a first sequence; S32, counting the frequency of occurrence of each first sequence to obtain the first sequence support, deleting the first sequence whose first sequence support is lower than the minimum preset support threshold, and obtaining a first frequent sequence; S33. Perform pairwise merging based on the first frequent sequences to generate a first candidate sequence; S34. Count the frequency of occurrence of each first candidate sequence to obtain the support of the first candidate sequence, delete the first candidate sequence whose support is lower than the minimum preset support threshold, and obtain a new first sequence; S35. Loop through steps S32 to S34 until a longer first sequence cannot be generated. Perform frequency calculation on the first sequence finally obtained, and output the first sequence whose support is higher than a minimum preset support threshold as the final target sequence. The final target sequence is a possible value of a fixed-value, fixed-length field.

Images