A network protocol reverse analysis method

Abstract

The present invention relates to a network protocol reverse parsing method. First, for each message in the target network interactive traffic, the positions of each global candidate key field are determined through numerical conversion and keyword position label analysis; then, probabilistic reasoning is applied, combined with loop iteration, to obtain the positions of each target key field; finally, the numerical vectors corresponding to each message in the target network interactive traffic are obtained, and autoencoders and clustering algorithms are alternately optimized and trained to implement network protocol format parsing for each message under each cluster; the design effectively solves the problem of network protocol reverse parsing, innovatively constructs keyword recognition as a probabilistic reasoning problem, overcomes the shortcomings of traditional methods in uncertainty processing, and thus outputs a more accurate message format, providing an efficient and comprehensive solution for network protocol reverse parsing engineering.

Description

Technical Field

[0001] The invention relates to a network protocol reverse parsing method, belonging to the technical field of network flow parsing. Background Art

[0002] With the rapid development of modern information technology, network protocols play a crucial role in data communication. Network protocols define the format, sequence, and rules for information exchange between communicating parties, covering all levels from low-level physical link communication to high-level application data transmission, such as the TCP / IP protocol suite. However, the communication protocols involved in many security applications are often poorly documented, such as CAN bus and FlexRay in autonomous driving, Modbus and DNP3 in control systems, and custom protocols for online chat / conferencing applications. Security analysis tasks such as static / symbolic vulnerability scanning, exploit generation, fuzz testing, attack detection, and malware behavioral analysis all require accurate modeling of network protocols. For example, in fuzz testing, understanding the network application protocol is crucial for generating valid input seeds; malware analysis often requires sending correctly formatted messages to command and control servers to trigger hidden behaviors; and static / symbolic analysis can generate numerous false positives if it fails to correctly model network functionality. In summary, research on network protocol reverse engineering is crucial for ensuring network security and is a key component in addressing various security challenges.

[0003] Protocol reverse engineering (PRE) refers to the process of inferring the parameters, format, and semantics of unknown or proprietary network protocols by analyzing network communication data or program binary code in the absence of formal protocol specifications. Protocol reverse engineering was originally performed manually by security analysts. However, manual protocol reverse engineering is extremely tedious and time-consuming, sometimes taking years to fully discover certain protocol specifications, making it unfeasible. Existing protocol reverse engineering techniques fall into two categories: program analysis-based protocol reverse engineering and network tracing-based protocol reverse engineering. Most of the former require access to program binaries, but some IoT firmware has protection mechanisms or binaries are packed or obfuscated, making this technique generally impractical in practice. The latter mainly falls into two categories: alignment-based protocol reverse engineering and token-based protocol reverse engineering. Alignment-based protocol reverse engineering utilizes various sequence alignment algorithms to align message pairs, calculate similarity scores, cluster them based on these scores, and then derive the format by analyzing the commonalities among messages within the clusters. However, the diversity of message content significantly reduces the quality of the alignment, which can cause problems for downstream analysis. Token-based methods propose to tokenize messages (e.g., separating text and binary fields) before alignment to reduce variation. However, existing techniques often rely on delimiters to identify tokens, which may not exist in binary protocols. Furthermore, these existing methods, based on deterministic heuristics, tend to generate too many clusters and often produce incorrect results.

[0004] To summarize the existing protocol reverse engineering technologies, there are two shortcomings: (1) in actual execution, due to the inability to obtain the delimiters in the binary file or binary protocol, the technology based on program analysis and tokens may not be able to be executed; (2) the existing reverse parsing technology does not perform well in the case of message diversity. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a network protocol reverse parsing method, which adopts bidirectional GRU sequence labeling and probabilistic reasoning to perform network protocol parsing on the target network interaction traffic, significantly reducing the false alarm rate and improving the accuracy.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solution: the present invention designs a network protocol reverse parsing method for performing network protocol parsing on target network interaction traffic extracted between a client and a server, comprising the following steps:

[0007] Step A. For each message in the target network's interactive traffic, the corresponding values ​​of each byte position in the message are converted, and each byte position in the message is analyzed to determine whether it is a keyword position label. The position of each global candidate key field is determined, and then the process proceeds to Step B.

[0008] Step B. Based on the positions of each global candidate key field, apply probabilistic reasoning and loop iteration to obtain the positions of each target key field, and then proceed to Step C.

[0009] Step C. Based on the location of each target key field, obtain the numerical vector corresponding to each message in the target network interaction traffic. Then, use the alternating optimization training method of the autoencoder and clustering algorithm to parse the network protocol format for each message under each cluster.

[0010] As a preferred technical solution of the present invention: Step A includes the following steps A1 to A3;

[0011] Step A1. Group each message in the target network interaction traffic by grouping messages with the same originating IP address, the same destination IP address, and messages with a time interval between consecutive timestamps less than a preset time interval threshold. This grouping is then performed to form message groups, and then proceed to Step A2.

[0012] Step A2. For each message group, first update the bit length of each message in the message group to a multiple of 8, and unify the bit length of each message in the message group. Then, for each message in the message group, execute steps A2-1 to A2-3 to determine the position of each candidate key field in the message, and then proceed to step A3.

[0013] Step A2-1. Sequentially convert each 8-bit single byte corresponding to the message into a corresponding value in the integer sequence 0-255, obtain the value corresponding to each byte position in the message, and then proceed to step A2-2;

[0014] Step A2-2. First, a bidirectional GRU encoder reads the value corresponding to each byte position in the message to obtain the hidden state corresponding to each byte position in the message. Then, a fully connected layer with a softmax activation function is used to map the hidden state to the label space. The probability of each byte position in the message corresponding to the keyword position label is obtained, and then the process proceeds to step A2-3.

[0015] Step A2-3. Determine the position of each candidate key field in the message based on the probability that each byte position in the message corresponds to the keyword position tag;

[0016] Step A3. Calculate the union of the positions of the candidate key fields in each message in each message group, that is, obtain the positions of the global candidate key fields.

[0017] As a preferred technical solution of the present invention: in the step A2, for each message group, the bit length of the message with the longest number of bits in the message group is first updated to a multiple of 8 by padding the end with at least 0, and the target bit length corresponding to the message group is formed. Then, the bit length of the remaining messages in the message group is updated to the target bit length by padding the end with 0, thereby achieving the unification of the bit length of each message in the message group.

[0018] As a preferred technical solution of the present invention: in the step A2-3, the conditional random field (CRF) is used to analyze the probability of each byte position in the message corresponding to a keyword position label, and the independent byte positions corresponding to the keyword position label and the adjacent continuous byte group positions are determined to constitute the candidate key field positions in the message.

[0019] As a preferred technical solution of the present invention: in step B, the positions of the global candidate key fields are initialized to form a candidate set, and the number of iterations is , then execute the following steps B1 to B3 to obtain the position of each target key field;

[0020] Step B1. Apply probabilistic reasoning to calculate the probability that each global candidate key field position in the candidate set constitutes the target key field position, and proceed to step B2;

[0021] Step B2. Select the maximum probability as the The probability corresponding to the iteration is If it is greater than 1, then the Is the probability corresponding to the iteration greater than the If the probability corresponding to the iteration is , the global candidate key field position corresponding to the maximum probability in the current iteration is selected to form the target key field position, and the global candidate key field position corresponding to the maximum probability is deleted from the candidate set, the candidate set is updated, and then the process goes to step B3; otherwise, the process of step B ends;

[0022] like If it is equal to 1, the global candidate key field position corresponding to the maximum probability in the current iteration is selected to form the target key field position, and the global candidate key field position corresponding to the maximum probability is deleted from the candidate set, the candidate set is updated, and then the process goes to step B3;

[0023] Step B3. Determine whether there is a global candidate key field position in the candidate set, if so, The value of is updated by adding 1, and the process returns to step B1.

[0024] As a preferred technical solution of the present invention: in step B1, for each global candidate key field position in the candidate set, the following steps B1-1 to B1-3 are performed to obtain the probability that the global candidate key field position constitutes the target key field position;

[0025] Step B1-1. Temporarily cluster each message in the target network interactive traffic that has a non-zero numerical sequence corresponding to the global candidate key field position based on the numerical sequence of each message in the target network interactive traffic. Messages with the same non-zero numerical sequence are grouped into the same cluster. Temporary clusters corresponding to the global candidate key field position are obtained, and then the process proceeds to Step B1-2.

[0026] Step B1-2. Calculate the message similarity probability corresponding to each temporary cluster corresponding to the global candidate key field position , telecoupling probability , structural consistency probability , and calculate the dimension predicate probability corresponding to the global candidate key field position , then proceed to step B1-3;

[0027] Step B1-3. Use the following formula:

[0028] ;

[0029] ;

[0030] Get intermediate parameters 、 , and further according to the following formula:

[0031] ;

[0032] Obtain the probability that the global candidate key field position constitutes the target key field position ,in, Indicates the global candidate key field position, Indicates the probability of similarity of preset messages; represents the preset telecoupling probability; represents the probability of consistency of the preset structure; Indicates the first position of the global candidate key field The message similarity probability corresponding to the temporary clusters, Indicates the first position in the global candidate key field The probability of telecoupling corresponding to a temporary cluster, Indicates the first position in the global candidate key field The structural consistency probability corresponding to the temporary clusters, Represents the probability of a preset dimension predicate.

[0033] As a preferred technical solution of the present invention, in step B1-2, for each temporary cluster corresponding to the position of the global candidate key field, the following is performed:

[0034] About message similarity probability : Group each message in the temporary cluster into pairs to obtain each first subgroup. Based on the values ​​corresponding to each byte position in the message, calculate the ratio of the number of identical values ​​between the two messages in the first subgroup to the total number of bytes in the two messages to form the score corresponding to the first subgroup and construct the error mismatch rate. The function is as follows:

[0035] ;

[0036] Among the other temporary clusters corresponding to the global candidate key field positions, select the one with the smallest difference in the number of messages in the temporary cluster. Group the messages in the temporary cluster with the messages in the selected other temporary clusters in pairs to form each second sub-group. According to the values ​​corresponding to the byte positions in the messages, calculate the ratio of the number of identical values ​​between the two messages in the second sub-group to the total number of bytes of the two messages to form the score corresponding to the second sub-group, and construct the error matching rate. The function is as follows:

[0037] ;

[0038] based on ,get Function curve and Intersection value of the function curve , then press , obtain the message similarity probability corresponding to the temporary cluster ;

[0039] On the probability of telecoupling :First, define each message in the temporary cluster whose starting IP address is the client as each message a, and use the number of message a groups as parameters ; Then, the starting address of each message a corresponding to the server is determined from the target network interaction traffic, and defined as each message b; then, based on the temporary clusters corresponding to the positions of the global candidate key fields, the maximum number of messages belonging to the same temporary cluster in each message b is determined as the parameter , and finally according to the following formula:

[0040] ;

[0041] Obtain the probability of telecoupling corresponding to the temporary clustering ;

[0042] On the probability of structural consistency : Get the average original length of each message in the temporary cluster , and obtain the average value of the difference between the original length of each message in the temporary cluster and the longest length , and then according to the following formula:

[0043] ;

[0044] Obtain the structural consistency probability corresponding to the temporary clustering .

[0045] As a preferred technical solution of the present invention: in step B1-2, for the global candidate key field position, the following is performed:

[0046] First, according to the following formula:

[0047] ; ;

[0048] get and ,in, It represents the reciprocal of the average number of messages contained in a cluster. Indicates the proportion of clusters containing only one message among the temporary clusters. Indicates the number of temporary clusters corresponding to the global candidate key field position, Indicates the total number of messages in each temporary cluster corresponding to each global candidate key field position, Indicates the number of temporary clusters containing only one message corresponding to the position of the global candidate key field;

[0049] Then according to the following formula:

[0050] ;

[0051] Get the dimension predicate probability corresponding to the global candidate key field position ,in, 、 These are all preset threshold parameters.

[0052] As a preferred technical solution of the present invention: Step C includes the following steps C1 to C7;

[0053] Step C1. For each message in the target network interaction traffic, obtain the value of each target key field position of the message, sequentially construct the numerical vector corresponding to the message, and then obtain the numerical vector corresponding to each message, and then use the mean square error as the reconstruction loss function , train the autoencoder and then proceed to step C2;

[0054] Step C2. Apply the encoder in the trained autoencoder to obtain the low-dimensional embedding vector corresponding to each message's numerical vector. Then, based on a preset number of clusters, use the K-means clustering algorithm to cluster these low-dimensional embedding vectors, obtaining cluster centers. The process then proceeds to Step C3.

[0055] Step C3. Connect a clustering layer after the encoder in the autoencoder and define the parameters in the clustering layer as the cluster centers, then proceed to Step C4.

[0056] Step C4. Fix the autoencoder parameters and apply the encoder in the autoencoder to obtain the low-dimensional embedding vector corresponding to each message's numerical vector. Then, apply the K-means clustering algorithm to cluster the low-dimensional embedding vectors corresponding to each message, obtain the cluster centers, and update the cluster centers in the clustering layer. Then, proceed to Step C5.

[0057] Step C5. Fix the cluster centers in the clustering layer and use the reconstruction loss function of the autoencoder Clustering loss with K-means clustering algorithm The sum of the two messages is used to construct a joint loss to minimize the joint loss. The numerical vectors corresponding to the messages are used to train the autoencoder to obtain the trained autoencoder, and then step C6 is entered.

[0058] Step C6. Determine whether the preset iteration exit condition is met, if so, proceed to step C7; otherwise, return to step C4;

[0059] Step C7. For each cluster, align the messages in the cluster starting from the first position in order to obtain the fixed fields that are the same among the messages and the variable fields that are different from each other. Based on the positions of each target key field, determine the key field Q in the fixed field that corresponds to the corresponding target key field position, and determine the key field L in the variable field that corresponds to the corresponding target key field position for each different content. Then, based on the key field Q and each key field L, construct the same network protocol format corresponding to each message in the cluster; then obtain the network protocol format corresponding to each cluster, that is, realize the parsing of the network protocol format of each message in the target network interaction traffic.

[0060] As a preferred technical solution of the present invention: in step C6, the preset iteration exit condition is a preset maximum number of iterations, or the joint loss no longer continues to decrease.

[0061] The network protocol reverse parsing method of the present invention, using the above technical solution, has the following technical effects compared with the existing technology:

[0062] The present invention designs a network protocol reverse parsing method. First, for each message in the target network interaction traffic, the position of each global candidate key field is determined through numerical conversion and keyword position label analysis; then, probabilistic reasoning is applied, combined with loop iteration, to obtain the position of each target key field; finally, the numerical vector corresponding to each message in the target network interaction traffic is obtained, and the autoencoder and clustering algorithm are alternately optimized and trained to implement the network protocol format parsing for each message under each cluster; the design scheme effectively solves the problem of network protocol reverse parsing, and based on coarse-grained candidate key fields, it innovatively constructs keyword recognition as a probabilistic reasoning problem, applies bidirectional GRU sequence labeling and probabilistic reasoning in collaboration, introduces predicates to indicate whether a field is a keyword, and combines multiple observation results to form a joint probability distribution, overcoming the shortcomings of traditional methods in uncertainty processing, thereby outputting a more accurate message format, and providing an efficient and comprehensive solution for network protocol reverse parsing engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a system application architecture diagram of the network protocol reverse parsing method designed by the present invention;

[0064] Figure 2 This is an application flow chart of the key field candidate generation module in the design application of the present invention;

[0065] Figure 3 This is an application flow chart of the keyword recognition module in the design and application of the present invention;

[0066] Figure 4 This is an application flow chart of the clustering and protocol recovery module in the design and application of the present invention. DETAILED DESCRIPTION

[0067] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0068] The present invention designs a network protocol reverse parsing method, in practical application, such as Figure 1 As shown, a key field candidate generation module, a keyword recognition module, and a clustering and protocol recovery module are designed, wherein the key field candidate generation module generates a candidate field list to improve the accuracy and efficiency of key field recognition; the keyword recognition module performs probabilistic analysis on the candidate fields to determine the keywords; the clustering and protocol recovery module clusters the key fields and recovers the protocol format according to the clustering results. Specifically, the three modules sequentially execute the following steps A to C to perform network protocol parsing on the target network interaction traffic extracted between the client and the server.

[0069] Step A. The key field candidate generation module converts the values ​​corresponding to each byte position in each message in the target network interaction traffic and analyzes whether each byte position in each message is a keyword position label to determine the global candidate key field position, and then proceeds to Step B.

[0070] In practical application, the above step A is as follows: Figure 2 As shown, the key field candidate generation module is designed to specifically perform the following steps A1 to A3.

[0071] Step A1. Group each message in the target network's interactive traffic by grouping it into groups based on the same originating IP address, the same destination IP address, and the duration between consecutive timestamps less than a preset duration threshold. Messages are then grouped together to form message groups, and then proceed to Step A2.

[0072] Step A2. For each message group, first update the bit length of the longest message in the message group to a multiple of 8 by padding the end with a minimum of 0, and form the target bit length corresponding to the message group. Then, update the bit length of the remaining messages in the message group to the target bit length by padding the end with 0, so as to unify the bit length of each message in the message group. Then, execute the following steps A2-1 to A2-3 for each message in the message group to determine the position of each candidate key field in the message, and then proceed to step A3.

[0073] Step A2-1. Sequentially convert the single byte corresponding to each 8 bits in the message into the corresponding value in the integer sequence 0-255, obtain the value corresponding to each byte position in the message, and then proceed to step A2-2.

[0074] Step A2-2. First, use a bidirectional GRU encoder to read the numerical value corresponding to each byte position in the message to obtain the hidden state corresponding to each byte position in the message. Specifically, the bidirectional GRU encoder reads the sequence from left to right, gradually updates the hidden state, and records the "past" context; reads the sequence from right to left, gradually updates the hidden state, and records the "future" context, thus obtaining the hidden state of each byte position. Then, use a fully connected layer with a softmax activation function to map the hidden state to the label space, obtain the probability of each byte position in the message corresponding to the keyword position label, and then proceed to step A2-3.

[0075] Step A2-3. Analyze the probability of each byte position in the message corresponding to a keyword position tag using a conditional random field (CRF). Determine the positions of each independent byte position and adjacent, continuous byte group that correspond to the keyword position tag, forming the candidate key field positions in the message.

[0076] Using CRF (conditional random field) as the optimization layer for sequence labeling obtains the global optimal label sequence, which is a conservative and rough result. The subsequent key field recognition module will provide a high-precision recognition result.

[0077] As a variant of the recurrent neural network (RNN), the Gated Recurrent Unit (GRU) excels at processing variable-length sequences and capturing long-distance dependencies, making it suitable for analyzing contextual features in network messages. The CRF, on the other hand, uses a transfer matrix to model dependencies between labels. It then uses the Viterbi algorithm to dynamically plan all possible label sequences and select the highest-scoring legal path. The resulting label sequence conforms to the protocol's grammatical rules, is highly consistent, and can correct local Softmax misjudgments using global information. This design, combining single-byte processing with the lightweight GRU, is well-suited for real-time scenarios. Furthermore, the CRF's global optimization reduces noise interference and improves the reliability of candidate fields.

[0078] Step A3. Calculate the union of the positions of the candidate key fields in each message in each message group, that is, obtain the positions of the global candidate key fields.

[0079] Step B. The keyword identification module applies probabilistic reasoning based on the positions of each global candidate keyword field, combined with loop iteration, to obtain the positions of each target keyword field, and then proceeds to step C;

[0080] In the actual application of step B above, the keyword recognition module first initializes the positions of the global candidate key fields to form a candidate set, and the number of iterations , then as Figure 3As shown, the design specifically executes the following steps B1 to B3 to obtain the position of each target key field.

[0081] Step B1. Apply probabilistic reasoning to calculate the probability that each global candidate key field position in the candidate set constitutes the target key field position, and then proceed to step B2.

[0082] In practical applications, in step B1, for each global candidate key field position in the candidate set, the following steps B1-1 to B1-3 are specifically performed to obtain the probability that the global candidate key field position constitutes the target key field position.

[0083] Step B1-1. Temporarily cluster the messages in the target network interactive traffic that have non-zero numerical sequences corresponding to the global candidate key field positions based on the numerical sequences of the messages in the target network interactive traffic. Messages with the same non-zero numerical sequences are grouped into the same cluster. Temporary clusters corresponding to the global candidate key field positions are obtained, and then the process proceeds to Step B1-2.

[0084] Step B1-2. Calculate the message similarity probability corresponding to each temporary cluster corresponding to the global candidate key field position , telecoupling probability , structural consistency probability , and calculate the dimension predicate probability corresponding to the global candidate key field position , then go to step B1-3.

[0085] Regarding the above 、 、 ,In actual applications, the following procedures are performed for each temporary cluster corresponding to the ,position of the global candidate key field.

[0086] About message similarity probability : Group each message in the temporary cluster into pairs to obtain each first subgroup. Based on the values ​​corresponding to each byte position in the message, calculate the ratio of the number of identical values ​​between the two messages in the first subgroup to the total number of bytes in the two messages to form the score corresponding to the first subgroup and construct the error mismatch rate. The function is as follows:

[0087] ;

[0088] Among the other temporary clusters corresponding to the global candidate key field positions, select the one with the smallest difference in the number of messages in the temporary cluster. Group the messages in the temporary cluster with the messages in the selected other temporary clusters in pairs to form each second sub-group. According to the values ​​corresponding to the byte positions in the messages, calculate the ratio of the number of identical values ​​between the two messages in the second sub-group to the total number of bytes of the two messages to form the score corresponding to the second sub-group, and construct the error matching rate. The function is as follows:

[0089] ;

[0090] based on ,get Function curve and Intersection value of the function curve , then press , obtain the message similarity probability corresponding to the temporary cluster .

[0091] On the probability of telecoupling :First, define each message in the temporary cluster whose starting IP address is the client as each message a, and use the number of message a groups as parameters ; Then, the starting address of each message a corresponding to the server is determined from the target network interaction traffic, and defined as each message b; then, based on the temporary clusters corresponding to the positions of the global candidate key fields, the maximum number of messages belonging to the same temporary cluster in each message b is determined as the parameter , and finally according to the following formula:

[0092] ;

[0093] Obtain the probability of telecoupling corresponding to the temporary clustering .

[0094] On the probability of structural consistency : Get the average original length of each message in the temporary cluster , and obtain the average value of the difference between the original length of each message in the temporary cluster and the longest length , and then according to the following formula:

[0095] ;

[0096] Obtain the structural consistency probability corresponding to the temporary clustering .

[0097] About the dimension predicate probability corresponding to the position of the global candidate key field In actual application, for the global candidate key field position, the following is performed:

[0098] First, according to the following formula:

[0099] ; ;

[0100] get and ,in, It represents the reciprocal of the average number of messages contained in a cluster. Indicates the proportion of clusters containing only one message among the temporary clusters, where Indicates the number of temporary clusters corresponding to the global candidate key field position, Indicates the total number of messages in each temporary cluster corresponding to each global candidate key field position, Indicates the number of temporary clusters containing only one message corresponding to the global candidate key field position.

[0101] Then according to the following formula:

[0102] ;

[0103] Get the dimension predicate probability corresponding to the global candidate key field position ,in, 、 These are all preset threshold parameters. In actual application, set .

[0104] The key step designed here is to model the uncertainty in keyword recognition as the joint distribution of observations and predicates and the details of probabilistic inference, where the definition of predicates is shown in Table 1 below.

[0105] Table 1 Predicate and constraint definition table

[0106]

[0107] The last column in the table is the constraint set of each predicate, which intuitively represents the correlation between predicates. There are two types of constraints for each predicate. One is the observation constraint, which is as follows: , meaning have The prior probability is true, and the other is an inference constraint, such as: , which means if is the keyword, cluster Chinese news The probability of having higher internal similarity than inter-cluster messages.

[0108] Following the execution of the above step B1-2, continue to execute the following step B1-3.

[0109] Step B1-3. Use the following formula:

[0110] ;

[0111] ;

[0112] Get intermediate parameters 、 , and further according to the following formula:

[0113] ;

[0114] Obtain the probability that the global candidate key field position constitutes the target key field position ,in, Indicates the global candidate key field position, Indicates the probability of similarity of preset messages; represents the preset telecoupling probability; represents the probability of consistency of the preset structure; If Constitute the target key field position, then The corresponding temporary clusters The message similarity will be based on the preset message similarity probability Established; If Constitute the target key field position, then The corresponding temporary clusters The medium and long-distance coupling will be based on the preset long-distance coupling probability Established; If Constitute the target key field position, then The corresponding temporary clusters The structural consistency will be based on the preset structural consistency probability Established; Indicates the first position in the global candidate key field The message similarity probability corresponding to the temporary clusters, Indicates the first position in the global candidate key field The probability of telecoupling corresponding to a temporary cluster, Indicates the first position in the global candidate key field The structural consistency probability corresponding to the temporary clusters, Represents the probability of a preset dimension predicate.

[0115] Step B2. Select the maximum probability as the The probability corresponding to the iteration is If it is greater than 1, then the Is the probability corresponding to the iteration greater than the If the probability corresponding to the iteration is , then the global candidate key field position corresponding to the maximum probability in the current iteration is selected to form the target key field position, and the global candidate key field position corresponding to the maximum probability is deleted from the candidate set, the candidate set is updated, and then step B3 is entered; otherwise, the processing of step B is ended.

[0116] like If it is equal to 1, the global candidate key field position corresponding to the maximum probability in the current iteration is selected to form the target key field position, and the global candidate key field position corresponding to the maximum probability is deleted from the candidate set, the candidate set is updated, and then step B3 is entered.

[0117] Step B3. Determine whether there is a global candidate key field position in the candidate set, if so, The value of is updated by adding 1, and the process returns to step B1.

[0118] Step C. The clustering and protocol recovery module obtains the numerical vector corresponding to each message in the target network interaction traffic based on the location of each target key field. It then uses an alternating optimization training method of the autoencoder and clustering algorithm to parse the network protocol format for each message in each cluster.

[0119] In practical applications, in the above step C, the clustering and protocol recovery modules are as follows: Figure 4 As shown, the specific design performs the following steps C1 to C7.

[0120] Step C1. For each message in the target network interaction traffic, obtain the value of each target key field position of the message, sequentially construct the numerical vector corresponding to the message, and then obtain the numerical vector corresponding to each message, and then use the mean square error as the reconstruction loss function , train the autoencoder and then go to step C2.

[0121] Step C2. Apply the encoder in the trained autoencoder to obtain the low-dimensional embedding vector corresponding to the numerical vector of each message, that is, the low-dimensional embedding vector corresponding to each message. Then, based on the preset number of clusters, use the K-means clustering algorithm to cluster the low-dimensional embedding vectors corresponding to each message to obtain the cluster centers, and then proceed to Step C3.

[0122] Step C3. Connect a clustering layer after the encoder in the autoencoder and define the parameters in the clustering layer as the cluster centers, then proceed to step C4.

[0123] Step C4. Fix the parameters of the autoencoder and apply the encoder in the autoencoder to obtain the low-dimensional embedding vector corresponding to the numerical vector of each message, that is, the low-dimensional embedding vector corresponding to each message. Then apply the K-means clustering algorithm to cluster the low-dimensional embedding vectors corresponding to each message to obtain the cluster centers. Update the cluster centers in the clustering layer and then proceed to step C5.

[0124] Step C5. Fix the cluster centers in the clustering layer and use the reconstruction loss function of the autoencoder Clustering loss with K-means clustering algorithm The sum of , constructs a joint loss to minimize the joint loss, applies the numerical vectors corresponding to each message, trains the autoencoder, obtains the trained autoencoder, and then enters step C6.

[0125] Step C6: Determine whether the preset iteration exit condition is met. If so, proceed to Step C7; otherwise, return to Step C4. The preset iteration exit condition is the preset maximum number of iterations or the joint loss no longer decreases.

[0126] Step C7. For each cluster, align the messages in the cluster starting from the first position in order to obtain the fixed fields that are the same among the messages and the variable fields that are different from each other. Based on the positions of each target key field, determine the key field Q in the fixed field that corresponds to the corresponding target key field position, and determine the key field L in the variable field that corresponds to the corresponding target key field position for each different content. Then, based on the key field Q and each key field L, construct the same network protocol format corresponding to each message in the cluster; then obtain the network protocol format corresponding to each cluster, that is, realize the parsing of the network protocol format of each message in the target network interaction traffic.

[0127] The network protocol reverse parsing method designed by the above technical solution first determines the position of each global candidate key field for each message in the target network interaction traffic through numerical conversion and keyword position label analysis; then applies probabilistic reasoning, combined with loop iteration, to obtain the position of each target key field; finally, obtains the numerical vector corresponding to each message in the target network interaction traffic, and alternately optimizes and trains with autoencoders and clustering algorithms to implement network protocol format parsing for each message under each cluster; the design scheme effectively solves the problem of network protocol reverse parsing, and based on coarse-grained candidate key fields, it innovatively constructs keyword recognition as a probabilistic reasoning problem, applies bidirectional GRU sequence labeling and probabilistic reasoning in collaboration, introduces predicates to indicate whether a field is a keyword, and combines multiple observation results to form a joint probability distribution, overcoming the shortcomings of traditional methods in uncertainty processing, thereby outputting a more accurate message format, and providing an efficient and comprehensive solution for network protocol reverse parsing engineering.

[0128] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in this field without departing from the spirit of the present invention.

Claims

1. A network protocol reverse parsing method for performing network protocol parsing on target network interaction traffic extracted between a client and a server, characterized in that: The steps include: Step A. For each message in the target network's interactive traffic, the corresponding values ​​of each byte position in the message are converted, and each byte position in the message is analyzed to determine whether it is a keyword position label. The position of each global candidate key field is determined, and then the process proceeds to Step B. The above step A includes the following steps A1 to A3; Step A1. Group each message in the target network interaction traffic by grouping messages with the same originating IP address, the same destination IP address, and messages with a time interval between consecutive timestamps less than a preset time interval threshold. This grouping is then performed to form message groups, and then proceed to Step A2. Step A2. For each message group, first update the bit length of each message in the message group to a multiple of 8, and unify the bit length of each message in the message group. Then, for each message in the message group, execute steps A2-1 to A2-3 to determine the position of each candidate key field in the message, and then proceed to step A3. Step A2-1. Sequentially convert each 8-bit single byte corresponding to the message into a corresponding value in the integer sequence 0-255, obtain the value corresponding to each byte position in the message, and then proceed to step A2-2; Step A2-2. First, a bidirectional GRU encoder reads the value corresponding to each byte position in the message to obtain the hidden state corresponding to each byte position in the message. Then, a fully connected layer with a softmax activation function is used to map the hidden state to the label space. The probability of each byte position in the message corresponding to the keyword position label is obtained, and then the process proceeds to step A2-3. Step A2-3. Determine the position of each candidate key field in the message based on the probability that each byte position in the message corresponds to the keyword position tag; Step A3. For each message group in each candidate key field position in each message is obtained by union, that is, to obtain each global candidate key field position; Step B. Based on the positions of each global candidate key field, apply probabilistic reasoning and loop iteration to obtain the positions of each target key field, and then proceed to Step C. In the above step B, the positions of the global candidate key fields are initialized to form a candidate set, and the number of iterations is , then execute the following steps B1 to B3 to obtain the position of each target key field; Step B1. Apply probabilistic reasoning to calculate the probability that each global candidate key field position in the candidate set constitutes the target key field position, and proceed to step B2; Step B2. Select the maximum probability as the The probability corresponding to the iteration is If it is greater than 1, then the Is the probability corresponding to the iteration greater than the If the probability corresponding to the iteration is [1], the global candidate key field position corresponding to the maximum probability in the current iteration is selected to form the target key field position, and the global candidate key field position corresponding to the maximum probability is deleted from the candidate set, the candidate set is updated, and then the process goes to step B3; Otherwise, end the processing of step B; like If it is equal to 1, the global candidate key field position corresponding to the maximum probability in the current iteration is selected to form the target key field position, and the global candidate key field position corresponding to the maximum probability is deleted from the candidate set, the candidate set is updated, and then the process goes to step B3; Step B3. Determine whether there is a global candidate key field position in the candidate set, if so, The value of is updated by adding 1, and the process returns to step B1; Step C. Based on the location of each target key field, obtain the numerical vector corresponding to each message in the target network interaction traffic. Then, use the alternating optimization training method of the autoencoder and clustering algorithm to parse the network protocol format for each message under each cluster.

2. The network protocol reverse analysis method according to claim 1, characterized in that: In step A2, for each message group, the bit length of the message with the longest number of bits in the message group is first updated to a multiple of 8 by padding the end with at least 0, and the target bit length corresponding to the message group is formed. Then, the bit lengths of the remaining messages in the message group are updated to the target bit length by padding the end with 0, thereby achieving the unification of the bit lengths of the messages in the message group.

3. The network protocol reverse analysis method according to claim 1, characterized in that: In step A2-3, the probability of each byte position in the message corresponding to a keyword position label is analyzed using a conditional random field (CRF) to determine the independent byte positions corresponding to the keyword position label and the adjacent continuous byte group positions, which constitute the candidate key field positions in the message.

4. The network protocol reverse analysis method according to claim 1, characterized in that: In step B1, for each global candidate key field position in the candidate set, the following steps B1-1 to B1-3 are executed to obtain the probability that the global candidate key field position constitutes the target key field position; Step B1-1. Temporarily cluster each message in the target network interactive traffic that has a non-zero numerical sequence corresponding to the global candidate key field position based on the numerical sequence of each message in the target network interactive traffic. Messages with the same non-zero numerical sequence are grouped into the same cluster. Temporary clusters corresponding to the global candidate key field position are obtained, and then the process proceeds to Step B1-2. Step B1-2. Calculate the message similarity probability corresponding to each temporary cluster corresponding to the global candidate key field position , telecoupling probability , structural consistency probability , and calculate the dimension predicate probability corresponding to the global candidate key field position , then proceed to step B1-3; Step B1-3. Use the following formula: ； ； Get intermediate parameters 、 , and further according to the following formula: ； Obtain the probability that the global candidate key field position constitutes the target key field position ,in, Indicates the global candidate key field position, Indicates the probability of similarity of preset messages; represents the preset telecoupling probability; represents the probability of consistency of the preset structure; Indicates the first position of the global candidate key field The message similarity probability corresponding to the temporary clusters, Indicates the first position of the global candidate key field The probability of telecoupling corresponding to a temporary cluster, Indicates the first position of the global candidate key field The structural consistency probability corresponding to the temporary clusters, Represents the probability of a preset dimension predicate.

5. A network protocol reverse parsing method according to claim 4, characterized in that: In step B1-2, for each temporary cluster corresponding to the global candidate key field position, the following is performed: About message similarity probability : Group each message in the temporary cluster into pairs to obtain each first subgroup. Based on the values ​​corresponding to each byte position in the message, calculate the ratio of the number of identical values ​​between the two messages in the first subgroup to the total number of bytes in the two messages to form the score corresponding to the first subgroup and construct the error mismatch rate. The function is as follows: ； Among the other temporary clusters corresponding to the global candidate key field positions, select the one with the smallest difference in the number of messages in the temporary cluster. Group the messages in the temporary cluster with the messages in the selected other temporary clusters in pairs to form each second sub-group. According to the values ​​corresponding to the byte positions in the messages, calculate the ratio of the number of identical values ​​between the two messages in the second sub-group to the total number of bytes of the two messages to form the score corresponding to the second sub-group, and construct the error matching rate. The function is as follows: ； based on ,get Function curve and Intersection value of the function curve , then press , obtain the message similarity probability corresponding to the temporary cluster ; On the probability of telecoupling :First, define each message in the temporary cluster whose starting IP address is the client as each message a, and use the number of message a groups as parameters ; Then, the starting address of each message a corresponding to the server is determined from the target network interaction traffic, and defined as each message b; then, based on the temporary clusters corresponding to the positions of the global candidate key fields, the maximum number of messages belonging to the same temporary cluster in each message b is determined as the parameter , and finally according to the following formula: ； Obtain the remote coupling probability corresponding to the temporary clustering ; On the probability of structural consistency : Get the average original length of each message in the temporary cluster , and obtain the average value of the difference between the original length of each message in the temporary cluster and the longest length , and then according to the following formula: ； Obtain the structural consistency probability corresponding to the temporary clustering .

6. A network protocol reverse parsing method according to claim 4, characterized in that: In step B1-2, for the global candidate key field position, the following is performed: First, according to the following formula: ； ； get and ,in, It represents the reciprocal of the average number of messages contained in a cluster. Indicates the proportion of clusters containing only one message among the temporary clusters. Indicates the number of temporary clusters corresponding to the global candidate key field position, Indicates the total number of messages in each temporary cluster corresponding to each global candidate key field position, Indicates the number of temporary clusters containing only one message corresponding to the position of the global candidate key field; Then according to the following formula: ； Get the dimension predicate probability corresponding to the global candidate key field position ,in, 、 These are all preset threshold parameters.

7. A network protocol reverse parsing method according to claim 1, characterized in that: The step C includes the following steps C1 to C7; Step C1. For each message in the target network interaction traffic, obtain the value of each target key field position of the message, sequentially construct the numerical vector corresponding to the message, and then obtain the numerical vector corresponding to each message, and then use the mean square error as the reconstruction loss function , train the autoencoder and then proceed to step C2; Step C2. Apply the encoder in the trained autoencoder to obtain the low-dimensional embedding vector corresponding to each message's numerical vector. Then, based on a preset number of clusters, use the K-means clustering algorithm to cluster these low-dimensional embedding vectors, obtaining cluster centers. The process then proceeds to Step C3. Step C3. Connect a clustering layer after the encoder in the autoencoder and define the parameters in the clustering layer as the cluster centers, then proceed to Step C4. Step C4. Fix the autoencoder parameters and apply the encoder in the autoencoder to obtain the low-dimensional embedding vector corresponding to each message's numerical vector. Then, apply the K-means clustering algorithm to cluster the low-dimensional embedding vectors corresponding to each message, obtain the cluster centers, and update the cluster centers in the clustering layer. Then, proceed to Step C5. Step C5. Fix the cluster centers in the clustering layer and use the reconstruction loss function of the autoencoder Clustering loss with K-means clustering algorithm The sum of the two messages is used to construct a joint loss to minimize the joint loss. The numerical vectors corresponding to the messages are used to train the autoencoder to obtain the trained autoencoder, and then step C6 is entered. Step C6. Determine whether the preset iteration exit condition is met, if so, proceed to step C7; otherwise, return to step C4; Step C7. For each cluster, align the messages in the cluster starting from the first position in order to obtain the fixed fields that are the same among the messages and the variable fields that are different from each other. Based on the positions of each target key field, determine the key field Q in the fixed field that corresponds to the corresponding target key field position, and determine the key field L in the variable field that corresponds to the corresponding target key field position for each different content. Then, based on the key field Q and each key field L, construct the same network protocol format corresponding to each message in the cluster; then obtain the network protocol format corresponding to each cluster, that is, realize the parsing of the network protocol format of each message in the target network interaction traffic.

8. A network protocol reverse parsing method according to claim 7, characterized in that: In step C6, the preset iteration exit condition is a preset maximum number of iterations, or the joint loss no longer continues to decrease.

Images