A network protocol reverse parsing method based on deep learning and graph neural network

Abstract

This paper discloses a network protocol reverse parsing method based on deep learning and graph neural networks. The method includes basic field detection: using sliding window embedding, bidirectional LSTM encoding and knowledge-enhanced CRF decoding to extract byte-level features from binary data streams and output a structured field annotation sequence; protocol format clustering based on the sequence: calculating multi-dimensional similarity using an improved Needleman-Wunsch algorithm, combining it with an LSH-optimized dynamic density clustering algorithm to automatically classify unknown protocols and output protocol clusters; and composite structure parsing based on the protocol clusters: constructing a protocol syntax tree based on a graph neural network, performing multiple rounds of message passing through a graph attention network (GAT) attention mechanism, identifying nested structures and recursively parsing them to generate a multi-level protocol syntax tree. This method can significantly improve the automation and accuracy of complex protocol parsing.

Description

Technical Field

[0001] The present invention belongs to the technical field of network protocol reverse parsing, and in particular relates to a network protocol reverse parsing method based on deep learning and graph neural networks. Background Art

[0002] With the rapid development of modern communications technology, network protocol reverse engineering (PRE) has become a vital tool in network security research, revealing the format, semantics, and behavior of unknown protocols, especially in the absence of protocol specifications. Existing PRE research primarily focuses on parsing textual and binary protocols. Binary protocols are considered more challenging due to their compact format, lack of distinct field delimiters, and difficulty in inferring semantics.

[0003] Traditional methods for protocol format segmentation primarily include alignment-based and probability-based techniques. Alignment-based methods extract field keywords through sequence comparison techniques (such as multiple sequence alignment algorithms), but are susceptible to noise and have high computational complexity. Probability-based methods leverage the frequency distribution of fields in messages, locating them through techniques such as frequent item set mining and information entropy. However, these methods are inefficient when processing large-scale messages and struggle to extract the complete protocol format.

[0004] Semantic inference is another important task in protocol reverse engineering. Existing methods primarily include type matching and heuristic mining. Type matching infers semantics by identifying common data types (such as integers, floating-point numbers, and timestamps) in protocol fields, but this approach is limited to common, reusable field types. Heuristic mining, on the other hand, discovers fields such as length and checksum by calculating relationships between fields. However, this requires specialized algorithm design and is time-consuming.

[0005] In recent years, deep learning models have been increasingly used in protocol reverse engineering. For example, by leveraging multi-scale feature extraction and knowledge-driven traffic simulation techniques, deep learning models can be used to segment and infer the format of binary protocols. These models have demonstrated high precision and recall in segmenting and inferring the format of some unknown protocols. However, this approach relies heavily on high-quality training data and has limitations when processing complex semantically related fields such as length, offset, and checksum.

[0006] In general, existing technologies still have problems in protocol reverse parsing, such as strong data dependence, insufficient adaptability, and limited semantic parsing depth. There is an urgent need for more efficient and general technical solutions to meet the diverse needs of protocol reverse engineering. Summary of the Invention

[0007] To address the above-mentioned defects, the present invention provides a network protocol reverse parsing method based on deep learning and graph neural networks, and proposes a three-level progressive parsing architecture. Through the improved Needleman-Wunsch algorithm, knowledge-enhanced CRF and graph neural network recursive parsing mechanism, the automation level and accuracy of complex protocol parsing are significantly improved.

[0008] A network protocol reverse parsing method based on deep learning and graph neural network, including:

[0009] Step 1: Basic field detection: Sliding window embedding, bidirectional LSTM encoding, and knowledge-enhanced CRF decoding are used to extract byte-level features from the binary data stream and output a structured field annotation sequence.

[0010] Step 2: Clustering the protocol formats based on the sequence: Calculating multi-dimensional similarity using the improved Needleman-Wunsch algorithm, combining it with the LSH-optimized dynamic density clustering algorithm to automatically classify unknown protocols and output a protocol cluster;

[0011] Step 3: Perform composite structure analysis based on the protocol cluster: construct a protocol syntax tree based on the graph neural network, perform multiple rounds of message passing through the graph attention network (GAT) attention mechanism, identify nested structures and recursively parse them, and generate a multi-level protocol syntax tree to characterize the protocol structure.

[0012] Preferably, in step 1.1, a sliding window process is performed on the input binary data stream, and the embedding vectors of the two bytes before and after each central byte are concatenated to generate a 640-dimensional combined feature vector to construct a local context; in step 1.2, global context features are extracted through a bidirectional LSTM network, and a 512-dimensional latent state sequence is output; in step 1.3, a knowledge-enhanced CRF decoder is used to adjust the label transfer weight through a dynamic priority injection mechanism, and a four-tuple sequence of field starting offset, length, type and confidence is output.

[0013] As a preferred method, the bidirectional LSTM network adopts a bidirectional encoding mechanism, and the forward LSTM generates a hidden state sequence along the direction of the data flow. , backward LSTM reverse processing generates , by splicing the two-way hidden state, a comprehensive feature representation containing global context information is obtained .

[0014] Preferably, the CRF decoder defines the state transition matrix , K is the number of label categories; the dynamic priority injection mechanism is introduced to improve the transfer score calculation ,in is the predefined domain knowledge weight matrix, is a learnable scaling factor; label prediction uses the BIO annotation system, and the conditional probability calculation is expressed as:

[0015] ,

[0016] Where, is the tag-related weight vector; finally, the optimal tag sequence is obtained through Viterbi algorithm decoding, and after merging consecutive tags of the same type, a structured field description tuple (start offset, field length, type identifier, confidence) is output.

[0017] Preferably, step 2 specifically includes: step 2.1, converting the field annotation sequence into a mixed representation of normalized position and type; step 2.2, performing multi-dimensional alignment based on the improved Needleman-Wunsch algorithm, and calculating the protocol similarity by combining the type matching score function and the nonlinear length penalty mechanism; step 2.3, using the dynamic density clustering algorithm optimized by local sensitive hashing (LSH), and realizing fine-grained division of protocol clusters through adaptive adjustment of the neighborhood radius.

[0018] As a preferred method, the construction of the multi-feature fusion similarity function in step 2.2 specifically includes: step 2.21, using the improved Needleman-Wunsch algorithm to quantify the similarity between the two labeled sequences. For global alignment, the dynamic programming recursive formula is:

[0019] ,

[0020] The type matching score function is defined as:

[0021] ,

[0022] Where, the gap penalty coefficient is .

[0023] Step 2.22: After alignment is completed, calculate the comprehensive similarity using the following formula:

[0024] ,

[0025] Where, is the position weight, is the length penalty factor, is the type matching score function, K is the number of matching field pairs obtained after field alignment, and Respectively represent Match fields in and The length of the field in and Respectively represent Match fields in sequence and The field type in .

[0026] As a preference, in step 2.3, the adaptive neighborhood adjustment mechanism based on local sensitive hashing (LSH) includes: initial neighborhood radius Take the i-th percentile of the similarity between two samples in the data set, and shrink dynamically according to the cluster density during the iteration process:

[0027] ,

[0028] in is the shrinkage rate, is the number of samples of identified dense clusters, is the total number of samples; by dynamically adjusting the strategy, dense protocols are divided into fine-grained segments while avoiding over-segmentation of sparse protocol clusters.

[0029] Preferably, step 3 includes:

[0030] Step 3.1: Convert the field annotation sequence output in step 2 into the initial graph structure , where the node set V includes four tuples: offset, length, type, confidence, and edge set Contains three types of connection relationships: physical connection edges of adjacent fields, type transition probability exceeding the threshold logically associated edges, nested edges generated by predefined nested structure patterns; Step 3.2, multiple rounds of message updates of node features are performed through the graph attention network GAT, The feature update formula of layer node i is:

[0031] ,

[0032] Where, Indicates features, superscript Indicates the Layer, subscript Indicates the nodes, Indicates the The set of neighboring nodes connected to each node, Representing the graph neural network The learnable weight matrix of the layer;

[0033] Attention weight Calculated by the bidirectional attention mechanism:

[0034] ;

[0035] In step 3.3, the basic detection module is recursively called to parse the nested fields and the multi-level syntax tree is integrated through the graph fusion operator.

[0036] As a preference, in step 3, the recursive parsing process includes: step 3.31, when the node type confidence When it is greater than the set threshold, it is determined to be a composite structure node and its value part byte stream is extracted ; Step 3.32, recursively call the basic detection module to Parse and generate subgraphs ; Step 3.33, integrate the subgraph into the parent graph structure through the graph fusion operator: , where the projection function Convert the subgraph node offset to the local coordinate relative to the parent node; step 3.34, the recursive process continues until all leaf nodes meet the basic type conditions , generating a protocol syntax tree that supports arbitrary nesting depth.

[0037] The present invention also discloses a network protocol reverse parsing system based on deep learning and graph neural network, comprising:

[0038] The basic field detection module uses a sliding window embedding unit, a bidirectional LSTM encoding unit, and a knowledge-enhanced CRF decoding unit to form a processing pipeline. It is used to extract byte-level features from the input binary data stream and output a structured field annotation sequence containing the starting offset, length, type, and confidence level.

[0039] The protocol format clustering module includes a multi-sequence alignment unit and a dynamic density clustering unit. The multi-sequence alignment unit uses an improved Needleman-Wunsch algorithm to calculate the multi-dimensional similarity of the annotated sequences, and the dynamic density clustering unit uses the LSH-optimized DBSCAN algorithm to automatically classify protocol clusters.

[0040] The composite structure parsing module includes a protocol syntax tree construction unit, a graph attention network unit and a recursive parsing unit. The protocol syntax tree construction unit converts the protocol cluster into an initial graph structure. The graph attention network unit updates node features through multiple rounds of message passing. The recursive parsing unit implements recursive detection and subgraph fusion on nested structures.

[0041] The beneficial effects of the present invention are:

[0042] The protocol parsing system / method disclosed in the present invention significantly improves the automation level and parsing accuracy of protocol analysis by introducing deep learning and graph neural network technology on the basis of traditional protocol parsing technology.

[0043] (1) This invention adopts a three-level progressive architecture, covering three core modules: basic field detection, protocol format clustering, and composite structure analysis, forming an efficient and accurate protocol analysis closed loop. This architecture can share knowledge across multiple analysis levels and optimize the protocol analysis process through repeated feedback, thereby improving the accuracy and consistency of analysis results.

[0044] (2) This invention uses a protocol format clustering method based on a combination of an improved Needleman-Wunsch algorithm and an adaptive density clustering algorithm. By comprehensively comparing cross-field features, the system can automatically discover the format of unknown protocols and effectively identify version differences, thereby providing prior information for protocol parsing and significantly improving the automation and accuracy of parsing.

[0045] (3) The composite structure parsing process of the present invention uses a graph neural network (GNN) architecture combined with a graph attention network (GAT) mechanism to achieve recursive parsing of complex nested structures such as TLVs and LVs. This innovative solution allows the system to automatically rebuild the protocol syntax tree when faced with nested structures and continuously adjust the parsing strategy when the nesting level is uncertain, supporting the parsing of protocols with arbitrarily deep nesting.

[0046] (4) To improve the accuracy of field type prediction, this paper uses a knowledge-enhanced conditional random field (CRF) for label sequence decoding. By introducing a domain knowledge injection mechanism based on the traditional CRF model, the system can dynamically adjust the field type priority in complex protocol environments, further improving the accuracy of complex data stream parsing.

[0047] (5) The present invention uses a cross-layer feature fusion mechanism to enable outputs from different layers to be mutually communicated and optimized, ensuring that the system can extract information layer by layer, from low-level byte features to high-level protocol structures. At each stage of protocol parsing, higher-level semantic information extracted in the previous stage can be utilized, thereby maintaining parsing accuracy and consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments.

[0049] Figure 1 It is a three-level progressive framework diagram of an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of a basic field detection process according to an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of a protocol format clustering process according to an embodiment of the present invention;

[0052] Figure 4It is a schematic diagram of the composite structure analysis process of an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention provide a method for making the purpose, technical solutions and advantages of the present invention more clearly understood. The present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Example 1

[0054] like Figures 1 to 4 As shown, this embodiment discloses a network protocol reverse parsing method based on deep learning and graph neural network, including the following steps:

[0055] Step 1, basic field detection: Sliding window embedding, bidirectional LSTM encoding and knowledge-enhanced CRF decoding are used to extract byte-level features from the binary data stream and output a structured field annotation sequence.

[0056] like Figure 2 As shown, the basic field detection module proposed in the present invention first performs byte-level segmentation processing on the input binary data stream to generate an ordered byte sequence , where each byte value By constructing a trainable embedding matrix of dimension 256×128 , mapping each byte value to a 128-dimensional vector representation To capture the characteristic patterns of data types across bytes, a five-byte sliding window mechanism is used to construct the local context: for each central byte , concatenate the embedding vectors of the two bytes before and after it to form a 640-dimensional combined feature , zero padding is used at the boundary position to keep the window size constant.

[0057] In the context feature extraction stage, a bidirectional long short-term memory network (BiLSTM) is used for sequence modeling, and the forward LSTM generates a hidden state sequence along the direction of the data flow. , Represents a subsequence from the 1st element to the tth element, generated by backward LSTM reverse processing , Represents the subsequence from the tth element to the nth element. By concatenating the two-way hidden state, a comprehensive feature representation containing global context information is obtained. This bidirectional encoding mechanism can simultaneously utilize the type characteristics of the preceding field and the structural information of the subsequent fields. For example, when identifying floating-point data appearing after the timestamp field, the forward LSTM retains the timestamp features, and the backward LSTM captures the subsequent floating-point patterns.

[0058] In the label sequence decoding stage, the conditional random field (CRF) layer models the legal transition rules of the label sequence through the state transition matrix. Define the state transition matrix (K is the number of label categories), and a dynamic priority injection mechanism is introduced to improve the transfer score calculation .in is the improved state transfer matrix, that is, the updated state transfer matrix; is the matrix index value, is the predefined domain knowledge weight matrix, is a learnable scaling factor. This mechanism automatically adjusts the type priority according to the protocol semantics when the four-byte data satisfies both integer and floating-point characteristics. Label prediction uses the BIO annotation system, and the conditional probability is calculated as follows, where is the label-related weight vector.

[0059] ,

[0060] Where, To get a sequence from a specific tag The label of the previous time step The label transferred to the current time step transfer score; For all possible tag sequences The label of the previous time step The label transferred to the current time step transfer score; is the current time step Input features With a specific tag sequence The label at the current time step Observation score of is the current time step Input features With all possible tag sequences The label at the current time step Observation score. is the tag-related weight vector; finally, the optimal tag sequence is obtained by decoding with the Viterbi algorithm, and the structured field description tuple is output after merging consecutive tags of the same type. Represents the currently predicted label sequence, corresponding to the actually observed protocol field, represents any label sequence in the candidate label sequence space, corresponding to a set of unobserved potential output states, is the total time step of the sequence, that is, the length of the byte sequence to be annotated, is the current time step, Finally, the optimal tag sequence is obtained through Viterbi decoding. Consecutive tags of the same type are merged to output a structured field description tuple (start offset, field length, type identifier, confidence level), for example (0x0004, 4, "Float32", 0.92). This step significantly improves the parsing accuracy of complex protocol data through the synergy of the context window mechanism, dual-stream feature fusion, and knowledge-enhanced CRF.

[0061] Step 2: perform protocol format clustering based on the sequence: calculate multi-dimensional similarity through the improved Needleman-Wunsch algorithm, combine the LSH optimized dynamic density clustering algorithm to realize automatic classification of unknown protocols, and output protocol clusters.

[0062] like Figure 3 As shown, first convert the input four-tuple sequence (starting offset, field length, type identifier, confidence) into a standardized representation , where the unknown type field (UNK) is treated as a wildcard in the type dimension while retaining its length characteristics, forming an intermediate representation that balances structural characteristics and type compatibility. For example, the quadruple (0x0004,4,"Float32",0.92) is converted to ("Float32",0.25), where 0.25 represents the normalized ratio of the field start position to the total protocol length.

[0063] The core of the clustering process is to define the multi-feature fusion similarity function between protocol formats. Given two labeled sequences and , the improved Needleman-Wunsch algorithm is used for global sequence alignment, and its dynamic programming recursive formula is:

[0064] ,

[0065] Represents two protocol annotation sequences Before elements and of The maximum similarity score when globally aligning sequences of elements. Representative sequence The index of The fields, Similarly; : table sequence The index of The fields, Same reason.

[0066] The type matching score function is defined as:

[0067] ,

[0068] Representation sequence The The type identifier of each field. For example, if a field is identified as "Float32", then It's "Float32". Representation sequence The The type identifier of the field. Gap penalty coefficient .

[0069] This method extends the field to include the wildcard UNK to adapt to the results of the aforementioned BiLSTM-CRF algorithm. This method also accommodates custom types that are not basic types. Compared to the traditional NW algorithm, it is more suitable for fuzzy classification in protocol syntax scenarios. The input is no longer a direct message byte sequence, but a four-tuple sequence (start offset, field length, type identifier, confidence level) derived through a deep learning algorithm. Based on the improved NW algorithm, the scoring mechanism is reconstructed and structural awareness is enhanced.

[0070] After alignment is completed, the comprehensive similarity calculation introduces a nonlinear length penalty and position weighting mechanism: ,

[0071] Where, is the position weight, is the length penalty factor, is the type matching score function, K is the number of matching field pairs obtained after field alignment, and Respectively represent Match fields in and The length of the field in and Respectively represent Match fields in sequence and In this embodiment, the position weight Make the protocol header field matching have higher weight, length penalty factor Sensitivity to control structure differences.

[0072] The comprehensive similarity calculation formula disclosed in this application realizes difference perception through an exponential decay function: it accurately reflects the impact of slight differences in field length on the protocol structure, is sensitive to changes in the length of key fields, and tolerates differences in irrelevant fields. For example, the penalty for the difference between 1 byte and 2 bytes (0.24) is significantly higher than that between 100 bytes and 101 bytes (0.005), which is in line with the domain characteristics of protocol analysis; and it has excellent anti-interference ability: through structural penalty balancing the offset caused by field detection errors, the robustness of the overall similarity evaluation can be maintained even when field boundary identification is not completely accurate.

[0073] The clustering algorithm adopts an improved density clustering framework and proposes an adaptive neighborhood adjustment mechanism based on locality sensitive hashing (LSH). A composite hash key is designed based on the mixed characteristics (type, length, position) of the protocol field, and a mathematical correlation is established between the LSH collision probability and the neighborhood radius through the similarity percentile threshold to achieve dynamic adjustment. Initial neighborhood radius Take the 15th percentile of the similarity between two samples in the data set, and dynamically shrink it according to the cluster density during the iteration process:

[0074] ,

[0075] in is the new neighborhood radius, which defines the range of the “neighbors” around a point. As becomes smaller, the clustering becomes more fine-grained. The old (current) neighborhood radius is used. During the iteration process, this value is dynamically adjusted according to the density of the cluster. It is the 15th percentile of the similarity between two samples in the data set. is the shrinkage rate, is the number of samples of identified dense clusters, This dynamic adjustment strategy enables finer-grained segmentation in dense areas (such as the HTTP protocol cluster containing multiple subversions) while avoiding over-segmentation in sparse areas (such as independent protocols such as SSH).

[0076] Step 3: Perform composite structure parsing based on the protocol cluster: Build a protocol syntax tree based on a graph neural network, perform multiple rounds of message passing through the graph attention network (GAT) attention mechanism, identify nested structures, and recursively parse to generate a multi-level protocol syntax tree; the parsing results are fed back to steps 1 and 2 to form a cross-layer closed-loop optimization. The syntax tree can represent the structure of the protocol, including the location, type, length, and nested relationship of the fields. Generating a syntax tree provides a structured representation of the protocol to support applications such as security detection and vulnerability mining. Step 3 specifically includes:

[0077] Step 3.1: Convert the field annotation sequence output in step 2 into the initial graph structure , where the node set V includes four tuples: offset, length, type, confidence, and edge set Contains three types of connection relationships: physical connection edges of adjacent fields, type transition probability exceeding the threshold Logical association edges and nested edges generated by predefined nested structure patterns.

[0078] Step 3.2, use the graph attention network GAT to update node features through multiple rounds of messages, The feature update formula of layer node i is:

[0079] ,

[0080] Where, Indicates features, superscript Indicates the Layer, subscript Indicates the nodes, Indicates the The set of neighboring nodes connected to each node, Representing the graph neural network The learnable weight matrix of the layer, For the Layer Node The eigenvector of is represented by, which is updated as .

[0081] Attention weight Calculated by the bidirectional attention mechanism:

[0082] ,

[0083] Where, The transpose of a learnable attention vector is dot-producted with the concatenation of the transformed node features to calculate the attention score. The weights of this vector are learned during training to determine which feature combinations contribute more to the attention score. Representation node The eigenvector of After a linear transformation (by the weight matrix The result after completion). is a learnable weight matrix that transforms the features of the nodes Projected into a new feature space for better attention calculation. Here is a node In the current GAT layer ( layer). Representation node The eigenvector of After the same linear transformation The result after Similarly, it is a neighbor node The linear transformation result of the features. When calculating the attention, the center node is considered and its neighboring nodes The relationship between them. Representation node The eigenvector of After the same linear transformation This term appears in the summation of the denominator, where Representation node All neighbor nodes. The denominator is used to compare all neighbor nodes with the central node. Normalize the attention score to ensure the attention weight The sum of is 1.

[0084] Step 3.3, recursively call the basic detection module to parse the nested fields and integrate the multi-level syntax tree through the graph fusion operator. The recursive parsing process includes: when the node type confidence When it is greater than the set threshold, it is determined to be a composite structure node and its value part byte stream is extracted ; Recursively call the basic detection module Parse and generate subgraphs ; Integrate the subgraph into the parent graph structure through the graph fusion operator: , Represents the first The parent graph (or current graph) structure in this iteration. This graph contains the parsed protocol fields and the relationships between them. When a nested structure (such as a TLV field) is detected, its value portion needs to be recursively parsed to generate a subgraph. It is the existing graph structure into which this subgraph will be integrated. Represents the first The updated graph structure in the round iteration. This is the newly parsed subgraph (through Indicates successful integration into the parent graph The result is a more complete, multi-level protocol syntax tree. Represents a graph neural network model with parameter set θ. Projection function Convert the subgraph node offset to the local coordinate relative to the parent node; the recursive process continues until all leaf nodes meet the basic type conditions , generating a protocol syntax tree that supports arbitrary nesting depth. Example 2

[0085] This embodiment discloses a network protocol reverse parsing system based on deep learning and graph neural networks. It adopts a three-level progressive architecture and is composed of three core modules: basic field detection, protocol format clustering, and composite structure parsing. The system first performs byte-level feature extraction on the input binary data stream through the basic field detection module. It then uses a three-stage process of sliding window embedding, bidirectional LSTM encoding, and knowledge-enhanced CRF decoding to output structured annotation sequences (including starting offset, length, type, and confidence). These annotation sequences are then passed to the protocol format clustering module, which calculates multi-dimensional similarity using an improved Needleman-Wunsch algorithm and combines it with an LSH-optimized dynamic density clustering algorithm to achieve automatic classification and version identification of unknown protocols. Finally, the composite structure parsing module constructs a protocol syntax tree based on the graph neural network, identifies nested structures such as TLV / LV through the GAT attention mechanism, and uses a recursive parsing engine to achieve multi-level structure reconstruction, supporting the generation of complete syntax trees for protocols with arbitrary depth of nesting.

[0086] The entire system operates through a closed-loop optimization process of "detection → clustering → parsing → re-detection." The basic field detection module provides a four-tuple sequence (including the field's starting offset, length, type, and confidence level) to the clustering and parsing modules. The clustering module leverages the protocol template library to provide the composite parsing module with prior knowledge of protocol structure. The parsing module, in turn, triggers recursive calls when nested structures are detected. This closed-loop process ensures accuracy and consistency throughout the protocol parsing process.

[0087] The system uses layered feature abstraction, gradually extracting and integrating information from the byte level to the field level, then to the protocol level and composite structure level. The output of each layer provides higher-level semantic information to the next layer, ensuring the accuracy of parsing complex protocols. Through cross-layer knowledge sharing, the system can achieve automated parsing from the raw bitstream to the final syntax tree, greatly improving the efficiency and accuracy of protocol parsing.

[0088] This method improves the accuracy of network protocol reverse parsing and reduces the false positive rate. It has important applications in security testing, vulnerability discovery, and network defense. It can help security researchers more quickly and accurately identify and parse unknown protocols, improving overall network security. The innovative and practical nature of this method provides a new technical solution for the field of network security and has broad application prospects.

[0089] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A network protocol reverse parsing method based on deep learning and graph neural network, characterized in that: include: Step 1, basic field detection: Use sliding window embedding, bidirectional LSTM encoding and knowledge-enhanced CRF decoding to extract byte-level features from binary data streams and output structured field annotation sequences. Specifically, Step 1.1: Perform sliding window processing on the input binary data stream, concatenate the embedding vectors of the two bytes before and after each central byte, generate a 640-dimensional combined feature vector, and construct the local context; Step 1.2: extract global context features through a bidirectional LSTM network and output a 512-dimensional latent state sequence; In step 1.3, a knowledge-enhanced CRF decoder is used to adjust the label transfer weight through a dynamic priority injection mechanism, and output a four-tuple sequence of field start offset, length, type, and confidence; Step 2: Clustering the protocol formats based on the sequence: Calculate the multi-dimensional similarity by using the improved Needleman-Wunsch algorithm, and automatically classify the unknown protocols by combining the LSH optimized dynamic density clustering algorithm, and output the protocol cluster. Specifically, Step 2.1: Convert the field annotation sequence into a mixed representation of normalized position and type; In step 2.2, multi-dimensional alignment is performed based on the improved Needleman-Wunsch algorithm, and the protocol similarity is calculated by combining the type matching score function and the nonlinear length penalty mechanism. The construction of the multi-feature fusion similarity function specifically includes: Step 2.21, use the improved Needleman-Wunsch algorithm to identify the two labeled sequences For global alignment, the dynamic programming recursive formula is: , Where, Represents two protocol annotation sequences Before elements and of The maximum similarity score when global sequence alignment is performed on elements, 、 For sequence The elements in are respectively where the type matching score function M is defined as: , In the formula Representation sequence The The type identifier of the field, Representation sequence The Type identifier of the field, gap penalty coefficient ; Step 2.22, after alignment is completed, calculate the comprehensive similarity using the following formula: , Where, is the position weight, is the length penalty factor, is the type matching score function, K is the number of matching field pairs obtained after field alignment, and Respectively represent Match fields in and The length of the field in and Respectively represent Match fields in sequence and The field type in; Step 2.3, using the dynamic density clustering algorithm optimized by local sensitive hashing (LSH), the fine-grained division of the protocol cluster is achieved through adaptive adjustment of the neighborhood radius, including: initial neighborhood radius Take the i-th percentile of the similarity between two samples in the data set, and shrink dynamically according to the cluster density during the iteration process: ,in, is the new neighborhood radius, is the current neighborhood radius, is the shrinkage rate, is the number of samples of identified dense clusters, is the total number of samples; by dynamically adjusting the strategy, dense protocols are divided into fine-grained segments while avoiding over-segmentation of sparse protocol clusters; Step 3: Perform composite structure analysis based on the protocol cluster: construct a protocol syntax tree based on the graph neural network, perform multiple rounds of message passing through the graph attention network (GAT) attention mechanism, identify nested structures and recursively parse them, and generate a multi-level protocol syntax tree to characterize the protocol structure.

2. The method according to claim 1, characterized in that The bidirectional LSTM network adopts a bidirectional encoding mechanism, and the forward LSTM generates a hidden state sequence along the direction of the data flow: , the backward LSTM reversely processes the generated sequence: , represents the subsequence from the 1st element to the tth element, Represents the subsequence from the tth element to the nth element, and obtains a comprehensive feature representation containing global context information by splicing the two-way hidden state .

3. The method according to claim 1, characterized in that The CRF decoder defines the state transition matrix , K is the number of label categories; the dynamic priority injection mechanism is introduced to improve the transfer score calculation ,in is the updated state transition matrix, is the predefined domain knowledge weight matrix, is the index value of the matrix, is a learnable scaling factor; label prediction uses the BIO annotation system, and the conditional probability calculation is expressed as: ， Where, To get a sequence from a specific tag The label of the previous time step The label transferred to the current time step transfer score; For all possible tag sequences The label of the previous time step The label transferred to the current time step transfer score; is the current time step Input features With a specific tag sequence The label at the current time step Observation score of is the current time step Input features With all possible tag sequences The label at the current time step Observation score of is the total time step of the sequence, that is, the length of the byte sequence to be annotated, is the current time step.

4. The method according to claim 1, wherein Step 3 includes: Step 3.1: Convert the field annotation sequence output in step 2 into the initial graph structure: , where the node set V includes four tuples: offset, length, type, confidence, and edge set Contains three types of connection relationships: physical connection edges of adjacent fields, type transition probability exceeding the threshold Logical association edges and nested edges generated by predefined nested structure patterns; Step 3.2, use the graph attention network GAT to update node features through multiple rounds of messages. The feature update formula of layer node i is: , where Indicates features, superscript Indicates the Layer, subscript Indicates the nodes, Indicates the The set of neighboring nodes connected to each node, Representing the graph neural network The learnable weight matrix of the layer, For the Layer Node The eigenvector representation of is updated to ; Attention weight Calculated by the bidirectional attention mechanism: Where, is the transpose of a learnable attention vector; For nodes The eigenvector of After a linear transformation, the result is For nodes The eigenvector of After the same linear transformation The result after for point The eigenvector of After the same linear transformation The result after In step 3.3, the basic detection module is recursively called to parse the nested fields and the multi-level syntax tree is integrated through the graph fusion operator.

5. The method according to claim 4, characterized in that In step 3, the recursive parsing process includes: Step 3.31, when the node type confidence When it is greater than the set threshold, it is determined to be a composite structure node and its value part byte stream is extracted ; Step 3.32, recursively call the basic detection module to Parse and generate subgraphs ; Step 3.33: Integrate the subgraph into the parent graph structure through the graph fusion operator: , where The recursive parsing process The parent graph structure in the round iteration, After being processed by the graph fusion operator, the The updated graph structure in the round iteration, is a graph neural network model with parameter set θ and projection function Convert the subgraph node offset to local coordinates relative to the parent node; Step 3.34, the recursive process continues until all leaf nodes meet the basic type conditions , generating a protocol syntax tree that supports arbitrary nesting depth.

6. A network protocol reverse parsing system based on deep learning and graph neural network, characterized by: Based on the method according to claim 1, it includes: a basic field detection module, which uses a sliding window embedding unit, a bidirectional LSTM encoding unit and a knowledge-enhanced CRF decoding unit to form a processing pipeline, which is used to extract byte-level features from the input binary data stream and output a structured field annotation sequence containing a starting offset, length, type and confidence; a protocol format clustering module, which includes a multi-sequence alignment unit and a dynamic density clustering unit, the multi-sequence alignment unit uses an improved Needleman-Wunsch algorithm to calculate the multi-dimensional similarity of the annotation sequence, and the dynamic density clustering unit realizes automatic classification of protocol clusters through the LSH-optimized DBSCAN algorithm; a composite structure parsing module, which includes a protocol syntax tree construction unit, a graph attention network unit and a recursive parsing unit, the protocol syntax tree construction unit converts the protocol cluster into an initial graph structure, the graph attention network unit updates the node features through multiple rounds of message passing, and the recursive parsing unit implements recursive detection and subgraph fusion on the nested structure.

Images