A network protocol format inference method and system based on field pattern

Abstract

The present invention provides a method and system for inferring network protocol formats based on field patterns, comprising: acquiring network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and generating sequence labels based on the target unknown protocol; calculating the covariance of a target domain of the target unknown protocol with a pre-acquired source domain of a known protocol based on the sequence labels; selecting the corresponding source domain as the optimal source for the target domain when the covariance is minimized; and inferring the message format of the target unknown protocol using a deep transfer learning model based on the selected optimal source. This invention solves the problems of inaccurate and complex reverse inference of network protocol formats currently encountered.

Description

Technical Field

[0001] The present invention relates to the technical field of communication protocol reverse engineering, and in particular to a network protocol format inference method and system based on field patterns. Background Art

[0002] In practice, protocols are a crucial component of industries such as communications, industry, and power systems. They play a crucial role in establishing communication links between various components within a system. In addition to basic functions such as exchanging data and information, some protocols even provide the ability to control remote devices. Protocol reverse engineering is the process of analyzing unknown or poorly documented communication protocols to uncover their internal syntax and semantics. Protocol reverse engineering is crucial for network security research, malware analysis, network traffic analysis, and network protocol interoperability testing. By understanding and analyzing protocols, researchers can better respond to cyberattacks, resolve communication issues, and improve the performance and security of network communications.

[0003] In the application of network protocols, international standards have been established for some commonly used protocols, such as FTP and SSL. This has led to the development of security tools specific to specific protocol standards. However, in actual use, vendors may customize standard protocols to meet their specific needs. Furthermore, there are many private protocols used internally by vendors that are not publicly available. With the increasing diversification of communication protocols and application scenarios, the closed-source heterogeneity of implementation methods, the increasing complexity of protocol behavior and functionality, and the evolving nature of protocol security attacks and defenses, protocol reverse engineering faces significant challenges.

[0004] In protocol reverse engineering, traffic-based protocol reverse engineering is a technique used to understand and restore communication protocols by analyzing network traffic without accessing the protocol's specific implementation source code or documentation. This process involves researchers attempting to understand and analyze the protocol's internal workings, typically through data interception and analysis, data parsing, state machine analysis, reverse engineering, and vulnerability analysis. This method effectively combats closed-source protocol implementations in diverse scenarios. This approach involves reconstructing the protocol's design using only traffic traces, without the support of protocol documentation or source code. It analyzes intercepted packets of the target protocol to identify components such as the protocol header, fields, and payload, and further determines the protocol type, message format, state machine, and communication mode. This provides a solid foundation for understanding the protocol's state transitions and behavior during communication, and for identifying and analyzing potential vulnerabilities or security issues within the protocol.

[0005] Traffic-based protocol reversal is a technique for inferring the structure and format of protocols by analyzing network traffic. It primarily relies on aligning and clustering captured network packets based on similarity. It then aligns byte sequences within the packets to discover recurring field patterns or structures (such as byte frequency, information entropy, and linear / nonlinear relationships). Currently, many protocol reversal tools are available for protocol format inference, such as Netzob and Netplier. Netzob aligns and clusters messages pairwise and then infers the protocol message format using field patterns such as field length, field order, and inter-field relationships. Netplier first builds a probabilistic keyword spotting system using four heuristic rules, then implements clustering. Finally, it infers the message format using field patterns (such as static fields with specific values ​​and dynamic fields with a list of potential values). BinaryInferno implements various atomic detectors (such as floating point detectors, timestamp detectors, and length detectors) derived from these heuristic rules. It then implements a detector ensemble for protocol field inference using a directed acyclic graph approach. Discoverer uses token patterns to tokenize bytes using simple semantics (e.g., text or binary), then clusters messages, and finally employs statistical analysis (e.g., byte frequency, information entropy, etc.) to identify relationships and structures between fields. Nemesys uses the intrinsic structure of individual messages to infer field boundaries from the distribution of value variations to identify fields in binary protocols. It introduces a format matching score, a quantitative measure of message format inference, which cannot handle the diversity of protocol formats. FieldHunter uses the general semantics and characteristics of specific common fields, such as message length or host ID, and identifies fields by establishing statistical correlations between these types and byte values.

[0006] The above methods all adopt a heuristic rule-based approach to classify messages into different clusters and infer the message syntax of the protocol based on preset and fixed field patterns or structures. Although these methods recover the syntax of unknown messages, their incomplete and inflexible field patterns lead to low accuracy. First, they use fixed patterns derived from heuristic rules to identify fields, limiting their applicability to specific types of protocols, as other protocols may lack predefined patterns. For example, since the timestamps used by the TCP protocol do not conform to standard time formats such as NTP or Unix timestamps, BinaryInferno's preset timestamp detector cannot recognize them. Second, fixed patterns with thresholds are inaccurate in protocol-typical field identification. Existing technologies that use a unified approach to standardize all protocols may lead to misunderstandings of protocol specifications.

[0007] Traditional methods use limited fixed patterns or heuristic rules to identify protocol formats. For example, Netplier uses four heuristic rules to identify key fields, while BinaryInferno has five built-in fixed patterns to identify typical fields. With the emergence of new protocols and the evolution of existing protocols, communication protocols have become extremely diverse, and each protocol can be designed with different fields and structures. Existing protocols contain a variety of field patterns. Therefore, rule-based heuristic methods cannot cover all field patterns. Existing protocols exhibit a wide range of field patterns, making it complex and labor-intensive to explore predefined patterns between these fields. For example, Figure 2 The syntax of a Modbus query message, a HART_IP direct request message, and an IEC60870_104 message segmentation and syntax is shown. The f1 field in a Modbus message and the f5 field in a HART_IP message can be accurately identified based on simple and clear patterns. However, the inherent patterns of the f3, f4, and f5 fields of the Modbus protocol are cryptic and difficult to grasp. Therefore, heuristic rule-based methods cannot explain all field patterns.

[0008] Because different protocols have different field patterns, these tools typically rely on preset and fixed field patterns to identify and analyze packets. This lacks flexibility and broad applicability, limiting the benefits of field segmentation. For example, because the TCP timestamp, used to improve network communication performance and reliability, does not conform to standard time formats like NTP or Unix timestamps, it cannot be identified using BinaryInferno's custom timestamp detector. Furthermore, some heuristic-based methods use thresholds to identify fields, which can increase recognition errors when working with unknown or non-standard protocols because these thresholds are not applicable in all cases. While all three protocols include the pattern y = kx + b, there are differences between the three patterns. In Hart_IP and IEC 60870_104, 'x' represents the length of the subsequent field, including the field itself, while Modbus represents the length of the sequence following the field. Fieldhunter uses a fixed threshold to fit a linear correlation between field values ​​and actual message size, which cannot accurately identify these fields. Protocol design and implementation typically follow the requirements of the functional application and are based on the specifications and knowledge of existing protocols. This allows for greater similarity or shared patterns between protocols. For example, the recognition pattern of the Length field in the Modbus protocol follows a linear fit between the message length and the length field value, which is the same in the AMQP and IEC104 protocols. Previous methods used statistical analysis to infer the protocol format based on general field patterns summarized from protocols with specifications. However, the significant differences between various protocols limit the expansion of field patterns, thereby reducing the accuracy of protocol understanding. Summary of the Invention

[0009] The present invention provides a network protocol format inference method and system based on field patterns, which are used to solve the problems of inaccurate and high complexity of the existing network protocol format reverse inference.

[0010] The present invention provides a network protocol format inference method based on field patterns, comprising:

[0011] Acquire network traffic, extract a target unknown protocol based on the network traffic, and perform decimal integer conversion and sequence label generation based on the target unknown protocol;

[0012] Based on the sequence tags, covariance calculation is performed on the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol, and when the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain;

[0013] The target unknown protocol message format is inferred through a deep transfer learning model based on the selected best source.

[0014] According to a field pattern-based network protocol format inference method provided by the present invention, obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol, specifically includes:

[0015] Obtaining network traffic, extracting data packets of a target unknown protocol from the network traffic, and grouping them according to protocol type;

[0016] Use a preset network protocol analysis tool to traverse each layer of the target unknown protocol data packet, use depth-first search to delete and sort duplicate data, extract information from the target layer, and find the field value and field position of the bytes in the target unknown protocol data packet;

[0017] The field value is converted to a decimal integer from 0 to 255, and a sequence tag is generated based on the field position and syntax.

[0018] According to a field pattern-based network protocol format inference method provided by the present invention, the covariance calculation is performed on the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol based on the sequence tag, and when the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain, specifically comprising:

[0019] Aligning the target domain of the target unknown protocol with the second-order statistical features of the source domain distribution of the pre-acquired known protocol through a linear transformation method based on the sequence tags;

[0020] Calculating the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result;

[0021] Put the calculated covariance into a pre-established empty list and find the minimum covariance value in the empty list;

[0022] In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain.

[0023] According to a field pattern-based network protocol format inference method provided by the present invention, the covariance between the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol is calculated based on the alignment result using a preset covariance calculation model, specifically including:

[0024] Based on the alignment results, the message length data related to the known protocol and the target unknown protocol is calculated through a preset covariance calculation model and cluster analysis is performed to generate cluster analysis results;

[0025] Based on the cluster analysis results, label information is generated for the protocol within the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connections between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data. The target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data;

[0026] A covariance is calculated based on the source domain data and the target domain data.

[0027] According to a field pattern-based network protocol format inference method provided by the present invention, the inference of the target unknown protocol message format based on the selected best source through a deep transfer learning model specifically includes:

[0028] Based on the selected best source, a deep transfer learning model is used to cluster the target unknown protocol message length to obtain clustered homogeneous messages;

[0029] After converting the bytes of the cluster homogeneous message into decimal integers, performing grammatical feature extraction through the base network of the deep transfer learning model;

[0030] The pre-extracted prior syntax of the known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

[0031] According to a method for inferring a network protocol format based on a field pattern provided by the present invention, the method of performing grammatical feature extraction through a basic network of a deep transfer learning model specifically includes:

[0032] The input vector is converted into hidden features through the linear layer in the basic network structure of the deep transfer learning model;

[0033] Based on the hidden features, a nonlinear transformation is introduced through the ReLU activation function in the basic network structure of the deep transfer learning model, and a higher-dimensional output representation in the output protocol field is output to obtain grammatical features.

[0034] The present invention also provides a network protocol format inference system based on field patterns, the system comprising:

[0035] a data processing module, configured to obtain network traffic, extract a target unknown protocol based on the network traffic, and perform decimal integer conversion and sequence label generation based on the target unknown protocol;

[0036] a source domain selection module, configured to calculate the covariance of the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol based on the sequence tag, and select the corresponding source domain as the optimal source of the target domain when the covariance is minimized;

[0037] The knowledge transfer module is used to infer the target unknown protocol message format through a deep transfer learning model based on the selected best source.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the network protocol format inference method based on field patterns as described above is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for inferring a network protocol format based on a field pattern.

[0040] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for inferring a network protocol format based on a field pattern.

[0041] This paper provides a field pattern-based network protocol format inference method and system. By combining prior grammatical knowledge of known protocols, deep transfer learning is used to more accurately understand unknown protocols. Unlike traditional methods, this approach first analyzes the domain differences between the knowledge base and the target unknown protocol to identify the optimal prior knowledge domain. Deep transfer learning then mines grammatical patterns of known protocols to improve the accuracy of inferring the unknown protocol grammar and reduce complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a flow chart of a network protocol format inference method based on field patterns provided by the present invention.

[0044] Figure 2 It is a schematic diagram of prior knowledge learning and transfer in the existing technology.

[0045] Figure 3 This is a diagram of the network protocol format inference system architecture based on field patterns provided by the present invention.

[0046] Figure 4 This is a module connection diagram of a network protocol format inference system based on field patterns provided by the present invention.

[0047] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention.

[0048] Reference numerals: 110: data processing module; 120: source domain selection module; 130: knowledge transfer module; 510: processor; 520: communication interface; 530: memory; 540: communication bus. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] The following combination Figure 1 The present invention describes a network protocol format inference method based on a field pattern, including: step 100, obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol.

[0051] In the present invention, by acquiring network traffic, data packets of target unknown protocol are extracted from the network traffic and grouped according to protocol type;

[0052] Use a preset network protocol analysis tool to traverse each layer of the target unknown protocol data packet, use depth-first search to delete and sort duplicate data, extract information from the target layer, and find the field value and field position of the bytes in the target unknown protocol data packet;

[0053] The field value is converted to a decimal integer from 0 to 255, and a sequence tag is generated based on the field position and syntax.

[0054] Specifically, the incoming network traffic is preprocessed to extract the target unknown protocol packets and group them by protocol type. Then, Tshark is used to traverse the various layers of the target protocol packets and extract information from the target layer through deduplication and sorting using depth-first search to obtain the field values ​​and positions of the bytes of the target unknown protocol packets. At the same time, the protocol messages are extracted and the field values ​​are converted to decimal integers from 0 to 255. Sequence labels are generated based on the field position and syntax, with the last byte of the message field marked as positive and all other bytes marked as negative. Known protocols require calling the entire process, while for unknown protocols, only the conversion from sequence to value is performed.

[0055] Among them, TShark is a network protocol analysis tool. It is based on the Wireshark packet capture tool and provides rich functions and flexible options to help users analyze and solve network problems more easily. TShark (The Shark) is a command-line tool for parsing, displaying, and analyzing network data packets. It supports multiple operating systems such as Windows, Linux, and macOS, and is compatible with Wireshark's file format. TShark can capture real-time data packets or parse data packets stored in files to provide users with detailed network protocol analysis results.

[0056] TShark supports a variety of capture file formats and can automatically detect file formats and compression methods. Packets can be filtered using capture filters and display filters, with display filters being more powerful than capture filters. Packets can be displayed as summaries, details, or statistics, depending on the report type. Packets can be saved or converted to other formats, and exported to formats that can be used by other tools. It can be used in conjunction with other tools or scripts, run on remote systems, or interact with other commands through pipes.

[0057] Step 200: Calculate the covariance of the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol based on the sequence tag, and select the corresponding source domain as the optimal source of the target domain when the covariance is minimized.

[0058] In the present invention, source domain selection is mainly for protocols without message syntax. First, the similarity of data distribution between the unknown protocol and the known protocols in the knowledge base is determined, and the most similar known protocol data packet is selected to improve the accuracy of the unknown protocol syntax inference task.

[0059] Specifically, based on the sequence tags, the target domain of the target unknown protocol is aligned with the second-order statistical features of the source domain distribution of the pre-acquired known protocol through a linear transformation method;

[0060] Calculating the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result;

[0061] Put the calculated covariance into a pre-established empty list and find the minimum covariance value in the empty list;

[0062] In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain.

[0063] The covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol is calculated based on the alignment result through a preset covariance calculation model, specifically including:

[0064] Based on the alignment results, the message length data related to the known protocol and the target unknown protocol is calculated through a preset covariance calculation model and cluster analysis is performed to generate cluster analysis results;

[0065] Based on the cluster analysis results, label information is generated for the protocol within the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connections between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data. The target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data;

[0066] A covariance is calculated based on the source domain data and the target domain data.

[0067] In this invention, source domain selection first sets up an empty list to store the covariance results of the unknown protocol and multiple source domains. Using linear transformation, the second-order statistical features of the source domain and the target domain distribution of the unknown protocol are aligned, and the agreement variance between them is calculated to minimize the covariance. The covariance distance between the two data domains is obtained using a covariance calculation model. When the source domain selection module calls this module, the variance distance between the target domain and the candidate source domains is written to the list.

[0068] To obtain matrix information for protocol messages in the data domain, an examination of publicly available industrial control system protocol datasets revealed that 12 protocols consist entirely of homogeneous clusters, while all other protocols contain at least 75% homogeneous clusters. Therefore, the covariance calculation model first calculates message length data associated with known and unknown protocols and then performs cluster analysis based on this information. Given the label information generated by protocols within a cluster, a transposition calculation is performed on the feature connections between clusters to generate matrix information and decompose it into eigenvectors. Unknown protocols are directly transposed to generate matrix information and decompose it into eigenvectors. Finally, the covariance calculation module generates labeled source domain data D_s and target domain data D_t to calculate their covariance.

[0069] In order to realize the domain difference calculation between known (protocol) and unknown protocol message data, the covariance matrix is ​​calculated by combining the obtained D_s and D_t with the identity matrix, as shown in Formula (1), where D is the matrix after feature transformation in the data domain, n is the size of the data domain, and E is the unit matrix.

[0070] (1)

[0071] In Equation (2), the covariance matrices Cs and Ct of the source and target protocols, respectively, are used to calculate the difference between the protocol data domains. Here, d represents the number of packets selected for protocol feature construction. The goal is to generate the optimal difference calculation result through gradient optimization. To reduce the resulting error, the covariance calculation is performed 100 times in a loop, and a covariance list is constructed to select the source protocol.

[0072] (2)

[0073] During source domain selection, the covariance calculation module is called to calculate the inter-domain differences between several known candidate protocols and the target protocol, constructing a list of inter-domain differences. Each element's position is iterated over, and the best element at that position in each list is found. If the list has a minimum value at that position, the count for that list is incremented. Finally, the column with the highest minimum count is found and used as the optimal source for target protocol selection.

[0074] Step 300: Infer the target unknown protocol message format through a deep transfer learning model based on the selected best source.

[0075] Source domain selection selects the optimal data source for the target unknown protocol, providing prior grammatical knowledge for understanding the unknown protocol. Messages from the unknown protocol are clustered based on message length, and multiple messages within each cluster are tested for homogeneity. If a cluster contains heterogeneous messages, the system removes and re-clusters these messages. This process is repeated iteratively until each cluster contains only homogeneous messages. After converting the bytes of the messages in the cluster to decimal integers, the base network of the deep transfer learning model performs grammatical feature extraction. Subsequently, a subdomain adaptation algorithm trains and optimizes the deep transfer learning model, leveraging prior syntactic knowledge of known protocols to infer the message format of the unknown protocol.

[0076] In the present invention, clustering is performed based on the target unknown protocol message length using a deep transfer learning model based on the selected best source to obtain cluster homogeneous messages;

[0077] After converting the bytes of the cluster homogeneous message into decimal integers, performing grammatical feature extraction through the base network of the deep transfer learning model;

[0078] The pre-extracted prior syntax of the known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

[0079] Specifically, the grammar transfer model is a neural network model designed for the protocol grammar transfer task. Its core consists of three parts: the basic network, the bottleneck layer and the classification layer.

[0080] The core of the deep transfer learning model, base_net, is a fully connected neural network. This basic network structure extracts features from input data and transforms them into higher-dimensional output representations. The model's network structure consists of a linear layer followed by a ReLU activation function, converting the input vector into hidden features. This layer is followed by a ReLU activation function to introduce nonlinearity, ultimately outputting a higher-dimensional representation of the protocol field. This feature representation helps the model learn the complex data patterns inherent in the protocol field.

[0081] The architecture of the deep transfer learning model includes a bottleneck layer, which aims to reduce the dimensionality of the feature space. This helps mitigate overfitting and improves the transferability of the model between different domains. The bottleneck layer consists of a linear transformation, batch normalization (to stabilize learning), a ReLU activation function (to introduce nonlinearity), and a dropout layer (for regularization, configured to drop 50% of activations to prevent co-adaptation).

[0082] The classification layer of the deep transfer learning model is designed to map the extracted bottlenecked features to the desired number of output categories. This classifier consists of two sequential linear transformation and ReLU activation blocks, with a dropout layer in between to enhance the generalization ability of the transfer model.

[0083] A unique feature of deep transfer learning models is their ability to handle domain adaptation losses for different types of protocol data. The loss of deep transfer learning models is divided into two parts: one is the loss incurred by training the model on known protocol data, and the other is the loss of evaluating the difference in data distribution between the feature representations of known protocols and the target unknown protocol.

[0084] The goal of the deep transfer learning model is to optimize two complementary objective functions: minimizing the error L of model training and minimizing the distribution difference Ltrans between known and unknown protocols. The final optimization objective is expressed in Equation (3).

[0085] (3)

[0086] The λ in formula (3) is a regularization parameter with a value of 0.01. The model learning error L of the deep transfer learning model trained on known protocol data is as follows:

[0087] (4)

[0088] Where J(.) is the cross entropy loss function, Θ(.) is the conditional probability that TransNet assigns Ds to Cs, and m represents the size of the dataset. Regarding the distribution divergence Ltrans, the local maximum mean difference (LMMD) is used, as shown in Formula 5, which can compare different distributions based on the distance between known and unknown protocol datasets in RKHS.

[0089] (5)

[0090] where ω and Φ(.) are the adaptive weights of the grammatical label and feature mapping function, respectively.

[0091] The deep transfer learning model processes both source and target data through the same pipeline, using cross-entropy to calculate the loss between labels and predicted structures. The network then uses features from the target data to infer the logical value of the target protocol. Weight parameters are obtained by combining packet data and grammatical labels from known protocols and then calculating the transfer loss using a Gaussian kernel. This determines the final learning loss of the transfer model. To optimize the transfer model, TransNet also uses a stochastic gradient descent (SGD) optimizer, setting different learning rates for different network layers to effectively balance feature extraction and domain adaptation during training. When faced with an unknown protocol, the base network of TransNet extracts high-dimensional features of the target protocol and further infers the grammatical format of the unknown protocol using a model trained on previously known grammatical formats. A key difference compared to traditional neural networks is that TransNet adds a measure of the data distribution difference between the known and unknown protocol data domains after the fully connected layers. This difference measure is included in the loss calculation during network training. When training TransNet with known protocol messages and their grammatical token vectors, TransNet also requires minimizing the data distribution difference between the known and target protocols in the base network's hidden representation.

[0092] In a specific embodiment, a prototype of a deep transfer learning model was implemented using Python 3. It consists of two modules: a source code selection module and a grammar transfer module. The source selection module implements inter-domain difference analysis and a source domain selection algorithm. This module does not limit the number of candidate unknown protocols and allows manual assignment of candidate protocols based on expert analysis of unknown protocols. It also supports manual review of protocol analysis results based on the selections made by the source selection module. Based on the selected source domain, the grammar transfer module learns knowledge of the selected protocol grammar and infers the unknown protocol grammar through domain difference analysis. Specifically, the knowledge transfer submodule is built on top of LMMD and provides inter-protocol data difference analysis.

[0093] This paper proposes a field-pattern-based network protocol format inference method that combines prior grammatical knowledge of known protocols with deep transfer learning to more accurately understand unknown protocols. Unlike traditional methods, this method first analyzes the domain differences between the knowledge base and the target unknown protocol to identify the optimal prior knowledge domain. Deep transfer learning then mines grammatical patterns of known protocols to improve the accuracy of inferring the unknown protocol syntax and reduce complexity.

[0094] refer to Figure 3 and Figure 4 The present invention also discloses a network protocol format inference system based on field patterns, the system comprising:

[0095] A data processing module 110 is configured to obtain network traffic, extract a target unknown protocol based on the network traffic, perform decimal integer conversion and generate a sequence label based on the target unknown protocol;

[0096] a source domain selection module 120 for performing covariance calculation between the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol based on the sequence tag, and selecting the corresponding source domain as the optimal source of the target domain when the covariance is minimized;

[0097] The knowledge transfer module 130 is used to infer the target unknown protocol message format through a deep transfer learning model based on the selected best source.

[0098] The steps of acquiring network traffic, extracting a target unknown protocol based on the network traffic, and performing decimal integer conversion and sequence label generation based on the target unknown protocol specifically include:

[0099] Obtaining network traffic, extracting data packets of a target unknown protocol from the network traffic, and grouping them according to protocol type;

[0100] Use a preset network protocol analysis tool to traverse each layer of the target unknown protocol data packet, use depth-first search to delete and sort duplicate data, extract information from the target layer, and find the field value and field position of the bytes in the target unknown protocol data packet;

[0101] The field value is converted to a decimal integer from 0 to 255, and a sequence tag is generated based on the field position and syntax.

[0102] Based on the sequence tags, covariance calculation is performed on the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol. When the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain, specifically including:

[0103] Aligning the target domain of the target unknown protocol with the second-order statistical features of the source domain distribution of the pre-acquired known protocol through a linear transformation method based on the sequence tags;

[0104] Calculating the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result;

[0105] Put the calculated covariance into a pre-established empty list and find the minimum covariance value in the empty list;

[0106] In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain.

[0107] Based on the alignment results, the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol is calculated using a preset covariance calculation model, specifically including:

[0108] Based on the alignment results, the message length data related to the known protocol and the target unknown protocol is calculated through a preset covariance calculation model and cluster analysis is performed to generate cluster analysis results;

[0109] Based on the cluster analysis results, label information is generated for the protocol within the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connections between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data. The target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data;

[0110] A covariance is calculated based on the source domain data and the target domain data.

[0111] Based on the selected best source, the target unknown protocol message format is inferred through a deep transfer learning model, specifically including:

[0112] Based on the selected best source, a deep transfer learning model is used to cluster the target unknown protocol message length to obtain clustered homogeneous messages;

[0113] After converting the bytes of the cluster homogeneous message into decimal integers, performing grammatical feature extraction through the base network of the deep transfer learning model;

[0114] The pre-extracted prior syntax of the known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

[0115] Perform grammatical feature extraction through the base network of the deep transfer learning model, specifically including:

[0116] The input vector is converted into hidden features through the linear layer in the basic network structure of the deep transfer learning model;

[0117] Based on the hidden features, a nonlinear transformation is introduced through the ReLU activation function in the basic network structure of the deep transfer learning model, and a higher-dimensional output representation in the output protocol field is output to obtain grammatical features.

[0118] The present invention provides a field pattern-based network protocol format inference system. By combining prior grammatical knowledge of known protocols, deep transfer learning is used to more accurately understand unknown protocols. Unlike traditional methods, this system first analyzes the domain differences between the knowledge base and the target unknown protocol to identify the optimal prior knowledge domain. Deep transfer learning then mines grammatical patterns of known protocols to improve the accuracy of inferring the unknown protocol grammar and reduce complexity.

[0119] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communications bus 540. The processor 510 may call logic instructions in the memory 530 to execute a network protocol format inference method based on a field pattern, the method comprising: acquiring network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol; calculating the covariance of a target domain of the target unknown protocol with a pre-acquired source domain of a known protocol based on the sequence label, and selecting the corresponding source domain as the optimal source of the target domain when the covariance is minimized; and inferring the message format of the target unknown protocol using a deep transfer learning model based on the selected optimal source.

[0120] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a network protocol format inference method based on a field pattern provided by the above methods, the method including: obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol; calculating the covariance of the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol based on the sequence label, and when the covariance is minimum, selecting the corresponding source domain as the optimal source of the target domain; and inferring the target unknown protocol message format through a deep transfer learning model based on the selected optimal source.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a network protocol format inference method based on a field pattern provided by the above-mentioned methods, the method comprising: obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol; performing covariance calculation on a target domain of the target unknown protocol and a source domain of a pre-acquired known protocol based on the sequence label, and selecting the corresponding source domain as the optimal source of the target domain when the covariance is minimized; and inferring the target unknown protocol message format through a deep transfer learning model based on the selected optimal source.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0124] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A network protocol format inference method based on field patterns, characterized in that: include: Acquire network traffic, extract a target unknown protocol based on the network traffic, and perform decimal integer conversion and sequence label generation based on the target unknown protocol; Based on the sequence tags, covariance calculation is performed on the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol, and when the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain; Inferring the target unknown protocol message format based on the selected best source through deep transfer learning model; The acquiring of network traffic, extracting a target unknown protocol based on the network traffic, and performing decimal integer conversion and sequence label generation based on the target unknown protocol specifically include: Obtaining network traffic, extracting data packets of a target unknown protocol from the network traffic, and grouping them according to protocol type; Use a preset network protocol analysis tool to traverse each layer of the target unknown protocol data packet, use depth-first search to delete and sort duplicate data, extract information from the target layer, and find the field value and field position of the bytes in the target unknown protocol data packet; Convert the field value to a decimal integer from 0 to 255, and generate a sequence tag based on the field position and syntax; The step of calculating the covariance of the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol based on the sequence tag, and selecting the corresponding source domain as the optimal source of the target domain when the covariance is minimized, specifically includes: Aligning the target domain of the target unknown protocol with the second-order statistical features of the source domain distribution of the pre-acquired known protocol through a linear transformation method based on the sequence tags; Calculating the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result; Put the calculated covariance into a pre-established empty list and find the minimum covariance value in the empty list; In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain; The covariance between the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol is calculated based on the alignment result by using a preset covariance calculation model, specifically including: Based on the alignment results, the message length data related to the known protocol and the target unknown protocol is calculated through a preset covariance calculation model and cluster analysis is performed to generate cluster analysis results; Based on the cluster analysis results, label information is generated for the protocol within the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connections between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data. The target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data; A covariance is calculated based on the source domain data and the target domain data.

2. The network protocol format inference method based on field pattern according to claim 1, characterized in that: The inference of the target unknown protocol message format based on the selected best source through the deep transfer learning model specifically includes: Based on the selected best source, a deep transfer learning model is used to cluster the target unknown protocol message length to obtain clustered homogeneous messages; After converting the bytes of the cluster homogeneous message into decimal integers, performing grammatical feature extraction through the base network of the deep transfer learning model; The pre-extracted prior syntax of the known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

3. The network protocol format inference method based on field pattern according to claim 2, characterized in that: The grammatical feature extraction is performed by the base network of the deep transfer learning model, specifically including: The input vector is converted into hidden features through the linear layer in the basic network structure of the deep transfer learning model; Based on the hidden features, a nonlinear transformation is introduced through the ReLU activation function in the basic network structure of the deep transfer learning model, and a higher-dimensional output representation in the output protocol field is output to obtain grammatical features.

4. A network protocol format inference system based on field patterns, characterized in that: The system comprises: a data processing module, configured to obtain network traffic, extract a target unknown protocol based on the network traffic, and perform decimal integer conversion and sequence label generation based on the target unknown protocol; a source domain selection module, configured to calculate the covariance of the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol based on the sequence tag, and select the corresponding source domain as the optimal source of the target domain when the covariance is minimized; The knowledge transfer module is used to infer the target unknown protocol message format through a deep transfer learning model based on the selected best source; The acquiring of network traffic, extracting a target unknown protocol based on the network traffic, and performing decimal integer conversion and sequence label generation based on the target unknown protocol specifically include: Obtaining network traffic, extracting data packets of a target unknown protocol from the network traffic, and grouping them according to protocol type; Use a preset network protocol analysis tool to traverse each layer of the target unknown protocol data packet, use depth-first search to delete and sort duplicate data, extract information from the target layer, and find the field value and field position of the bytes in the target unknown protocol data packet; Convert the field value to a decimal integer from 0 to 255, and generate a sequence tag based on the field position and syntax; The step of calculating the covariance of the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol based on the sequence tag, and selecting the corresponding source domain as the optimal source of the target domain when the covariance is minimized, specifically includes: Aligning the target domain of the target unknown protocol with the second-order statistical features of the source domain distribution of the pre-acquired known protocol through a linear transformation method based on the sequence tags; Calculating the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result; Put the calculated covariance into a pre-established empty list and find the minimum covariance value in the empty list; In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain; The covariance between the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol is calculated based on the alignment result by using a preset covariance calculation model, specifically including: Based on the alignment results, the message length data related to the known protocol and the target unknown protocol is calculated through a preset covariance calculation model and cluster analysis is performed to generate cluster analysis results; Based on the cluster analysis results, label information is generated for the protocol within the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connections between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data. The target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data; A covariance is calculated based on the source domain data and the target domain data.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the network protocol format inference method based on field patterns as described in any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the network protocol format inference method based on field patterns as described in any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the network protocol format inference method based on field patterns as described in any one of claims 1 to 3 is implemented.

Images