Network protocol format inference method and system based on field mode

Abstract

The invention provides a network protocol format inference method and system based on a field mode. The method comprises the following steps: acquiring network flow, extracting a target unknown protocol based on the network flow, and performing decimal integer conversion and sequence label generation based on the target unknown protocol; performing covariance calculation on a target domain of the target unknown protocol and a pre-acquired source domain of a known protocol based on the sequence label, and selecting the corresponding source domain as an optimal source of the target domain under the condition of a minimum covariance value; and inferring a target unknown protocol message format through a deep transfer learning model based on the selected optimal source. According to the invention, the problems of inaccurate reverse inference and high complexity of the existing network protocol format are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of communication protocol reverse engineering, and in particular to a network protocol format inference method and system based on field mode. Background Art

[0002] In actual use, protocols are an important part of industries such as communications, industry, and power systems. They play an important role in establishing communication connections for various components within the system. In addition to basic functions such as exchanging data and information, some protocols have the authority to control remote devices. Protocol reversal refers to the process of analyzing unknown or under-documented communication protocols to reveal their internal syntax-semantics. Protocol reversal is very important in areas such as network security research, malware analysis, network traffic analysis, and interoperability testing of network protocols. By understanding and analyzing protocols, researchers can better respond to network attacks, solve communication problems, and improve the performance and security of network communications.

[0003] In the application process of network protocols, the international community has formulated standards for some commonly used protocols, such as the famous FTP, SSL, etc., and thus some security assurance tools dedicated to a certain protocol standard have emerged. However, in the actual use of the protocol, manufacturers may customize the standard protocol according to their needs. In addition, there are many private protocols used internally by manufacturers that are not public. With the continuous diversification of communication protocol types and application scenarios, the closed-source heterogeneity of implementation methods, the increasing complexity of protocol behaviors and functions, and the continuous evolution of protocol security attacks and defense methods, protocol reverse engineering work faces severe challenges.

[0004] In protocol reverse engineering, the flow-based protocol reverse engineering method is a technique for understanding and restoring communication protocols by analyzing network communication traffic without accessing the specific implementation source code or documentation of the protocol. In this process, researchers try to understand and analyze the internal working principles of the protocol, which usually includes: data interception and analysis, data parsing, state machine analysis, reverse engineering, and vulnerability analysis. This method can effectively combat the problem of closed-source protocol implementation in diversified scenarios, that is, in the absence of protocol documentation or source code support, only the traffic trace is needed to restore the design of the protocol. It analyzes the data packets of the intercepted target protocol to identify the protocol header, field, and payload, and further determines the protocol type, message format, state machine, and communication mode of the protocol. It provides a solid foundation for understanding the state transition and behavior of the protocol in communication, and identifying and analyzing potential vulnerabilities or security issues in the protocol.

[0005] Traffic-based protocol reversal is a technique for inferring the structure and format of a protocol by analyzing network traffic. It mainly relies on aligning and clustering captured network packets according to similarity, and then discovering repeated field patterns or structures (such as byte frequency, information entropy, linear / nonlinear relationships, etc.) by aligning the byte sequences in the packets. Currently, there are many protocol reversal tools that can be used for protocol format inference, such as Netzob and Netplier. After aligning and clustering messages in pairs, Netzob uses field patterns such as field length, field order, and inter-field relationships to infer the protocol message format. Netplier first builds a probability-based keyword recognition system using four heuristic rules, then implements clustering, and finally uses field patterns (such as static fields with specific values ​​and dynamic fields with potential value lists) to infer the message format. BinaryInferno implements various atomic detectors (such as floating point detectors, timestamp detectors, and length detectors) derived from heuristic rules, and then implements detector integration for protocol field reasoning through a directed acyclic graph method. Discoverer uses token patterns to tokenize bytes using simple semantics (e.g., text or binary), then clusters messages, and finally employs statistical analysis (e.g., byte frequency, information entropy, etc.) to identify relationships and structures between fields. Nemesys infers field boundaries from value variation distributions through the intrinsic structure of a single message to identify fields for binary protocols. It introduces a format matching score, a quantitative measure of message format reasoning, which cannot handle the diversity of protocol formats. FieldHunter uses general semantics and characteristics of specific common fields, such as message length or host ID, and identifies fields by establishing statistical correlations between these types and byte values.

[0006] The above methods all adopt a heuristic rule-based approach to classify messages into different clusters and infer the message syntax of the protocol based on preset and fixed field patterns or structures. Although these methods recover the syntax of unknown messages, their incomplete and inflexible field patterns lead to low accuracy. First, they use fixed patterns derived from heuristic rules to identify fields, limiting their applicability to specific types of protocols because other protocols may lack predefined patterns. For example, since the timestamps used by the TCP protocol do not conform to standard time formats such as NTP or Unix timestamps, BinaryInferno's preset timestamp detector cannot recognize them. Second, the fixed patterns with thresholds are inaccurate in protocol-typical field identification. Existing techniques that use a unified approach to standardize all protocols may lead to misunderstandings of protocol specifications.

[0007] Traditional methods adopt limited fixed patterns or heuristic rules to identify protocol formats. For example, Netplier uses four heuristic rules to identify key fields, while BinaryInferno has five built-in fixed patterns to identify typical fields. With the emergence of new protocols and the evolution of existing protocols, communication protocols have become extremely diverse, and each protocol can be designed with different fields and structures. Existing protocols contain a variety of field patterns. Therefore, rule-based heuristic methods cannot cover all field patterns. Existing protocols exhibit a wide range of field patterns, making it complex and labor-intensive to explore predefined patterns between these fields. For example, Figure 2 The syntax of Modbus query message, Hart_IP pass-through request message and IEC60870_104 message slicing and syntax is shown. The f1 field in Modbus message and f5 in Hart_IP message can be accurately identified by simple and clear patterns. Nevertheless, the inherent patterns of f3, f4 and f5 fields of Modbus protocol are mysterious and difficult to capture. Therefore, the heuristic rule-based approach cannot explain all field patterns.

[0008] Since different protocols have different field patterns, these tools usually rely on preset and fixed field patterns to identify and analyze packets, lacking flexibility and wide applicability, resulting in limited benefits of field segmentation. For example, since the timestamp of the TCP protocol, which is used to improve network communication performance and reliability, does not conform to standard time formats such as NTP or Unix timestamps, it cannot be identified using BinaryInferno's custom timestamp detector. In addition, some heuristic-based methods adopt threshold settings to identify fields, which may increase identification errors when dealing with unknown or non-standard protocols because these thresholds are not applicable to all cases. Although all three protocols include the pattern of y=kx+b, there are differences between the three patterns. In Hart_IP and IEC 60870_104, 'x' indicates the length of the subsequent field containing the field itself, while Modbus indicates the length of the sequence after the field. Fieldhunter uses a fixed threshold to fit the linear correlation between the field value and the actual message size, which cannot accurately identify their fields. Protocol design and implementation usually follow the requirements of functional applications and are based on the specifications and knowledge of existing protocols. This allows for more similar knowledge or common patterns between protocols. For example, the recognition pattern of the Length field in the Modbus protocol follows a linear fit of the message length to the length field value, which is the same in the AMQP and IEC104 protocols. Previous methods used statistical analysis to infer the protocol format based on general field patterns summarized from protocols with specifications. However, the significant differences between various protocols limit the expansion of field patterns, thereby reducing the accuracy of protocol understanding. Summary of the invention

[0009] The present invention provides a network protocol format inference method and system based on field mode, which are used to solve the problems of inaccurate and high complexity of the existing network protocol format reverse inference.

[0010] The present invention provides a network protocol format inference method based on field mode, comprising: Acquire network traffic, extract a target unknown protocol based on the network traffic, and perform decimal integer conversion and sequence label generation based on the target unknown protocol; Based on the sequence tags, the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol are subjected to covariance calculation, and when the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain; The target unknown protocol message format is inferred through a deep transfer learning model based on the selected best source.

[0011] According to a network protocol format inference method based on field mode provided by the present invention, the network traffic is obtained, a target unknown protocol is extracted based on the network traffic, and a decimal integer conversion and sequence label generation are performed based on the target unknown protocol, specifically including: Obtaining network traffic, extracting data packets of a target unknown protocol from the network traffic, and grouping them according to protocol type; Traverse each layer in the data packet of the target unknown protocol through a preset network protocol analysis tool, use depth-first search to delete and sort duplicate data, extract information from the target layer, and the field value and field position of the byte in the data packet of the target unknown protocol; The field value is converted to a decimal integer from 0 to 255, and a sequence tag is generated based on the field position and syntax.

[0012] According to a network protocol format inference method based on field patterns provided by the present invention, the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol are subjected to covariance calculation based on the sequence tag, and the corresponding source domain is selected as the optimal source of the target domain when the covariance is minimum, specifically including: Based on the sequence tags, aligning the target domain of the target unknown protocol with the second-order statistical features of the source domain distribution of the pre-acquired known protocol by a linear transformation method; Calculate the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result; Put the calculated covariance into a pre-established empty list, and find the minimum covariance value in the empty list; In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain.

[0013] According to a network protocol format inference method based on field mode provided by the present invention, the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol is calculated based on the alignment result by using a preset covariance calculation model, specifically including: Based on the alignment results, the message length data related to the known protocol and the target unknown protocol are calculated through a preset covariance calculation model and cluster analysis is performed to generate a cluster analysis result; Based on the cluster analysis result, label information is generated for the protocol in the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connection between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data, and the target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data; A covariance is calculated based on the source domain data and the target domain data.

[0014] According to a network protocol format inference method based on field mode provided by the present invention, the inference of the target unknown protocol message format based on the selected best source through a deep transfer learning model specifically includes: Based on the selected best source, cluster the target unknown protocol message length through deep transfer learning model to obtain cluster homogeneous messages; After converting the bytes of the cluster homogeneous message into decimal integers, grammatical feature extraction is performed through the basic network of the deep transfer learning model; The prior syntax of the pre-extracted known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

[0015] According to a network protocol format inference method based on field mode provided by the present invention, the syntax feature extraction is performed by the basic network of the deep transfer learning model, specifically including: The input vector is converted into hidden features through the linear layer in the basic network structure of the deep transfer learning model; Based on the hidden features, a nonlinear transformation is introduced through the ReLU activation function in the basic network structure of the deep transfer learning model, and a higher-dimensional output representation in the output protocol field is output to obtain grammatical features.

[0016] The present invention also provides a network protocol format inference system based on field patterns, the system comprising: A data processing module, used for acquiring network traffic, extracting a target unknown protocol based on the network traffic, and performing decimal integer conversion and sequence label generation based on the target unknown protocol; A source domain selection module, configured to perform covariance calculation on the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol based on the sequence tag, and select the corresponding source domain as the best source of the target domain when the covariance is minimum; The knowledge transfer module is used to infer the target unknown protocol message format through a deep transfer learning model based on the selected best source.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the network protocol format inference method based on the field pattern as described above is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for inferring a network protocol format based on a field pattern.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for inferring a network protocol format based on a field pattern as described above is implemented.

[0020] The present invention provides a method and system for inferring network protocol formats based on field patterns, which can more accurately understand unknown protocols through deep transfer learning by combining prior grammatical knowledge of known protocols. Different from traditional methods, the best prior knowledge domain is first matched by analyzing the domain differences between the knowledge base and the target unknown protocol, and then the grammatical patterns of known protocols are mined through deep transfer learning to improve the accuracy of inferring unknown protocol grammar and reduce complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of a network protocol format inference method based on field pattern provided by the present invention.

[0023] Figure 2 It is a schematic diagram of prior knowledge learning and transfer in the prior art.

[0024] Figure 3 This is a network protocol format inference system architecture diagram based on field patterns provided by the present invention.

[0025] Figure 4 It is a module connection diagram of a network protocol format inference system based on field mode provided by the present invention.

[0026] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention.

[0027] Reference numerals: 110: data processing module; 120: source domain selection module; 130: knowledge transfer module; 510: processor; 520: communication interface; 530: memory; 540: communication bus. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Combine the following Figure 1 The present invention describes a method for inferring a network protocol format based on a field pattern, including: step 100, obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol.

[0030] In the present invention, by acquiring network traffic, data packets of target unknown protocol are extracted from the network traffic and grouped according to protocol type; Traverse each layer in the data packet of the target unknown protocol through a preset network protocol analysis tool, use depth-first search to delete and sort duplicate data, extract information from the target layer, and the field value and field position of the byte in the data packet of the target unknown protocol; The field value is converted to a decimal integer from 0 to 255, and a sequence tag is generated based on the field position and syntax.

[0031] Specifically, the incoming network traffic is preprocessed to extract the target unknown protocol packets and group them by protocol type. Then, Tshark is used to traverse the layers of the target protocol packets and extract information from the target layers by deduplication and sorting using depth-first search to obtain the field values ​​and positions of the bytes of the target unknown protocol packets. At the same time, the protocol messages are extracted and the field values ​​are then converted to decimal integers from 0 to 255. Sequence labels are generated based on field positions and syntax, with the last byte of the message field marked as positive and all other bytes marked as negative. Known protocols require the entire process to be called, while for unknown protocols, only the conversion from sequence to numerical value is performed.

[0032] Among them, TShark is a network protocol analysis tool. It is based on the Wireshark packet capture tool and provides rich functions and flexible options to help users analyze and solve network problems more easily. TShark (The Shark) is a command-line tool for parsing, displaying, and analyzing network packets. It supports multiple operating systems such as Windows, Linux, and macOS, and is compatible with Wireshark's file format. TShark can capture real-time packets, or parse packets stored in files, providing users with detailed network protocol analysis results.

[0033] TShark can support multiple capture file formats, and can also automatically detect file formats and compression methods. You can use capture filters and display filters to filter packets. Display filters are more powerful than capture filters. You can display summary information, detailed information, or statistical information of packets, which are displayed according to different report types. Packets can be saved or converted to other formats, or exported to formats that can be used by other tools. It can be used in conjunction with other tools or scripts, and can also be run on remote systems or interact with other commands through pipes.

[0034] Step 200: Based on the sequence tags, the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol are subjected to covariance calculation, and when the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain.

[0035] In the present invention, source domain selection is mainly for protocols without message syntax. First, the similarity of data distribution between the unknown protocol and the known protocols in the knowledge base is determined, and the most similar known protocol data packet is selected to improve the accuracy of the unknown protocol syntax inference task.

[0036] Specifically, based on the sequence tags, the target domain of the target unknown protocol is aligned with the second-order statistical features of the source domain distribution of the pre-acquired known protocol by a linear transformation method; Calculate the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result; Put the calculated covariance into a pre-established empty list, and find the minimum covariance value in the empty list; In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain.

[0037] Among them, the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol is calculated based on the alignment result through a preset covariance calculation model, specifically including: Based on the alignment results, the message length data related to the known protocol and the target unknown protocol are calculated through a preset covariance calculation model and cluster analysis is performed to generate a cluster analysis result; Based on the cluster analysis result, label information is generated for the protocol in the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connection between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data, and the target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data; A covariance is calculated based on the source domain data and the target domain data.

[0038] In the present invention, source domain selection first sets an empty list to store the covariance results of the unknown protocol and multiple source domains. The second-order statistical features of the source domain and the target domain distribution of the target unknown protocol are aligned using the linear transformation method and the protocol variance between them is calculated to minimize the covariance. The covariance distance between the two data domains is obtained through the covariance calculation model. When the source domain selection module calls this module, the variance distance between the target domain and the candidate source domain is written into the list.

[0039] In order to obtain the matrix information of protocol messages in the data domain, by examining the publicly available industrial control system protocol dataset, it was found that 12 protocols are completely composed of isomorphic clusters, and the other protocols contain more than 75% of isomorphic clusters. Therefore, the covariance calculation model first calculates the message length data related to known and unknown protocols, and then performs cluster analysis based on this information. Given the label information generated by the protocol within the cluster, the feature connection between clusters is combined for transposition calculation to generate matrix information and decompose it into feature vectors. The unknown protocol is directly transposed to generate matrix information and decompose it into feature vectors. Finally, the covariance calculation module generates labeled source domain data D_s and target domain data D_t to calculate their covariance.

[0040] In order to realize the domain difference calculation between known (protocol) and unknown protocol message data, the covariance matrix is ​​calculated by combining the obtained D_s and D_t with the identity matrix, as shown in formula (1), where D is the matrix after feature transformation in the data domain, n is the size of the data domain, and E is the unit matrix.

[0041] (1)

[0042] In formula (2), the covariance matrices Cs and Ct of the source and target protocols are used to calculate the difference between the protocol data domains. Here d represents the number of packets selected for protocol feature construction. The goal is to generate the optimal difference calculation result through gradient optimization. In order to reduce the error generated, 100 covariance calculations are performed in a loop, and a covariance list is constructed to select the source protocol.

[0043] (2)

[0044] In the source domain selection process, by calling the covariance calculation module, the inter-domain differences between several known candidate protocols and the target protocol can be calculated and a list of inter-domain differences can be constructed. The position of each element is traversed to find the best element at the current position in each list. If the list has a minimum value at that position, the count of that list is increased. Finally, the column with the most minimum counts is found and used as the best source for target protocol selection.

[0045] Step 300: Infer the target unknown protocol message format through a deep transfer learning model based on the selected best source.

[0046] The best data source is selected for the target unknown protocol through source domain selection, providing prior grammatical knowledge for understanding the unknown protocol. For messages of unknown protocols, the information of unknown protocols is clustered according to the message length, and then multiple messages within each cluster are tested for homogeneity. If the cluster contains heterogeneous messages, the system will delete and re-cluster these messages. This process is repeated iteratively until each cluster contains only homogeneous messages. After converting the bytes of the messages in the cluster to decimal integers, the base network of the deep transfer learning model performs grammatical feature extraction. Subsequently, the subdomain adaptation algorithm trains and optimizes the deep transfer learning model, using the prior syntactic knowledge of known protocols to infer the message format of unknown protocols.

[0047] In the present invention, clustering is performed based on the length of target unknown protocol messages through a deep transfer learning model based on the selected best source to obtain cluster homogeneous messages; After converting the bytes of the cluster homogeneous message into decimal integers, grammatical feature extraction is performed through the basic network of the deep transfer learning model; The prior syntax of the pre-extracted known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

[0048] Specifically, the grammar transfer model is a neural network model designed for the protocol grammar transfer task, and its core consists of three parts: the basic network, the bottleneck layer and the classification layer.

[0049] The core of the deep transfer learning model, base_net, is a fully connected neural network. The basic network structure is used to extract features from the input data and convert them into higher-dimensional output representations. The network structure of this model consists of a linear layer and a ReLU activation function, which converts the input vector into hidden features. This layer is followed by a ReLU activation function to introduce nonlinearity and finally output a higher-dimensional output representation in the protocol field. This feature representation helps the model learn the complex data patterns contained in the protocol field.

[0050] The architecture of the deep transfer learning model contains a bottleneck layer, which aims to reduce the dimensionality of the feature space, which helps to alleviate overfitting and improve the transferability of the model between different domains. The bottleneck layer consists of a linear transformation, batch normalization (to stabilize learning), a ReLU activation function (to introduce nonlinearity), and a dropout layer (for regularization, configured to drop 50% of the activations to prevent co-adaptation).

[0051] The classification layer of the deep transfer learning model aims to map the extracted bottlenecked features to the desired number of output categories. This classifier consists of two sequential linear transformation and ReLU activation blocks with a dropout layer in between to enhance the generalization ability of the transfer model.

[0052] A unique feature of the deep transfer learning model is that it can handle domain adaptation losses for different types of protocol data. The loss of the deep transfer learning model is divided into two parts: one is the loss generated by training the model with known protocol data, and the other is the loss of evaluating the difference in data distribution between the feature representation of the known protocol and the target unknown protocol.

[0053] The goal of the deep transfer learning model is to optimize two complementary objective functions: minimizing the error L of model training; minimizing the distribution difference Ltrans between known protocols and unknown protocols; the final optimization target is expressed in formula (3).

[0054] (3)

[0055] The λ in formula (3) is a regularization parameter with a value of 0.01. The model learning error L of the deep transfer learning model trained on known protocol data is as follows: (4)

[0056] Among them, J(.) is the cross entropy loss function, Θ(.) is the conditional probability that TransNet assigns Ds to Cs, and m represents the size of the dataset. Regarding the distribution divergence Ltrans, the local maximum mean difference (LMMD) is adopted, as shown in Formula 5, which can compare different distributions based on the distance between known and unknown protocol datasets in RKHS.

[0057] (5)

[0058] where ω and Φ(.) are the adaptive weights of the grammatical label and feature mapping function, respectively.

[0059] The deep transfer learning model processes the source and target data through the same pipeline, using cross entropy to calculate the loss between the label and the predicted structure. The network then uses the features of the target data to infer the logical value of the target protocol. The weight parameters are obtained by combining the packet data and grammatical labels from the known protocol and then calculating the transfer loss using a Gaussian kernel. This determines the final learning loss of the transfer model. To optimize the transfer model, TransNet also uses a stochastic gradient descent (SGD) optimizer to set different learning rates for different network layers to effectively balance feature extraction and domain adaptation during training. When facing an unknown protocol, the base network of TransNet extracts high-dimensional features of the target protocol and further infers the grammatical format of the unknown protocol using a model trained on the previous grammatical format of the known protocol. Compared with traditional neural networks, the key difference is that TransNet adds a measure of the difference in data distribution between the known and unknown protocol data domains after the fully connected layer. This difference measure is included in the loss calculation during network training. When training TransNet with known protocol messages and their grammatical tag vectors, TransNet also requires minimizing the difference in data distribution between the known and target protocols in the hidden representation of the base network.

[0060] In a specific embodiment, a prototype of a deep transfer learning model is implemented using Python 3. It consists of two modules: a source code selection module and a grammar transfer module. The source selection module implements the inter-domain difference analysis and source domain selection algorithm. This module does not limit the number of candidate unknown protocols and allows manual assignment of candidate protocols based on expert analysis of unknown protocols. It also supports manual inspection of protocol analysis results based on the selection of the source selection module. Based on the selected source domain, the grammar transfer module learns the selected protocol grammar knowledge and infers the unknown protocol grammar through domain difference analysis. Specifically, the knowledge transfer submodule is built on top of LMMD to provide inter-protocol data difference analysis.

[0061] Based on the field pattern-based network protocol format inference method provided by the present invention, by combining the prior grammatical knowledge of known protocols, the unknown protocol can be understood more accurately through deep transfer learning. Different from the traditional method, the best prior knowledge domain is first matched by analyzing the domain differences between the knowledge base and the target unknown protocol, and then the grammatical pattern of the known protocol is mined through deep transfer learning to improve the accuracy of inferring the unknown protocol grammar and reduce the complexity.

[0062] refer to Figure 3 and Figure 4 The present invention also discloses a network protocol format inference system based on field mode, the system comprising: A data processing module 110 is used to obtain network traffic, extract a target unknown protocol based on the network traffic, and perform decimal integer conversion and sequence label generation based on the target unknown protocol; A source domain selection module 120 is used to calculate the covariance of the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol based on the sequence tag, and select the corresponding source domain as the best source of the target domain when the covariance is minimum; The knowledge transfer module 130 is used to infer the target unknown protocol message format through a deep transfer learning model based on the selected best source.

[0063] The method of acquiring network traffic, extracting a target unknown protocol based on the network traffic, and performing decimal integer conversion and sequence label generation based on the target unknown protocol specifically includes: Obtaining network traffic, extracting data packets of a target unknown protocol from the network traffic, and grouping them according to protocol type; Traverse each layer in the data packet of the target unknown protocol through a preset network protocol analysis tool, use depth-first search to delete and sort duplicate data, extract information from the target layer, and the field value and field position of the byte in the data packet of the target unknown protocol; The field value is converted to a decimal integer from 0 to 255, and a sequence tag is generated based on the field position and syntax.

[0064] Based on the sequence tag, the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol are subjected to covariance calculation, and when the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain, specifically including: Based on the sequence tags, aligning the target domain of the target unknown protocol with the second-order statistical features of the source domain distribution of the pre-acquired known protocol by a linear transformation method; Calculate the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result; Put the calculated covariance into a pre-established empty list, and find the minimum covariance value in the empty list; In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain.

[0065] Based on the alignment result, the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol is calculated through a preset covariance calculation model, specifically including: Based on the alignment results, the message length data related to the known protocol and the target unknown protocol are calculated through a preset covariance calculation model and cluster analysis is performed to generate a cluster analysis result; Based on the cluster analysis result, label information is generated for the protocol in the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connection between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data, and the target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data; A covariance is calculated based on the source domain data and the target domain data.

[0066] Based on the selected best source, the target unknown protocol message format is inferred through a deep transfer learning model, including: Based on the selected best source, cluster the target unknown protocol message length through deep transfer learning model to obtain cluster homogeneous messages; After converting the bytes of the cluster homogeneous message into decimal integers, grammatical feature extraction is performed through the basic network of the deep transfer learning model; The prior syntax of the pre-extracted known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

[0067] Perform grammatical feature extraction through the base network of the deep transfer learning model, including: The input vector is converted into hidden features through the linear layer in the basic network structure of the deep transfer learning model; Based on the hidden features, a nonlinear transformation is introduced through the ReLU activation function in the basic network structure of the deep transfer learning model, and a higher-dimensional output representation in the output protocol field is output to obtain grammatical features.

[0068] Based on the field pattern-based network protocol format inference system provided by the present invention, by combining the prior grammatical knowledge of known protocols, the unknown protocol can be understood more accurately through deep transfer learning. Different from the traditional method, the best prior knowledge domain is first matched by analyzing the domain differences between the knowledge base and the target unknown protocol, and then the grammatical patterns of the known protocol are mined through deep transfer learning to improve the accuracy of inferring the unknown protocol grammar and reduce the complexity.

[0069] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a network protocol format inference method based on a field pattern, the method comprising: obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol; performing covariance calculation on the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol based on the sequence label, and selecting the corresponding source domain as the best source of the target domain in the case of the minimum covariance; and performing inference on the target unknown protocol message format through a deep transfer learning model based on the selected best source.

[0070] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0071] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a network protocol format inference method based on a field pattern provided by the above methods, and the method includes: obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol; based on the sequence label, performing covariance calculation on a target domain of the target unknown protocol and a source domain of a pre-acquired known protocol, and when the covariance is minimized, selecting the corresponding source domain as the optimal source of the target domain; and inferring the target unknown protocol message format through a deep transfer learning model based on the selected optimal source.

[0072] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a network protocol format inference method based on a field pattern provided by the above-mentioned methods, the method comprising: obtaining network traffic, extracting a target unknown protocol based on the network traffic, performing decimal integer conversion and sequence label generation based on the target unknown protocol; calculating the covariance of a target domain of the target unknown protocol and a source domain of a pre-acquired known protocol based on the sequence label, and selecting the corresponding source domain as the optimal source of the target domain when the covariance is minimized; and inferring the target unknown protocol message format through a deep transfer learning model based on the selected optimal source.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network protocol format inference method based on field pattern, characterized in that: include: Acquire network traffic, extract a target unknown protocol based on the network traffic, and perform decimal integer conversion and sequence label generation based on the target unknown protocol; Based on the sequence tags, the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol are subjected to covariance calculation, and when the covariance is minimized, the corresponding source domain is selected as the optimal source of the target domain; The target unknown protocol message format is inferred through a deep transfer learning model based on the selected best source.

2. The network protocol format inference method based on field pattern according to claim 1, characterized in that: The obtaining of network traffic, extracting a target unknown protocol based on the network traffic, and performing decimal integer conversion and sequence label generation based on the target unknown protocol specifically include: Obtaining network traffic, extracting data packets of a target unknown protocol from the network traffic, and grouping them according to protocol type; Traverse each layer in the data packet of the target unknown protocol through a preset network protocol analysis tool, use depth-first search to delete and sort duplicate data, extract information from the target layer, and the field value and field position of the byte in the data packet of the target unknown protocol; The field value is converted to a decimal integer from 0 to 255, and a sequence tag is generated based on the field position and syntax.

3. The network protocol format inference method based on field pattern according to claim 1, characterized in that: The step of calculating the covariance of the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol based on the sequence tag, and selecting the corresponding source domain as the best source of the target domain when the covariance is minimum, specifically includes: Based on the sequence tags, aligning the target domain of the target unknown protocol with the second-order statistical features of the source domain distribution of the pre-acquired known protocol by a linear transformation method; Calculate the covariance between the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol through a preset covariance calculation model based on the alignment result; Put the calculated covariance into a pre-established empty list, and find the minimum covariance value in the empty list; In the case of minimum covariance, the corresponding source domain is selected as the best source for the target domain.

4. The method for inferring a network protocol format based on a field pattern according to claim 3, characterized in that: The covariance between the target domain of the target unknown protocol and the pre-acquired source domain of the known protocol is calculated based on the alignment result by using a preset covariance calculation model, specifically including: Based on the alignment results, the message length data related to the known protocol and the target unknown protocol are calculated through a preset covariance calculation model and cluster analysis is performed to generate a cluster analysis result; Based on the cluster analysis result, label information is generated for the protocol in the given cluster of the source domain protocol message, and transposition calculation is performed in combination with the feature connection between clusters to generate matrix information and decompose it into source domain feature vectors to obtain source domain data, and the target unknown protocol is directly transposed to generate matrix information and decompose it into target domain feature vectors to obtain target domain data; A covariance is calculated based on the source domain data and the target domain data.

5. The method for inferring network protocol format based on field pattern according to claim 1, characterized in that: The inference of the target unknown protocol message format based on the selected best source through a deep transfer learning model specifically includes: Based on the selected best source, cluster the target unknown protocol message length through deep transfer learning model to obtain cluster homogeneous messages; After converting the bytes of the cluster homogeneous message into decimal integers, grammatical feature extraction is performed through the basic network of the deep transfer learning model; The prior syntax of the pre-extracted known protocol is mapped to the grammatical features of the target unknown protocol to infer the message format of the unknown protocol.

6. The method for inferring a network protocol format based on a field pattern according to claim 5, characterized in that: The grammatical feature extraction is performed by the basic network of the deep transfer learning model, specifically including: The input vector is converted into hidden features through the linear layer in the basic network structure of the deep transfer learning model; Based on the hidden features, a nonlinear transformation is introduced through the ReLU activation function in the basic network structure of the deep transfer learning model, and a higher-dimensional output representation in the output protocol field is output to obtain grammatical features.

7. A network protocol format inference system based on field patterns, characterized in that: The system comprises: A data processing module, used for acquiring network traffic, extracting a target unknown protocol based on the network traffic, and performing decimal integer conversion and sequence label generation based on the target unknown protocol; A source domain selection module, configured to perform covariance calculation on the target domain of the target unknown protocol and the source domain of the pre-acquired known protocol based on the sequence tag, and select the corresponding source domain as the best source of the target domain when the covariance is minimum; The knowledge transfer module is used to infer the target unknown protocol message format through a deep transfer learning model based on the selected best source.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the network protocol format inference method based on field pattern as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for inferring a network protocol format based on a field pattern as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for inferring a network protocol format based on a field pattern as described in any one of claims 1 to 6 is implemented.

Images