Bit stream-oriented protocol state machine inference method and device

Abstract

The invention provides a bit stream-oriented protocol state machine inference method and device, and the method comprises the steps: extracting messages of a plurality of unknown protocols from a plurality of data packets analyzed from bit stream network flow data, and carrying out the session segmentation of the plurality of data packets, and obtaining a session segmentation result; performing protocol format clustering on the messages of the plurality of unknown protocols to obtain a format clustering result, and clustering session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result; and for each unknown session protocol in the session clustering result, constructing a protocol type migration set according to the unknown session protocols, converting the protocol type migration set into a state migration set, and inferring a target protocol state machine according to the state migration set. According to the method, state machine inference of multiple unknown protocols is realized at the same time, and the analysis efficiency and accuracy of state machine inference of protocols mixed with multiple unknown protocol data sets are improved.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a bit stream-oriented protocol state machine inference method and device. Background Art

[0002] With the development of network technology and the widespread popularity of network applications, a large number of custom protocols have emerged, and network traffic has become more complex and diversified, bringing severe challenges to malware behavior analysis, intrusion detection, vulnerability scanning, fuzz testing, traffic analysis, etc.

[0003] In the context of unknown protocol research, the ultimate goal is to use protocol reverse engineering technology to analyze network traffic and build a protocol state machine to perform protocol behavior analysis; the protocol state machine can detect abnormal traffic, identify vulnerabilities, optimize scanning strategies, and classify and archive different types of traffic, helping to manage and analyze network traffic and identify abnormal patterns.

[0004] Existing protocol reverse engineering solutions can be divided into program analysis-oriented and network traffic analysis-oriented methods; the former requires access to the protocol binary program or communication with the protocol server, and analyzes the unknown protocol format and field semantics by sending captured or modified unknown protocol data packets to the server and analyzing the server's feedback information or the behavior of the communication entity. Due to the protection mechanism of the private protocol owner or the encryption or obfuscation of the binary program, the binary file of the unknown protocol is usually difficult to obtain in actual situations. This method is technically difficult and the implementation process is complicated; The latter solution only needs to obtain the network traffic of the unknown protocol, which is relatively simple to implement. However, the existing solution focuses on the inference of the protocol format, ignores the analysis of the unknown protocol interaction rounds and state machines, and only supports protocol reverse engineering of a single unknown protocol, resulting in low efficiency and accuracy of the state machine analysis of mixed multiple unknown protocols, and cannot meet the state machine inference requirements of data packets that usually mix multiple protocols in real network traffic. Summary of the invention

[0005] The present invention provides a bit stream-oriented protocol state machine inference method and device, which are used to solve the problems that the prior art uses a program-oriented analysis method to implement protocol state machine inference, which has great technical difficulty and complicated implementation process, and uses a network traffic analysis method to implement protocol state machine inference, which focuses on the inference of the protocol format and only supports a single unknown protocol, resulting in poor analysis effect of the state machine on a mixture of multiple unknown protocols, thereby improving the analysis efficiency and accuracy of the state machine on a mixture of multiple unknown protocols.

[0006] The present invention provides a bit stream-oriented protocol state machine inference method, comprising: Extracting multiple unknown protocol messages from multiple data packets parsed from the bitstream network traffic data, and performing session segmentation processing on the multiple data packets to obtain session segmentation results; Performing protocol format clustering on the messages of the multiple unknown protocols to obtain a format clustering result, and clustering the session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result; For each unknown session protocol in the session clustering result, a protocol type migration set is constructed according to the unknown session protocol, the protocol type migration set is converted into a state migration set, and a target protocol state machine is inferred according to the state migration set; wherein the state migration set is used to represent the interaction process between the communicating parties.

[0007] According to a bitstream-oriented protocol state machine inference method provided by the present invention, the protocol format clustering of the messages of the multiple unknown protocols to obtain the format clustering result includes: Analyze the unknown protocol and construct a feature vector to obtain a feature vector set; the length of each feature vector in the feature vector set is the size of the maximum frequent item set in the messages of the multiple unknown protocols; The feature vector set is clustered according to the target protocol format clustering algorithm to obtain the format clustering result.

[0008] According to a bitstream-oriented protocol state machine inference method provided by the present invention, clustering the session sequences in the session segmentation result according to the format clustering result to obtain the session clustering result includes: Marking all data packets in the session segmentation result by using the format clustering result to obtain the session sequence; A similarity matrix between the conversation sequences is calculated, and the conversations of multiple unknown protocols are clustered according to the similarity matrix to obtain the conversation clustering result.

[0009] According to a bitstream-oriented protocol state machine inference method provided by the present invention, extracting multiple unknown protocol messages from multiple data packets parsed from bitstream network traffic data includes: According to the target protocol format template, the data packets corresponding to different protocol layers of the bit stream network traffic data are nested and parsed and matched, and the matching results are processed as follows: Mark and field store the data packets or parts of data packets that meet the matching conditions; The unrecognizable application layer data is extracted to obtain the messages of the multiple unknown protocols.

[0010] According to a bitstream-oriented protocol state machine inference method provided by the present invention, after constructing a protocol type migration set according to the unknown session protocol, the method further includes: Misclassified data or noise data is removed from the protocol type migration set to obtain a new protocol type migration set.

[0011] According to a bit stream-oriented protocol state machine inference method provided by the present invention, the target protocol format clustering algorithm is an automatic convergence protocol format clustering algorithm; The clustering process of the feature vector set according to the target protocol format clustering algorithm includes: Create convergence function; Based on the silhouette coefficient of the convergence function, the quality of the clustering result output by the target protocol format clustering algorithm is evaluated, the clustering parameters are adjusted according to the variation range of the silhouette coefficient, and multiple iterations are performed until the silhouette coefficient of the clustering result converges to a preset target value to obtain the format clustering result.

[0012] The present invention also provides a bit stream-oriented protocol state machine inference device, comprising: The protocol extraction module is used to extract multiple unknown protocol messages from multiple data packets parsed from the bitstream network traffic data, and perform session segmentation processing on the multiple data packets to obtain session segmentation results; A clustering module, configured to perform protocol format clustering on the messages of the plurality of unknown protocols to obtain a format clustering result, and cluster the session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result; An inference module is used to construct a protocol type migration set according to each unknown session protocol in the session clustering result, convert the protocol type migration set into a state migration set, and infer a target protocol state machine according to the state migration set; wherein the state migration set is used to represent the interaction process between the two communicating parties.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the bitstream-oriented protocol state machine inference method described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for inferring a bitstream-oriented protocol state machine as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned bitstream-oriented protocol state machine inference methods.

[0016] The bitstream-oriented protocol state machine inference method and device provided by the present invention obtain messages and session segmentation results of multiple unknown protocols through data packets parsed in bitstream network traffic data, and perform protocol format clustering on the messages of multiple unknown protocols, and then cluster the session sequences in the session segmentation results according to the format clustering results to obtain the session clustering results; finally, a protocol type migration set is constructed according to the unknown session protocol, and converted into a state migration set to infer the target protocol state machine, thereby realizing the state machine inference of multiple unknown protocols at the same time, and improving the analysis efficiency and accuracy of the protocol state machine inference for a mixed data set of multiple unknown protocols. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the bitstream-oriented protocol state machine inference method provided by the present invention.

[0019] Figure 2 It is a flowchart of an algorithm for extracting the maximum frequent itemset based on the fast Apriori algorithm provided by the present invention.

[0020] Figure 3 It is a schematic diagram of aligning the conversation clustering sequences and calculating the similarity between conversations provided by the present invention.

[0021] Figure 4 This is the second flow chart of the bitstream-oriented protocol state machine inference method provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of a bit stream-oriented protocol state machine inference device provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Combine the following Figure 1-Figure 5 The present invention describes a bitstream-oriented protocol state machine inference method and device.

[0026] Figure 1 This is one of the flow charts of the bit stream-oriented protocol state machine inference method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: extract multiple unknown protocol messages from multiple data packets parsed from the bitstream network traffic data, and perform session segmentation processing on the multiple data packets to obtain session segmentation results.

[0027] It should be noted that the principle of bit stream is a data communication technology based on binary coding and transmission. It converts various types of data into binary form, and then transmits and stores them through data streams in a certain order. During the transmission process, the bit stream can be transmitted serially or in parallel to achieve high-speed data transmission and sharing.

[0028] In this step, in terms of network traffic data, the bit stream can represent any type of data, such as text, audio, video, etc., and convert these data into binary form for transmission and storage through a specific encoding method; the bit stream network traffic data can be represented as "1011001...", the network traffic data.

[0029] In this embodiment, multiple data packets are extracted from the bitstream network traffic data, each data packet includes a packet header and an application layer message, and after each data packet is identified, the data packets of unrecognizable protocol layer messages are marked, thereby extracting an unknown protocol message set, which contains messages of multiple unknown protocols.

[0030] In this embodiment, a plurality of data packets in the bitstream network traffic data are divided according to different session types to obtain a plurality of different session segmentation results, and different session segmentation results include at least one data packet.

[0031] Step 120: cluster the protocol formats of the messages of the multiple unknown protocols to obtain a format clustering result, and cluster the session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result.

[0032] In this step, for the multiple messages in the above unknown protocol message set, the multiple messages can be further divided by constructing feature vectors.

[0033] For example, by constructing a fuzzy membership feature vector, the messages of multiple unknown protocols are clustered into protocol formats, and clustering results of different unknown protocol messages based on protocol formats are obtained.

[0034] In this step, after obtaining the above-mentioned protocol format clustering results, the sessions containing unknown protocol messages are clustered according to the protocol type, realizing protocol reverse engineering for multiple unknown protocols at the same time, and then obtaining clustering results of different unknown protocol messages based on the session content.

[0035] In this embodiment, unknown protocol messages are first clustered according to different protocol formats, and then session clustering is performed based on the clustering results and the session segmentation results. Sessions are clustered according to their protocol types to achieve state machine inference for multiple unknown protocols at the same time.

[0036] Specifically, this embodiment obtains 10 data packets from the bit stream network traffic data parsing, wherein the 10 data packets are clustered according to the protocol format, and the obtained format clustering results include format 1 (corresponding to data packet 1, data packet 3, data packet 5, data packet 7, data packet 9), format 2 (corresponding to data packet 2, data packet 4, data packet 6, data packet 8, data packet 10), and before clustering the above session segmentation results (corresponding to data packets 1-10), all data packets in the session are marked with the results of the protocol format clustering to obtain an unknown protocol session represented by a protocol format class cluster number sequence.

[0037] For example, the above session segmentation results include session 1 (corresponding to data packets 1-4) and session 2 (corresponding to data packets 5-10), that is, session 1 can be represented as a sequence (1,2,1,2), and session 2 can be represented as a sequence (1,2,1,2,1,2). Then, according to the relevant clustering algorithm, clustering is performed according to the corresponding sequences of session 1 and session 2 to obtain the session clustering results.

[0038] In this embodiment, the marking methods for the above different protocol formats include but are not limited to numbers, letters or symbols.

[0039] Step 130: For each unknown session protocol in the session clustering result, a protocol type transition set is constructed according to the unknown session protocol, the protocol type transition set is converted into a state transition set, and a target protocol state machine is inferred according to the state transition set; wherein the state transition set is used to represent the interaction process between the communicating parties.

[0040] In this step, the above-mentioned communicating parties may be a protocol client and a server; or may be an interaction process between other sending ends and receiving ends in the communication system.

[0041] In this embodiment, a protocol state machine is constructed in each session cluster in the above-mentioned different session clustering results, and two adjacent data packets in each session are taken as a protocol format migration to construct a protocol format migration set. Since the protocol format migration set describes the order between data packets, it is necessary to convert the set into a state machine that describes the interaction process between the protocol client and the server, with the state of the communication entity as the state of the state machine and the data packet as the migration condition, and finally the protocol state machine is constructed accordingly.

[0042] It should be noted that real network traffic may contain uncertainty and noise, and misclassification is inevitable in the process of protocol format clustering and session clustering.

[0043] To address this problem, after constructing a protocol type migration set based on the unknown session protocol, the bitstream-oriented protocol state machine inference method adopted in this embodiment also includes: removing misclassified data or noise data from the protocol type migration set to obtain a new protocol type migration set.

[0044] In this embodiment, the method for eliminating misclassified data or noise data includes but is not limited to one or a combination of a probabilistic state machine algorithm, a threshold-based noise filtering algorithm, a sliding window-based noise removal algorithm, and a noise removal algorithm based on machine learning or deep learning.

[0045] In this embodiment, the probabilistic state machine naturally handles the uncertainty of misclassification in real network traffic by means of probabilistic transfer, thereby enhancing the accuracy and robustness of the protocol state machine construction and accurately capturing the behavior pattern of the protocol; therefore, the overall migration probability is increased on the basis of calculating the same initial state migration probability of each migration, and any protocol format migration with a probability less than a threshold is regarded as noise data and eliminated.

[0046] Specifically, assuming that the session clusters in the above different session clustering results include two categories, named protocol 1 and protocol 2 respectively; the automatic state machine construction module specifically includes the following steps: (1) Construct a protocol type migration set in protocol 1 and protocol 2 respectively, and regard every two adjacent protocol format pairs in the session as a migration.

[0047] (2) Perform noise removal, set the probability thresholds Ps = 0.05 and Pt = 0.05, and record the protocol format migration with a probability less than Ps and an overall probability less than Pt as noise data and remove them.

[0048] (3) Convert the protocol format transition set into a state transition set that describes the interaction process between the client and the server and formalize the state machine of each protocol.

[0049] In this embodiment, the target protocol state machine inferred through the above steps only needs the support of unknown protocol network traffic, and there is no need to access the server or communication entity of the unknown protocol. The data is easier to obtain, the scope of application is wider, and the universality is stronger. It can simultaneously infer the protocol state machine of network traffic containing multiple unknown protocols without the need to perform data classification and other steps in advance. It is more practical and has great advantages in protocol behavior analysis and intrusion detection.

[0050] The bitstream-oriented protocol state machine inference method provided by the embodiment of the present invention obtains messages and session segmentation results of multiple unknown protocols through data packets parsed in bitstream network traffic data, and performs protocol format clustering on the messages of multiple unknown protocols, and then clusters the session sequences in the session segmentation results according to the format clustering results to obtain session clustering results; finally, a protocol type migration set is constructed according to the unknown session protocol, and converted into a state migration set to infer the target protocol state machine, thereby realizing state machine inference of multiple unknown protocols at the same time, thereby improving the analysis efficiency and accuracy of protocol state machine inference for mixed multiple unknown protocol data sets.

[0051] In some embodiments, protocol format clustering is performed on messages of multiple unknown protocols to obtain format clustering results, including: analyzing each unknown protocol and constructing a feature vector to obtain a feature vector set; the length of each feature vector in the feature vector set is the size of the maximum frequent item set in the messages of the multiple unknown protocols; and the feature vector set is clustered according to a target protocol format clustering algorithm to obtain a format clustering result.

[0052] In this embodiment, the feature vector construction method includes but is not limited to one of a fuzzy membership function calculation method based on a maximum frequent item set and a data packet feature vector calculation method based on TFD and MFD.

[0053] In this embodiment, the target protocol format clustering algorithm includes but is not limited to one of DBSCAN, K-means, spectral clustering, and hierarchical clustering.

[0054] In this embodiment, a frequent item extraction algorithm is first used to extract the maximum frequent item set from the unknown protocol message data set input, and the minimum support parameter is set to adjust the frequency of occurrence of frequent items, thereby controlling the size of the maximum frequent item set.

[0055] In this embodiment, the frequent item extraction algorithm includes but is not limited to algorithms such as Apriori and FP-Growth.

[0056] In this embodiment, since the frequent items are continuous bit streams, it is ensured that each frequent item is a substring of the unknown protocol message; the length of the frequent items extracted in this embodiment can be defined by oneself, for example, the length of the frequent items can be set to n (n can be equal to 1, 2, 4, 8, etc.).

[0057] Since cardinality byte fields usually do not appear in protocol design, and field types are usually represented by integers or floating-point numbers, fields larger than 8 bytes are usually transmitted as random numbers or strings, which is meaningless for protocol format clustering. This embodiment can be dynamically adjusted according to specific protocol design specifications. If frequent items in the set have an inclusion relationship, the included frequent items are removed to obtain the maximum frequent item set.

[0058] Figure 2 is a flowchart of an algorithm for extracting the maximum frequent itemset according to the fast Apriori algorithm provided by the present invention. Figure 2 In the illustrated embodiment, the step of extracting the maximum frequent item set based on the fast Apriori algorithm includes: setting the initial value of the maximum frequent item extraction algorithm parameter according to the input unknown protocol message set, and setting the initial frequent item set in bytes; when traversing the unknown protocol message set, only retaining the frequent items that meet the parameter requirements, and combining the frequent items in the current set in pairs; when the length of the combined items meets the length requirement and there is an inclusion relationship between the items, deleting the included items, and outputting the maximum frequent item set.

[0059] In this embodiment, a predefined fuzzy membership function can be used to construct a feature vector for each unknown protocol message, and then based on protocol format clustering (including but not limited to DBSCAN, K-means, spectral clustering, hierarchical clustering), a relevant distance formula, such as Euclidean distance, is used to calculate the distance between the feature vectors to obtain the format clustering result; each element in the feature vector reflects the membership of the message to the frequent item at this position.

[0060] Furthermore, the target protocol format clustering algorithm is an automatic convergence protocol format clustering algorithm; clustering the feature vector set according to the target protocol format clustering algorithm includes: creating a convergence function; performing quality assessment on the clustering results output by the target protocol format clustering algorithm based on the contour coefficient of the convergence function, adjusting the clustering parameters according to the range of variation of the contour coefficient, and performing multiple iterations until the contour coefficient of the clustering result converges to a preset target value to obtain a format clustering result.

[0061] Specifically, an automatic convergence DBSCAN algorithm is designed based on the convergence function and the DBSCAN algorithm, and the automatic convergence DBSCAN algorithm performs the following steps to achieve protocol format clustering: (1) The unknown protocol message data set is input into the fast Apriori algorithm to extract the maximum frequent itemset, and the minsupport parameter value is set to 0.35; (2) Use the fuzzy membership function to calculate the feature vector of each unknown protocol message. The formula of the fuzzy membership function is: ; Among them, LCSS (frequent item j, message i) represents the maximum common substring length between frequent item j and message i, and length (frequent item j) represents the length of frequent item j. The length of the feature vector is the size of the maximum frequent item set, which ensures that the feature vectors of all unknown protocol messages are of the same dimension, which is convenient for distance calculation; (3) Set the eps parameter range of the automatic convergence DBSCAN algorithm to 0.1-2.0, the initial value range of the min samples parameter to 5-50, the step size of eps to 0.1, the step size of min samples to 5, and the convergence threshold of the convergence function to 0.1.

[0062] (4) The feature vector set of the unknown protocol message set is input into the automatically converged DBSCAN protocol format clustering algorithm. The algorithm undergoes three iterations. The optimal results of the second and third iterations are equal, with a silhouette coefficient of 0.97. The final result is eps = 0.42, min samples = 17. Compared with the true label, the Rand coefficient is 0.94. A total of 9 protocol format clusters (i.e., format clustering results) are generated, which are consistent with the true label.

[0063] The bitstream-oriented protocol state machine inference method provided by the embodiment of the present invention calculates the feature vectors corresponding to the messages of each unknown protocol by adopting a fuzzy membership function to obtain a feature vector set; then clusters the feature vector set according to an automatic convergence protocol format clustering algorithm to obtain a format clustering result, and automatically converges to the optimal clustering result through multiple iterations, thereby improving the convergence speed, and further improving the efficiency and accuracy of protocol format clustering.

[0064] In some embodiments, clustering the session sequences in the session segmentation results according to the format clustering results to obtain the session clustering results includes: marking all data packets in the session segmentation results using the format clustering results to obtain the session sequences; calculating the similarity matrix between the session sequences, and clustering multiple sessions of unknown protocols according to the similarity matrix to obtain the session clustering results.

[0065] In this embodiment, the similarity matrix calculation method includes but is not limited to one or a combination of sequence alignment algorithm, Euclidean distance, Manhattan distance, cosine similarity, Jaccard similarity coefficient, Hamming distance, and Chebyshev distance.

[0066] In this embodiment, the target protocol format clustering algorithm includes but is not limited to one of K-medoids, K-means, and spectral clustering.

[0067] In this embodiment, the original data packet sequence number in each session is matched one-to-one with the cluster number obtained by clustering the protocol format, and each unknown protocol message session is represented as a sequence of protocol format cluster numbers.

[0068] For example, 10 data packets are obtained from the bitstream network traffic data parsing. The 10 data packets are clustered according to the protocol format, and the obtained format clustering results include format 1 (corresponding to data packet 1, data packet 3, data packet 5, data packet 7, data packet 9) and format 2 (corresponding to data packet 2, data packet 4, data packet 6, data packet 8, and data packet 10). The above session segmentation results include session 1 (corresponding to data packets 1-4) and session 2 (corresponding to data packets 5-10), that is, session 1 can be represented as a sequence (1,2,1,2) and session 2 can be represented as a sequence (1,2,1,2,1,2).

[0069] It should be noted that the protocol format cluster numbers contained in sessions of different protocols will not overlap with sessions of other protocols, while the protocol format clusters contained in sessions of the same protocol have high similarity in type, and because they follow the same protocol process, their sequence orders also have strong similarity; although the number of data packets or processes between different sessions may be inconsistent, by using sequence alignment algorithms, including but not limited to Needleman-Wunsch (NW) and Smith-Waterman (SW), the similarity between sequences is made more obvious, and the number of the same protocol format cluster numbers at the same position is used as the similarity between the two sessions. The higher the similarity, the closer the distance, and vice versa.

[0070] Specifically, this embodiment uses cluster number input type clustering algorithms, including but not limited to K-means and K-medoids, to perform session clustering, and uses the silhouette coefficient to evaluate the quality of the clustering results. Considering that the number of protocols must be less than the number of protocol formats, the number of clusters k of the given clustering algorithm ranges from 1 to the number of protocol format clusters, and the result with the highest silhouette coefficient is selected as the final clustering result. Each session cluster is a protocol.

[0071] Specifically, this embodiment implements clustering of multiple unknown protocol sessions through the following steps: (1) Use the results of protocol format clustering to mark all data packets in the session and obtain an unknown protocol session represented by a sequence of protocol format cluster numbers; (2) Figure 3 is a schematic diagram of aligning the conversation clustering sequences and calculating the similarity between conversations provided by the present invention. Figure 3 In the illustrated embodiment, all sessions are aligned pairwise using the NW algorithm, and the number of the same protocol format cluster number (corresponding to the labeled clusters 4, 6 and 2) at the same position is taken as the similarity between sessions to construct a similarity matrix for the session set.

[0072] (3) Use the K-medoids algorithm to cluster the sessions, grouping the sessions belonging to the same protocol into one category to obtain the final session clustering result.

[0073] The bitstream-oriented protocol state machine inference method provided by the embodiment of the present invention marks all data packets in the session segmentation result through format clustering results, obtains the session sequence, and uses a sequence comparison algorithm to calculate the similarity matrix corresponding to the session sequence, and clusters multiple sessions of unknown protocols according to the similarity matrix, thereby realizing the clustering of sessions of different protocol types and improving the accuracy of session clustering.

[0074] In some embodiments, extracting multiple unknown protocol messages from multiple data packets parsed from bitstream network traffic data includes: nested parsing and matching data packets corresponding to different protocol layers of the bitstream network traffic data according to a target protocol format template, and processing the matching results as follows: marking and field storing data packets or partial data packets that meet the matching conditions; extracting unrecognizable application layer data to obtain multiple unknown protocol messages.

[0075] In this embodiment, the idea of ​​template matching is adopted and a formal language is used to segment the above-mentioned session and extract the unknown protocol message set.

[0076] In this embodiment, the target protocol format template may be a template in a format such as JSON or XML, which is used to describe the protocol format, important field functions, and state machine of a known protocol; In this embodiment, the target protocol format template is used to implement nested parsing of each layer of protocol in a data packet through keyword recognition or multi-threaded template matching.

[0077] Specifically, the template matching system can accurately identify data packets at the data link layer, network layer, transport layer and some application layers in network traffic; then perform template matching on all data packets, mark and store fields for successfully matched data packets or parts of data packets, extract unrecognizable application layer data, and define it as a set of unknown protocol messages.

[0078] The bitstream-oriented protocol state machine inference method provided in the embodiment of the present invention performs nested parsing and matching of data packets corresponding to different protocol layers of bitstream network traffic data through a target protocol format template, marks and stores fields for data packets or partial data packets that meet the matching conditions; extracts unrecognizable application layer data to obtain messages of multiple unknown protocols, thereby improving the parsing and session segmentation efficiency of bitstream network traffic data, and providing data support for subsequent protocol format clustering and session clustering.

[0079] Figure 4 This is the second flow chart of the bit stream-oriented protocol state machine inference method provided by the present invention. Figure 4 In the illustrated embodiment, data preprocessing is performed on the input bit stream network data (1011001…), including extracting an unknown protocol message set from the application layer messages of the n data packets obtained by parsing, performing session segmentation on the n data packets according to the session type, and obtaining a session segmentation result; then, performing protocol format clustering on the unknown protocol message set (including extracting the maximum frequent item from multiple messages, calculating the eigenvector, and performing clustering based on the self-convergent DBSCA algorithm) to obtain a protocol format clustering result; at the same time, performing session clustering on the session segmentation result (including data packet marking, NW similarity matrix calculation, and protocol session clustering) to obtain a session clustering result, and finally performing protocol state machine inference based on the session clustering result (including constructing a probabilistic protocol format migration set based on the session clustering result, removing noise, and converting a state migration set) to obtain a target protocol state machine.

[0080] The bitstream-oriented protocol state machine inference device provided by the present invention is described below. The bitstream-oriented protocol state machine inference device described below and the bitstream-oriented protocol state machine inference method described above can correspond to each other.

[0081] Figure 5 is a schematic diagram of the structure of the bit stream-oriented protocol state machine inference device provided by the present invention, such as Figure 5 As shown, the device includes: a protocol extraction module 510, a clustering module 520 and an inference module 530.

[0082] The protocol extraction module 510 is used to extract multiple unknown protocol messages from multiple data packets parsed from the bitstream network traffic data, and perform session segmentation processing on the multiple data packets to obtain session segmentation results; The clustering module 520 is used to perform protocol format clustering on messages of multiple unknown protocols to obtain a format clustering result, and cluster the session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result; The inference module 530 is used to construct a protocol type migration set according to each unknown session protocol in the session clustering result, convert the protocol type migration set into a state migration set, and infer the target protocol state machine according to the state migration set; wherein the state migration set is used to represent the interaction process between the communicating parties.

[0083] The bitstream-oriented protocol state machine inference device provided by the embodiment of the present invention obtains messages and session segmentation results of multiple unknown protocols through data packets parsed in bitstream network traffic data, and performs protocol format clustering on the messages of multiple unknown protocols, and then clusters the session sequences in the session segmentation results according to the format clustering results to obtain session clustering results; finally, a protocol type migration set is constructed according to the unknown session protocol, and converted into a state migration set to infer the target protocol state machine, thereby realizing state machine inference of multiple unknown protocols at the same time, thereby improving the analysis efficiency and accuracy of protocol state machine inference for mixed multiple unknown protocol data sets.

[0084] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a protocol state machine inference method for bitstream, the method comprising: extracting multiple unknown protocol messages from multiple data packets parsed from the bitstream network traffic data, and performing session segmentation processing on the multiple data packets to obtain a session segmentation result; performing protocol format clustering on the messages of the multiple unknown protocols to obtain a format clustering result, clustering the session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result; for each unknown session protocol in the session clustering result, constructing a protocol type migration set according to the unknown session protocol, converting the protocol type migration set into a state migration set, and inferring the target protocol state machine according to the state migration set; wherein the state migration set is used to represent the interaction process between the two communicating parties.

[0085] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the bitstream-oriented protocol state machine inference method provided by the above-mentioned methods, and the method includes: extracting multiple messages of unknown protocols from multiple data packets parsed from bitstream network traffic data, and performing session segmentation processing on the multiple data packets to obtain session segmentation results; performing protocol format clustering on the messages of the multiple unknown protocols to obtain format clustering results, and clustering the session sequences in the session segmentation results according to the format clustering results to obtain session clustering results; for each unknown session protocol in the session clustering results, constructing a protocol type migration set according to the unknown session protocol, converting the protocol type migration set into a state migration set, and inferring the target protocol state machine according to the state migration set; wherein, the state migration set is used to represent the interaction process between the communicating parties.

[0087] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the bitstream-oriented protocol state machine inference method provided by the above-mentioned methods, the method comprising: extracting multiple messages of unknown protocols from multiple data packets parsed from bitstream network traffic data, and performing session segmentation processing on the multiple data packets to obtain session segmentation results; performing protocol format clustering on the messages of the multiple unknown protocols to obtain format clustering results, clustering the session sequences in the session segmentation results according to the format clustering results to obtain session clustering results; for each unknown session protocol in the session clustering results, constructing a protocol type migration set according to the unknown session protocol, converting the protocol type migration set into a state migration set, and inferring the target protocol state machine according to the state migration set; wherein the state migration set is used to represent the interaction process between the communicating parties.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bitstream-oriented protocol state machine inference method, characterized in that: include: Extracting multiple unknown protocol messages from multiple data packets parsed from the bitstream network traffic data, and performing session segmentation processing on the multiple data packets to obtain session segmentation results; Performing protocol format clustering on the messages of the multiple unknown protocols to obtain a format clustering result, and clustering the session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result; For each unknown session protocol in the session clustering result, a protocol type migration set is constructed according to the unknown session protocol, the protocol type migration set is converted into a state migration set, and a target protocol state machine is inferred according to the state migration set; wherein the state migration set is used to represent the interaction process between the communicating parties.

2. The bitstream-oriented protocol state machine inference method according to claim 1, characterized in that: The performing protocol format clustering on the messages of the multiple unknown protocols to obtain the format clustering result includes: Analyze the unknown protocol and construct a feature vector to obtain a feature vector set; the length of each feature vector in the feature vector set is the size of the maximum frequent item set in the messages of the multiple unknown protocols; The feature vector set is clustered according to the target protocol format clustering algorithm to obtain the format clustering result.

3. The bitstream-oriented protocol state machine inference method according to claim 1, characterized in that: The clustering of the session sequences in the session segmentation result according to the format clustering result to obtain the session clustering result includes: Marking all data packets in the session segmentation result by using the format clustering result to obtain the session sequence; A similarity matrix between the conversation sequences is calculated, and the conversations of multiple unknown protocols are clustered according to the similarity matrix to obtain the conversation clustering result.

4. The bitstream-oriented protocol state machine inference method according to claim 1, characterized in that: Extracting a plurality of unknown protocol messages from a plurality of data packets parsed from the bitstream network traffic data comprises: According to the target protocol format template, the data packets corresponding to different protocol layers of the bit stream network traffic data are nested and parsed and matched, and the matching results are processed as follows: Mark and field store the data packets or parts of data packets that meet the matching conditions; The unrecognizable application layer data is extracted to obtain the messages of the multiple unknown protocols.

5. The bitstream-oriented protocol state machine inference method according to claim 1, characterized in that: After constructing a protocol type migration set according to the unknown session protocol, the method further includes: Misclassified data or noise data is removed from the protocol type migration set to obtain a new protocol type migration set.

6. The bitstream-oriented protocol state machine inference method according to claim 2, characterized in that: The target protocol format clustering algorithm is an automatic convergence protocol format clustering algorithm; The clustering process of the feature vector set according to the target protocol format clustering algorithm includes: Create convergence function; Based on the silhouette coefficient of the convergence function, the quality of the clustering result output by the target protocol format clustering algorithm is evaluated, the clustering parameters are adjusted according to the variation range of the silhouette coefficient, and multiple iterations are performed until the silhouette coefficient of the clustering result converges to a preset target value to obtain the format clustering result.

7. A protocol state machine inference device for bit stream, characterized in that: include: The protocol extraction module is used to extract multiple unknown protocol messages from multiple data packets parsed from the bitstream network traffic data, and perform session segmentation processing on the multiple data packets to obtain session segmentation results; A clustering module, configured to perform protocol format clustering on the messages of the plurality of unknown protocols to obtain a format clustering result, and cluster the session sequences in the session segmentation result according to the format clustering result to obtain a session clustering result; An inference module is used to construct a protocol type migration set according to each unknown session protocol in the session clustering result, convert the protocol type migration set into a state migration set, and infer a target protocol state machine according to the state migration set; wherein the state migration set is used to represent the interaction process between the communicating parties.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the bitstream-oriented protocol state machine inference method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the bitstream-oriented protocol state machine inference method as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the bitstream-oriented protocol state machine inference method as claimed in any one of claims 1 to 6 is implemented.

Images