Method for encrypted anonymous network traffic analysis identification based on traffic reconstruction

Abstract

The present application relates to the technical field of data processing, and discloses an encrypted anonymous network flow analysis and identification method based on flow reconstruction, which comprises the following steps: obtaining message sequences of unknown protocols of a plurality of sessions; obtaining a plurality of initial clusters of the sessions; marking entropy mutation points of each message sequence; obtaining a plurality of message clusters of each initial cluster; obtaining static feature vectors of each message sequence; obtaining mapping symbols of each message sequence, and obtaining receiving probabilities between different message sequences; performing semantic analysis on the message sequences, and constructing semantic annotation sequences of the message sequences; constructing a protocol semantic graph based on entity types and the ownership relationship between the message sequences, unknown protocols and sessions; and analyzing the unknown protocols through the protocol semantic graph. The present application aims to solve the problem that network flow relies on matching with a known feature library, thereby reducing identification efficiency and identification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an encrypted anonymous network traffic analysis and identification method based on traffic reconstruction. BACKGROUND

[0002] In today's digital society, as a basic component in network communication, the security of the cryptographic protocol is directly related to the confidentiality, integrity and availability of the information system; although standard protocols (such as SSL / TLS, IPSec, SSH, etc.) have been widely studied and protected mechanisms, in scenarios such as APT attacks, malware, private industrial communication systems, etc., more and more communications use unknown or non-standard protocols for information exchange. These protocols have high specificity, confusion and even encryption, and often cannot be identified and parsed through traditional methods.

[0003] The current mainstream network protocol identification technology mostly relies on known feature libraries (such as protocol signatures, port numbers, byte sequences, etc.), and is almost powerless for unknown protocols, variant protocols, and encrypted or obfuscated protocols; at the same time, protocol reverse analysis still relies on manual deconstruction of fields, comparison of message characteristics, and analysis of control flow logic, which is time-consuming and prone to errors, and is difficult to meet the real combat needs of "fast identification, fast response" in modern network attack and defense.

[0004] And the knowledge graph (Knowledge Graph) as a semantic network expressing entities and their relationships has good structured, inferable and extensible nature; in recent years, knowledge graph has been gradually popularized in the security field, such as threat intelligence graph, malware graph, vulnerability knowledge graph, etc., which has proven its significant value in complex security entity modeling and semantic reasoning; introducing knowledge graph into unknown cryptographic protocol identification can build a semantic-driven protocol understanding framework, breaking the traditional analysis mode of "message as a black box", realizing the structured expression, graphical management, semantic association and security tracing of knowledge such as protocol fields, message structure, communication logic and key usage, and greatly improving the identification efficiency and system intelligence level. SUMMARY

[0005] The present application provides an encrypted anonymous network traffic analysis and identification method based on traffic reconstruction to solve the problem of existing network traffic relying on matching with known feature libraries, which reduces identification efficiency and identification accuracy, and the technical solution adopted is as follows:

[0006] The present application provides an encrypted anonymous network traffic analysis and identification method based on traffic reconstruction, which comprises the following steps:

[0007] Collecting original network traffic and preprocessing to obtain message sequences of unknown protocols for a plurality of sessions;

[0008] obtaining a plurality of initial clusters of the session based on clustering of lengths of the message sequences in the same session; performing byte window division on the message sequences in the same initial cluster, analyzing entropy value changes of adjacent byte windows, and marking entropy value mutation points of the message sequences; obtaining a plurality of message clusters of the initial cluster based on distribution of the entropy value mutation points and message byte contents before the entropy value mutation points in the message sequences; obtaining a coefficient of variation of each byte in the message clusters, and then obtaining a static feature vector of the message sequences;

[0009] obtaining mapping symbols of the message sequences based on the message clusters corresponding to the message sequences in the same session, obtaining receiving probabilities between different message sequences in combination with order relationships between the message sequences in the session, constructing a semantic annotation sequence of the message sequences in combination with the static feature vector of the message sequences and semantic analysis of the message sequences, and constructing a protocol semantic graph based on entity types and belonging relationships between the message sequences and unknown protocols and sessions.

[0010] obtaining the static feature vector and the receiving probability for the message sequences of the unknown protocol, and analyzing the unknown protocol through the protocol semantic graph.

[0011] Optionally, the method for obtaining the plurality of initial clusters of the session includes the following specific method:

[0012] For a plurality of message sequences of any session, lengths of the message sequences are obtained, a number of clusters is determined through an elbow method and used as a K value in a K-means clustering algorithm, all message sequences of the session are clustered based on the lengths of the message sequences, a distance measurement adopts an absolute value of a difference between the lengths of the message sequences, a plurality of clusters are obtained, and the clusters are used as the plurality of initial clusters of the session.

[0013] Optionally, the method for performing byte window division on the message sequences in the same initial cluster includes the following specific method:

[0014] For a plurality of message sequences in any initial cluster of any session, a length of a message sequence with the smallest length in the message sequences is used as a reference length, message sequences with lengths smaller than the reference length are supplemented, the supplement is performed in byte units, and contents of the supplemented bytes are 0, the message sequences with lengths smaller than the reference length are used as a plurality of processed message sequences of the initial cluster, and message sequences with lengths equal to the reference length are directly used as the processed message sequences.

[0015] A preset window is used to segment any processed message sequence based on a window size, and a plurality of byte windows of the processed message sequence are obtained.

[0016] Optionally, the method for obtaining the entropy value mutation points of the message sequences includes the following specific method:

[0017] The entropy value is calculated for each byte window respectively; an empty byte window is supplemented before the first byte window in any processed message sequence, and the entropy value of the empty byte window is 0; the difference between the entropy value of the next byte window and the entropy value of the previous byte window is calculated for adjacent two byte windows in the processed message sequence, and the difference is taken as the entropy value change amount of the next byte window;

[0018] A preset change threshold is set, and the first byte window is taken as a starting point for judgment; when a byte window with a first entropy value change amount greater than the change threshold appears, the first element in the byte window is marked as an entropy value mutation point of the message sequence corresponding to the processed message sequence;

[0019] The entropy value mutation points of the message sequences in the initial cluster of the session are obtained.

[0020] Optionally, the specific method for obtaining the message class clusters of each initial cluster includes the following steps:

[0021] For the entropy value mutation points of the message sequences in any initial cluster of any session, all the entropy value mutation points are sorted according to the order values of the corresponding bytes in the message sequences;

[0022] For the first entropy value mutation point, a local sequence composed of all the elements before the position corresponding to the first entropy value mutation point in any processed message sequence is obtained as a local sequence of the message sequence corresponding to the processed message sequence at the first entropy value mutation point; the local sequences of all the message sequences in the initial cluster are subjected to DBSCAN clustering, the Euclidean distance between the local sequences of the message sequences is taken as a distance measure, and a plurality of class clusters and a plurality of outliers are obtained; all the outliers are combined to form a to-be-analyzed class cluster, and the other class clusters are taken as message class clusters;

[0023] For the message sequences in the to-be-analyzed class cluster, the local sequences of the message sequences at the second entropy value mutation point are obtained based on the second entropy value mutation point, and the message sequences in the to-be-analyzed class cluster are subjected to DBSCAN clustering to obtain a plurality of class clusters and a plurality of outliers; all the outliers are combined to form a to-be-analyzed class cluster again, and the other class clusters are taken as message class clusters; in this way, the clustering is continued until no outlier exists after clustering at an entropy value mutation point, and the subsequent clustering is stopped, thereby obtaining the message class clusters of the initial cluster of the session.

[0024] Optionally, the specific method for obtaining the variation coefficients of the bytes in each message class cluster and then obtaining the static feature vectors of the message sequences includes the following steps:

[0025] For any message sequence in any message class of any initial cluster of any session, based on the processed message sequence corresponding to each message sequence, calculate the coefficient of variation for bytes at the same position to obtain the coefficient of variation of each byte in the message class; preset a mutation threshold, if the coefficient of variation is less than the mutation threshold, the corresponding byte is taken as a byte to be marked; mark consecutive bytes to be marked in the message class to obtain a number of marked bytes in the message class.

[0026] For any message sequence in the message cluster, the message cluster corresponding to the message sequence, the order value of the bytes corresponding to the entropy mutation point, and the starting order value of each segment marker byte together constitute the static feature vector of the message sequence.

[0027] Optionally, the specific method for obtaining the reception probability between different message sequences is as follows:

[0028] The mapping symbols for each message sequence are arranged in order to form the message sequence of the session; a preset high-frequency threshold is used.

[0029] Several subsequences of the message sequence are obtained. The frequency of occurrence of any subsequence in the message sequence is obtained. The ratio of the frequency of occurrence to the number of elements in the message sequence is used as the frequency of occurrence of the subsequence. If the frequency of occurrence is greater than a high-frequency threshold, the subsequence is used as an initial high-frequency subsequence of the session. Several initial high-frequency subsequences of the session are filtered in this way. Initial high-frequency subsequences that are not fully included are retained. All retained initial high-frequency subsequences are used as several high-frequency subsequences.

[0030] For any message cluster corresponding to a mapping symbol, obtain the number of times the mapping symbol acts as a sender in all high-frequency subsequences, obtain the number of times the mapping symbol corresponding to any other message cluster (excluding the message cluster) acts as a receiver of the mapping symbol of the message cluster in all high-frequency subsequences, and take the ratio of the number of receivers to the number of senders as the probability of the message cluster receiving the other message cluster.

[0031] Obtain the reception probability between different message clusters and use it as the reception probability between message sequences in each message cluster.

[0032] Optionally, the specific method for constructing the semantically labeled sequence of the message sequence includes:

[0033] For any packet sequence, obtain the entropy value of the packet sequence, mark the semantics of the packet sequence through the rule engine, obtain the semantics of each packet sequence and record; according to the elements corresponding to the bytes in each packet sequence, convert each packet sequence into a vector form to obtain the original vector of each packet sequence; for the session in which the packet sequence is located, according to the packet order sequence of the session, obtain the mapping symbol of the packet sequence, and the mapping symbol of the previous packet sequence corresponding to the packet sequence in the packet order sequence as the prefix symbol of the packet sequence; according to the receiving probability corresponding to the packet sequence and other packet sequences, obtain several successor symbols and their probability distribution of the packet sequence, and form a binary tuple for each successor symbol and its receiving probability, and all the binary tuples corresponding to the successor symbols form a vector as the successor symbol vector of the packet sequence;

[0034] For several packet sequences in any packet class cluster, based on the processed packet sequences corresponding to each packet sequence, calculate the entropy value of the bytes at the same position, and arrange them in the order of the bytes to form the entropy value vector of the packet class cluster; arrange the coefficients of variation of each byte of the packet class cluster in the order of the bytes to form the coefficient of variation vector of the packet class cluster;

[0035] The semantics of the packet sequence is extracted using a BiLSTM-CRF joint network, and the semantics of the packet sequence is obtained; the original vector of the packet sequence is processed through the BiLSTM layer, and the entropy value vector and the coefficient of variation vector of the packet class cluster to which the packet sequence belongs are used to capture the internal dependency relationship of the field; the attention mechanism is used to focus on the key position, and the predecessor symbol and the successor symbol vector of the packet sequence are used to weight the dynamic context features; the semantics annotation sequence of the packet sequence is output through the CRF annotation.

[0036] Optionally, the method for constructing the protocol semantic graph includes the following specific method:

[0037] For several packet sequences (unknown protocol) in any session, based on entity type division, extract several different entities to obtain several entity nodes corresponding to the packet sequence; the similarity between different entity nodes is calculated using Jaccard similarity, and a preset similarity threshold is used; if the similarity is greater than the similarity threshold, the corresponding two entity nodes are merged, and after merging all entity nodes of the session, several final entity nodes of the session are obtained;

[0038] For several packet sequences in any session, the final entity nodes are generated according to the belonging relationship of bytes, packets and sessions.

[0039] For each packet sequence in the session, several final entity nodes and entity relationship edges are obtained, and a protocol semantic graph composed of unknown protocols in the session is constructed.

[0040] Optionally, the method for analyzing unknown protocols through protocol semantic graphs comprises the following specific steps:

[0041] Obtain the added traffic data, and obtain the corresponding session and unknown protocol therein as an added session and an added unknown protocol, obtain the message sequence and static feature vector of the unknown protocol, obtain the receiving probability of the mapping symbol corresponding to each message sequence according to the corresponding message cluster and the session to which the unknown protocol belongs, and obtain the protocol semantic graph constituted by the unknown protocol in the added session;

[0042] For any known protocol semantic graph and any added unknown protocol in the added session, take any final entity node in the known protocol semantic graph as a known node, obtain the cosine similarity between the static feature vector of the added unknown protocol and the known node and the cosine similarity between the successor symbol vectors, take the average of the two cosine similarities as the node matching degree between the added unknown protocol and the known node;

[0043] Obtain a plurality of entity relationship edges of the known node, obtain a plurality of entity relationship edges of the added unknown protocol, calculate the cosine similarity between any entity relationship edge of the known node and any entity relationship edge of the added unknown protocol, take the average of all cosine similarities obtained from the two groups of entity relationship edges as the entity matching degree between the added unknown protocol and the known node, match all added unknown protocols in the protocol semantic graph of the added session with each final entity node in the known protocol semantic graph, adopt KM matching, the matching rule is that the greater the entity matching degree, the more matched, obtain the matching node pair, take the average of the node matching degrees corresponding to all matching node pairs as the protocol matching degree between the added session and the known protocol semantic graph;

[0044] Obtain the protocol matching degree between the added session and each known protocol semantic graph, take the maximum value of the protocol matching degrees as the final matching degree of the added session;

[0045] Pre-set a first matching threshold and a second matching threshold, if the final matching degree is greater than or equal to the first matching threshold, output the session traffic protocol corresponding to the known protocol semantic graph corresponding to the final matching degree as the session traffic protocol corresponding to the added session; if the final matching degree is less than the first matching threshold and greater than or equal to the second matching threshold, mark the added session as a variant of the session traffic protocol corresponding to the known protocol semantic graph corresponding to the final matching degree; if the final matching degree is less than the second matching threshold, mark the added session as a new unknown session traffic protocol.

[0046] The beneficial effects of the present application are: the present application is based on the static division of protocol instruction according to entropy value mutation, the entropy value mutation point is detected through the entropy value calculation of the byte window, the boundary of the protocol instruction field and the payload field is identified, and the preliminary structured analysis of the protocol message is realized; the same length message is subdivided through clustering for multiple rounds, and the instruction types with different functions are identified; the fixed field is further identified through the coefficient of variation, and the accuracy of the protocol field identification is enhanced; the dynamic interaction behavior characteristics of the protocol are captured, the message sequence, the entropy value, the coefficient of variation and the context characteristics of the message sequence in the session are fused, the joint semantic labeling is carried out by using the BiLSTM-CRF model, and the protocol semantic graph is constructed, and the entity fusion and conflict resolution are carried out by using the Jaccard similarity, so that the structured storage and semantic association of the protocol knowledge are realized; finally, the unknown protocol, the protocol variant and the known protocol are accurately identified and classified by combining the static and dynamic characteristics; through the semantic feature modeling and the knowledge graph technology, the limitation of the traditional dependence on the feature library is broken through, the unknown, private or encrypted protocol is effectively analyzed and identified; combined with the machine learning and the graph computing technology, the artificial intervention is reduced, the efficiency and the accuracy of the protocol reverse analysis are improved, the system is suitable for a large-scale network flow environment, and can identify the slight variant and the upgraded version of the protocol, enhance the adaptability and the expansibility of the system, and has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0048] Figure 1 The flowchart of the encrypted anonymous network flow analysis and identification method provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] Please refer to Figure 1 , which shows the flowchart of the encrypted anonymous network flow analysis and identification method provided by an embodiment of the present application. The method comprises the following steps:

[0051] Step S001, collect original network traffic and pre-process to obtain message sequences of several unknown protocols of several sessions.

[0052] The purpose of the embodiment is to build a set of intelligent identification system for unknown cryptographic protocols, which combines protocol reverse analysis, machine learning, graph computing and knowledge graph technology, realizes feature extraction, classification identification, semantic modeling and graph construction of unknown or encrypted communication protocols; first, the network traffic needs to be captured, and the session is shunted to extract several unknown protocols, and the message sequence of the content of the unknown protocol is obtained for subsequent analysis.

[0053] Specifically, the original traffic is captured by Scapy or libpcap; the original traffic is shunted according to the five-tuple (source IP address, destination IP address, source port number, destination port number and protocol type), and the retransmission packets and error check packets are removed to obtain several unknown protocols of several sessions. The content in the unknown protocol is sequenced to obtain the message sequence of the unknown protocol. The message sequences of several unknown protocols in the same session are sorted according to the time stamp corresponding to the time sequence relationship, while the direction relationship of each unknown protocol in the session is retained, that is, the direction relationship of the sender and receiver of the unknown protocol. Therefore, the message sequences of several unknown protocols of several sessions are obtained.

[0054] It should be noted that unknown protocols often lack public documents and field definitions, and their structural features must be identified from the original network traffic, so the semantic features of the unknown protocol traffic are extracted first; the traffic features of the communication protocol can be divided into two types: static features and dynamic features; among them, the static feature refers to the protocol statement structure; and the dynamic feature is related to the behavior logic set by the protocol, that is, the static feature is the statement structure of a single sentence, and the dynamic feature is the interaction feature and relationship of multiple sentences.

[0055] Step S002, clustering based on the length of the message sequence in the same session, obtaining several initial clusters of the session; dividing the byte window of each message sequence in the same initial cluster, analyzing the entropy value change of adjacent byte windows, and marking the entropy value mutation point of each message sequence; according to the distribution of the entropy value mutation point in each message sequence and the message byte content before it, clustering analysis obtains several message clusters of each initial cluster; obtaining the coefficient of variation of each byte in each message cluster, and then obtaining the static feature vector of each message sequence.

[0056] It should be noted that in the message of the communication protocol, a large number of messages with specific length values will appear, which usually correspond to control messages or fixed format messages in the protocol; for example, heartbeat packets, acknowledgement packets, command request packets usually have fixed lengths, and long communication packets; therefore, first of all, the message sequence is divided based on the message length by using a clustering algorithm, and subsequent analysis is performed in different initial clusters.

[0057] Preferably, in an embodiment of the present application, clustering is performed based on the lengths of the message sequences in the same session to obtain a plurality of initial clusters of the session, and the specific method includes the following steps:

[0058] For a plurality of message sequences of any session, the lengths of the message sequences are obtained, the number of clusters is determined by the elbow method and used as the K value in the K-means clustering algorithm, the message sequences of the session are clustered based on their lengths, the distance metric uses the absolute value of the difference between the lengths of the message sequences, a plurality of clusters are obtained, and each cluster is used as an initial cluster of the session.

[0059] It should be further noted that in a type of protocol messages, the lengths are the same or similar, but this does not mean that this type of message only contains one type of protocol message, but can also contain a plurality of messages with the same length; and the instruction bytes of messages with different purposes are different, and are usually composed of a message type (instruction byte) and a length field (actual payload byte), and the message types of the same type of instruction message are the same; and the actual payload byte after the instruction byte has a large change in entropy value because its content is the actual communication content, so that the entropy value between the instruction byte and the actual payload byte will have a sudden change, and the instruction static division is performed by detecting the sudden change in the entropy value.

[0060] Preferably, in an embodiment of the present application, the message sequences in the same initial cluster are divided into byte windows, the changes in the entropy values of adjacent byte windows are analyzed, and the entropy sudden change points of the message sequences are marked, and the specific method includes the following steps:

[0061] For a plurality of message sequences in any initial cluster of any session, the length (the number of elements in the message sequence) of the message sequence with the maximum length is used as a reference length, the message sequences with lengths less than the reference length are supplemented, the supplement is performed in byte units, and the contents of the supplemented bytes are 0, which are used as a plurality of processed message sequences of the initial cluster, and the message sequences with lengths equal to the reference length are directly used as the processed message sequences; a preset window is used, and in this embodiment, the window size is described as 4 bytes, any processed message sequence is segmented according to the window size to obtain a plurality of byte windows of the processed message sequence, and the entropy value of each byte window is calculated.

[0062] Further, an empty byte window is supplemented before the first byte window in the processed packet sequence, and the entropy value of the empty byte window is 0 (there is a window and elements, but the elements are all bytes with content 0); the difference between the entropy value of the next byte window and the entropy value of the previous byte window is calculated for two adjacent byte windows in the processed packet sequence, and the difference is taken as the entropy value change of the next byte window; a preset change threshold is used, and the change threshold is 1.5 in the embodiment; starting from the first byte window, it is judged whether the first entropy value change of the byte window is greater than the change threshold; when the first entropy value change of the byte window is greater than the change threshold, the first element (byte) in the byte window is marked as an entropy value mutation point of the packet sequence corresponding to the processed packet sequence; the entropy value mutation points of the packet sequences in the initial cluster of the session are obtained according to the above method.

[0063] Further, it needs to be further explained that for the same length of the cluster, there may be a plurality of different purpose packets in the cluster, and the instruction byte length of the packets with different purposes is also different, so it is necessary to further cluster and divide, so as to accurately identify the instruction packets with different purposes, such as the following three different packets:

[0064] 01 03 0D DD ** ** ** ** 00;

[0065] 01 03 ** ** ** ** ** ** 00;

[0066] 01 05 03 DD ** ** ** ** 00;

[0067] The mutation points detected by the entropy value mutation detection are two, that is, window 2 (between window 3) and window 4 (between window 5).

[0068] Preferably, in an embodiment of the present application, according to the distribution of the entropy value mutation points in each packet sequence and the byte content before the entropy value mutation points, a plurality of packet clusters of each initial cluster are obtained by cluster analysis, and the specific method includes:

[0069] For any initial cluster of any session, a number of entropy value mutation points of each message sequence are sorted according to the order values of the bytes corresponding to the entropy value mutation points in the message sequence (the order in the message sequence); for the first entropy value mutation point, a local sequence formed by all elements before the position corresponding to the entropy value mutation point in any processed message sequence is obtained as the local sequence of the message sequence corresponding to the processed message sequence at the first entropy value mutation point; the local sequences of all message sequences in the initial cluster are subjected to DBSCAN clustering, and the Euclidean distance between the local sequences of the message sequences is used as the distance measure, thereby obtaining a number of clusters and a number of outliers, and all outliers form a to-be-analyzed cluster, and the other clusters are taken as message clusters; for the message sequences in the to-be-analyzed cluster, based on the second entropy value mutation point, the local sequences of the message sequences at the second entropy value mutation point are obtained, and the message sequences in the to-be-analyzed cluster are subjected to DBSCAN clustering, thereby obtaining a number of clusters and a number of outliers, and all outliers are reformed into a to-be-analyzed cluster, and the other clusters are taken as message clusters; the process is repeated until, after clustering at an entropy value mutation point, there are no outliers, and subsequent clustering is stopped, thereby obtaining a number of message clusters of the initial cluster of the session.

[0070] It should be further explained that, in each message cluster, in addition to the fixed bytes of the message header, there can be other fixed and small fluctuation bytes, such as IP, counter, timestamp, etc.; the changes of the bytes relative to other bytes are small, and therefore the coefficient of variation is used to identify and mark the continuous bytes in each type of message.

[0071] Preferably, in an embodiment of the present application, the coefficient of variation of each byte in each message cluster is obtained, and then the static feature vector of each message sequence is obtained, and the specific method includes:

[0072] For a number of message sequences in any message cluster of any initial cluster of any session, the coefficient of variation (the ratio of the mean value to the variance) of the bytes at the same position in the processed message sequences corresponding to the message sequences is calculated, thereby obtaining the coefficient of variation of each byte in the message cluster; a variation threshold is preset, and the variation threshold is 0.1 in this embodiment, and if the coefficient of variation is less than the variation threshold, the corresponding byte is taken as a to-be-marked byte; the continuous to-be-marked bytes (the number of continuous to-be-marked bytes is greater than or equal to 2) in the message cluster are marked, thereby obtaining a number of marked bytes of the message cluster.

[0073] Further, for any message sequence in the message cluster, the message cluster corresponding to the message sequence, the order value of the byte corresponding to the entropy value mutation point, and the starting order value of each segment of the marked bytes (the order value of the starting marked byte and the order value of the ending marked byte) jointly form the static feature vector of the message sequence.

[0074] At this point, the static feature vector of each message sequence is obtained.

[0075] In step S003, the mapping symbol of each message sequence is obtained based on the message class cluster corresponding to the message sequence in the same session, the receiving probability between different message sequences is obtained in combination with the order relationship between the message sequences in the session, the semantic annotation sequence of the message sequence is constructed in combination with the static feature vector of the message sequence and the semantic analysis of the message sequence, and the protocol semantic graph is constructed based on the entity type and the belonging relationship between the message sequence and the unknown protocol and the session.

[0076] It should be noted that the dynamic behavior feature in the protocol refers to the interaction behavior between different instructions, such as handshake behavior (firstly exchanging messages with each other), packet sequence alternation mode, and feature sequence in the encryption communication stage; and the interaction behavior of the instructions is often reflected in the length interaction of the message, the packet sequence mode coding, and the traffic change, so the dynamic features of each message are calculated; the protocol is a state machine, and any session is a series of state transitions generated by the state machine; different operations (such as uploading, downloading, heartbeat, and command query) of the same protocol will trigger different state transition paths, thereby generating different packet sequence modes, i.e., different transmission orders of messages.

[0077] Preferably, in an embodiment of the present application, the mapping symbol of each message sequence is obtained based on the message class cluster corresponding to the message sequence in the same session, the receiving probability between different message sequences is obtained in combination with the order relationship between the message sequences in the session, and the specific method comprises:

[0078] For a plurality of message sequences of any session, different message class clusters are corresponded, and the message sequences in the same message class cluster are mapped to the same symbol through abstract processing of all message sequences, so that the mapping symbol of each message sequence is obtained.

[0079] Further, the mapping symbols of the message sequences are arranged in order to form a message order sequence of the session; a high-frequency threshold is preset, and the high-frequency threshold is 0.2 in the embodiment; a plurality of subsequences of the message order sequence are obtained, i.e., a small sequence composed of two or more adjacent elements, the occurrence frequency of any subsequence in the message order sequence is obtained, the ratio of the occurrence frequency to the number of elements in the message order sequence is taken as the occurrence frequency of the subsequence, if the occurrence frequency is greater than the high-frequency threshold, the subsequence is taken as an initial high-frequency subsequence of the session, and a plurality of initial high-frequency subsequences of the session are screened in this way; if any initial high-frequency subsequence is a subsequence of another initial high-frequency subsequence, the initial high-frequency subsequence is deleted, i.e., only the initial high-frequency subsequence not completely included is reserved, and all the reserved initial high-frequency subsequences are taken as a plurality of high-frequency subsequences.

[0080] Further, for any mapping symbol corresponding to a message class cluster, the number of times the mapping symbol is a sender in all high-frequency sub-sequences is obtained, the number of times the mapping symbol corresponding to any other message class cluster is a receiver of the mapping symbol of the message class cluster in all high-frequency sub-sequences is obtained, and the ratio of the number of times of the receiver to the number of times of the sender is taken as the receiving probability of the message class cluster to the other message class cluster. The receiving probability between different message class clusters is obtained according to the above method and taken as the receiving probability between message sequences in each message class cluster.

[0081] It needs to be further explained that different messages have different characteristic performances in the bytes of the messages, and therefore obvious semantic annotation is performed by using a rule engine.

[0082] Preferably, in an embodiment of the present application, the static characteristic vector of the message sequence is combined, semantic analysis of the message sequence is performed, and a semantic annotation sequence of the message sequence is constructed, including the following specific method:

[0083] For any message sequence, the entropy value of the message sequence is obtained, the semantic of the message sequence is annotated by using a rule engine, for example, a high-entropy message (entropy value greater than 6.5) is an encryption field, a monotonically increasing fixed-length field is a timestamp, and a self-correlation function has an obvious peak value, which is a synchronization code. The semantic of the message sequence is annotated by using an entropy value by a rule engine, which is a known technology, and the embodiment will not be described again. The semantic of each message sequence is obtained and recorded. Meanwhile, each message sequence is converted into a vector form according to the elements corresponding to the bytes in the message sequence, and the original vector of each message sequence is obtained. For the session in which the message sequence is located, the mapping symbol of the message sequence is obtained according to the message order sequence of the session, and the mapping symbol of the previous message sequence corresponding to the message sequence in the message order sequence is taken as the prefix symbol of the message sequence. Meanwhile, the receiving probability of the message sequence to other message sequences, i.e., the receiving probability between the mapping symbols corresponding to the message sequence and other message class clusters, is obtained, and the several successor symbols and the probability distribution of the message sequence are obtained. A binary tuple is formed by the successor symbol and the receiving probability, and a vector is formed by all the binary tuples corresponding to the successor symbols, which is taken as the successor symbol vector of the message sequence.

[0084] Further, for several message sequences in any message class cluster, the entropy value of the bytes at the same position in the processed message sequences corresponding to the message sequences is calculated, i.e., the entropy value of the elements corresponding to the same bytes is calculated, and the entropy value vector of the message class cluster is arranged in the order of the bytes. Similarly, the coefficient of variation of each byte of the message class cluster is arranged in the order of the bytes, and the coefficient of variation vector of the message class cluster is constructed.

[0085] Further, the semantic extraction of the message sequence is performed by using a BiLSTM-CRF combined network to obtain the semantic of the message sequence; the original vector of the message sequence is processed by a BiLSTM layer, and the internal dependency of the field is captured based on the entropy vector and the coefficient of variation vector of the message cluster to which the message sequence belongs; the attention mechanism is used to focus on the key position, and the dynamic context feature is weighted based on the pre-symbol and post-symbol vector of the message sequence, that is, the context analysis is performed based on the pre-symbol and post-symbol vector; the semantic annotation sequence (such as identity, used to indicate the semantic performance of the message sequence) of the message sequence is output by performing annotation by CRF, wherein the BiLSTM-CRF combined network is prior art, and details are not described herein.

[0086] It should be further explained that after the semantic annotation sequence of a large number of unknown protocol message sequences is obtained, a knowledge graph is constructed based on the semantic annotation sequence and the entity type of the message sequence; meanwhile, the ownership relationship among the sessions, messages and bytes corresponding to the original flow is considered, so as to generate the corresponding entity relationship edge of the protocol, thereby providing a basis for subsequent analysis and recognition based on the protocol semantic graph.

[0087] Preferably, in an embodiment of the present application, the protocol semantic graph is constructed based on the entity type and the ownership relationship between the message sequence and the unknown protocol and the session, and the specific method comprises:

[0088] The entity recognition is performed on each message sequence according to the semantic annotation sequence thereof, and the entity type is divided in the embodiment as follows:

[0089] ;

[0090] For a plurality of message sequences (unknown protocol) in any session, a plurality of different entities are extracted based on the entity type division, to obtain a plurality of entity nodes corresponding to the message sequences in the session, wherein the entity extraction and the entity type division are prior art, and details are not described herein; the Jaccard similarity is used to calculate the similarity between different entity nodes, a preset similarity threshold is used, and the similarity threshold is 0.9 in the embodiment; if the similarity is greater than the similarity threshold, the corresponding two entity nodes are merged; after the merging of all entity nodes in the session is judged, a plurality of final entity nodes of the session are obtained.

[0091] It should be explained that the entity is extracted by the semantic annotation sequence (such as flow statistics, byte sequence and context behavior), and the protocol entity information is redundant, ambiguous and contradictory, and needs to be fused and conflict resolved to form a unified, consistent and accurate protocol knowledge graph, otherwise the broken and wrong knowledge will lead to wrong recognition results.

[0092] Further, for any message sequence (unknown protocol) in a session, the final entity node needs to be generated according to the byte, message and session belonging relationship, and the entity relationship edge is generated, as shown in the following table:

[0093] ;

[0094] The semantic constraints of different entity nodes are added by using OWL, and the entity relationship edge of each final entity node is obtained, and then the protocol semantic graph of the unknown protocol in the session is constructed.

[0095] At this point, the construction of the protocol semantic graph is completed.

[0096] Step S004, the static feature vector and the receiving probability of the message sequence of the unknown protocol are obtained, the unknown protocol is analyzed through the protocol semantic graph, and the protocol analysis and identification of the network traffic are realized.

[0097] It should be noted that for the newly added traffic data, after obtaining the static feature vector and the receiving probability of the message sequence, the corresponding protocol semantic graph is constructed; if the generated protocol semantic graph is the same as the known protocol semantic graph, it is a known protocol; if there are a few different entity nodes, it is an unknown variant of the known protocol; if there is a large difference, it needs to be marked as an unknown protocol.

[0098] Specifically, the newly added traffic data is obtained, and the corresponding session and unknown protocol are obtained as the newly added session and the newly added unknown protocol. The message sequence and its static feature vector of the unknown protocol are obtained according to the above method, and the receiving probability of the mapping symbol corresponding to each message sequence is obtained according to the corresponding message cluster and the session to which it belongs. The protocol semantic graph of the unknown protocol in the newly added session is obtained according to the above method.

[0099] Further, for any known protocol semantic graph and any newly added unknown protocol in the newly added session, any final entity node in the known protocol semantic graph is taken as a known node, the static feature vector and the successor symbol vector corresponding to the known node are obtained, the cosine similarity between the static feature vector of the newly added unknown protocol and the static feature vector of the known node is obtained, the cosine similarity between the successor symbol vector of the newly added unknown protocol and the successor symbol vector of the known node is obtained, and the average of the two cosine similarities is taken as the node matching degree between the newly added unknown protocol and the known node.

[0100] Further, a plurality of entity relationship edges of the known node are acquired, a plurality of entity relationship edges of the added unknown protocol are acquired, cosine similarity is calculated for any entity relationship edge of the known node and any entity relationship edge of the added unknown protocol (where the entity relationship edges correspond to vectors of different lengths, the cosine similarity is calculated after the vectors are made equal in length by supplementing 0), and the average of all cosine similarities obtained by the two groups of entity relationship edges and the average of the node matching degrees are taken as the entity matching degree of the added unknown protocol and the known node; all added unknown protocols in the protocol semantic graph of the added session are matched with each final entity node in the known protocol semantic graph, KM matching is adopted, the matching rule is that the greater the entity matching degree, the more matched, a matching node pair is obtained, and the average of the node matching degrees corresponding to all matching node pairs is taken as the protocol matching degree of the added session and the known protocol semantic graph.

[0101] Further, the protocol matching degree of the added session and each known protocol semantic graph is acquired according to the above method, the maximum value of the protocol matching degrees is taken as the final matching degree of the added session, a first matching threshold and a second matching threshold are preset, the first matching threshold is described by using 0.9 in this embodiment, and the second matching threshold is described by using 0.6; if the final matching degree is greater than or equal to the first matching threshold, the session traffic protocol corresponding to the known protocol semantic graph corresponding to the final matching degree is directly output as the session traffic protocol corresponding to the added session; if the final matching degree is less than the first matching threshold and greater than or equal to the second matching threshold, the added session is marked as a variant of the session traffic protocol corresponding to the known protocol semantic graph corresponding to the final matching degree, the added session is directly output as a session traffic protocol, and the protocol semantic graph of the added session is added to all known protocol semantic graph libraries; if the final matching degree is less than the second matching threshold, the added session is marked as a new unknown session traffic protocol, and the added session is directly output as a session traffic protocol, and the protocol semantic graph of the added session is added to all known protocol semantic graph libraries; and thus protocol analysis and identification of sessions in added traffic are realized.

[0102] Thus, the embodiment is completed.

[0103] The above merely describes preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for analyzing and identifying encrypted and anonymous network traffic based on traffic reconstruction, characterized in that, The method includes the following steps: Collect raw network traffic and preprocess it to obtain message sequences of several unknown protocols from several sessions; Clustering is performed based on the length of message sequences within the same session to obtain several initial clusters of the session. Byte windows are then divided into segments for each message sequence within the same initial cluster. The entropy changes between adjacent byte windows are analyzed, and entropy abrupt change points in each message sequence are marked. Based on the distribution of entropy abrupt change points in each message sequence and the preceding message byte content, clustering analysis is performed to obtain several message clusters within each initial cluster. The coefficient of variation of each byte in each message cluster is obtained, thus yielding the static feature vector of each message sequence. Based on the message clusters corresponding to message sequences in the same session, the mapping symbols of each message sequence are obtained. Combined with the order relationship between message sequences in the session, the reception probability between different message sequences is obtained. Combined with the static feature vectors of message sequences and semantic analysis of message sequences, a semantic annotation sequence of message sequences is constructed. Based on entity type and the attribution relationship between message sequences and unknown protocols and sessions, a protocol semantic graph is constructed. Static feature vectors and reception probabilities are obtained for message sequences of unknown protocols, and the unknown protocols are analyzed through protocol semantic graphs; The specific method for dividing the message sequences in the same initial cluster into byte windows includes: For any initial cluster of messages in any session, the length of the longest message sequence is used as the base length. Message sequences with a length less than the base length are supplemented by byte by byte, with the content of the supplemented bytes being 0. These are used as several processed message sequences of the initial cluster. Message sequences with a length equal to the base length are directly used as processed message sequences. A preset window is used to segment any processed message sequence according to the window size, resulting in a window of several bytes for that processed message sequence. The specific method for obtaining the entropy abrupt change points of each message sequence is as follows: Calculate the entropy value for each byte window; add an empty byte window before the first byte window in any processed message sequence, the entropy value of the empty byte window is 0; calculate the difference between the entropy value of the next byte window and the entropy value of the previous byte window for two adjacent byte windows in the processed message sequence, and use it as the change in entropy value of the next byte window. A preset change threshold is set, and the judgment starts from the first byte window. When the first byte window with an entropy value change greater than the change threshold appears, the first element in the byte window is marked as the entropy value mutation point of the message sequence corresponding to the processed message sequence. Obtain the entropy mutation point of each message sequence in the initial cluster of this session.

2. The encrypted anonymous network traffic analysis and identification method based on traffic reconstruction according to claim 1, characterized in that, The specific methods for obtaining several initial clusters of the session are as follows: For any given session, obtain the length of each message sequence, determine the number of clusters using the elbow method and use it as the K value in the K-means clustering algorithm, cluster all message sequences of the session according to their lengths, and use the absolute value of the difference between the lengths of the message sequences as the distance metric to obtain several clusters, and use each cluster as a set of initial clusters for the session.

3. The encrypted anonymous network traffic analysis and identification method based on traffic reconstruction according to claim 1, characterized in that, The specific method for obtaining several message clusters for each initial cluster is as follows: For any initial cluster of any session, several entropy value mutation points in each message sequence are sorted according to the order of the bytes corresponding to each entropy value mutation point in the message sequence. For the first entropy mutation point, obtain the local sequence consisting of all elements before the corresponding position of the entropy mutation point in any processed message sequence, and use it as the local sequence of the message sequence corresponding to the processed message sequence under the first entropy mutation point; perform DBSCAN clustering on the local sequences of all message sequences in the initial cluster, and use the Euclidean distance between the local sequences of each message sequence as the distance metric to obtain several clusters and several outliers. All outliers are combined into a cluster to be analyzed, and the other clusters are used as message clusters; For the message sequences in the cluster to be analyzed, based on the second entropy value mutation point, the local sequence of each message sequence under the second entropy value mutation point is obtained. DBSCAN clustering is performed on the message sequences in the cluster to be analyzed to obtain several clusters and several outliers. All outliers are reconstructed into a cluster to be analyzed, and the other clusters are used as message clusters. This process is repeated until an entropy value mutation point is reached and no outliers are found. Then, the subsequent clustering stops, and several message clusters of the initial cluster of the session are obtained.

4. The encrypted anonymous network traffic analysis and identification method based on traffic reconstruction according to claim 1, characterized in that, The specific method for obtaining the variation coefficient of each byte in each message cluster, and then obtaining the static feature vector of each message sequence, includes: For any message sequence in any message class of any initial cluster of any session, based on the processed message sequence corresponding to each message sequence, calculate the coefficient of variation for bytes at the same position to obtain the coefficient of variation of each byte in the message class; preset a mutation threshold, if the coefficient of variation is less than the mutation threshold, the corresponding byte is taken as a byte to be marked; mark consecutive bytes to be marked in the message class to obtain a number of marked bytes in the message class. For any message sequence in the message cluster, the message cluster corresponding to the message sequence, the order value of the bytes corresponding to the entropy mutation point, and the starting order value of each segment marker byte together constitute the static feature vector of the message sequence.

5. The encrypted anonymous network traffic analysis and identification method based on traffic reconstruction according to claim 3, characterized in that, The specific method for obtaining the reception probability between different message sequences is as follows: The mapping symbols for each message sequence are arranged in order to form the message sequence of the session; a preset high-frequency threshold is used. Several subsequences of the message sequence are obtained. The frequency of occurrence of any subsequence in the message sequence is obtained. The ratio of the frequency of occurrence to the number of elements in the message sequence is used as the frequency of occurrence of the subsequence. If the frequency of occurrence is greater than a high-frequency threshold, the subsequence is used as an initial high-frequency subsequence of the session. Several initial high-frequency subsequences of the session are filtered in this way. Initial high-frequency subsequences that are not fully included are retained. All retained initial high-frequency subsequences are used as several high-frequency subsequences. For any message cluster corresponding to a mapping symbol, obtain the number of times the mapping symbol acts as a sender in all high-frequency subsequences, obtain the number of times the mapping symbol corresponding to any other message cluster (excluding the message cluster) acts as a receiver of the mapping symbol of the message cluster in all high-frequency subsequences, and take the ratio of the number of receivers to the number of senders as the probability of the message cluster receiving the other message cluster. Obtain the reception probability between different message clusters and use it as the reception probability between message sequences in each message cluster.

6. The encrypted anonymous network traffic analysis and identification method based on traffic reconstruction according to claim 5, characterized in that, The specific methods for constructing the semantically labeled sequence of the message sequence are as follows: For any message sequence, obtain its entropy value, annotate its semantics using a rule engine, and record the semantics of each message sequence. Convert each message sequence into a vector based on the elements corresponding to its bytes, obtaining the original vector for each message sequence. For the session to which the message sequence belongs, obtain its mapping symbol and the mapping symbol of the preceding message sequence in the message sequence, based on the message order of the session, as the preceding symbol for the message sequence. Based on the reception probabilities of the message sequence and other message sequences, obtain several successor symbols and their probability distributions for the message sequence. Form a tuple for each successor symbol and its reception probability, and construct a vector from all the tuples corresponding to the successor symbols, as the successor symbol vector for the message sequence. For any message sequence in a message cluster, based on the processed message sequence corresponding to each message sequence, the entropy value is calculated for the bytes at the same position, and arranged in the order of bytes to form the entropy value vector of the message cluster; the coefficient of variation of each byte of the message cluster is arranged in the order of bytes to form the coefficient of variation vector of the message cluster. The semantics of the message sequence are extracted using a BiLSTM-CRF joint network. The original vector of the message sequence is processed by the BiLSTM layer, and the internal dependencies of the fields are captured based on the entropy vector and the coefficient of variation vector of the message cluster to which the message sequence belongs. The attention mechanism is used to focus on key positions, and dynamic context features are weighted based on the preceding and succeeding symbol vectors of the message sequence. The message sequence is annotated using CRF and outputs a semantically annotated sequence.

7. The encrypted anonymous network traffic analysis and identification method based on traffic reconstruction according to claim 1, characterized in that, The specific methods for constructing the protocol semantic graph are as follows: For any given session, several message sequences are divided based on entity type, and several different entities are extracted to obtain several entity nodes corresponding to the message sequence. The similarity between different entity nodes is calculated using Jaccard similarity, and a preset similarity threshold is set. If the similarity is greater than the similarity threshold, the corresponding two entity nodes are merged. After merging and judging all entity nodes in the session, several final entity nodes of the session are obtained. For any sequence of messages in a session, entity relationship edges are generated for the final entity node based on the byte, message, and session affiliation. By obtaining several final entity nodes and their entity relationship edges from each message sequence in the session, a protocol semantic graph composed of unknown protocols in the session is constructed.

8. The encrypted anonymous network traffic analysis and identification method based on traffic reconstruction according to claim 6, characterized in that, The specific methods for analyzing unknown protocols using protocol semantic graphs include: The system acquires new traffic data and obtains the corresponding sessions and their unknown protocols. As new sessions and new unknown protocols, it acquires the message sequence and its static feature vector of the unknown protocol. Based on its corresponding message cluster and session, it obtains the reception probability of the mapping symbol corresponding to each message sequence and acquires the protocol semantic graph composed of unknown protocols in the new sessions. For any known protocol semantic graph and any newly added unknown protocol in the new session, any final entity node in the known protocol semantic graph is taken as a known node. The cosine similarity between the static feature vector of the newly added unknown protocol and the known node and the cosine similarity between the subsequent symbol vector are obtained respectively. The mean of the two cosine similarities is taken as the node matching degree between the newly added unknown protocol and the known node. Obtain several entity relationship edges of the known node and several entity relationship edges of the newly added unknown protocol. Calculate the cosine similarity between any entity relationship edge of the known node and any entity relationship edge of the newly added unknown protocol. Take the mean of all cosine similarities obtained from the two sets of entity relationship edges and the mean of the node matching degree as the entity matching degree between the newly added unknown protocol and the known node. Match all newly added unknown protocols in the protocol semantic graph of the newly added session with each final entity node in the known protocol semantic graph using KM matching. The matching rule is that the higher the entity matching degree, the better the match. Obtain matching node pairs. Take the mean of the node matching degrees corresponding to all matching node pairs as the protocol matching degree between the newly added session and the known protocol semantic graph. Obtain the protocol matching degree between the newly added session and each known protocol semantic graph, and take the maximum protocol matching degree as the final matching degree of the newly added session; A first matching threshold and a second matching threshold are preset. If the final matching degree is greater than or equal to the first matching threshold, the session traffic protocol corresponding to the known protocol semantic graph corresponding to the final matching degree is output as the session traffic protocol corresponding to the newly added session. If the final matching degree is less than the first matching threshold and greater than or equal to the second matching threshold, the newly added session is marked as a variant of the session traffic protocol corresponding to the known protocol semantic graph corresponding to the final matching degree; if the final matching degree is less than the second matching threshold, the newly added session is marked as a new unknown session traffic protocol.

Images