Encrypted tunnel identification method and system based on feature manifold coherence score

Abstract

The invention discloses an encrypted tunnel identification method and system based on a feature manifold coherence score, and aims to solve the problems that an existing encrypted traffic identification technology is not high in accuracy and cannot effectively utilize the internal structure of traffic data. The method comprises the following steps: acquiring flow data in an encrypted tunnel; extracting a time sequence feature vector used for representing a flow instantaneous state; for each candidate service type, loading a pre-trained service feature manifold; calculating the dynamic entropy weight of the time sequence feature vector; calculating a service coherence score of the traffic data for each candidate service type through an integral formula; and finally, judging the candidate service type with the highest service coherence score as a borne service type. According to the method, the service traffic is modeled into the low-dimensional manifold, and the dynamic entropy weight is introduced to weight the information amount of the traffic, so that the internal structure and dynamic characteristics of the traffic can be more deeply captured, and the accuracy and robustness of service identification in the encrypted tunnel are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer networks, in particular, to a network security and traffic analysis technology, and more specifically, to an encrypted traffic identification method and system for identifying specific bearer services in encrypted network tunnels. BACKGROUND

[0002] With the rapid development of Internet applications and the increasing awareness of privacy protection among users, encryption technology has been widely used in network communication, forming a large number of encrypted network tunnels (such as HTTPS, VPN, QUIC, etc.). Although encryption effectively protects the confidentiality of data content, it also brings great challenges to network management, quality of service (QoS) guarantee, and legal network supervision. Traditional service identification methods, such as methods based on port numbers, are ineffective due to the use of non-standard ports or dynamic ports by a large number of applications; while deep packet inspection (DPI) technology is completely ineffective because it cannot decrypt message content.

[0003] Therefore, identification technology for encrypted traffic has emerged as the times require, and the mainstream is a traffic analysis method based on machine learning. This kind of method does not rely on message content, but extracts external statistical features of encrypted traffic, such as packet size sequence, arrival time interval, burst features, etc., to train a classifier to identify different applications or services.

[0004] However, the existing encrypted traffic identification methods based on machine learning usually have the following defects: Limitations of feature representation: Traditional methods usually treat the extracted traffic feature vector as an isolated data point in a high-dimensional Euclidean space, and use standard classifiers (such as support vector machines, random forests, etc.) for division. This way ignores the possible internal low-dimensional geometric structure of the same service traffic data in the feature space, resulting in limited generalization ability and expression ability of complex patterns of the model.

[0005] Static information weight: Existing methods often treat all traffic features within the observation time equally, i.e., assign the same weight. However, the information content at different times in a piece of traffic is different. For example, the traffic pattern when a video stream starts playing or the resolution switches is more distinguishable than the pattern when it is playing stably. The existing technology lacks a mechanism to dynamically evaluate and utilize these "high information content" key moments, resulting in limited recognition accuracy.

[0006] Poor adaptability to dynamic patterns: Due to the above defects, when the traffic pattern of a service changes dynamically or a new pattern appears that has never been seen before, the recognition accuracy of the existing method will decrease significantly.

[0007] Therefore, how to provide a technical scheme capable of more deeply understanding the internal structure of encrypted traffic, dynamically focusing on traffic segments with rich information, and improving service identification accuracy and robustness is a problem to be solved by those skilled in the art. SUMMARY

[0008] The present application aims to overcome the defects in the background art, and provides an encrypted tunnel identification method and system based on feature manifold coherence scoring. The scheme can more accurately and robustly identify specific service types in encrypted traffic by introducing the concepts of service feature manifold and dynamic entropy weight.

[0009] To achieve the above-mentioned purpose, the present application provides an encrypted tunnel identification method based on feature manifold coherence scoring, comprising the following steps: Obtaining the traffic data of the encrypted tunnel to be identified within a preset observation time length (T); Extracting a series of time-ordered time series feature vectors v(t) for representing the instantaneous state of traffic from the traffic data; Loading the service feature manifold (Mk) generated by pre-training for each preset candidate service type (k); Calculating the dynamic entropy weight w(t) of the time series feature vector v(t); Based on the service feature manifold (Mk) and the dynamic entropy weight w(t), the service coherence score C(k) of the traffic data for each candidate service type (k) is obtained according to the integral operation; Determining the candidate service type with the highest service coherence score C(k) as the service type carried in the encrypted tunnel. As a preferred technical scheme, the time series feature vector v(t) includes features selected from at least one of the following or a combination thereof: packet size, packet transmission direction, packet arrival time interval, burst length, and burst duration.

[0010] As a preferred technical scheme, the service feature manifold (Mk) is a low-dimensional representation generated by a deep generative model trained on a large number of known traffic samples of the corresponding service type (k). Further, the deep generative model can be a variational autoencoder (VAE) or a generative adversarial network (GAN).

[0011] As a preferred technical scheme, the service feature manifold (Mk) is a low-dimensional representation generated by a deep generative model trained on a large number of known traffic samples of the corresponding service type (k). Further, the deep generative model can be a variational autoencoder (VAE) or a generative adversarial network (GAN).

[0012] ​​As a preferred technical solution, the calculation step of the dynamic entropy weight w(t) comprises: taking a sliding time window around time t, obtaining a part of time sequence feature vectors in the window; calculating the Shannon entropy H(v(t)) based on the feature vectors in the window; calculating the gradient of the change of the Shannon entropy H(v(t)) with time t ; and taking the modulus of the gradient as the dynamic entropy weight w(t).

[0013] As a preferred technical solution, the calculation formula of the integral operation is: wherein D_M(v(t), ) is the distance of the feature vector v(t) at time t to the service feature manifold .

[0014] As a preferred technical solution, the distance D_M(v(t), ) is the minimum value of the Euclidean distance of the feature point v(t) to all points on the service feature manifold , or an approximate value thereof.

[0015] As a preferred technical solution, the integral operation is realized as a discrete summation operation in a digital computing device: wherein t_i is a discrete time sampling point, is a time step.

[0016] As a preferred technical solution, before the determination, it is firstly verified whether the maximum service coherence score C(k)_max exceeds a preset confidence threshold; only when the threshold is exceeded, the determination result is output, otherwise the traffic is marked as unknown type.

[0017] The application also provides an encrypted tunnel identification system based on feature manifold coherence score, comprising: a traffic acquisition and feature extraction module, configured to acquire traffic data of the encrypted tunnel and extract time sequence feature vectors therefrom; a manifold loading module, configured to load a pre-trained service feature manifold according to a candidate service type; a coherence score calculation module, configured to calculate a dynamic entropy weight of the time sequence feature vectors and calculate a service coherence score of each candidate service according to an integral formula; a service discrimination module, configured to determine a final service type according to the calculated service coherence score.

[0018] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method.

[0019] Compared with the prior art, the application has the following remarkable beneficial effects: Stronger feature expression capability: The application innovatively no longer regards the traffic features of a specific service as a scattered point set in a high-dimensional space, but models an intrinsic, low-dimensional "service feature manifold". This way can better capture the essential geometric structure of the specific service traffic pattern, making the feature representation more compact and robust.

[0020] Dynamic focusing on information is achieved: By introducing "dynamic entropy weight", the application can quantify the "information amount" of traffic at different times. High weight is given to the time when the traffic pattern changes dramatically (large entropy gradient), and low weight is given to the time when it is smooth (small entropy gradient). This makes the model automatically focus on the most discriminative traffic segment for decision-making, significantly improving the accuracy of recognition.

[0021] Higher recognition accuracy and robustness: The "service coherence score" formula organically combines the "fitting degree" (distance) of the feature vector to the service manifold and the "information amount" (dynamic entropy weight) at this time. A high score means that the traffic to be tested not only highly coincides with a certain service manifold in pattern, but also such coincidence occurs at the key moment with the most information amount. This double consideration mechanism makes the recognition accuracy and robustness to noise and pattern drift of the application superior to traditional methods.

[0022] Better unknown traffic processing capability: By setting a confidence threshold, the application can effectively identify traffic that does not belong to any known service model and mark it as "unknown", thereby avoiding forced misclassification and reducing the false positive rate of the system. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced as follows.

[0024] Figure 1 is a flowchart of an encrypted tunnel recognition method based on feature manifold coherence scoring according to an embodiment of the application.

[0025] Figure 2 is a structural block diagram of an encrypted tunnel recognition system based on feature manifold coherence scoring according to an embodiment of the application. DETAILED DESCRIPTION

[0026] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0027] Embodiment one: Please refer to Figure 1 , which shows a flowchart of an encrypted tunnel identification method based on feature manifold coherence scoring provided by an embodiment of the present application. The method can be realized by a computer program and run on a general computing device.

[0028] Step 101: obtaining traffic data of an encrypted tunnel to be identified within a preset observation time length.

[0029] First, the original network traffic flowing through a specific encrypted tunnel is captured through a network interface (such as a network splitter TAP or a mirror port configured on a switch) and stored as a file in, for example, pcap format. A preset observation time length T, for example, 60 seconds, is set to intercept a traffic session to be identified.

[0030] Step 102: extracting a series of time-ordered time series feature vectors for representing traffic instantaneous state from the traffic data.

[0031] The traffic data obtained in step 101 is parsed to extract a series of statistical features capable of representing the traffic instantaneous state. These features are arranged in time sequence to form a time series feature vector v(t). In this embodiment, v(t) can be a multi-dimensional vector, and the feature dimensions contained therein can be selected from: Packet size: the number of payload bytes of each packet at the IP layer or the TCP / UDP layer.

[0032] Packet transmission direction: defined as uplink (client→server) or downlink (server→client) relative to the client.

[0033] Packet arrival time interval: the time difference between the arrival of adjacent two packets at the observation point.

[0034] Burst length: defined as the number of a set of packets continuously transmitted in one direction.

[0035] Burst duration: the time span from the first packet to the last packet of a burst.

[0036] At time t (or for the i-th packet), these features together form a feature vector .

[0037] Step 103: Load the pre-trained generated service feature manifold for each candidate service type.

[0038] This step involves an offline training phase and an online loading phase.

[0039] Offline training phase: For each candidate service type k that needs to be identified (e.g., k = 1 for video conferencing, k = 2 for web browsing, k = 3 for file downloading, etc.), a large number of known, pure encrypted traffic samples of the service are collected. Then, a deep generative model is used to train these samples. In this embodiment, a variational autoencoder (VAE) can be used. The traffic feature vectors of service k are input into the VAE for training. After training, the decoder part of the VAE has learned how to generate high-dimensional data conforming to the service traffic feature distribution from a low-dimensional latent space. This low-dimensional latent space constitutes the feature manifold of service k . Similarly, a generative adversarial network (GAN) can also be used, and the space where the generator is located can be regarded as the feature manifold. The trained manifold model of each service is stored.

[0040] Online loading phase: When performing the identification task, according to the pre-set list of candidate service types, all pre-trained service feature manifolds are loaded from storage into memory for subsequent calculation and use.

[0041] Step 104: Calculate the dynamic entropy weight of the time series feature vector.

[0042] In order to quantify the "amount of information" contained in the feature vector v(t) at each time t, the dynamic entropy weight of v(t) is calculated. The specific calculation process is as follows: a. In the vicinity of time t, a sliding time window is defined, for example, containing a total of 20 time series feature vectors before and after time t.

[0043] b. Based on the feature vector set in this window, it is regarded as a local feature distribution, and its Shannon entropy H(v(t)) is calculated. Shannon entropy measures the complexity or uncertainty of the local traffic pattern.

[0044] c. Calculate the gradient of Shannon entropy H(v(t)) with respect to time t , which reflects the rate of change of traffic pattern complexity.

[0045] d. Take the modulus (i.e., the size) of the gradient as the dynamic entropy weight w(t) of time t.

[0046] Intuitively, when the traffic pattern changes from smooth to complex (e.g. a web page finishes loading and starts playing a video), or from complex to smooth, the rate of change of entropy will be large, and the value of w(t) will be high, indicating that this moment is a key node for identifying the service type. Conversely, in the stable transmission phase, the value of w(t) is low.

[0047] Step 105: Based on the service feature manifold and the dynamic entropy weight, obtain the service coherence score of the traffic data for each candidate service type according to the integral operation.

[0048] For each candidate service k, calculate the coherence score C(k) of the traffic under test with this service. The core is the integral formula: .

[0049] Distance D_M(v(t), k): For the feature vector v(t) at time t, calculate its distance to the service feature manifold of service k. This distance can be defined as the minimum value of the Euclidean distance from this point v(t) to all points on the manifold of service k. This distance measures to what extent the current traffic pattern v(t) conforms to the "typical pattern" of service k. The smaller the distance, the more conforming. exp(-D_M(...)): Convert the distance into a similarity score in the range (0, 1] through an exponential function. The smaller the distance D_M, the closer the value to 1.

[0050] w(t) ⋅ exp(...): Multiply the above similarity score with the dynamic entropy weight w(t). This means that only when the traffic pattern at a moment both "conforms" to the typical pattern of service k (high value of exp(-D_M)), and is at a "key moment" with large amount of information (high value of w(t)), will their product be significantly large.

[0051] Integral / sum: In actual digital devices, the integral is realized through discrete summation:

[0052] where is the discrete time sampling point, is the time step. That is, the weighted similarity scores of all sampling points in the observation time T are added up to obtain the final total score C(k).

[0053] Step 106: Determine the candidate service type with the highest service coherence score as the service type carried in the encrypted tunnel.

[0054] ​After calculating the coherence scores C(1), C(2),... of all candidate services k, find the highest score C(k)_max and its corresponding service .

[0055] Confidence judgment: Preferably, C(k)_max is compared with a preset confidence threshold. The threshold can be determined by experiment on the validation set.

[0056] Output result: If C(k)_max is greater than the threshold, it is determined that the service type carried in the encrypted tunnel is , and the result is output. If C(k)_max is not greater than the threshold, it means that the matching degree of the traffic with all known service models is low, and it is marked as "unknown type", thereby avoiding false guesses.

[0057] Embodiment two: Please refer to Figure 2 , which shows the structure block diagram of an encrypted tunnel identification system based on feature manifold coherence score provided by the embodiment of the application. The system can be a software system or a combination of software and hardware system deployed on a network gateway, server or dedicated hardware. The system comprises: Traffic acquisition and feature extraction module 201: responsible for performing steps 101 and 102 in the method flow. The module usually contains a network interface for listening and capturing data packets, and a processing unit for parsing and calculating the time sequence feature vector v(t) from the original data packet.

[0058] Manifold loading module 202: responsible for performing step 103 in the method flow. The module has a storage unit (such as hard disk, database) inside or connected, which stores the feature manifold models of various services trained offline When the identification task starts, the module loads the required manifold model into the working memory according to the instruction.

[0059] Coherence score calculation module 203: the core computing unit of the system, responsible for performing steps 104 and 105 in the method flow. The module receives the feature vector v(t) from module 201 and the manifold from module 202, and then performs the calculation of dynamic entropy weight w(t) and the integration / summation operation of the final service coherence score C(k). Considering the complexity of the calculation, the module can be realized by high-performance CPU or GPU.

[0060] The service identification module 204 is responsible for performing step 106 in the method flow. The module receives the scores C(k) of all candidate services calculated from module 203, compares them to find the maximum value, and performs a comparison operation with the confidence threshold, and finally outputs the identification result (specific service type or "unknown type").

[0061] In addition, the present application also provides a computer readable storage medium, such as a hard disk, a solid state disk (SSD) or a cloud storage of a server. The medium stores computer program codes, and when the program is executed by one or more processors, all or part of the steps in the above method embodiment are implemented.

[0062] Another embodiment of the present application provides a computer device (computer, server, etc.) comprising a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program comprises instructions for executing the steps of the method of the present application.

[0063] Another embodiment of the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a computer to implement the steps of the method of the present application.

[0064] In summary, the present application analyzes encrypted traffic from a completely new perspective combining data geometry and information theory. The method and system based on feature manifold coherence scoring proposed by the present application can significantly improve the performance of encrypted tunnel identification, and has high practical application value.

[0065] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying encrypted tunnels based on feature manifold coherence scoring, characterized in that, Includes the following steps: Obtain traffic data of the encrypted tunnel to be identified within a preset observation period; Extract a series of time-ordered feature vectors from the traffic data to characterize the instantaneous state of the traffic. For each preset candidate service type, load the pre-trained service feature manifold; Calculate the dynamic entropy weight of the time-series feature vector; Based on the service feature manifold and the dynamic entropy weight, the service coherence score of the traffic data for each candidate service type is obtained by integral calculation. The candidate service type with the highest service coherence score is determined as the service type carried within the encrypted tunnel.

2. The method according to claim 1, characterized in that, The timing feature vector includes features selected from at least one or a combination of the following: packet size, packet transmission direction, packet arrival time interval, burst length, and burst duration.

3. The method according to claim 1, characterized in that, The service feature manifold is a low-dimensional representation generated by training a deep generative model on a large number of known traffic samples of the corresponding service type.

4. The method according to claim 3, characterized in that, The deep generative model is a variational autoencoder or a generative adversarial network.

5. The method according to claim 1, characterized in that, The calculation steps for the dynamic entropy weight include: Take a sliding time window around time t and obtain a partial time series feature vector within the window; Calculate the Shannon entropy H(v(t)) based on the feature vectors within the window; Calculate the gradient of the Shannon entropy H(v(t)) as a function of time t. ; The magnitude of the gradient As the dynamic entropy weight w(t).

6. The method according to claim 1, characterized in that, The formula for calculating the integral is as follows: Where v(t) is the time-series feature vector, To serve the feature manifold, w(t) is the dynamic entropy weight, T is the preset observation duration, and D_M(v(t)). Let v(t) be the eigenvector v(t) at time t and the service feature manifold. The distance; the distance D_M(v(t), Let be the feature points of v(t) and the service feature manifold. The minimum Euclidean distance between all points on the given surface, or an approximation thereof.

7. The method according to claim 1, characterized in that, The integration operation is implemented as a discrete summation operation in the digital computing device: ,in For discrete time sampling points, For time step.

8. The method according to claim 1, characterized in that, Before making the determination, it is first verified whether the maximum service coherence score exceeds a preset confidence threshold; the determination result is only output when it exceeds the threshold, otherwise the traffic is marked as an unknown type.

9. A cryptographic tunnel identification system based on characteristic manifold coherence scoring, characterized in that, include: The traffic acquisition and feature extraction module is used to acquire traffic data from the encrypted tunnel and extract time-series feature vectors from it. The manifold loading module is used to load pre-trained service feature manifolds based on the candidate service type; The coherence score calculation module is configured to calculate the dynamic entropy weight of the time-series feature vector and calculate the service coherence score for each candidate service according to the integral formula. The service identification module is used to determine the final service type based on the calculated service coherence score.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images