A lightweight real-time identification method for dark web services based on Gaussian Bayesian model

Abstract

The present invention discloses a lightweight real-time identification method for dark web services based on a Gaussian Bayesian model. The method includes: cleaning and labeling the public dark web traffic data collected from the Internet to obtain dark web traffic data labeled with dark web system categories and dark web service type labels; segmenting the labeled dark web traffic data into data streams according to five-tuple information, extracting data stream feature vectors, and dividing the data streams into training sets and validation sets; using the SMOTE method to generate inter-class samples, balance minority class samples in the training set data, and enhance the distribution boundaries between unbalanced traffic data; and iteratively training the Gaussian Bayesian model using the divided training set. The present invention provides a lightweight real-time identification method for dark web services based on a Gaussian Bayesian model, which improves the efficiency of traffic identification, and can iteratively train the model, thereby improving the accuracy of model identification.

Description

Technical Field

[0001] The present invention relates to the field of network security, and specifically designs a lightweight dark web business real-time identification method based on a Gaussian Bayesian model. Background Art

[0002] The dark web poses a serious threat due to its strong concealment, high anonymity and anti-tracing characteristics.

[0003] At present, the common dark web encrypted traffic identification methods are mainly divided into: port-based methods, DPI-based methods, statistics-based methods and behavior-based methods. Both the statistics-based and behavior-based methods can adopt the idea of ​​machine learning, but how to design a set of feature sets that can accurately reflect the characteristics of the traffic has become the difficulty of using machine learning to detect dark web encrypted traffic. As for deep learning, although high-level features can be automatically learned from the input raw data, its disadvantage is that the deep learning model relies on high-quality data sets and has a large demand for computing resources. When faced with large traffic data, there is a problem of lagging detection results, so it is difficult to ensure the real-time detection on large traffic nodes.

[0004] In summary, in order to be able to perform real-time identification of different business types of dark web traffic on large traffic nodes, the present invention proposes a lightweight dark web business real-time identification method based on the Gaussian Bayesian model, which characterizes traffic characteristics from three levels: time, size, and content. The SMOTE algorithm is used to generate inter-class samples, that is, samples between different dark web business types, to enhance the distribution boundaries between unbalanced traffic in the data set, and on this basis, iteratively trains a lightweight machine learning model based on the Gaussian Bayesian model to achieve real-time and accurate identification of different dark web system business categories. Summary of the invention

[0005] In view of this, the present invention provides a lightweight dark web business real-time identification method based on a Gaussian Bayesian model, aiming to identify dark web traffic of different business types in network nodes in real time.

[0006] A lightweight dark web business real-time identification method based on a Gaussian Bayesian model, the method comprising:

[0007] Step 1: Clean and label the public dark web traffic data collected from the Internet to obtain dark web traffic data labeled with dark web system categories and dark web business types;

[0008] Step 2: Preprocessing: segment the labeled dark web traffic data into data streams, extract data stream feature vectors, and divide them into training sets and validation sets;

[0009] Step 3: Data enhancement: Use the SMOTE method to generate inter-class samples, balance the minority class samples in the training set data, and enhance the distribution boundary between unbalanced traffic data;

[0010] Step 4: Iteratively train the Gaussian Bayes model using the divided training set.

[0011] Preferably, the process of cleaning and labeling public dark web traffic data includes:

[0012] Collect and organize the dark web traffic data sets publicly available on the Internet, save the data sets in pcap file format, and delete the data that cannot be labeled with the dark web system category and dark web business type. Further, label the data packets according to the dark web system category and dark web business type. Among them, according to the dark web system category, there are Tor, I2P, ZeroNet and Freenet. According to the dark web business type, there are Audio-Stream, Chat, Email, P2P, Transfer, Video-Stream and VOIP.

[0013] Preferably, the dark web traffic data preprocessing process includes:

[0014] Use the five-tuple {source IP, destination IP, source port, destination port, protocol type} to divide the dark web traffic data into data streams;

[0015] By counting the timestamps of the data packets in the data flow, we can construct the transmission delay sequence T of the data flow = {t1, t2, ..., t n Each bit represents the absolute value of the difference between the timestamps of two adjacent data packets.

[0016] By counting the TCP data packet payload sizes in the data flow, we construct a TCP data packet payload size sequence R = {r1, r2, ..., r m},Due to the fact that darknet communication can be regarded as the transmission of upper-layer application data mainly in the form of fixed-length data encapsulation, the data packet payload size can only be distinguished by analyzing it from the transport layer;

[0017] The data packet protocol feature sequence F = {f1, f2, ..., f m};

[0018] By extracting the statistical features in the data stream, a statistical feature sequence of the data stream is constructed. The minimum value, lower quartile, upper quartile, median, maximum value, average value, variance and standard deviation of the time interval of the data packets in the upstream and downstream directions in the statistical data stream are counted. The maximum load, minimum load, average load, variance and standard value of the TCP data packets in the statistical data stream are counted. Then all the statistical features are combined to form a synthetic feature sequence Z = {z1,z2,...,z m};

[0019] All feature sequences of the data stream are merged in sequence, and the feature vector S = T || R || F || Z = {t1, t2, ..., t n ,r1,r2,...,r m ,f1,f2,...,f m ,z1,z2,...,z m};

[0020] Normalize the feature vector of the data stream. Since the machine learning model needs to input vector data of a fixed length, the length of the feature vector extracted from the data stream is set to L. If the length of the data stream is less than L, the feature vector is filled with 0; if the length of the data stream is greater than L, the excess part is discarded.

[0021] Normalize the eigenvector and process it using the Z standardization method. The Z standardization method is defined as Where x* represents the normalized eigenvalue, x represents the original eigenvalue, and u and σ represent the mean and variance of the eigenvalue of the corresponding dimension of the eigenvector, respectively.

[0022] Preferably, the process of balancing the minority class samples in the training data using the SMOTE method includes:

[0023] Apply the Chebyshev distance formula D Chebyshev (P,Q)=max(|p1-q1|,|p2-q2|,...,|p i -q i |) Calculate the minority class samples N i The nearest neighbor N, i , where P and Q represent two sample points in the minority class sample, {p1, p2, ..., p i} and {q1,q2,...,q i} respectively represent the eigenvalues ​​of different dimensions in the eigenvectors of the two samples. By calculating the absolute value of the eigenvalue of the corresponding dimension in the eigenvector of the sample point, the maximum value is taken as the distance between the two sample points.

[0024] In each sample pair (N i ,N, i ) to generate new inter-class samples and optimize the imbalance problem of training data, where the single linear interpolation method is defined as: K = N i (1-t)+N′ i t, K represents the generated inter-class samples. The variable t is a random value with a value range of (0,1). By randomly generating the variable t, it is ensured that the generated new samples are between the sample pairs, and each new sample generated has a certain degree of randomness.

[0025] Preferably, the process of iteratively training the Gaussian Bayesian model includes:

[0026] The Gaussian Bayesian model is constructed based on the Gaussian distribution, where the probability distribution of the darknet feature vector is defined as xi represents the i-th feature dimension, yc represents the business category of different dark web systems, σci and uci represent the standard deviation and expectation corresponding to feature xi under the business category y=c of the dark web system;

[0027] After calculating the conditional probability of each feature dimension, the posterior probability is maximized. Where n represents the dimension of the feature vector;

[0028] After calculating the maximum posterior probability of each feature vector in the business category of different dark web systems, a size comparison is performed to classify the feature vector into the business category of the dark web system with the highest probability;

[0029] In the training of the model, after each round of training, the data set is manually checked and re-labeled, and then steps 2 to 3 are iteratively executed, the SMOTE method is used to balance the minority class sample data, and the model is iteratively trained;

[0030] The model is evaluated, and when the model achieves an accuracy rate of no less than 90% and an identification response time of no more than 0.2 seconds on the test set, it is deployed in an actual network environment for real-time identification and classification of dark web traffic.

[0031] The present application provides a lightweight real-time identification method for dark web services based on a Gaussian Bayesian model. Compared with the prior art, the present invention has the following beneficial effects: firstly, based on the Gaussian Bayesian model, real-time identification of business type traffic for different dark web systems is achieved, significantly improving the efficiency of traffic identification; secondly, based on the SMOTE method, an inter-class sample enhanced data set is generated, which can iteratively train the model and improve the accuracy of model recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the design scheme in this embodiment and the prior art, the drawings required for use in the embodiment and the description of the prior art will be briefly introduced below. The drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of the framework of a lightweight dark web business real-time identification method based on a Gaussian Bayesian model provided in an embodiment of the present application.

[0034] Figure 2 A schematic diagram of the process of preprocessing dark web traffic data provided in an embodiment of the present application.

[0035] Figure 3 A schematic diagram of a process for generating inter-class samples for data enhancement using the SMOTE method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following is a further detailed description of the specific implementation of the present invention in conjunction with the accompanying drawings and specific implementations. The following examples or drawings are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0037] See also Figure 1 , Figure 1 The flowchart of the lightweight dark web business real-time identification method based on the Gaussian Bayesian model provided in the embodiment of the present application includes the following specific steps:

[0038] Step 1: Clean and label the public dark web traffic data collected from the Internet to obtain dark web traffic data labeled with dark web system categories and dark web business types;

[0039] Step 2: Preprocessing: segment the labeled dark web traffic data into data streams, extract feature vectors, and divide them into training and validation sets;

[0040] Step 3: Data enhancement: Use the SMOTE method to balance the minority class samples in the training set data and enhance the distribution boundary between the unbalanced traffic data;

[0041] Step 4: Iteratively train the Gaussian Bayes model using the divided training set.

[0042] The specific steps for cleaning and labeling the public dark web traffic data collected from the Internet include:

[0043] Step 1a: Collect and organize the dark web traffic data packets publicly available on the Internet and save all data packets in pcap file format;

[0044] Step 1b: Delete data that cannot be labeled with dark web system categories and dark web business types based on the relevant descriptions of the public data set;

[0045] Step 1c: Label the data packets according to the dark web system category and dark web service type. According to the dark web system category, there are Tor, I2P, ZeroNet and Freenet. According to the dark web service type, there are Audio-Stream, Chat, Email, P2P, Transfer, Video-Stream and VOIP.

[0046] See also Figure 2 , Figure 2 The flowchart provided in the embodiment of the present application for segmenting the labeled dark web traffic data into data streams, extracting feature vectors, and dividing the training set and the validation set includes the following specific steps:

[0047] Step 2a: Use the five-tuple {source IP, destination IP, source port, destination port, protocol type} to divide the data into data streams;

[0048] Step 2b: Construct the transmission delay sequence T of the data stream by counting the timestamps of the data packets in the data stream. n Each bit represents the absolute value of the difference between the timestamps of two adjacent data packets.

[0049] Step 2c: By counting the TCP data packet payload sizes in the data flow, construct a TCP data packet payload size sequence R = {r1, r2, ..., r m},Since the dark web network can be regarded as a fixed-length data encapsulation method to transmit upper-layer application data, the data packet payload size can only be distinguished by analyzing it from the transport layer;

[0050] Step 2d: Construct the protocol feature sequence F = {f1, f2, ..., f m};

[0051] Step 2e: By extracting statistical features from the data stream, construct a statistical feature sequence of the data stream, calculate the minimum value, lower quartile, upper quartile, median, maximum value, average value, variance and standard deviation of the time interval of the data packets in the upstream and downstream directions in the statistical data stream, calculate the maximum load, minimum load, average load, variance and standard value of the TCP data packets in the statistical data stream, and then combine all the statistical features to form a synthetic feature sequence Z = {z1, z2, ..., z m}

[0052] Step 2f: Merge all feature sequences of the data stream in sequence, and construct a feature vector S = T || R || F || Z = {t1, t2, ..., t n ,r1,r2,...,r m ,f1,f2,...,f m ,z1,z2,...,z m};

[0053] Step 2g: Normalize the feature vector of the data stream. Since the machine learning model needs to input vector data of a fixed length, the length of the feature vector extracted from the data stream is set to L. If the length of the data stream is less than L, the feature vector is filled with 0; if the length of the data stream is greater than L, the excess part is discarded.

[0054] Step 2h: Normalize the feature vector using the Z-standardization method, which is defined as where x * represents the normalized eigenvalue, x represents the original eigenvalue, u and σ represent the mean and variance of the eigenvalue of the corresponding dimension of the eigenvector, respectively.

[0055] See also Figure 3 , Figure 3 The flowchart provided in the embodiment of the present application about using the SMOTE method to balance the minority class samples of the training set data and enhance the distribution boundary between the unbalanced traffic data includes the following specific steps:

[0056] Step 3a: Apply the Chebyshev distance formula D Chebyshev (P,Q)=max(|p1-q1|,|p2-q2|,...,|p i -q i |) Calculate the minority class samples N i The nearest neighbor N, i , where P and Q represent two sample points in the minority class sample. By calculating the absolute value of the corresponding dimension feature value in the feature vector of the sample point, the maximum value is taken as the distance between the two sample points;

[0057] Step 3b: In each sample pair (N i ,N, i ) to generate new inter-class samples and optimize the imbalance problem of training data, where the single linear interpolation method is defined as: K = N i (1-t)+N′ it, K represents the generated inter-class samples. The variable t is a random value with a value range of (0,1). By randomly generating the variable t, it is ensured that the generated new samples are between the sample pairs, and each new sample generated has a certain degree of randomness.

[0058] The specific steps of iteratively training the Gaussian Bayesian model using the divided training set include:

[0059] Step 4a: Construct a Gaussian Bayesian model based on the Gaussian distribution, where the probability distribution of the darknet feature vector is defined as xi represents the i-th feature dimension, yc represents the business category of different dark web systems, σci and uci represent the standard deviation and expectation corresponding to feature xi under the business category y=c of the dark web system;

[0060] Step 4b: After calculating the conditional probability of each feature dimension, maximize the posterior probability calculation Where n represents the dimension of the feature vector;

[0061] Step 4c: After calculating the maximum posterior probability of each feature vector in the service category of different dark web systems, compare all probability values ​​and classify the feature vector into the dark web system service category with the largest probability value;

[0062] Step 4d: During model training, after each round of training, the data set is manually checked and relabeled, and then steps 2 to 3 are iteratively executed to iteratively train the model using the SMOTE method to balance the minority class sample data;

[0063] Step 4e: Evaluate the model. When the model achieves an accuracy rate of no less than 90% and an identification response time of no more than 0.2 seconds on the test set, deploy it in an actual network environment to identify the business traffic of different dark web systems in real time.

[0064] It should be noted that, for the above method embodiments, for the convenience of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and processes involved are not necessarily required by the present application.

[0065] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of complete software embodiments or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0066] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent replacements for some of the technical features therein.

[0067] Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight real-time identification method for dark web services based on a Gaussian Bayesian model. This method is based on a Gaussian Bayesian model, realizes real-time identification of different dark web service flows, and uses the SMOTE method to generate inter-class samples to enhance the data set, thereby realizing iterative optimization of the model and improving the accuracy of model recognition. The method is characterized in that: The method comprises: Step 1: Clean and label the public dark web traffic data collected from the Internet to obtain dark web traffic data labeled with dark web system categories and dark web business types; Step 2: Preprocessing: segment the labeled dark web traffic data into data streams in the form of five-tuple {source IP, destination IP, source port, destination port, protocol type}, extract the transmission delay sequence, TCP packet payload size sequence, data stream protocol feature sequence and data stream statistical feature sequence in the data stream, and further merge all feature sequences into a synthetic feature vector of the data stream, then normalize the feature vector through the Z standardization method, and finally divide it into training set and validation set; Step 3: Data enhancement: Use the SMOTE method to balance the minority class samples of the training set data and enhance the distribution boundary between the unbalanced traffic data. The SMOTE method is used to balance the minority class samples of the training set data by applying the Chebyshev distance formula to calculate the minority class samples N i The nearest neighbor N, i And in each sample pair (N i ,N, i ) use the single linear random interpolation method to generate new inter-class samples; Step 4: Use the SMOTE method to balance the minority class sample data in the training set and iteratively train the Gaussian Bayes model.

2. According to the method of claim 1, a lightweight dark web business real-time identification method based on Gaussian Bayesian model is characterized in that: In step 1: The sample data in pcap format are cleaned and labeled, and the data that cannot be labeled with dark web system categories and dark web business types based on the relevant introduction of the public data set are deleted.

3. According to the method of claim 1, a lightweight dark web business real-time identification method based on Gaussian Bayesian model is characterized in that: In step 2: Divide the data into data streams according to the five-tuple {source IP, destination IP, source port, destination port, protocol type} characteristics of the data packet; By counting the timestamps of the data packets in the data flow, we can construct the transmission delay sequence T of the data flow = {t1, t2, ..., t n }, where each bit represents the absolute value of the difference between the timestamps of two adjacent data packets. By counting the TCP data packet payload sizes in the data flow, we construct a TCP data packet payload size sequence R = {r1, r2, ..., r m }; The cumulative number of FIN, SYN, RST, PSH, ACK, URG, ECE, and CWR in the Flag bit of the TCP data packet header in the statistical data flow is used to construct the protocol feature sequence F of the data flow = {f1, f2, ..., f m }; Extract statistical features from the data stream, construct a statistical feature sequence of the data stream, count the minimum, lower quartile, upper quartile, median, maximum, average, variance and standard deviation of the time interval of packets in the upstream and downstream directions in the statistical data stream, count the maximum load, minimum load, average load, variance and standard value of the TCP data packets in the statistical data stream, and then combine all statistical features to form a statistical feature sequence Z = {z1,z2,...,z m }; All feature sequences of the data stream are merged in sequence, and the transmission delay sequence of the data stream, the TCP packet payload size sequence, the data stream protocol feature sequence and the data stream statistical feature sequence are combined in sequence to form a synthetic feature vector S = T || R || F || Z = {t1, t2, ..., t n ,r1,r2,...,r m ,f1,f2,...,f m ,z1,z2,...,z m }; The data stream feature vector is normalized. The length of the feature vector extracted from the data stream is set to L. If the data stream length is less than L, the feature vector is filled with 0. The part of the data stream longer than L is discarded, and the feature vector is processed using the Z normalization method.

4. According to claim 1, a lightweight dark web business real-time identification method based on Gaussian Bayesian model is characterized in that: In step 3: Apply the Chebyshev distance formula D Chebyshev (P,Q)=max(|p1-q1|,|p2-q2|,...,|p i -q i |) Calculate the minority class samples N i The nearest neighbor N, i , where P and Q represent two sample points in the minority class sample, {p1, p2, ..., p i } and {q1, q2, ..., q i } respectively represent the eigenvalues ​​of different dimensions in the eigenvectors of the two samples; In each sample pair (N i ,N, i ) to generate new inter-class samples, where the single linear interpolation method is defined as: K = N i (1-t)+N′ i t, K represents the generated inter-class samples, and the variable t is a random value with a value range of (0,1).

5. According to claim 1, a lightweight dark web business real-time identification method based on Gaussian Bayesian model is characterized in that: In step 4: The probability distribution of the darknet feature vector is defined as xi represents the i-th feature dimension, yc represents the business category of different dark web systems, σci and uci represent the standard deviation and expectation corresponding to feature xi under the business category y=c of the dark web system; After calculating the conditional probability of each feature dimension, the posterior probability is maximized. Where n represents the dimension of the feature vector; After calculating the maximum posterior probability of each feature vector in the business category of different dark web systems, all probability values ​​are compared and the feature vector is classified into the dark web system business category with the largest probability value; In the training of the model, after each round of training, the data set is manually checked and re-labeled, and then steps 2 to 3 are iteratively executed, and the SMOTE method is used to balance the minority class sample data to iteratively train the model; The model is evaluated, and when the model achieves an accuracy rate of no less than 90% and an identification response time of no more than 0.2 seconds on the test set, it is deployed in an actual network environment to identify the business traffic of different dark web systems in real time.

Images