A malicious encrypted traffic detection method based on variational autoencoder
By using a variational autoencoder-based method, the characteristic distribution of normal encrypted traffic is learned and the identification threshold range is determined. This solves the generalization and transfer problems of malicious encrypted traffic detection in existing technologies, and achieves the ability to identify unknown threats with high accuracy and speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XINGTANG TELECOMM TECH CO LTD
- Filing Date
- 2021-12-24
- Publication Date
- 2026-05-29
AI Technical Summary
Existing methods for detecting malicious encrypted traffic are difficult to generalize and transfer, and have poor detection capabilities for unknown types of malicious encrypted traffic.
A variational autoencoder-based method is adopted. By collecting and processing encrypted network traffic data, the variational autoencoder is trained using normal traffic sample data to learn the characteristic distribution of normal traffic. The recognition threshold range of each category is determined by the reconstruction error, thereby realizing the detection of encrypted traffic.
It achieves highly generalizable and accurate encrypted traffic identification and detection, effectively detects known types of malicious encrypted traffic, and quickly identifies and detects unknown threats, with strong generalization and transfer capabilities.
Smart Images
Figure CN116346384B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer network security technology, and in particular to a method for detecting malicious encrypted traffic based on a variational autoencoder. Background Technology
[0002] With increased awareness of cybersecurity, the internet has widely adopted encryption protocols for communication, leading to a rapid increase in encrypted traffic. However, while encryption technology protects user privacy and information security, it has also become a significant tool for cyberattacks and information theft. For HTTPS encrypted traffic used in web communication, the use of encryption disrupts the unique statistical characteristics and data format of plaintext data. This allows attackers to exploit SSL encrypted channels to circumvent border device filtering and protection, delivering and distributing malicious payloads, enabling scanning, brute-force attacks, and other malicious attacks, facilitating communication between infected hosts and servers, and posing a significant threat to network information security. Because the characteristics of encrypted traffic change, traditional traffic detection methods are difficult to reuse in encrypted environments. Effectively identifying malicious encrypted traffic has become a major challenge in the field of cybersecurity.
[0003] Currently, researchers both domestically and internationally primarily employ machine learning-based detection methods to detect malicious encrypted traffic. Machine learning-based malicious traffic detection methods leverage the prior knowledge of cybersecurity experts, using feature engineering to extract statistically significant features of the target. These extracted features are then used as training samples in a machine learning algorithm model. Different machine learning algorithms are then used for training and parameter tuning to optimize the model, ultimately achieving the detection of malicious encrypted traffic. The core of machine learning-based malicious traffic detection technology is feature engineering of network traffic data. Targeted feature engineering leads to better identification results; however, feature engineering heavily relies on the prior knowledge of domain experts, making existing detection methods difficult to generalize and transfer. Furthermore, due to this reliance on empirical knowledge, the detection capability for unknown types of malicious encrypted traffic is poor.
[0004] Therefore, existing methods for detecting malicious encrypted traffic are difficult to generalize and transfer, and have poor detection capabilities for unknown types of malicious encrypted traffic. Summary of the Invention
[0005] Based on the above analysis, the embodiments of the present invention aim to provide a malicious encrypted traffic detection method based on variational autoencoder, in order to solve the problems that existing malicious encrypted traffic detection methods are difficult to generalize and transfer, and have poor detection capabilities for unknown malicious encrypted traffic types.
[0006] This invention provides a method for detecting malicious encrypted traffic based on a variational autoencoder, comprising the following steps:
[0007] Collect and process encrypted network traffic data to obtain feature vector data, add category labels to the feature vector data to obtain total sample data, wherein the category labels include normal traffic data and traffic data of each malicious category;
[0008] Normal traffic sample data is input into the variational autoencoder for training to learn the feature distribution of normal traffic data and obtain the trained variational autoencoder; the total sample data is input into the trained variational autoencoder to obtain the reconstruction error of the sample data of each detection category, and the recognition threshold range of each detection category is determined based on the reconstruction error; where each detection category includes normal and malicious categories.
[0009] The feature vector data of the encrypted traffic data to be tested is input into the trained variational autoencoder to obtain the reconstruction error. Based on the reconstruction error and the recognition threshold range of each detection category, the detection result of the encrypted traffic is obtained.
[0010] Furthermore, the feature vector data is obtained by performing the following steps:
[0011] The data features of the encrypted traffic data are extracted, and the data features are cleaned, filled with null values, and numerically processed. Then, they are normalized to zero mean to obtain feature vector data. The data features include statistical features, time series features, and message load features.
[0012] Furthermore, the statistical features include the number of data packets, the number of bytes transmitted, the data packet transmission rate, and the byte transmission rate; the time series features include the average packet interval time, the average stream duration, the standard deviation of the packet interval time, and the standard deviation of the stream duration; and the message payload features include the protocol version number, the protocol entropy, the number of TLS cipher suites, and the TLS extension length.
[0013] Further, the step of inputting the total sample data into the trained variational autoencoder to obtain the reconstruction error of the sample data for each detection category, and determining the recognition threshold range for each detection category based on the reconstruction error, includes:
[0014] The total sample data is input into the trained variational autoencoder to obtain the reconstructed input data.
[0015] The reconstruction error is obtained based on the total sample data of the original input and the reconstructed input data. The distribution of reconstruction error for each category of traffic data is then statistically analyzed.
[0016] Based on the reconstruction error distribution and the detection accuracy of each type of traffic data, the identification thresholds for each type are adaptively adjusted to obtain the identification threshold ranges for each type.
[0017] Furthermore, the reconstruction error MSE is obtained using the following formula:
[0018]
[0019] In the formula, x i 、x′ i Let i and n represent the i-th dimension features of the input data and the reconstructed input data, respectively, where n is the number of dimension features of the input data and the reconstructed input data.
[0020] Furthermore, the recognition thresholds for each category are adaptively adjusted by performing the following steps to obtain the recognition threshold ranges for each category:
[0021] Step 1: Set the initial value of LBound to the reconstruction error with the highest frequency in the reconstruction error distribution of the normal traffic category, and the initial value of RBound to the reconstruction error with the highest frequency in the reconstruction error distribution of the malicious traffic category closest to the normal traffic category, with an initial step size of 0.1.
[0022] Step 2: Take mid = (LBound + RBound) / 2. In the reconstruction error distribution, traverse from the mid position to the left with Step as the step size and LBound as the endpoint. Determine the reconstruction error with the maximum classification accuracy within the interval [LBound, mid] as the left threshold point Lmax; traverse from the mid position to the right with Step as the step size and RBound as the endpoint. Determine the reconstruction error with the maximum classification accuracy within the interval [mid, RBound] as the right threshold point Rmax.
[0023] Step 3: If (Rmax-Lmax) is less than or equal to the tolerable error, stop the iteration and use the median of Lmax and Rmax as the current recognition threshold for the two categories; otherwise, set LBound=Lmax, RBound=Rmax, Step=Step*0.1, return to step 2, and proceed to the next iteration.
[0024] Step 4: If there are no other malicious traffic categories to the right of the category to which the initial value of the current RBound belongs, stop calculating the identification threshold and use the maximum reconstruction error of the category as the final identification threshold; otherwise, use the initial value of the current RBound as the initial value of the next LBound, and use the reconstruction error with the highest frequency in the malicious traffic category closest to the category to the right of the reconstruction error distribution as the initial value of the next RBound. Then execute steps 2 to 3 to confirm the identification thresholds for the next two categories.
[0025] Step 5: Sort the obtained identification thresholds to obtain the identification threshold ranges for each category of traffic data, where 0 is taken as the initial identification threshold.
[0026] Furthermore, the classification accuracy is the average of the detection accuracy of the two categories; wherein, the detection accuracy of the left category is the ratio of the number of samples of that category to the left of the threshold point to the total number of samples of that category; and the detection accuracy of the right category is the ratio of the number of samples of that category to the right of the threshold point to the total number of samples of that category.
[0027] Furthermore, the tolerance error is set to 0.001.
[0028] Furthermore, obtaining the detection result of encrypted traffic based on the reconstruction error and the identification threshold range of each category includes:
[0029] If the reconstruction error of the encrypted traffic to be tested falls within the identification threshold range of any category, then the detection result of the encrypted traffic is the corresponding category.
[0030] Otherwise, the encrypted traffic is determined to be unknown malicious traffic.
[0031] Further, the step of inputting normal traffic sample data into the variational autoencoder for training to obtain a trained variational autoencoder includes:
[0032] Normal traffic sample data is input into a variational autoencoder to obtain reconstructed normal traffic sample data;
[0033] The loss function Loss is derived from the input normal traffic sample data and the reconstructed normal traffic sample data, and is expressed as:
[0034] Loss=KL(N(μ(X),σ 2 (X))|N(0,1))-L(X,P(X′=X|Z));
[0035] In the formula, X represents the input normal traffic sample data, and X′ represents the reconstructed normal traffic sample data; N(μ(X),σ 2 (X) represents a variable with mean μ(X) and variance σ. 2 The normal distribution of (X); N(0,1) represents a standard normal distribution with a mean of 0 and a variance of 1; KL() represents the KL divergence; P(X′=X|Z) represents the probability that the reconstructed value of X′ is equal to the value of the input X under the condition that the latent vector Z distribution is satisfied; -L() represents the negative log-likelihood loss function;
[0036] The parameters of the variational autoencoder are updated using the backpropagation algorithm and the stochastic gradient descent optimization algorithm to minimize the loss function, thus obtaining the trained variational autoencoder.
[0037] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0038] This invention provides a method for detecting malicious encrypted traffic based on a variational autoencoder.
[0039] 1. By collecting and processing encrypted network traffic data, a total sample data is obtained. A variational autoencoder is then trained using normal traffic sample data to obtain the feature distribution of normal encrypted traffic. The total sample data is then input into the variational autoencoder to obtain the reconstruction error of the sample data, and then the identification threshold range of each category is obtained. The type is detected by using the reconstruction error and identification threshold range of the encrypted traffic to be tested. This achieves encrypted traffic identification and detection with high generalization ability and high accuracy, which is of great practical significance for ensuring network information security and maintaining the normal operation of the network.
[0040] 2. By employing a variational autoencoder to learn the feature distribution of normal encrypted traffic samples and using a probabilistic encoder to simulate the distribution of latent vectors, the expressive power of the input features is expanded, and the model's anti-interference ability and universality are improved.
[0041] 3. The detection method based on a deep learning model using a variational autoencoder can effectively and accurately detect and classify known types of malicious encrypted traffic;
[0042] 4. By reconstructing errors and threshold values for each category of identification intervals, encrypted traffic can be detected. It can flexibly expand the detection types without retraining the variational autoencoder, and has strong generalization and transfer capabilities, enabling rapid identification and detection of unknown threats.
[0043] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0044] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0045] Figure 1 A flowchart illustrating the malicious encrypted traffic detection method based on a variational autoencoder provided in this embodiment of the invention;
[0046] Figure 2 A schematic diagram of the process for obtaining the recognition threshold ranges for various categories provided in an embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram of the structure of a variational autoencoder provided in an embodiment of the present invention. Detailed Implementation
[0048] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0049] A specific embodiment of the present invention discloses a method for detecting malicious encrypted traffic based on a variational autoencoder, such as... Figure 1 As shown, it includes the following steps:
[0050] S1. Collect and process network encrypted traffic data to obtain feature vector data, add category labels to the feature vector data to obtain total sample data, wherein the category labels include normal traffic data and traffic data of each malicious category.
[0051] Specifically, encrypted network traffic data is traffic generated by encryption algorithms. It refers to the encrypted plaintext content transmitted during communication, which can guarantee the confidentiality of the communication content.
[0052] During implementation, feature vector data is obtained by performing the following steps:
[0053] Data features are extracted from encrypted network traffic data. These data features are then processed sequentially through cleaning, null padding, and numericalization, followed by zero-mean standardization to obtain feature vector data. The data features include statistical features, time series features, and packet load features.
[0054] Specifically, a web-based testbed was built using LAMP (Linux, Apache, MySQL, Php) technology, and a network attack and defense simulation platform was built using Kali Linux. Wireshark network packet capture and analysis software was used to collect intermediate traffic data. Useless initial traffic data was filtered out, and pcap packets were collected for each detection category to simulate abnormal traffic conditions when network devices are under attack in a real environment. The collected pcap packets were parsed to extract data features. The detection categories included normal traffic data and various malicious traffic categories, including DDoS attacks, brute-force attacks, and SQL injection attacks; each category contained 5000 records.
[0055] Specifically, the statistical features include the number of data packets, the number of bytes transmitted, the data packet transmission rate, and the byte transmission rate; the time series features include the average packet interval time, the average stream duration, the standard deviation of the packet interval time, and the standard deviation of the stream duration; and the message payload features include the protocol version number, the protocol entropy, the number of TLS cipher suites, and the TLS extension length.
[0056] Specifically, the data features are cleaned and null values are filled in as follows:
[0057] The source and destination IPs in the extracted data features are filtered. Considering that most network traffic is currently IPv4, only IPv4 addresses are retained, and IPv6 network addresses are filtered out to ensure the data type of the collected sample traffic is consistent. The columns containing missing values in the extracted data features are averaged, and data with empty values in the column are set to the average, such as received data size and number of received packets. A cipher suite length column and an extension suite length column are added. For TLS fields that are empty, their cipher suite and extension suite lengths are set to 0. Other irrelevant fields are set to 0.
[0058] It is understandable that cleaning and filling in missing values in the data features can make the extracted data features more usable, avoid the impact of data noise and outliers on generalization, and help with the later training and determination of the category recognition threshold.
[0059] Specifically, the data features after cleaning and null filling are numerically processed in the following way:
[0060] One-hot encoding is used to quantify non-numerical data in the extracted data features, such as message payload features; then, all data features are standardized to zero mean. It is understandable that quantifying and standardizing the data to zero mean helps the variational autoencoder to accelerate convergence and improve robustness later on.
[0061] S2. Input the normal traffic sample data into the variational autoencoder for training to learn the feature distribution of the normal traffic data, thus obtaining the trained variational autoencoder; input the total sample data into the trained variational autoencoder to obtain the reconstruction error of the sample data for each detection category, and determine the recognition threshold range for each detection category based on the reconstruction error, such as... Figure 2 As shown; among them, each detection category includes normal and malicious categories.
[0062] Specifically, the unsupervised learning module based on variational autoencoders employs a variational autoencoder deep learning model to learn the feature distribution of normal traffic data and save the model weights. Processed and standardized normal traffic sample data is input into the variational autoencoder (VAE) deep learning model. A variational autoencoder is a directed probabilistic graphical model based on variational inference; by combining deep learning with probability statistics, it can learn the feature distribution of the data.
[0063] Specifically, such as Figure 3 As shown, the variational autoencoder includes a probabilistic encoder and a probabilistic decoder;
[0064] The probability encoder is used to encode the input data into a probability distribution in the latent space, and to sample the corresponding low-dimensional latent vector from the probability distribution.
[0065] The probabilistic encoder is used to map low-dimensional latent vectors to reconstructed input data.
[0066] Specifically, the probabilistic encoder is defined by p(Z|X), which represents the distribution of the encoded vector given by the original input vector, and the probabilistic decoder is defined by p(X'|Z), which represents the distribution from the given encoded vector to the decoded vector; where X is the original high-dimensional input feature vector, Z is the low-dimensional latent vector in the latent space, and X' is the reconstructed high-dimensional output vector.
[0067] Assume the original input feature vector is Where, x i Let represent the i-th dimension feature of the original input feature vector X, and n represent the number of dimensional features of the original input feature vector X; each data sample x i These are all randomly generated, independent, and discrete distribution vectors; the generated output feature vector is... Where, x′ i Let X' represent the i-th dimension feature of the output feature vector X', and let X' be the dimensional feature of the original input feature vector. Furthermore, it is assumed that the prior distribution p(Z) of the low-dimensional latent vector Z follows a standard Gaussian distribution, and the likelihood distribution p(X|Z) follows a Gaussian distribution. By Bayes' theorem, the prior distribution p(Z), the likelihood distribution p(X|Z), and the posterior distribution p(Z|X) have the following relationship:
[0068]
[0069] Therefore, the variational autoencoder deep learning model can be divided into two processes: the probabilistic encoder used for approximate inference of the posterior distribution of the low-dimensional latent vector Z. That is, the inference network; the probabilistic decoder is used to generate the conditional distribution of vector X'. θ (Z)p θ (X′|Z), which is the generator network.
[0070] In implementation, the step of inputting normal traffic sample data into the variational autoencoder for training to obtain a trained variational autoencoder includes:
[0071] S21. Input the normal flow sample data into the variational autoencoder to obtain the reconstructed normal flow sample data;
[0072] Specifically, the reconstructed normal traffic sample data is obtained by performing the following steps:
[0073] The initial weights of the variational autoencoder are set to random values that conform to a normal distribution; the input data is encoded as a normal distribution in the latent space; the probabilistic encoder maps the input sample data to the mean μ and standard deviation σ. 2 Two sets of parameters are used to determine the conditional probability distribution of the low-dimensional latent vector Z based on the input data X. The probability decoder samples a low-dimensional latent vector Z belonging to the probability distribution from the probability distribution;
[0074] Through probability decoder p θ (Z)p θ (X′|Z) maps the low-dimensional latent vector Z to the reconstructed input sample data X′.
[0075] Preferably, setting the dimension of the latent vector Z to 20 will optimize the performance of the variational autoencoder.
[0076] Understandably, by employing a variational autoencoder to learn the feature distribution of normal encrypted traffic samples and using a probabilistic encoder to simulate the distribution of latent vectors, the expressive power of the input features is expanded, and the model's anti-interference ability and generalization ability are improved.
[0077] S22. Based on the input and reconstructed normal traffic sample data, the loss function Loss is obtained, expressed as:
[0078] Loss=KL(N(μ(X),σ 2 (X))|N(0,1))-L(X,P(X′=X|Z));
[0079] In the formula, X represents the input normal traffic sample data, i.e., the original input feature vector, and X′ represents the reconstructed normal traffic sample data, i.e., the output feature vector; N(μ(X),σ 2 (X) represents a variable with mean μ(X) and variance σ. 2 The normal distribution of (X); N(0,1) represents a standard normal distribution with a mean of 0 and a variance of 1; KL() represents the KL divergence; P(X′=X|Z) represents the probability that the reconstructed value of X' is equal to the value of the input X under the condition that the latent vector Z distribution is satisfied; -L() represents the negative log-likelihood loss function.
[0080] Specifically, KL divergence representation The distance between the reference probability distribution, i.e., the standard Gaussian distribution N(0,1); the negative log-likelihood loss function is the probability that the value of the reconstructed output X' is equal to that of the input sample X, under the condition that the latent vector Z distribution is satisfied. The negative log-likelihood loss function represents the degree of overlap between the distribution of the reconstructed input X' and the distribution of the input sample data X.
[0081] Specifically, the KL divergence is calculated using the following formula:
[0082]
[0083] In the formula, d represents the dimension of the latent vector Z, and μ i and Let Z represent the mean and variance of the approximate posterior distribution of the i-th dimension in the latent vector Z.
[0084] Specifically, the negative log-likelihood loss function is calculated using the following formula:
[0085]
[0086] In the formula, P(x′) i =x i |Z) represents the value of the i-th dimension feature of the reconstructed X' under the condition of satisfying the latent vector Z. i It equals the value of the i-th dimension feature of input X. i The probability of.
[0087] S23. The parameters of the variational autoencoder are updated using the backpropagation algorithm and the stochastic gradient descent optimization algorithm to minimize the loss function and obtain the trained variational autoencoder.
[0088] Understandably, designing a loss function and adding a backpropagation algorithm during variational encoder training aims to minimize the distance between the reconstructed input data X' and the original input data X while ensuring the variational autoencoder possesses generative capabilities. Try to satisfy the standard Gaussian distribution N(0,1) as much as possible.
[0089] During implementation, the total sample data is input into the trained variational autoencoder to obtain the reconstruction error of the sample data for each detection category. Based on the reconstruction error, the recognition threshold range for each detection category is determined, including:
[0090] The total sample data is input into the trained variational autoencoder to obtain the reconstructed input data.
[0091] The reconstruction error is obtained based on the total sample data of the original input and the reconstructed input data. The reconstruction error distribution of each category of traffic data is statistically analyzed. The categories include normal categories and malicious categories.
[0092] Based on the reconstruction error distribution and the detection accuracy of each type of traffic data, the identification thresholds for each type are adaptively adjusted to obtain the identification threshold ranges for each type.
[0093] In practice, the reconstruction error MSE is obtained using the following formula:
[0094]
[0095] In the formula, x i 、x′ i Let i and n represent the i-th dimension features of the input data and the reconstructed input data, respectively, where n is the number of dimension features of the input data and the reconstructed input data.
[0096] In practice, the recognition thresholds for each category are adaptively adjusted by performing the following steps to obtain the recognition threshold ranges for each category:
[0097] Step 1: Set the initial value of LBound to the reconstruction error with the highest frequency in the reconstruction error distribution of the normal traffic category, and the initial value of RBound to the reconstruction error with the highest frequency in the reconstruction error distribution of the malicious traffic category closest to the normal traffic category, with an initial step size of 0.1.
[0098] Step 2: Take mid = (LBound + RBound) / 2. In the reconstruction error distribution, traverse from the mid position to the left with Step as the step size and LBound as the endpoint. Determine the reconstruction error with the maximum classification accuracy within the interval [LBound, mid] as the left threshold point Lmax; traverse from the mid position to the right with Step as the step size and RBound as the endpoint. Determine the reconstruction error with the maximum classification accuracy within the interval [mid, RBound] as the right threshold point Rmax.
[0099] Step 3: If (Rmax-Lmax) is less than or equal to the tolerable error, stop the iteration and use the median of Lmax and Rmax as the current recognition threshold for the two categories; otherwise, set LBound=Lmax, RBound=Rmax, Step=Step*0.1, return to step 2, and proceed to the next iteration.
[0100] Step 4: If there are no other malicious traffic categories to the right of the category to which the initial value of the current RBound belongs, stop calculating the identification threshold and use the maximum reconstruction error of the category as the final identification threshold; otherwise, use the initial value of the current RBound as the initial value of the next LBound, and use the reconstruction error with the highest frequency in the malicious traffic category closest to the category to the right of the reconstruction error distribution as the initial value of the next RBound. Then execute steps 2 to 3 to confirm the identification thresholds for the next two categories.
[0101] Step 5: Sort the obtained identification thresholds to obtain the identification threshold ranges for each category of traffic data, where 0 is used as the initial identification threshold. For example, if the identification thresholds are determined sequentially as 0.33, 0.45, and 0.55, the initial identification threshold is 0, and the final identification threshold is 0.7, then the obtained identification threshold ranges are [0, 0.33], (0.33, 0.45], (0.45, 0.55], and (0.55, 0.7], respectively; where [0, 0.33] is the identification threshold range for normal traffic categories, and the others correspond to the respective malicious traffic categories.
[0102] Understandably, variational autoencoders learn the data features of normal traffic categories, so the reconstruction error is relatively small compared to other malicious categories when reconstructing the input. Therefore, the identification thresholds of normal traffic categories and their adjacent malicious traffic categories are determined first, and then the identification thresholds of subsequent adjacent malicious traffic categories are determined in turn.
[0103] Specifically, the classification accuracy is the average of the detection accuracies of the two categories; where the detection accuracy of the left category is the ratio of the number of samples of that category to the left of the threshold point to the total number of samples of that category; and the detection accuracy of the right category is the ratio of the number of samples of that category to the right of the threshold point to the total number of samples of that category. It can be understood that when determining the left threshold point Lmax, the threshold point used to calculate the detection accuracy of the two categories is the left threshold point; and when determining the right threshold point Rmax, the threshold point used to calculate the detection accuracy of the two categories is the right threshold point.
[0104] Specifically, the tolerable error can be set according to actual needs. Preferably, the tolerable error is 0.001, which makes the obtained recognition threshold more accurate and reliable.
[0105] S3. Input the feature vector data of the encrypted traffic data to be tested into the trained variational autoencoder to obtain the reconstruction error. Based on the reconstruction error and the recognition threshold range of each category, the detection result of the encrypted traffic is obtained.
[0106] In implementation, obtaining the detection result of encrypted traffic based on the reconstruction error and the identification threshold range of each category includes:
[0107] If the reconstruction error of the encrypted traffic to be tested falls within the identification threshold range of any category, then the detection result of the encrypted traffic is the corresponding category.
[0108] Otherwise, the encrypted traffic is determined to be unknown malicious traffic.
[0109] Understandably, this method of detection can quickly identify malicious traffic of unknown types, provide timely warnings, and allow for further analysis of such traffic, thereby improving the detection model's ability to resist unknown threats.
[0110] Compared with existing technologies, this embodiment provides a malicious encrypted traffic detection method based on variational autoencoders. It collects and processes network encrypted traffic data to obtain total sample data, and trains a variational autoencoder using normal traffic sample data to obtain the feature distribution of normal encrypted traffic. The total sample data is then input into the variational autoencoder to obtain the reconstruction error of the sample data, and subsequently, the identification threshold ranges for each category are obtained. Type detection is performed using the reconstruction error and identification threshold ranges of the encrypted traffic to be tested, achieving high generalization ability and high accuracy in encrypted traffic identification and detection. This is of significant practical importance for ensuring network information security and maintaining normal network operation. The deep learning model detection method based on variational autoencoders can effectively and accurately detect and classify known types of malicious encrypted traffic. By detecting encrypted traffic using reconstruction error and identification threshold ranges for each category, the detection types can be flexibly expanded without retraining the variational autoencoder, exhibiting strong generalization and transfer capabilities, and enabling rapid identification and detection of unknown threats.
[0111] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0112] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for detecting malicious encrypted traffic based on variational autoencoders, characterized in that, Includes the following steps: Collect and process encrypted network traffic data to obtain feature vector data, add category labels to the feature vector data to obtain total sample data, wherein the category labels include normal traffic data and traffic data of each malicious category; Normal traffic sample data is input into the variational autoencoder for training. The feature distribution of normal traffic data is learned to obtain the trained variational autoencoder. The total sample data is input into the trained variational autoencoder to obtain the reconstruction error of the sample data for each detection category. Based on the reconstruction error, the recognition threshold range for each detection category is determined. Each detection category includes normal and malicious categories, including: The total sample data is input into the trained variational autoencoder to obtain the reconstructed input data. The reconstruction error is obtained based on the total sample data of the original input and the reconstructed input data. The distribution of reconstruction error for each category of traffic data is then statistically analyzed. Based on the reconstruction error distribution and the detection accuracy of each category of traffic data, the identification threshold of each category is adaptively adjusted to obtain the identification threshold range of each category. The feature vector data of the encrypted traffic data to be tested is input into the trained variational autoencoder to obtain the reconstruction error. Based on the reconstruction error and the recognition threshold range of each detection category, the detection result of the encrypted traffic is obtained.
2. The malicious encrypted traffic detection method based on variational autoencoder according to claim 1, characterized in that, Feature vector data is obtained by performing the following steps: The data features of the encrypted traffic data are extracted, and the data features are cleaned, filled with null values, and numerically processed. Then, they are normalized to zero mean to obtain feature vector data. The data features include statistical features, time series features, and message load features.
3. The malicious encrypted traffic detection method based on variational autoencoder according to claim 2, characterized in that, The statistical features include the number of data packets, the number of bytes transmitted, the data packet transmission rate, and the byte transmission rate; the time series features include the average packet interval time, the average stream duration, the standard deviation of the packet interval time, and the standard deviation of the stream duration; the message payload features include the protocol version number, the protocol entropy, the number of TLS cipher suites, and the TLS extension length.
4. The malicious encrypted traffic detection method based on variational autoencoder according to claim 1, characterized in that, The reconstruction error is obtained using the following formula. : ， In the formula, , These represent the first and second parts of the input data and the reconstructed input data, respectively. Features in each dimension The number of dimensional features for the input data and the reconstructed input data.
5. The malicious encrypted traffic detection method based on variational autoencoder according to claim 4, characterized in that, The recognition thresholds for each category are adaptively adjusted by performing the following steps to obtain the recognition threshold ranges for each category: Step 1: Set the initial value of LBound to the reconstruction error with the highest frequency in the reconstruction error distribution of the normal traffic category, and the initial value of RBound to the reconstruction error with the highest frequency in the reconstruction error distribution of the malicious traffic category closest to the normal traffic category, with an initial step size of 0.
1. Step 2: Take mid = (LBound + RBound) / 2. In the reconstruction error distribution, traverse from the mid position to the left with Step as the step size and LBound as the endpoint. Determine the reconstruction error with the maximum classification accuracy within the interval [LBound, mid] as the left threshold point Lmax; traverse from the mid position to the right with Step as the step size and RBound as the endpoint. Determine the reconstruction error with the maximum classification accuracy within the interval [mid, RBound] as the right threshold point Rmax. Step 3: If (Rmax-Lmax) is less than or equal to the tolerable error, stop the iteration and use the median of Lmax and Rmax as the current recognition threshold for the two categories. Otherwise, set LBound = Lmax, RBound = Rmax, and Step = Step. 0.1, Return to step 2 and proceed to the next iteration; Step 4: If the category to which the current RBound's initial value belongs does not have any other malicious traffic categories on the right, then stop calculating the identification threshold and use the maximum reconstruction error of that category as the final identification threshold. Otherwise, the initial value of the current RBound is used as the initial value of the next LBound. The reconstruction error with the highest frequency in the malicious traffic category closest to this category on the right side of the reconstruction error distribution is used as the initial value of the next RBound. Steps 2 and 3 are executed to confirm the identification thresholds for the next two categories. Step 5: Sort the obtained identification thresholds to obtain the identification threshold ranges for each category of traffic data, where 0 is taken as the initial identification threshold.
6. The malicious encrypted traffic detection method based on variational autoencoder according to claim 5, characterized in that, The classification accuracy is the average of the detection accuracy of the two categories; wherein, the detection accuracy of the left category is the ratio of the number of samples of that category to the left of the threshold point to the total number of samples of that category; the detection accuracy of the right category is the ratio of the number of samples of that category to the right of the threshold point to the total number of samples of that category.
7. The malicious encrypted traffic detection method based on variational autoencoder according to claim 5, characterized in that, The tolerance error is set to 0.
001.
8. The malicious encrypted traffic detection method based on variational autoencoder according to claim 1, characterized in that, The step of obtaining the detection result of encrypted traffic based on the reconstruction error and the identification threshold range of each category includes: If the reconstruction error of the encrypted traffic to be tested falls within the identification threshold range of any category, then the detection result of the encrypted traffic is the corresponding category. Otherwise, the encrypted traffic is determined to be unknown malicious traffic.
9. The malicious encrypted traffic detection method based on variational autoencoder according to claim 2, characterized in that, The step of inputting normal traffic sample data into the variational autoencoder for training to obtain a trained variational autoencoder includes: Normal traffic sample data is input into a variational autoencoder to obtain reconstructed normal traffic sample data; The loss function is obtained based on the input normal traffic sample data and the reconstructed normal traffic sample data. , is represented as: ； In the formula, This represents the input normal traffic sample data. This indicates the reconstruction of normal traffic sample data; The mean is The variance is The normal distribution; This represents a standard normal distribution with a mean of 0 and a variance of 1. express KL Divergence; This indicates that the implicit vector is satisfied. Under the condition of distribution, reconstruction The value is equal to the input. The probability of the value of ; This represents the negative log-likelihood loss function; The parameters of the variational autoencoder are updated using the backpropagation algorithm and the stochastic gradient descent optimization algorithm to minimize the loss function, thus obtaining the trained variational autoencoder.