Robust unknown flow detection method and system based on feature alignment and random disturbance injection

Abstract

The invention discloses a robust unknown traffic detection method and system based on feature alignment and random disturbance injection. The robust unknown traffic detection method comprises a known traffic fingerprint construction stage and an unknown traffic detection stage. In a known traffic fingerprint construction stage, a neural network model is trained by using a multi-scale feature extractor and a feature alignment mechanism according to a simulated fluctuation network environment, so that a known category traffic fingerprint with environmental adaptability is constructed. And an unknown flow detection stage: performing multi-scale feature fusion on the network flow captured in real time based on the trained model parameters and flow fingerprints, and calculating a comprehensive score through a random disturbance injection strategy to complete unknown flow detection. According to the invention, a unidirectional feature alignment mechanism based on gradient truncation is designed, and the mechanism can align the fluctuation traffic features to the normal traffic features, thereby eliminating the influence caused by network fluctuation. According to the invention, high-precision and high-robustness detection of unknown traffic in a complex fluctuation network environment can be realized.

Description

Technical Field

[0001] This invention relates to a robust unknown traffic detection method and system based on feature alignment and random perturbation injection, which extracts features from the sequence information of network traffic using a multi-scale feature extractor, optimizes the feature space based on deep learning technology, and enhances the distinction between known and unknown traffic by employing random perturbation injection technology. Specifically, it relates to a robust unknown traffic detection method and system based on feature alignment and random perturbation injection. Background Technology

[0002] Unknown traffic detection refers to identifying unknown traffic not currently on the known list by analyzing the characteristics of network packets. This traffic could be an attack or a new application. This work is crucial for network resource management, Quality of Service (QoS) assurance, and network security monitoring (such as DDoS attack detection and malware identification). With the evolution of network technology, traditional port-based or deep packet inspection (DPI) methods have gradually become ineffective due to dynamic ports and the widespread use of encryption technologies. Automated traffic analysis techniques based on machine learning and deep learning have become the mainstream research approach.

[0003] However, in practical industrial deployments and applications, unknown traffic detection faces severe challenges. First, real-world network environments are highly volatile and uncertain. The packet loss, out-of-order delivery, and retransmission phenomena prevalent in real networks are no longer merely transport layer issues, but severely disrupt the sequential characteristics of traffic. For example, packet loss leads to missing byte stream content, causing severe misalignment of features in the sequence; out-of-order delivery breaks the semantic dependencies of the context. This physical layer volatility manifests as a dramatic distribution shift in the feature space, causing models to fail to match the static features used during training. More seriously, environmental volatility blurs the boundary between "known" and "unknown." Changes in normal known traffic caused by volatility can easily be misjudged as "unknown traffic" by the model, leading to a high false alarm rate. Therefore, how to overcome environmental volatility interference and accurately identify unknown traffic in open-set scenarios is a pressing problem that needs to be solved.

[0004] This patent designs and implements a robust unknown traffic detection method and system based on feature alignment and random perturbation injection. The method simulates the fluctuations and disturbances of real networks, utilizes a feature alignment mechanism to enhance the model's adaptability to environmental fluctuations, and combines this with a random perturbation injection strategy to leverage the difference in sensitivity of known and unknown traffic to parameter perturbations, thereby achieving robust detection of unknown traffic in fluctuating network environments. Summary of the Invention

[0005] The purpose of this invention is to design and implement a robust unknown traffic detection method and system based on feature alignment and random perturbation injection. This method and system can effectively address both the challenges of fluctuating interference (such as packet loss, out-of-order delivery, and retransmission) and the difficulty of identifying unknown traffic in real-world network traffic detection. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0006] A robust unknown traffic detection method based on feature alignment and random perturbation injection is characterized by including a known traffic fingerprint construction stage and an unknown traffic detection stage;

[0007] The known traffic fingerprint construction phase includes the following steps:

[0008] 1) Using a set of known network traffic data with category labels as input, clean and standardize the data to obtain a known traffic sample sequence;

[0009] 2) Based on the known flow sample sequence obtained in step 1), a fluctuation simulation mechanism is introduced to generate a paired training dataset containing normal flow and fluctuating flow;

[0010] 3) Using the paired training dataset obtained in step 2) as input, construct a neural network model based on multi-scale feature fusion, and use classification loss and feature alignment loss for joint training to obtain the parameters of the feature extractor model with fluctuation adaptability, and construct the traffic fingerprint of known categories;

[0011] The unknown traffic detection phase includes the following steps:

[0012] 4) Using the real-time captured traffic data to be detected as input, perform the same preprocessing operation as in step 1) to obtain a standardized sequence of the traffic to be detected;

[0013] 5) Load the solidified feature extractor parameters from step 3) and extract the fused feature vector of the flow to be detected;

[0014] 6) Apply random perturbation injection to the classifier weight matrix solidified in step 3), and calculate the output distribution and response score of the traffic to be detected under the perturbation weight;

[0015] 7) Combine the known category traffic fingerprints constructed in step 3) to calculate the comprehensive discrimination score, compare the comprehensive discrimination score with the preset threshold, and output the traffic category determination result.

[0016] A robust unknown traffic detection system based on feature alignment and random perturbation injection is characterized by comprising a data preprocessing and fluctuation simulation module, a multi-scale feature extraction module, and a joint training module for classification and feature alignment loss in the known traffic fingerprint construction stage, as well as a data preprocessing and feature extraction module, a random perturbation injection module, and an unknown traffic detection module in the unknown traffic detection stage.

[0017] The data preprocessing and fluctuation simulation module is responsible for cleaning and serializing the input known category traffic set, and randomly applying packet loss, out-of-order and retransmission interference to generate an enhanced training dataset containing normal and fluctuating traffic pairs.

[0018] The multi-scale feature extraction module is responsible for extracting byte stream and packet length sequence features using a neural network based on a self-attention mechanism, generating high-dimensional feature vectors through adaptive fusion, and optimizing the neural network model by combining classification loss and alignment loss.

[0019] The known category traffic fingerprint construction module is responsible for calculating and storing the average activation vector and average probability distribution of all known traffic samples in the training set after the model parameters are fixed.

[0020] The data preprocessing and feature extraction module is responsible for taking the traffic set to be detected as input, performing cleaning and serialization operations, and loading solidified model parameters to extract robust feature vectors;

[0021] The random perturbation injection module is responsible for applying random perturbations to the classifier weights, calculating the density estimate and maximum probability score of the input features under the perturbation weights, and calling the known category traffic fingerprint to calculate the KL divergence between the density estimate of the sample input features and the known category average density estimate.

[0022] The unknown flow detection module uses the maximum probability score and KL divergence of the samples to calculate a comprehensive discrimination score through weighting and compare it with a preset threshold, and outputs a discrimination result of whether the flow is known or unknown.

[0023] The key technical point of this invention is:

[0024] 1. A multi-scale self-attention feature fusion network architecture is adopted to achieve complementary enhancement and accurate representation of network traffic content information and behavioral statistical features.

[0025] 2. A feature alignment mechanism based on gradient truncation is designed. This mechanism forces traffic features under fluctuating environments to unidirectionally align to the normal feature space, thereby giving the model the ability to adapt to fluctuating network environments such as packet loss, out-of-order delivery, and retransmission.

[0026] 3. A strategy for detecting unknown traffic based on random perturbation injection is proposed. This strategy constructs a perturbation space by applying small random perturbations to the classifier weights, which increases the difference between known and unknown traffic and achieves high-precision detection of unknown traffic.

[0027] The method of this invention can achieve robust and accurate detection of unknown traffic in complex and fluctuating network environments. Compared with previously disclosed related technical solutions, it has the following practical advantages:

[0028] 1. This invention designs a unidirectional feature alignment mechanism based on gradient truncation, which significantly enhances the model's adaptability to network environment fluctuations. This mechanism, by introducing fluctuation simulation and gradient truncation operations during the training phase, forces the features of fluctuating traffic to unidirectionally align with the "anchor" features of normal traffic.

[0029] 2. This invention proposes an unknown traffic detection strategy based on random perturbation injection, which significantly improves the detection accuracy in open set scenarios. It leverages the difference in sensitivity between known and unknown traffic to model parameter perturbations to amplify the differences between known and unknown traffic in the perturbation feature space.

[0030] 3. This invention constructs a feature extractor based on a multi-scale self-attention mechanism, achieving a comprehensive representation of traffic information. This multimodal complementary approach can more accurately capture the global patterns of traffic, further improving the model's feature discrimination when facing novel, unknown applications or complex encrypted traffic. Attached Figure Description

[0031] Figure 1 This is a flowchart of the known traffic fingerprint construction phase.

[0032] Figure 2 This is a flowchart of the unknown traffic detection phase.

[0033] Figure 3 This is an architecture diagram of a robust unknown traffic detection system based on feature alignment and random perturbation injection. Detailed Implementation

[0034] The workflow of this invention is mainly divided into two stages: the first stage is the known traffic fingerprint construction stage, and the second stage is the unknown traffic detection stage.

[0035] Known traffic fingerprint construction stages: such as Figure 1 As shown, this stage aims to build and train a feature extractor and classifier capable of adapting to network fluctuations. By simulating network fluctuations in an offline environment and utilizing a feature alignment mechanism, the model is made capable of resisting interference such as packet loss and out-of-order delivery during training. The specific implementation steps are as follows:

[0036] 1. Offline capture and standardized preprocessing of known network traffic data. First, this invention acquires a known network traffic dataset with accurate category labels. ,in Representing the A sample of network traffic. For the corresponding category label.

[0037] Secondly, data cleaning is performed to remove invalid data packets with incomplete header information or abnormal payloads from the original traffic, ensuring the validity of the input data. Afterwards, sequence construction is performed on the cleaned data. Data packets are then split into streams based on five-tuples, and each stream is processed... The parsing process involves representing the sequence at two scales. On one hand, the payloads of all data packets in the traffic are extracted and concatenated sequentially to form a one-dimensional byte sequence. On the other hand, the length information of each data packet in the traffic (i.e., the total length of the IP packets) is extracted to form a packet length sequence. .

[0038] Finally, a fixed-length processing operation is performed. Based on a preset maximum length threshold, the generated byte sequence and packet length sequence are truncated or padded with zeros to meet the fixed input dimension requirements of the deep neural network.

[0039] 2. Fluctuation in Network Environment Simulation and Paired Sample Generation. First, to impart robustness to the model against network fluctuations, this invention introduces a fluctuation simulation mechanism during the training phase. The preprocessed first... The original data packet sequence of normal traffic is ,in This represents the total number of data packets. Next, simulated packet loss, out-of-order delivery, and retransmission operations are performed on the original data packet sequence. The specific implementation steps are as follows:

[0040] For the original data packet sequence Perform packet loss simulation operation: Define packet loss rate Randomly select a set of data packet indexes Discard the packets. Generate the post-drop sequence. Defined as Its sequence length becomes This operation simulates the loss of features caused by information loss during network transmission.

[0041] Perform out-of-order simulation on the data packet sequence: Define the out-of-order window Through permutation function The packet index is rearranged. The resulting disordered sequence is... Defined as This operation preserves the total length of the sequence. It remains unchanged, but the temporal dependency between the byte stream and the packet-length sequence has been altered.

[0042] Perform a retransmission simulation operation on the data packet sequence: Define the retransmission rate. Randomly select the set of indices The data packets in the sequence are copied and inserted into the original sequence. The resulting retransmitted sequence... The sequence length of the data packet, which includes both the original and copied data packets, is increased to [number]. This operation introduces redundant noise into the feature sequence.

[0043] Finally, through a random combination of one or more of the above fluctuation operations, a corresponding fluctuation byte stream sequence is generated. and fluctuation packet length sequence And construct a paired training dataset containing "(normal traffic, fluctuating traffic, label)". .

[0044] 3. Feature extraction based on a multi-scale feature fusion network. First, this invention constructs a dual-stream feature extraction network, including a byte-stream feature extractor. Packet-long sequence feature extractor Both adopt the Transformer architecture based on the self-attention mechanism.

[0045] Next, normal feature extraction is performed, and normal traffic is input into the network, utilizing... Extracting feature vectors from normal byte streams ,use Extracting feature vectors from normal packet length sequences .

[0046] Similar to normal features, fluctuation feature extraction is performed by using the same network with shared parameters to extract the feature vector of fluctuating flow. and .

[0047] 4. Feature Alignment Based on Gradient Truncation. First, to eliminate feature shifts caused by network fluctuations, this invention designs a feature alignment module. An aligner composed of a multilayer perceptron (MLP) is used. and The calibrated wave characteristics are obtained by performing nonlinear mapping on the wave characteristics respectively.

[0048]

[0049]

[0050] in, For the first Byte sequence features of fluctuating flow samples after multilayer perceptron calibration. For the first Packet length sequence features of fluctuating flow samples after multilayer perceptron calibration.

[0051] Secondly, to prevent normal features from compromising with fluctuating features, this invention introduces gradient stopping. The operation treats normal, unfluctuating flow features as fixed "anchor points," blocking the back propagation of gradients towards the normal feature extraction path. Finally, the unidirectional alignment loss is calculated. It is defined as the weighted sum of the alignment loss of the byte stream and the packet-length sequence:

[0052]

[0053] in, A regression loss function based on cosine similarity is adopted to maximize the consistency of the calibrated fluctuation features with the normal features in terms of direction.

[0054] 5. Feature Fusion and Classification Prediction. This invention utilizes an adaptive feature fusion module. The byte stream features and packet length sequence features of normal traffic are fused, and high-dimensional fused features are generated by learning the weights of different modalities. :

[0055]

[0056] Next, the fused features are input into the classifier (a fully connected layer combined with a softmax activation function) to obtain the predicted class probability distribution. .

[0057] 6. Calculate the joint total loss. First, calculate the classification loss. (e.g., cross-entropy loss), used to measure the distribution of predicted classes. With real labels The differences between them. Next, the total loss function used to optimize the entire neural network is calculated. The loss consists of the classification loss and the feature alignment loss calculated in step 4. Weighted composition:

[0058]

[0059] in, Hyperparameters are used to balance classification accuracy and feature robustness.

[0060] 7. Neural Network Parameter Update and Iterative Optimization. First, determine whether the current training state of the neural network meets the preset termination condition. The termination condition Including but not limited to reaching the maximum iteration period (Epochs) and the total loss function value The convergence occurs when the target value is below the preset threshold, or when the statistical evaluation metric on the validation set reaches the expected value.

[0061] (a) If the termination condition is not met Then, based on the total loss calculated in step 6 The gradient is calculated using the backpropagation algorithm, and the feature extractor is updated. ), feature fusion module ( ), Feature alignment module ( ) and classifier ( All trainable parameters of the dataset are updated. After updating the parameters, return to step 3, read in the next batch of data, and repeat the subsequent process.

[0062] (b) If the termination condition is met If the training process stops, the trained neural network parameter set ( The model is then solidified and output as the final offline model.

[0063] 8. Construction of Known Category Traffic Fingerprints (Benchmark). After model training is completed and parameters are fixed, this invention establishes a statistical benchmark for subsequent unknown traffic detection. This involves statistically analyzing the feature vectors of all known traffic samples in the training set at the penultimate layer of the model. These statistics are stored as "known category traffic fingerprints" and used to calculate the difference between input traffic and known traffic during the online detection phase.

[0064] Unknown traffic detection phase: such as Figure 2 As shown, this stage aims to analyze real-time captured network traffic using the model parameters trained offline and the constructed traffic fingerprint. By introducing a random perturbation mechanism, and leveraging the difference in sensitivity of known and unknown traffic to parameter perturbations, robust unknown traffic detection in fluctuating network environments is achieved. The specific implementation steps are as follows:

[0065] 1. Online capture of test traffic data and standardized preprocessing. This invention uses a traffic capture interface deployed on network nodes to capture raw network traffic to be tested in real time. For each captured real-time traffic... Perform data cleaning operations identical to the offline phase, removing invalid or corrupted data packets to ensure consistent data quality. Then, perform sequence construction, parsing the cleaned traffic into sequence representations at two scales. Extract the payloads of all data packets in the traffic and concatenate them sequentially to form a byte stream sequence. Simultaneously, the length information of each data packet is extracted to form a packet length sequence. Finally, a fixed-length processing operation is performed. Based on the maximum length threshold set during offline training, the above sequence is truncated or padded with zeros to generate standardized input data pairs. .

[0066] 2. Multi-scale feature extraction and feature fusion. This invention loads a feature extractor that has been trained offline and whose parameters have been fixed (parameters are...). The preprocessed data from step 1 is input into the model, and a byte stream feature extractor is used. Packet-long sequence feature extractor Extract byte feature vectors respectively and bag length feature vector Then, the adaptive feature fusion module is used. (parameter is) The feature vectors of the two modalities mentioned above are then deeply integrated. Finally, a high-dimensional fused feature vector of the real-time traffic sample is generated. Since the feature extractor has been trained offline using fluctuation simulation, the feature vector is robust to network fluctuations and does not require additional online alignment.

[0067] 3. Random perturbation injection of classifier weights. This invention obtains the offline-trained classifier weight matrix. ,in Given the number of categories, Let be the feature dimension. Then, define a random perturbation operation. For each class projection vector in the weight matrix... (corresponding to the first) (Class), by adding a random noise vector sampled from a standard normal distribution. ,generate A tiny perturbation vector. Weights after the second perturbation The calculation formula is:

[0068]

[0069] in, To control the hyperparameters of disturbance intensity, .

[0070] Construct a set of perturbation weights This is used to generate a perturbed feature space when performing unknown traffic detection later.

[0071] 4. Calculate the maximum probability after the perturbation ( Score. The fused feature vector obtained in step 2... Input them into the generated step 3 respectively A perturbation weight matrix In the middle, calculate the corresponding output. Apply the output for each disturbance The function yields the category probability distribution. And extract the highest probability value. Finally, calculate... The average of the maximum probability values ​​under each perturbation is used to obtain the perturbation. Score

[0072]

[0073] The known flow rate usually has a high The score is given for unknown traffic, which is sensitive to disturbances and is usually lower.

[0074] 5. Calculate fingerprint-based matching Divergence score. First, construct the offline baseline distribution. This is based on the penultimate layer feature vector set of the training set saved during the offline phase. and generated through step 3 A perturbation weight matrix For Each sample feature vector Calculate its output in the perturbation space ( ):

[0075]

[0076] Will All elements are treated as scalar samples, and the density estimate of the sample is calculated through histogram statistics and smoothing. Finally, the density estimates of all training samples are averaged to obtain the offline baseline average density distribution:

[0077]

[0078] Next, the density estimate of the current sample is calculated. For the current sample to be detected, its density is calculated in... Output under perturbation weights Similarly, through histogram statistics and smoothing, the density estimate of the current sample is obtained. Finally, the distribution difference score of the perturbation space is calculated. Using symmetry Divergence measures the current sample density Compared with offline benchmarks Differences between them:

[0079]

[0080] The above The divergence score quantifies the degree of deviation between the current traffic and the known traffic patterns in the activation distribution; a higher score indicates that the traffic is more likely to be unknown.

[0081] 6. Unknown Traffic Detection, Judgment, and Result Output. First, this invention combines the MSP score obtained in step 4 with the score obtained in step 5... The divergence scores are weighted and combined to calculate the final comprehensive discriminant score.

[0082]

[0083] in, These are preset weighting coefficients used to balance the distance-based (KL) and confidence-based (MSP) detection metrics. Then, the overall discrimination score will be calculated. With the preset detection threshold Compare. If If the classification is correct, the real-time traffic is determined to be of an unknown category; otherwise, it is determined to be of a known category, and its specific classification result is output.

[0084] Combining the aforementioned robust unknown traffic detection method based on feature alignment and random perturbation injection, this invention also discloses a robust unknown traffic detection system based on feature alignment and random perturbation injection. This system mainly consists of two important stages: a known traffic fingerprint construction stage and an unknown traffic detection stage. The system architecture is as follows: Figure 3 As shown.

[0085] 1. Known Traffic Fingerprint Construction Stage: First, using a known set of traffic categories as input, the original traffic is cleaned and serialized by the (1) Data Preprocessing and Fluctuation Simulation module, and random packet loss, out-of-order, and retransmission interference is applied to generate an enhanced training dataset containing "normal-fluctuation" traffic pairs. The (2) Multi-scale Feature Extraction module uses a neural network based on a self-attention mechanism to extract byte stream and packet length sequence features respectively, and generates high-dimensional feature vectors through adaptive fusion. Finally, the (3) Classification and Feature Alignment Loss Joint Training Module calculates the classification and alignment loss in conjunction with the gradient truncation mechanism, iteratively optimizes the model parameters, and makes them adaptable to fluctuations. The output of this module is the solidified feature extractor model parameters and the statistically obtained known traffic anti-fluctuation fingerprint.

[0086] 2. Unknown Traffic Detection Stage: The detection stage takes the set of traffic to be detected as input. The (1) data preprocessing and feature extraction module performs the same cleaning and serialization as the offline stage, loads the offline solidified model parameters, and extracts the robust feature vector of the real-time traffic. (2) The random perturbation injection module does not classify directly, but applies multiple random perturbations to the classifier weights and calculates the response change and distribution of the input features under the perturbation weights. (3) The unknown traffic detection module takes the feature response as input, calls the known traffic anti-fluctuation fingerprint, and calculates the sample to be tested. The traffic category is determined by combining the divergence and the maximum probability score after perturbation.

[0087] This invention underwent systematic validation on three representative network traffic datasets. Dataset-I (CSNET) contains 120 types of encrypted traffic with 46,270 samples; Dataset-II (C-P215) contains 215 types of mobile application traffic with 25,879 samples; and Dataset-III (USTC) contains 10 types of malicious attack traffic with 5,312 samples. These three datasets cover various typical scenarios including encrypted communication, mobile internet, and network attacks, as detailed in Table 1.

[0088] Table 1: Network traffic dataset used for verification in this invention

[0089] Dataset source Category Number of categories Number of samples (streams) Dataset-I CSNET Encrypted traffic 120 46,270 Dataset-II C-P215 Mobile application 215 25,879 Dataset-III USTC attack traffic 10 5,312

[0090] Based on the aforementioned dataset, this invention designs three unknown traffic detection scenarios and introduces different levels of network fluctuation (0%, 15%, 30%) to simulate real-world challenges. Scenario-A, based on Dataset-I, halves the categories into known and unknown, focusing on verifying the detection and identification of encrypted unknown traffic. Scenario-B, based on Dataset-II, also halves the categories, examining the unknown detection capability in mobile application scenarios. Scenario-C uses all categories from Dataset-I as known and all categories from Dataset-III as unknown, simulating a real-world scenario of defending against unknown malicious attacks. Detailed settings are shown in Table 2.

[0091] Table 2: Experimental Verification Scenario Setting

[0092]

[0093] To systematically evaluate the performance of unknown flow detection, this invention employs three main metrics: AUROC, AUPR, and FPR95.

[0094] AUROC: The area under the receiver operating characteristic curve (AUROC) measures the model's ability to distinguish between known and unknown categories. AUROC is calculated as follows:

[0095]

[0096] AUPR: The area under the precision-recall curve (AUPR) is particularly effective for evaluating performance on imbalanced datasets, a common scenario in unknown traffic detection. Its calculation formula is:

[0097]

[0098] FPR95: False positive rate at a 95% true positive rate (FPR95), representing the proportion of false positives when the true positive rate is fixed at 95%. A lower FPR95 indicates better performance in identifying unknown flows while maintaining high detection accuracy for known flows. It is defined as:

[0099]

[0100] As shown in Table 3, in Scenario-A (encrypted traffic), when network fluctuations reach a severe level of 30%, the AUROC of AutoUA-MSP drops to 0.8074, and GradBP-Max drops to 0.7818, while the present invention still maintains a high level of 0.9276; at the same time, the FPR95 of the present invention is only 0.3212, which is significantly better than the 0.5332 and 0.5661 of the comparison methods, proving the stable unknown detection capability of encrypted traffic in a fluctuating environment.

[0101] Table 3: Comparison of this invention with the other two methods in scenario-A

[0102]

[0103] As shown in Table 4, under the 30% fluctuation environment of Scenario-B (mobile applications), this invention achieved an AUROC of 0.9201 and an AUPR of 0.9227, far exceeding AutoUA-MSP (AUROC 0.8146) and GradBP-Max (AUROC 0.8041). This demonstrates that this invention can effectively address common instability characteristics in mobile networks and accurately identify unknown mobile applications.

[0104] Table 4: Comparison of this invention with the other two methods in scenario-B

[0105]

[0106] As shown in Table 5, in Scenario-C, even with a 30% network fluctuation, this invention still achieved an AUROC score of 0.9619 and an FPR95 as low as 0.0879. In contrast, AutoUA-MSP achieved an FPR95 as high as 0.1853. The data demonstrates that this invention can accurately intercept unknown malicious attack traffic with an extremely low false positive rate, even in highly volatile network environments.

[0107] Table 5: Comparison of this invention with the other two methods in scenario-C

[0108]

[0109] In summary, the experimental data fully demonstrate that the present invention significantly outperforms existing technologies in different types of open set scenarios and under various network fluctuation intensities, successfully solving the problem of significant performance degradation of traditional models in real fluctuating networks, and providing strong technical support for the field of unknown traffic detection in fluctuating network environments.

Claims

1. A robust unknown flow detection method based on feature alignment and random perturbation injection, characterized in that, It includes the known traffic fingerprinting stage and the unknown traffic detection stage; The known traffic fingerprint construction phase includes the following steps: Step 1) Take a set of known network traffic data with category labels as input, clean and standardize it to obtain a known traffic sample sequence; Step 2) Based on the known flow sample sequence obtained in Step 1), a fluctuation simulation mechanism is introduced to generate a paired training dataset containing normal flow and fluctuating flow. Step 3) Using the paired training dataset obtained in Step 2) as input, construct a neural network model based on multi-scale feature fusion, and use classification loss and feature alignment loss for joint training to obtain the parameters of the feature extractor model with fluctuation adaptability, and construct the traffic fingerprint of known categories; The unknown traffic detection phase includes the following steps: Step 4) Using the real-time captured traffic data to be detected as input, perform the same preprocessing operation as in Step 1) to obtain a standardized sequence of the traffic to be detected; Step 5) Load the solidified feature extractor parameters from Step 3) and extract the fused feature vector of the flow to be detected; Step 6) Apply random perturbation injection to the classifier weight matrix solidified in Step 3), and calculate the output distribution and response score of the traffic to be detected under the perturbation weight; Step 7) Combine the known category traffic fingerprints constructed in Step 3) to calculate the comprehensive discrimination score, compare the comprehensive discrimination score with the preset threshold, and output the traffic category determination result.

2. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 1, characterized in that, Step 1) The specific operation method for standardization preprocessing is as follows: Step 1-1) For each captured data packet, remove invalid data packets with incomplete header information or empty payload to ensure the validity of the input data, and split the data packets according to the five-tuple to form network traffic data; Steps 1-2) The cleaned traffic is parsed into two scales of sequence representation: First, the payload of all data packets is extracted and concatenated in order to form a byte stream sequence; Second, the length information of each data packet is extracted to form a packet length sequence. Steps 1-3) Based on the preset maximum length threshold, truncate or pad the generated byte stream sequence and packet length sequence with zeros to meet the fixed input dimension requirements of the neural network.

3. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 1, characterized in that, The specific operational method for introducing the fluctuation simulation mechanism in step 2) is as follows: Step 2-1) Define the original data packet sequence and perform packet loss, out-of-order delivery, and retransmission simulation operations on it respectively; Step 2-2) The packet loss simulation refers to randomly selecting a set of data packet indices to discard according to a defined packet loss rate, and generating a sequence of packets after loss. Steps 2-3) refer to the out-of-order simulation, which means rearranging the data packet index according to the defined out-of-order window or ratio using a permutation function to generate a out-of-order sequence. Steps 2-4) The retransmission simulation refers to randomly selecting a set of data packet indexes according to the defined retransmission rate, copying them, and inserting them into the original sequence to generate the retransmitted sequence.

4. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 3, characterized in that, Step 2) The specific steps for generating the paired training dataset are as follows: Steps 2-5) For each normal traffic, randomly combine one or more fluctuation operations described in steps 2-2) to 2-4) to generate the corresponding fluctuation byte stream sequence and fluctuation packet length sequence; Steps 2-6) Construct a paired training dataset containing normal traffic byte stream sequences, normal traffic packet length sequences, fluctuating traffic byte stream sequences, fluctuating traffic packet length sequences, and category labels, which will serve as input for subsequent models.

5. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 1, characterized in that, The specific steps for constructing the neural network model in step 3) are as follows: Step 3-1) Construct a multi-scale feature extractor, including a byte stream feature extractor and a packet-length sequence feature extractor, both of which adopt a neural network architecture based on a self-attention mechanism; Step 3-2) Set up an adaptive feature fusion module, taking the feature vectors output by the byte stream feature extractor and the feature vectors output by the packet-length sequence feature extractor as inputs, and deeply integrating them by learning the weights of different modalities to generate high-dimensional fusion features; Step 3) The specific operation method for training using feature alignment mechanism and gradient truncation strategy is as follows: Step 3-3) Use a multi-scale feature extractor with shared parameters to extract normal flow characteristics and fluctuating flow characteristics respectively; Steps 3-4) Introduce a feature alignment module composed of multilayer sensing mechanisms to calibrate the fluctuating flow characteristics, making them closer to the normal feature space; Steps 3-5) Introduce a gradient cutoff mechanism, treating normal flow features as fixed anchor points, blocking the propagation of gradients to the normal flow feature extraction path during backpropagation, and forcing fluctuating features to unidirectionally align to normal features.

6. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 5, characterized in that, The specific operation method for model training and optimization in step 3) is as follows: Steps 3-6) Calculate the joint total loss function, which is a weighted sum of the classification loss and the one-way alignment loss; Steps 3-7) The unidirectional alignment loss adopts a regression loss function based on cosine similarity, which aims to maximize the consistency between the calibrated fluctuation features and normal features in the direction, and the alignment loss includes a weighted sum of byte stream sequence alignment loss and packet length sequence alignment loss; Steps 3-8) Based on the joint total loss function, the network parameters are iteratively updated using the backpropagation algorithm until the training termination condition is met, and the fixed model parameters are output.

7. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 1, characterized in that, The specific steps for constructing known category traffic fingerprints in step 3) are as follows: Steps 3-9) After the model training is completed and the parameters are fixed, input all known traffic samples in the training set into the model; Step 3-10) Calculate the feature vectors of all known traffic samples in the penultimate layer of the model, and use them as the feature vector set of known traffic. Step 3-11) Store the feature vector set obtained in step 3-10) as a known category traffic fingerprint, as a statistical benchmark for subsequent unknown traffic detection.

8. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 1, characterized in that, The specific operation method for injecting random perturbation in step 6) is as follows: Step 6-1) Obtain the offline trained classifier weight matrix, which contains multiple class projection vectors; Step 6-2) For each class projection vector in the weight matrix, add a random noise vector sampled from the standard normal distribution to generate multiple small perturbation weights; Step 6-3) Construct a set of perturbation weights containing multiple perturbation weights, which will be used to generate a perturbed feature space when performing unknown traffic detection later.

9. The robust unknown flow detection method based on feature alignment and random perturbation injection as described in claim 8, characterized in that, The specific steps for calculating the response score in step 6) are as follows: Step 6-4) Input the fused feature vector of the traffic to be detected into the multiple perturbation weight matrices generated in step 6-3, and calculate the output distribution under each perturbation; Step 6-5) Apply the Softmax function to the output of each perturbation to obtain the class probability distribution, and extract the maximum probability value; Step 6-6) Calculate the average of the maximum probability values ​​under multiple perturbations to obtain the maximum probability score of the perturbation, which is used to measure the sensitivity of the flow to perturbations of the model parameters; Step 7) The specific operation method for comprehensive judgment and result output is as follows: Step 7-1) Under perturbation weights, calculate the KL divergence between the current sample feature density estimate and the known class average density estimate, which is used to measure the distribution difference between the traffic to be detected and the known class traffic fingerprint under multiple random perturbations; Step 7-2) The maximum probability score obtained in step 6-6) and the KL divergence score obtained in step 7-1) are weighted and combined to obtain the final comprehensive discrimination score; Step 7-3) Compare the comprehensive discrimination score with the preset detection threshold. If the comprehensive discrimination score is greater than the threshold, the traffic is determined to be unknown traffic; otherwise, it is determined to be identified traffic. Finally, the discrimination results of unknown traffic and known traffic are output.

10. A robust unknown flow detection system based on feature alignment and random perturbation injection, characterized in that, It includes a data preprocessing and fluctuation simulation module, a multi-scale feature extraction module, and a joint training module for classification and feature alignment loss in the known traffic fingerprint construction stage; and a data preprocessing and feature extraction module, a random perturbation injection module, and an unknown traffic detection module in the unknown traffic detection stage. The data preprocessing and fluctuation simulation module is responsible for cleaning and serializing the input known category traffic set, and randomly applying packet loss, out-of-order and retransmission interference to generate an enhanced training dataset containing normal and fluctuating traffic pairs. The multi-scale feature extraction module is responsible for extracting byte stream and packet length sequence features using a neural network based on a self-attention mechanism, generating high-dimensional feature vectors through adaptive fusion, and optimizing the neural network model by combining classification loss and alignment loss. The known category traffic fingerprint construction module is responsible for calculating and storing the average activation vector and average probability distribution of all known traffic samples in the training set after the model parameters are fixed. The data preprocessing and feature extraction module is responsible for taking the traffic set to be detected as input, performing cleaning and serialization operations, and loading solidified model parameters to extract robust feature vectors; The random perturbation injection module is responsible for applying random perturbations to the classifier weights, calculating the density estimate and maximum probability score of the input features under the perturbation weights, and calling the known category flow fingerprint to calculate the KL divergence between the density estimate of the sample input features and the known category average density estimate. The unknown flow detection module uses the maximum probability score and KL divergence of the samples to calculate a comprehensive discrimination score through weighting and compare it with a preset threshold, and outputs a discrimination result of whether the flow is known or unknown.

Images