A lightweight network traffic intrusion detection method fusing TrafficFormer-knowledge distillation

Abstract

The application discloses a kind of fusion TrafficFormer-knowledge distillation's light network traffic intrusion detection method, to solve the model light weight demand and network traffic data imbalance problem in resource limited scene.First, the original traffic data of pcap format is segmented and filtered, after data set is divided according to preset proportion, data balancing is carried out to training set using RIFA method;Subsequently, effective data extraction and byte-level processing are carried out on all data sets to generate standardized token sequence;Then, based on TrafficFormer pre-training model fine-tuning is constructed teacher model, the model is pre-trained by MBM, SODF dual self-supervised task, has excellent traffic feature representation ability;Then, 4 layers of transformer encoder are used as light student model, and the knowledge of teacher model is transferred to student model by KL divergence distillation loss and focal supervision loss double-loss function;Finally, the performance of the obtained student model is evaluated.

Description

Technical Field

[0001] This invention belongs to the field of network security and relates to a lightweight network traffic intrusion detection method that integrates TrafficFormer and knowledge distillation. Addressing the need for lightweight intrusion detection models in resource-constrained scenarios and the prevalent data imbalance problem in network traffic datasets, this invention employs knowledge distillation technology to achieve a lightweight detection model, while combining a data generation strategy with a knowledge distillation loss function to resolve the data imbalance issue. Background Technology

[0002] Network traffic intrusion detection is a core component of a network security protection system. Its core objective is to identify malicious intrusion behaviors (such as brute-force attacks, DDoS attacks, and covert penetration) in network traffic in real time, providing critical decision support for network threat interception and ensuring the security of network infrastructure and data. With the rapid development of 5G, IoT, and edge computing, the number of network terminal devices is surging, and resource-constrained scenarios such as edge nodes and vehicle terminals are increasing, making the need for lightweight, low-latency deployment of intrusion detection models increasingly urgent.

[0003] Traditional network traffic intrusion detection methods mainly rely on rule matching or manually designed statistical features, combined with machine learning algorithms for classification. However, such methods are highly dependent on domain expert knowledge and struggle to capture dynamically changing attack characteristics when faced with new intrusion patterns such as encrypted traffic and zero-day attacks. Their lack of generalization ability leads to a significant decrease in detection accuracy, failing to meet the protection needs of complex network environments.

[0004] In recent years, deep learning technology has made groundbreaking progress in the field of intrusion detection thanks to its end-to-end feature learning advantages. Convolutional neural networks can effectively extract local features in the traffic space, recurrent neural networks are good at modeling traffic temporal dependencies, while the Transformer model based on the self-attention mechanism has a stronger ability to model global features, demonstrating excellent performance in intrusion detection tasks. However, such deep learning models (especially Transformer models) typically have hundreds of millions of parameters, resulting in large storage consumption and high computational costs. When deployed on resource-constrained devices, they face severe challenges of high latency and high memory consumption, making them difficult to adapt to real-time and resource-constrained scenarios, highlighting the urgent need for lightweight models.

[0005] Existing lightweight model techniques mainly include model structure optimization (such as depthwise separable convolution), model compression (such as pruning and quantization), and knowledge distillation. Among them, knowledge distillation, by transferring knowledge from a complex teacher model to a lightweight student model, provides an effective approach to balancing detection performance and resource efficiency. However, existing research often directly transfers knowledge distillation schemes from the image domain, failing to fully adapt to the unique temporal characteristics and complex semantic structures of network traffic data. This makes it difficult for the student model to learn general intrusion traffic representations, resulting in a struggle to balance detection accuracy and generalization ability in multi-class intrusion detection tasks.

[0006] TrafficFormer is a pre-trained model designed specifically for network traffic scenarios. Through masked packet cluster modeling (MBM) and same-origin flow direction discrimination (SODF) dual self-supervised tasks, it can accurately capture the temporal dependence, local semantics and flow association information of traffic. Its feature representation capabilities are more in line with the needs of intrusion detection. However, its complex structure of 12-layer Transformer encoder still has the problems of large number of parameters and high computational consumption, which cannot be directly adapted to resource-constrained scenarios. It needs to be optimized for lightweighting through knowledge distillation technology to meet deployment requirements.

[0007] Meanwhile, network traffic intrusion detection scenarios commonly suffer from data imbalance: normal traffic samples account for a very high proportion, while malicious intrusion traffic (especially long-tail categories such as covert attacks and novel intrusions) samples are scarce. This further exacerbates the detection difficulty of lightweight models, making existing methods unable to meet the actual deployment needs in complex network environments. Summary of the Invention

[0008] To improve the lightweight nature and accuracy of network traffic intrusion detection for resource-constrained devices while addressing the issue of data imbalance, this invention proposes a lightweight network traffic intrusion detection method that integrates TrafficFormer and knowledge distillation. First, the data is preprocessed and the dataset is balanced. Then, a teacher model is constructed by fine-tuning a TrafficFormer pre-trained model. Finally, knowledge from the teacher model is transferred to a lightweight student model based on a dual-supervised knowledge distillation mechanism, ultimately achieving efficient intrusion detection for resource-constrained devices.

[0009] Its characteristics include the following steps:

[0010] (1) Dataset preprocessing and data balancing

[0011] Obtain the raw network traffic dataset in pcap format, generate packet clusters through traffic segmentation, filter and divide it into training set, validation set and test set according to preset ratio; perform data balancing on the training set to obtain a balanced training set; finally, extract effective data and perform byte-level processing on each dataset to generate a standardized token sequence.

[0012] (2) Teacher Model Construction and Training

[0013] The teacher model is obtained by fine-tuning the TrafficFormer pre-trained model. This pre-trained model is pre-trained using large-scale unlabeled general traffic data through dual self-supervised tasks of Masked Message Cluster Modeling (MBM) and Same Origin Direction Discrimination (SODF). The training and validation sets are balanced as labeled datasets, and the pre-trained model is fine-tuned to obtain a teacher model adapted to the intrusion detection task.

[0014] (3) Student model construction and distillation training

[0015] A lightweight student model is constructed based on the knowledge distillation mechanism. The parameters of the teacher model are fixed after training, and the student model is trained using a balanced training set and validation set. The model parameters are optimized through a dual loss function of "distillation loss + supervision loss". An early stopping strategy is combined to avoid overfitting, resulting in the trained student model.

[0016] (4) Model performance testing

[0017] The trained student model is tested using a test set. The intrusion detection capability of the model is objectively evaluated through four core indicators: accuracy, precision, recall, and F1 score, to ensure that it meets the deployment requirements of edge devices. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the present invention will be further described below with reference to the accompanying drawings and specific implementation schemes:

[0019] Figure 1 Overall flowchart.

[0020] Figure 2 Teacher model structure diagram.

[0021] Figure 3 Knowledge distillation flowchart. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. The present invention mainly includes the following steps:

[0023] (1) Data preprocessing and data balancing

[0024] 1) Traffic data collection, segmentation and filtering: Obtain the raw network traffic dataset in pcap format (the standard file format for storing network data packets) through network traffic collection tools. Based on the five-tuple (source IP, destination IP, source port, destination port, protocol type), the traffic is segmented into independent flows. Then, consecutive data packets in the same transmission direction in each flow are divided into bursts of the same direction. At the same time, invalid data packets and corresponding bursts containing empty payloads and malformed packets are removed to eliminate data redundancy and abnormal interference.

[0025] 2) Dataset partitioning and balancing: The split traffic data is divided into training, validation and test sets in a ratio of 8:1:1; the training set is balanced using the Random Initialization Field Augmentation (RIFA) method, and different data augmentation factors are used for minority classes to obtain a balanced dataset.

[0026] Some fields in network protocols are randomly initialized, and their values ​​do not have inherent semantic meaning and have no practical use for traffic classification tasks. Common randomly initialized fields in protocols include the IPID field (the 16-bit identifier in the IP datagram header) in the IP protocol (Internet Protocol), the source port number in the UDP protocol (User Datagram Protocol), the source port number, sequence number, acknowledgment number, and timestamp in the TCP protocol (Transmission Control Protocol), and the client and server greeting messages in the TLS protocol (Transport Layer Security).

[0027] RIFA can generate multiple copies of traffic data while retaining the original labels by randomly modifying randomly initialized fields in data packets, without altering the original semantics of the data. Considering that the pattern of field changes is often more important than the actual value, RIFA maintains the pattern of change for a field in subsequent data packets after modifying its initial value. For example, after modifying the sequence number of the first data packet in the TCP protocol, the sequence numbers of subsequent data packets will be calculated based on the difference between that random value and the original value.

[0028] To better describe the network traffic data imbalance problem in a multi-class intrusion detection scenario, let's assume a network traffic dataset... For a finite dataset, Let i be a subset of the dataset representing the i-th class. The number of categories. When there are... At that time, D is considered to be an imbalanced dataset, where Indicate category The total number of samples included. Based on the above definition, assuming the imbalance of dataset D is denoted as IR, its calculation method is shown in equation (1):

[0029] (1)

[0030] As shown in equation (1), a larger IR indicates a more severe imbalance in the dataset. When IR > 10, the dataset needs to be rebalanced. Perform data balancing.

[0031] To accommodate the imbalance in the differentiation between different categories, it is necessary to have a minority category of the k-th category. Determine the RIFA data enhancement factor Using the rounding function The calculation is performed using the following formula:

[0032] (2)

[0033] in, The category with the largest sample size in the dataset. To enhance the post-minority class Sample size and The target percentage of the sample size ranges from 0.5 to 1.

[0034] 3) Effective Data Extraction and Standardization: All packet cluster data in the balanced training, validation, and test sets are uniformly standardized to adapt to the model input. The 64 bytes of data after the Ethernet layer of each packet are extracted (excluding irrelevant Ethernet header information), converted to hexadecimal strings, and then split using a two-byte segmentation strategy. For example, 0x4504008bd0 is split into 0x4504, 0x0400, 0x008b, and 0x8bd0. A vocabulary of up to 65535 bytes is constructed using the Byte-Pair Encoding (BPE) algorithm. Five special tokens are introduced: [CLS], [SEP], [PAD], [UNK], and [MASK]. [CLS] is placed at the beginning of the sequence, and its corresponding output vector serves as the aggregate representation for the classification task; [SEP] is used to separate different bursts; [PAD] is used to pad sequences within a batch to the same length; [UNK] is used to represent tokens not appearing in the vocabulary; and [MASK] is used for masking operations in the pre-training task. Each packet cluster is converted into a fixed-length sequence of 512 tokens (the smallest input unit of the model) to obtain standardized traffic data that can be directly input into the model.

[0035] (2) Teacher Model Construction and Training

[0036] The teacher model is fine-tuned based on the TrafficFormer pre-trained model. TrafficFormer is a pre-trained model based on the Transformer architecture specifically designed for network traffic data. Its core structure is a 12-layer Transformer encoder (768 hidden layer dimensions and 12 attention heads). This pre-trained model adopts a two-stage training paradigm, using large-scale unlabeled general traffic data to complete pre-training through dual self-supervised tasks of Masked Message Cluster Modeling (MBM) and Same Source Direction of Flow (SODF). The specific task design is as follows:

[0037] The MBM task addresses the temporal characteristics of network traffic. Using a burst as the input unit, it masks 15% of the input tokens (80% replaced with [MASK] tokens, 10% with random tokens from the vocabulary, and 10% remaining unchanged). The model is required to predict the original token at the masked position based on contextual information. This task aims to teach the model the temporal dependencies and local semantic relationships of traffic data, optimized through a cross-entropy loss function. As shown in the formula:

[0038] (3)

[0039] in, The number of masked tokens; For the first one-hot encoded The true probability of each token is in the range {0,1}. For the first The predicted probability of each token and satisfying The value range is [0,1].

[0040] To uncover the direction, sequence, and flow attribution relationships of traffic, a SODF five-class classification task is designed. Training samples are generated by constructing different types of message cluster combinations, with each category accounting for 20% of the samples.

[0041] Category 1: Two segments of a single burst split joined together in normal order, separated by [SEP];

[0042] Category 2: Two segments from a single burst are swapped and concatenated, separated by [SEP];

[0043] Category 3: Two consecutive bursts in the same stream are joined in normal order, separated by [SEP];

[0044] Category 4: Two consecutive bursts in the same stream are swapped and concatenated, separated by [SEP];

[0045] Category 5: Two bursts from different streams are randomly concatenated and separated by [SEP].

[0046] This task also uses the cross-entropy loss function, the loss function... As shown in the formula:

[0047] (4)

[0048] in, Batch size; For the sample The true probability distribution of , with values ​​ranging from {0,1}; For the sample The predicted probability distributions range from [0,1]. Both distributions have a dimension of 5, corresponding to the number of classes in the SODF task.

[0049] Total pre-training loss The weighted sum of the losses from the two tasks:

[0050] (5)

[0051] Among them, hyperparameters The training weights used to balance the two tasks enhance the model's ability to learn basic traffic semantics, and their values ​​range from [0,1].

[0052] The teacher model uses a Transformer encoder as its core architecture, and consists of an embedding layer, a Transformer encoder layer, a classification head, and a pooling layer. The parameters and structural configurations of each module are as follows:

[0053] The embedding layer consists of three parts: word embedding, position embedding, and segment embedding. Word embedding is implemented based on a vocabulary of size 65535 built by BPE. Position embedding is used to model the temporal position information of the token sequence. Segment embedding is used to distinguish different burst segments. The three are superimposed to form the initial vector representation of the input token.

[0054] The Transformer encoder layer consists of 12 stacked layers, each containing a multi-head attention mechanism and a feedforward network. The hidden layer dimension is set to 768, the number of attention heads is 12, the feedforward network dimension is 3072, the activation function is GELU (Gaussian error linear unit), and the dropout rate (random deactivation, a regularization technique used to prevent model overfitting, which reduces the dependency redundancy between neurons by randomly discarding the output of some neurons during training) is set to 0.1 to ensure the model's feature extraction capability and generalization performance.

[0055] The classification head adopts a two-layer linear network structure. The first layer maps the 768-dimensional feature vector output by the encoder to 768-dimensional intermediate features and introduces a non-linear transformation through the Tanh (hyperbolic tangent function) activation function. The second layer maps the intermediate features to an output vector that matches the number of intrusion detection categories.

[0056] The pooling method employs a first token pooling strategy, extracting the output vector corresponding to the first [CLS] token in the input token sequence as the global feature representation of the entire traffic sequence. [CLS] is a special token that is fixedly inserted at the beginning of the input token sequence and has no actual traffic semantics. During the TrafficFormer pre-training stage (especially for the SODF five-class classification task), the global information of the sequence has been aggregated through a multi-head attention mechanism. During fine-tuning, its 768-dimensional feature vector is directly extracted as the input classification head for the global representation of the traffic sequence to complete the intrusion detection category determination.

[0057] The pre-trained model was fine-tuned by using a balanced training set and an initial validation set as labeled datasets and employing an early stopping strategy (stopping if the F1 score on the validation set does not improve after 5 consecutive rounds). After training, a teacher model adapted to the intrusion detection task was obtained.

[0058] (3) Student model construction and distillation training

[0059] The student model adopts a lightweight design, with a 4-layer stacked Transformer encoder, a hidden layer dimension of 384, 6 attention heads, and a feedforward network dimension of 1536. The activation function and dropout rate are kept consistent with the teacher model. The embedding layer shares the BPE vocabulary and embedding structure of the teacher model. The classification head is simplified to a 1-layer linear network, and the pooling method is also first token pooling. The 384-dimensional feature vector corresponding to the [CLS] token is extracted as the global representation.

[0060] During distillation training, all parameters of the teacher model are fixed after training, and only the student model is trained. The early stopping strategy consistent with the teacher model is adopted. The objective function consists of two parts: distillation loss and supervision loss.

[0061] To guide the student model to better fit the predicted distribution of the teacher model, a distillation loss based on KL divergence is introduced. KL divergence, also known as relative entropy, is an indicator used to measure the similarity between two probability distributions. The loss function... As shown in the formula:

[0062] (6)

[0063] (7)

[0064] (8)

[0065] in, This represents the total number of classification categories for intrusion detection tasks. , These are the logits (the raw numerical vectors directly output by the last layer of the neural network without normalization) distributions for the teacher model and the student model, respectively. This is a temperature coefficient used to adjust the smoothness of the probability distribution, with a value range of [1, 10]. As a scaling factor, it ensures the stability of the gradient at different temperatures; For the teacher model at temperature Under the condition of the first The soft probability distribution of the class, i.e. the soft label, takes values ​​in the range [0,1]. For student models at the same temperature Next to the The soft probability distribution of the class, i.e., soft prediction, takes values ​​in the range [0,1].

[0066] To address the data class imbalance issue in traffic detection scenarios, a Focal loss is introduced instead of the traditional cross-entropy loss, incorporating supervisory signals from real labels to enhance robustness. (Supervisory Loss) As shown in the formula:

[0067] (9)

[0068] in, The total number of samples in a single training batch; for The student model for the true category of the i-th sample The predicted probability, i.e., hard prediction, has a value range of [0,1]. This is the category weight parameter, with a value range of (0,1); The focus parameter is used to reduce the weight of easily classified samples, alleviate the class imbalance problem, and improve the overall performance and robustness of the detection model. Its value range is [0,5].

[0069] Total loss function The weighted sum of distillation loss and monitoring loss is calculated as shown in the formula:

[0070] (10)

[0071] in, As a trade-off factor, used to balance the supervised learning signal. With knowledge distillation signals The contribution value ranges from [0,1]. A larger value implies a greater emphasis on knowledge distillation.

[0072] (4) Model performance testing

[0073] To comprehensively evaluate the detection performance, accuracy was used on the test set. Accuracy Recall rate and The metrics used to measure the effectiveness of the student model detection are as follows:

[0074] (11)

[0075] (12)

[0076] (13)

[0077] (14)

[0078] in, This represents true positives, which is the number of actual attack samples successfully identified by the model. This represents the number of true negatives, which is the number of samples that the model accurately identifies as normal traffic. This represents the number of false positives, which are samples of normal network traffic that are incorrectly labeled as attack behavior. This represents false negatives, which are records that are actually attack samples but are incorrectly identified as normal traffic by the model.

Claims

1. A lightweight network traffic intrusion detection method integrating TrafficFormer and knowledge distillation, characterized in that, Includes the following steps: a. Obtain the network traffic dataset and preprocess the raw traffic data: First, segment the traffic into independent flows based on the five-tuple, which includes source IP, destination IP, source port, destination port, and protocol type. Then, divide consecutive data packets in the same transmission direction within each flow into bursts of the same direction, while removing invalid data packets and corresponding bursts containing empty payloads or malformed packets to eliminate data redundancy and abnormal interference. Divide the segmented and filtered traffic data into training, validation, and test sets according to a preset ratio, and perform data balancing on the training set: Use the RIFA (Real-Initialized Field Enhancement) method to randomly modify the randomized initialization fields in the data packets and maintain the change pattern of these fields in subsequent data packets to generate a copy of the traffic data that retains the original labels. Apply different data enhancement multiples to minority classes to obtain a balanced dataset. The randomized initialization fields include the IPID field of the IP protocol and the source field of the UDP protocol. The dataset includes port number, source port number, sequence number, acknowledgment number, and timestamp for TCP, as well as client and server greeting messages for TLS. The dataset extracts 64 bytes of data after the Ethernet layer from each packet in the training, validation, and test sets. This data is converted to hexadecimal strings and split using a double-byte segmentation strategy. A vocabulary of up to 65535 bytes is constructed using the Byte-Pair Encoding (BPE) algorithm. Five special tokens—[CLS], [MASK], [SEP], [PAD], and [UNK]—are introduced. Each packet cluster is converted into a fixed-length 512-byte token sequence to obtain standardized traffic data. [CLS] is placed at the beginning of the sequence, and its corresponding output vector serves as the aggregation representation for the classification task. [MASK] is used for masking operations in the pre-training task. [SEP] is used to separate different bursts. [PAD] is used to pad sequences within a batch to the same length. [UNK] represents tokens not appearing in the vocabulary. b. Constructing and training the teacher model: The teacher model is obtained by fine-tuning the TrafficFormer pre-trained model. The pre-trained model has a structure of 12 Transformer encoder layers, 768 hidden layers, and 12 attention heads. This pre-trained model is pre-trained using large-scale unlabeled general traffic data through masked message cluster modeling (MBM) and source flow direction discrimination (SODF) dual self-supervised tasks. In the MBM task, 15% of the input tokens are masked. The masking process replaces 80% with [MASK] tokens, 10% with random tokens, and 10% with the original tokens. The model predicts the original tokens based on the context. The SODF task is a five-class classification task, with each class accounting for 20% of the samples. By constructing message cluster combinations with different flow directions, sequences, and flow affiliations, the model learns the direction, timing, and flow association information of the traffic. Category 1: Two segments of a single burst split joined together in normal order, separated by [SEP]; Category 2: Two segments from a single burst are swapped and concatenated, separated by [SEP]; Category 3: Two consecutive bursts in the same stream are joined in normal order, separated by [SEP]; Category 4: Two consecutive bursts in the same stream are swapped and concatenated, separated by [SEP]; Category 5: Two bursts from different streams are randomly concatenated, separated by [SEP]; The training set and validation set are used as the annotation task dataset. An early stopping strategy is adopted. If the F1 score on the validation set does not improve for 5 consecutive rounds, training is stopped. The TrafficFormer pre-trained model is fine-tuned to obtain a teacher model adapted to the network traffic intrusion detection task. c. Constructing and training the student model using knowledge distillation: A 4-layer Transformer encoder is used as a lightweight student model. The hidden layer dimension of the student model is 384, the number of attention heads is 6, and the feedforward network dimension is 1536. The activation function and dropout rate are consistent with the teacher model. The embedding layer shares the BPE vocabulary and embedding structure of the teacher model. The classification head is simplified to a 1-layer linear network, and the pooling method is also first token pooling. The 384-dimensional feature vector corresponding to the [CLS] token is extracted as the global representation. Based on the knowledge distillation mechanism, the parameters of the teacher model are fixed, and the student model is trained using the trained teacher model, the training set, and the validation set. The early stopping strategy consistent with the teacher model is adopted during training. A weighted total loss function is constructed using KL divergence (relative entropy loss) as the distillation loss and Focal focus loss as the supervision loss to obtain the trained student model. The formulas for the distillation loss, supervision loss, and total loss function are as follows: in, This represents the total number of classification categories for intrusion detection tasks. This is a temperature coefficient used to adjust the smoothness of the probability distribution. As a scaling factor, it ensures the stability of the gradient at different temperatures; For the teacher model at temperature Under the condition of the first The soft probability distribution of a class, i.e., soft labels; For student models at the same temperature Next to the The soft probability distribution of the class, i.e., soft prediction; for The student model for the true category of the i-th sample The predicted probability, i.e., hard prediction; The total number of samples in a single training batch; For category weight parameters, To focus parameters, the weight of easily classified samples is reduced to alleviate the class imbalance problem; As a trade-off factor, used to balance monitoring losses With distillation loss Contributions; d. The trained student model is tested using the test set. The test metrics include accuracy, precision, recall, and F1 score.

Images