Lightweight intrusion detection method based on multi-teacher knowledge distillation

Abstract

The invention provides a lightweight intrusion detection method based on multi-teacher knowledge distillation. The lightweight intrusion detection method is suitable for resource-limited scenes such as an industrial control system. The method comprises the following steps: firstly, cleaning original industrial control network flow data, carrying out classification feature numeralization and normalization processing, and screening important features; and then constructing a teacher group composed of a space-time convolutional network, a gating circulation unit and a bidirectional LSTM network model, and designing a 1DCNN-LSTM student model. In the distillation training stage, knowledge migration is realized by dynamically calculating the prediction confidence of each teacher model and combining a temperature scaling mechanism to generate a soft label. After training is completed, the student model is deployed to the edge device, and intrusion detection is carried out on real-time network traffic. The method is small in calculation overhead, good in detection performance and suitable for intrusion detection tasks in resource-limited scenes.

Description

Technical Field

[0001] This invention relates to network intrusion detection, specifically a lightweight intrusion detection method based on multi-teacher knowledge distillation. Background Technology

[0002] With the digitalization, networking, and intelligentization of industrial control systems (ICS), these systems face increasingly severe cybersecurity threats. ICS are widely used in critical national infrastructure sectors such as transportation, petroleum, chemical, and power, and their security is directly related to national security and economic stability. However, the increasingly open architecture and interconnectedness of current ICS systems have led to an exponential expansion of the attack surface. Traditional cybersecurity measures, such as firewalls and intrusion detection systems (IDS), often struggle to effectively detect and defend against new types of cyberattacks in complex ICS network environments. Furthermore, ICS typically have complex network topologies and extensive device interconnections, resulting in network traffic data that is high-dimensional, noisy, and dynamically changing. These complex characteristics have made ICS intrusion algorithm research a common research topic in both industry and academia.

[0003] Intrusion detection methods in the industrial control field can be mainly divided into detection algorithms based on traditional machine learning and detection algorithms based on deep learning. Chen combined SVM with ANN and verified the effectiveness of the detection algorithm on the DARPA 1998 dataset. Oanh combined Naive Bayes and genetic algorithms, using the genetic algorithm for data dimensionality reduction and then using Naive Bayes for classification on a simplified dataset. The results showed that on the NSL-KDD dataset, although the accuracy of this model was higher than other classifiers such as SVM, random forest, and decision tree, its detection execution time was longer due to the added computational overhead. Yin et al. proposed an intrusion detection method based on RNN. The advantage of this method lies in its strong modeling ability for high-dimensional features and its excellent performance in both binary and multi-class classification tasks. It effectively solves the limitations of traditional machine learning methods in large-scale intrusion data classification problems, and its classification accuracy and detection rate on the NSL-KDD dataset are both superior to traditional machine learning methods. Laghrissi uses an intrusion detection algorithm based on LSTM. Compared to RNN, LSTM can effectively capture long-term dependencies in data and shows superior performance in sequence data processing tasks such as intrusion detection.

[0004] Traditional machine learning-based intrusion detection methods suffer from excessive reliance on feature engineering, insufficient ability to handle high-dimensional data, and inadequate handling of imbalanced data, resulting in detection performance and accuracy that cannot cope with the growing threats to industrial cybersecurity. While deep learning-based intrusion detection methods have developed rapidly in recent years, most existing research focuses only on metrics such as detection performance, classification accuracy, and precision, neglecting the limited computing power of industrial control equipment. The computational overhead and running speed of algorithms and models have not been thoroughly investigated and optimized. Therefore, designing an intrusion detection algorithm with low computational complexity suitable for deployment on industrial control equipment remains a challenge. Summary of the Invention

[0005] The purpose of this invention is to propose a lightweight intrusion detection method based on multi-teacher knowledge distillation, which aims to reduce computational complexity while maintaining detection accuracy through knowledge transfer between teacher and student models, so as to adapt to industrial control equipment.

[0006] The objective of this invention is achieved as follows:

[0007] A lightweight intrusion detection method based on multi-teacher knowledge distillation is proposed, suitable for resource-constrained scenarios such as industrial control systems. The method first integrates and cleans the raw industrial control network traffic data, quantifies the classification features through encoding transformation, and unifies the feature dimensions using normalization. A feature selection algorithm based on statistical metrics is then used to select a subset of highly important features. Subsequently, a teacher group consisting of three heterogeneous models—a spatiotemporal convolutional network, a gated recurrent unit, and a bidirectional long short-term memory network—is constructed, along with a lightweight 1DCNN-LSTM student model. During the distillation training phase, the prediction confidence of each teacher model is dynamically calculated and weighted, and soft labels are generated using a temperature scaling mechanism. Knowledge transfer is achieved through a weighted loss function. After training, the student model is deployed to edge devices, enabling feature selection and normalization of real-time network traffic. After model inference, the classification result is output and intrusion alarms are triggered. This method significantly reduces model complexity and computational overhead while maintaining high detection performance.

[0008] The specific steps are as follows:

[0009] Step 1: Data preprocessing. The original industrial control network traffic dataset is integrated, noise and outliers are removed, unstructured data is converted into numerical representation, feature units are unified through normalization, and feature subsets are selected using a feature selection algorithm based on statistical metrics.

[0010] Step 2: Construct a multi-teacher knowledge distillation framework, design three heterogeneous teacher models (TCN spatiotemporal convolutional network, GRU gated recurrent unit, and BiLSTM bidirectional long short-term memory network) and a lightweight student model (1DCNN-LSTM);

[0011] Step 3: Dynamic weight loss calculation. By calculating the cross-entropy confidence between the output of each teacher model and the true label, the distillation loss weights of the teacher models are dynamically allocated.

[0012] Step 4: Distillation training, combined with temperature scaling mechanism to generate teacher soft labels and student soft predictions, and realize knowledge transfer through dynamic weighted loss function;

[0013] Step 5: Deploy the trained student model to a computing device for real-time intrusion detection;

[0014] Step 6: Collect real-time network traffic data and perform feature selection and normalization operations:

[0015] Step 7: Input the processed real-time data into the student model for inference, and generate an intrusion alarm based on the output probability distribution.

[0016] 1. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that the numericalization method in step 1 uses the One-Hot encoding method to convert categorical features into numerical data. Using Min... - Max normalization maps data to the interval [0, 1].

[0017]

[0018] 2. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that the feature selection method in step 1 calculates the standard deviation of each feature. Where μ is the feature mean, N is the number of samples, and the ranking is generated by sorting the samples from high to low according to the σ value, R1; the absolute difference between the feature mean and the median is calculated as D = |Mean - Median|, and the ranking is generated by sorting the samples from high to low according to the D value, R2; the comprehensive ranking value R is calculated. c =R1+R2, press R c Select the top 50% of features in ascending order.

[0019] 3. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that the dynamic weight loss calculation in step 3 specifically involves calculating the teacher prediction confidence, whereby the confidence of the teacher model is measured by calculating the cross-entropy loss between its predicted distribution and the true label. For the k-th teacher model T... k , where z c This is the predicted Softmax output. The teacher's prediction confidence is calculated using the following formula:

[0020]

[0021] The weighting factor is calculated using the following formula:

[0022]

[0023] 4. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that the hyperparameter distillation temperature T in the distillation training process in step 4 is 3, and the learning rate is 5×10⁻⁶. -5 The preprocessed data is input into the teacher model to obtain the output soft labels.

[0024]

[0025] The soft prediction q of the student model output from the same input data to the student model (c) :

[0026]

[0027] Total distillation loss can be expressed as:

[0028]

[0029] By minimizing the total loss L total This is to bring the training of the student model to a convergent state.

[0030] 5. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that, in step 7, when the data is input into the student model for inference, it first undergoes spatial feature extraction through a two-layer 1DCNN:

[0031]

[0032] Next, the sequence dependencies are resolved using LSTM units, and finally, the classification result is output after passing through a Softmax layer.

[0033] P(y|X)=Softmax(W out ·h T +b out )

[0034] The positive effects of this invention are:

[0035] This paper proposes an algorithm based on a multi-teacher knowledge distillation framework, designing three heterogeneous teacher models and a lightweight student model architecture. Through knowledge transfer during the distillation process, the number of model parameters is reduced to adapt to the computing power limitations of industrial control equipment while preserving the detection accuracy of the teacher models as much as possible. In the knowledge distillation process, a dynamic weighted loss function is used instead of the average loss function. The distillation loss of each teacher is weighted by calculating the cross-entropy between the distribution of the teacher model output and the true labels, avoiding the misleading of the student model's learning of the true label distribution by low-quality teacher prediction bias, thereby improving the overall performance and generalization ability of the algorithm. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the feature selection method.

[0037] Figure 2 This is a schematic diagram of the data preprocessing and knowledge transfer process.

[0038] Figure 3 This is a diagram illustrating the multi-teacher knowledge distillation training process.

[0039] Figure 4 Schematic diagram of student model reasoning process Detailed Implementation

[0040] The implementation process of this algorithm begins with the data preprocessing stage, such as... Figure 1 As shown: Data cleaning and normalization are performed on the original industrial control network traffic dataset. An automated feature selection method based on statistical metrics (using indicators such as the difference between feature standard deviation, mean, and median) is used to select key feature subsets (e.g., F1, F2, F3). Based on this, a multi-teacher knowledge distillation framework is initiated. Figure 2 The training process involves three heterogeneous teacher models (TCN spatiotemporal convolutional network, BiLSTM bidirectional long short-term memory network, and GRU gated recurrent units) jointly transferring knowledge from the student model (a lightweight structure combining ID-CNN and LSTM). Figure 3 This is achieved through a dynamic coordination mechanism: preprocessed network traffic data is input into both the teacher model group and the student model for forward propagation. The soft-label knowledge generated by the teacher model is weighted and fused with the real labels to construct a loss function, guiding the student model to optimize its parameters. The final deployment phase employs... Figure 4 The student model architecture shown is as follows: the input data is processed by convolutional layers to extract spatial features, then by pooling layers to reduce dimensionality before being fed into the LSTM module for temporal modeling, and finally by the Softmax output layer to generate the intrusion detection classification result.

Claims

1. A lightweight intrusion detection method based on multi-teacher knowledge distillation, comprising the following steps: Step 1: Data preprocessing. The original industrial control network traffic dataset is integrated, noise and outliers are removed, unstructured data is converted into numerical representation, feature units are unified through normalization, and feature subsets are selected using a feature selection algorithm based on statistical metrics. Step 2: Construct a multi-teacher knowledge distillation framework, design three heterogeneous teacher models (TCN spatiotemporal convolutional network, GRU gated recurrent unit, and BiLSTM bidirectional long short-term memory network) and a lightweight student model (1DCNN-LSTM); Step 3: Dynamic weight loss calculation. By calculating the cross-entropy confidence between the output of each teacher model and the true label, the distillation loss weights of the teacher models are dynamically allocated. Step 4: Distillation training, combined with temperature scaling mechanism to generate teacher soft labels and student soft predictions, and realize knowledge transfer through dynamic weighted loss function; Step 5: Deploy the trained student model to a computing device for real-time intrusion detection; Step 6: Collect real-time network traffic data and perform feature selection and normalization operations; Step 7: Input the processed real-time data into the student model for inference, and generate an intrusion alarm based on the output probability distribution.

2. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that, The numerical method in step 1 uses the One-Hot encoding method to convert categorical features into numerical data, and uses Min... - Max normalization maps data to the interval [0, 1].

3. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that, The feature selection method in step 1 calculates the standard deviation of each feature. Where μ is the feature mean, N is the number of samples, and the ranking is generated by sorting the samples from high to low according to the σ value, R1; the absolute difference between the feature mean and the median is calculated as D = |Mean - Median|, and the ranking is generated by sorting the samples from high to low according to the D value, R2; the comprehensive ranking value R is calculated. c =R1+R2, press R c Select the top 50% of features in ascending order.

4. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that, The dynamic weight loss calculation in step 3 specifically involves calculating the teacher prediction confidence. The confidence of the teacher model is measured by calculating the cross-entropy loss between its predicted distribution and the true label. For the k-th teacher model T... k , where z c The predicted Softmax output; the teacher's prediction confidence is calculated using the following formula: The weighting factor is calculated using the following formula.

5. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that, In step 4, the hyperparameter distillation temperature T during the distillation training process is 3, and the learning rate is 5 × 10⁻⁶. -5 The preprocessed data is input into the teacher model to obtain the output soft labels. The soft prediction q of the student model output from the same input data. (c) : Total distillation loss can be expressed as: By minimizing the total loss L total This is to bring the training of the student model to a convergent state.

6. The lightweight intrusion detection method based on multi-teacher knowledge distillation as described in claim 1, characterized in that, In step 7, when the data is input into the student model for inference, it first undergoes spatial feature extraction through a two-layer 1DCNN: Next, the sequence dependencies are resolved using LSTM units, and finally the classification results are output through a Softmax layer. P(y|X)=Softmax(W out ·h T +b out )

Images