A lightweight intrusion detection method, system and storage medium based on frequency domain gating and dynamic head attention distillation

Abstract

The application provides a kind of light-weight intrusion detection method, system and storage medium based on frequency domain gating and dynamic head attention distillation, method includes: step 1: construct teacher model, the model includes frequency domain wavelet analysis module and dynamic head interactive attention module, frequency domain wavelet analysis module is decomposed and information is enhanced in frequency domain to network traffic features by two-stage Haar wavelet transform and adaptive gating mechanism, dynamic head interactive attention module regards feature dimension as token sequence, by introducing position perception embedding and multi-head self-attention mechanism, the global dependence relationship between features is modeled;Step 2: design dynamic entropy adaptive topology distillation strategy, so that the light-weight student model inherits the discriminative features and sample relationship structure of the teacher model while compressing the feature space;Step 3: transfer the knowledge of the teacher model to the light-weight student network.The beneficial effects of the present application are: more efficient and more sufficient knowledge transfer is realized.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a lightweight intrusion detection method, system and storage medium based on frequency domain gating and dynamic head attention distillation. Background Technology

[0002] With the rapid development of complex environments such as the Internet of Things (IoT) and vehicular networks, network intrusion detection systems have become a core line of defense for cybersecurity. Traditional signature-based detection methods struggle to cope with new types of attacks, while early machine learning methods have limitations when handling high-dimensional, dynamic data. Modern intrusion detection research is evolving in two main directions: first, by integrating multiple artificial intelligence technologies and optimization strategies to maximize the detection performance and accuracy of models; and second, by developing lightweight, deployable solutions that protect user privacy for resource-constrained or privacy-sensitive real-world scenarios such as IoT and vehicular networks.

[0003] In the area of ​​model performance optimization and hybrid architecture, researchers are dedicated to building high-performance detection models by integrating and optimizing multiple advanced algorithms. Shim et al. proposed a hybrid framework whose core approach is to automatically adjust the key hyperparameters of XGBoost and optimized sequence neural networks using Bayesian optimization to perform anomaly detection. The advantage of this method is that it automates hyperparameter tuning, significantly improving model optimization efficiency and final performance; the disadvantage is that its high performance may be sensitive to specific data distributions, and the complex computation of the hybrid model may limit its deployment on resource-constrained terminal devices. Wang et al. designed the FeCo framework, which integrates federated learning and contrastive learning. Participating devices only upload model parameters, and the contrastive learning mechanism brings normal samples closer together and separates anomalous samples from normal clusters in the feature representation space. The advantage of this method is that it can significantly improve the model's detection accuracy by learning more accurate benign category representations while protecting data privacy; the disadvantage is that the framework's effectiveness depends on the scale and data quality of collaborative training, and it is sensitive to non-independent and identically distributed data patterns among devices, which may affect the stability of the global model. Ghadermazi et al. proposed GTAE-IDS, which dynamically constructs network traffic into a graph structure based on packet sequences and uses an autoencoder with an integrated graph transformer to learn the structure and contextual patterns of normal traffic in an unsupervised manner. The advantage of this method is that it can achieve early and accurate anomaly detection without any labeled data, thus enhancing the model's generalization ability to novel attacks. The disadvantage is the high computational complexity of real-time packet-based graph construction and the transformer model, which may face real-time challenges in environments requiring ultra-high throughput.

[0004] In the area of ​​lightweight and privacy protection for practical deployment, research has shifted its focus to achieving efficient detection in edge environments such as the Internet of Things (IoT) and the Internet of Vehicles (IoV) where computing and storage resources are limited, and fundamentally preventing user privacy leaks caused by centralized data processing. For lightweight model deployment, Li et al. proposed the KDBC method, the core of which uses knowledge distillation to transfer the knowledge of a large pre-trained model (teacher) with a deep understanding of network traffic semantics to a streamlined, lightweight model (student). The advantage of this method is that it can significantly compress the model size and computational requirements, enabling it to run on resource-intensive edge devices such as in-vehicle gateways while maintaining high detection and discrimination capabilities. Its limitation is that the overall performance relies on a pre-trained, complex teacher model, and its design is primarily geared towards specific communication protocols within in-vehicle networks; migration to other network environments may require additional adaptation work. At the privacy protection level, Pei et al. designed the NIGNN model, which models the relationship between network hosts and traffic as a graph structure. During information aggregation and updating in the graph neural network, a noise addition mechanism that meets differential privacy requirements is integrated, and the impact of noise on model performance is optimized by introducing enhancement terms related to node connectivity. The pioneering aspect of this method is that it provides rigorously provable privacy protection for deep learning-based intrusion detection, effectively resisting inference attacks targeting training data. Its inherent challenge lies in the trade-off between privacy protection strength and model detection utility; the noise injected to achieve strict privacy standards inevitably leads to a loss of model accuracy, requiring a search for the optimal balance. These works, from the perspectives of model compression and algorithm protection, have jointly promoted the development of intrusion detection technology towards practicality and security.

[0005] The existing technology has two main drawbacks, the reasons for which are described below: 1. High-performance models have high computational overhead, making them difficult to deploy in real-time in resource-constrained environments. This is because, in pursuit of ultimate detection accuracy, existing high-performance solutions (such as GTAE-IDS) generally employ model architectures with extremely high computational complexity. For example, they model network traffic as a graph structure and process it using graph transformer models, or integrate multiple complex deep learning models. While these architectures can capture complex traffic patterns, their massive number of parameters and floating-point operation requirements make it difficult to meet the stringent requirements of low latency and low power consumption for edge devices such as IoT and vehicular networks.

[0006] 2. Existing lightweight distillation methods suffer from insufficient knowledge transfer depth, limiting the performance ceiling of student models. This is because existing knowledge distillation schemes, such as KDBC, employ relatively simplistic distillation loss functions. These methods typically use a fixed "temperature" parameter to soften the teacher model's output predictions and only use KL divergence loss to align the output distributions (soft labels) of the teacher and student models. This approach has two inherent limitations: First, the fixed temperature parameter cannot adaptively adjust according to the prediction difficulty (uncertainty) of the samples, leading to overly smooth knowledge transfer for simple samples and insufficient transfer for difficult samples, resulting in lower knowledge transfer efficiency. Second, it only aligns the final output distribution, completely ignoring the rich structural knowledge of the feature space learned by the intermediate layers of the teacher model (such as the relative relationships or manifold structures of different samples in the feature space). This causes the student model to merely mimic the teacher's "final answer" without inheriting the deep discriminative patterns inherent in its "reasoning process." Therefore, after significant model compression, its representational ability and generalization performance suffer a substantial loss.

[0007] The information disclosed in this background section is intended only to enhance understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0008] To address the common problems of existing deep learning-based intrusion detection models, such as high computational overhead and storage requirements leading to deployment difficulties in resource-constrained edge environments, insufficient feature extraction capabilities for complex, covert, or novel attack patterns, and poor model generalization and robustness, this invention provides a lightweight intrusion detection method, system, and storage medium based on frequency domain gating and dynamic head attention distillation. Specifically, the technical problem this invention aims to solve is how to significantly reduce model complexity and computational resource consumption (lightweighting) while enhancing the model's ability to extract key discriminative features from network traffic, its robustness against noise and adversarial interference, and effectively improving the detection accuracy and generalization ability against diverse and evolving network intrusion behaviors, thereby achieving efficient and accurate intrusion detection in resource-constrained environments.

[0009] This invention provides a lightweight intrusion detection method based on frequency domain gating and dynamic head attention distillation, comprising: Step 1: Construct a teacher model, which includes a frequency domain wavelet analysis module and a dynamic head interaction attention module. The frequency domain wavelet analysis module performs multi-scale decomposition and information enhancement of network traffic features in the frequency domain through a two-level Haar wavelet transform and an adaptive gating mechanism. The dynamic head interaction attention module treats the feature dimension as a token sequence and models the global dependency relationship between features by introducing position-aware embedding and multi-head self-attention mechanism. Step 2: Design a dynamic entropy adaptive topological distillation strategy. Soften teacher knowledge through entropy-aware temperature regulation and introduce topological manifold consistency loss. This allows the lightweight student model to inherit the discriminative features and sample relationship structure of the teacher model while compressing the feature space. Step 3: Transfer the knowledge of the teacher model to the lightweight student network through end-to-end training.

[0010] As a further improvement of the present invention, in step 1, the frequency domain wavelet analysis module is configured to perform the following steps: Step S201, Two-pole wavelet decomposition: For the input features Zero-padding is performed at the end of the feature dimension d to obtain , This represents the feature matrix after zero-padding of the 0th layer features; subsequently, a two-stage discrete Haar wavelet transform is performed: the first-stage decomposition splits the feature sequence into low-frequency components along the feature dimension according to parity indices. With high frequency components The calculation process involves linear combination and scaling; let... The kth feature element is The low-frequency coefficients and high-frequency coefficients generated by the first-order decomposition are as follows: in, and Each constitutes a first-order low-frequency component With first-order high-frequency components , This represents the i-th low-frequency coefficient of the first-level wavelet decomposition. This refers to the i-th high-frequency coefficient of the first-level wavelet decomposition. Indicates the feature after filling The feature element value at the 2i-th index position; for the first-order low-frequency component Performing the same Haar transform yields the low-frequency components of the second stage. With high frequency components All of their dimensions are This two-stage decomposition process decouples the original features in the frequency domain into three sub-bands with different scales and frequencies: the coarsest scale low frequency. Secondary coarse-scale high frequency and the finest high frequency ; Step S202, Gated Feature Enhancement: For the three frequency sub-bands obtained from the second-level decomposition, apply independent adaptive gating functions respectively, as follows: For the coarsest-scale low-frequency subband Sigmoid gate is used Weighted modulation is performed for the finest high-frequency scale. and secondary coarse-scale high frequency Gating using residual form Enhance; Step S203, Inverse wavelet reconstruction: The enhanced second-level low-frequency subband... With high-frequency subband Reconstruction is performed using inverse Haar wavelet transform to obtain the enhanced first-order low-frequency component. The inverse transform is achieved by interleaving the scaled sums and differences; this enhances the first-order low-frequency components. With the enhanced first-order high-frequency component The feature representation is then reconstructed again using the inverse Haar transform to obtain the final frequency-domain enhanced feature representation. ; Remove the padding added in step S201, truncate the first d dimensions, and restore its dimensions to normal. ; Through a hybrid network containing linear layers, layer normalization, and the GELU activation function. Perform a nonlinear transformation and compare it with the input features H of the frequency domain wavelet analysis module. (0) Perform residual connections to generate the final output of the frequency domain wavelet analysis module. .

[0011] The present invention also discloses a lightweight intrusion detection system based on frequency domain gating and dynamic head attention distillation, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the lightweight intrusion detection method of the present invention when invoked by the processor.

[0012] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the lightweight intrusion detection method described in the present invention when invoked by a processor.

[0013] The beneficial effects of this invention are: 1. This invention achieves more efficient and comprehensive knowledge transfer, thus maintaining high performance while keeping the model lightweight. This is mainly attributed to the dynamic entropy adaptive topological distillation strategy employed in this invention. Unlike the simple approach of traditional knowledge distillation using fixed temperature and aligned soft labels, the dynamic temperature mechanism of this invention adaptively adjusts the smoothness of knowledge based on the uncertainty of sample prediction, achieving more refined knowledge transfer; while the topological manifold consistency loss ensures that the student model learns the crucial inter-sample relationship structure in the feature space of the teacher model. These two techniques enable the student model to inherit deeper discriminative knowledge, resulting in a performance degradation far less than that of lightweight models using traditional distillation methods after significant parameter compression.

[0014] 2. Enhanced feature extraction capabilities and model robustness against complex and covert attacks. This is primarily due to the synergistic effect of the frequency domain wavelet analysis module and the dynamic head interactive attention module. Frequency domain wavelet analysis, through Haar transform and adaptive gating, explicitly separates and enhances frequency components at different scales in traffic features (such as stable low-frequency baselines and abrupt high-frequency anomalies), enabling the model to capture attack features more precisely. Dynamic head attention establishes a global dependency model along the feature dimension, enhancing feature discriminativity. These techniques, distinct from existing methods that operate directly in the temporal / spatial domain or use standard attention, collectively improve the model's robustness to noise and its ability to generalize detection of diverse attack patterns.

[0015] 3. Experimental verification shows that this invention achieves a superior balance between model efficiency and detection accuracy. Comparative tests on public network intrusion detection datasets demonstrate that the lightweight student model obtained using this method, with parameter and computational costs reduced to approximately 30% of the backbone teacher model, exhibits a significantly smaller decrease in detection accuracy (e.g., F1 score) compared to the lightweight benchmark model employing traditional fixed-temperature knowledge distillation. This directly proves that the dynamic entropy adaptive mechanism and topological consistency alignment can more effectively "condense" the knowledge of a large model into a small model. Simultaneously, the teacher model itself, integrating frequency domain gating and dynamic head attention, also outperforms conventional models of the same size in detection accuracy, providing a higher-quality knowledge source for subsequent distillation, thus forming a complete technological closed-loop advantage from high-performance teacher to high-efficiency student. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 This is a flowchart of the frequency domain wavelet analysis module of the present invention; Figure 3 This is a flowchart of the dynamic head interaction attention module of the present invention; Figure 4 This is a flowchart of the dynamic entropy adaptive topology distillation module of the present invention; Figure 5 This is a flowchart of the end-to-end model compression training process module of the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0018] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0019] For example Figure 1 As shown, this invention discloses a lightweight intrusion detection method based on frequency domain gating and dynamic head attention distillation, comprising: Step 1: Construct a teacher model, which includes a frequency domain wavelet analysis module and a dynamic head interaction attention module. The frequency domain wavelet analysis module performs multi-scale decomposition and information enhancement of network traffic features in the frequency domain through a two-level Haar wavelet transform and an adaptive gating mechanism. The dynamic head interaction attention module treats the feature dimension as a token sequence and models the global dependency relationship between features by introducing position-aware embedding and multi-head self-attention mechanism. Step 2: Design a dynamic entropy adaptive topological distillation strategy. Soften teacher knowledge through entropy-aware temperature regulation and introduce topological manifold consistency loss. This allows the lightweight student model to inherit the discriminative features and sample relationship structure of the teacher model while compressing the feature space. Step 3: Transfer the knowledge of the teacher model to the lightweight student network through end-to-end training.

[0020] The specific implementation process is as follows: Figure 1 As shown, the following are detailed steps: Frequency domain wavelet analysis module This section is responsible for multi-scale decomposition and gated enhancement of features in the frequency domain to capture and enhance key information within different frequency ranges. It includes steps S201 to S203 (second-order wavelet decomposition, gated feature enhancement, and inverse wavelet reconstruction), with the specific mechanism and flow as follows: Figure 2 As shown.

[0021] First, regarding the network traffic feature matrix (Includes traffic characteristics such as IP address, port, and protocol type, where m is the batch size,) (where d is the feature dimension size). In the teacher model, this is mapped to a feature representation with a uniform hidden dimension d=256 through an input projection layer. This projection layer consists of a linear transformation, layer normalization, and the GELU activation function connected sequentially, and its mathematical expression is as follows: in, Let be the projection weight matrix. This is the bias vector. The projected features... It will be used as input to the frequency domain wavelet analysis module.

[0022] Step S201, Two-pole wavelet decomposition: For the input features Zero padding is applied to the end of the feature dimension d, with a padding length of pad= To ensure its dimensions If it is a multiple of 4, then... , This represents the feature matrix after zero-padding of the 0th layer features. A two-stage discrete Haar wavelet transform is then performed. The first-stage decomposition splits the feature sequence into approximate (low-frequency) components along the feature dimension according to their parity indices. With detail (high frequency) components The calculation process involves linear combination and scaling. Let... The kth feature element is The low-frequency and high-frequency coefficients generated by the first-order decomposition are as follows: Among them, and Each constitutes a first-order low-frequency component With first-order high-frequency components , This represents the i-th low-frequency coefficient of the first-level wavelet decomposition. This refers to the i-th high-frequency coefficient of the first-level wavelet decomposition. Indicates the feature after filling The feature element value at the 2i-th index position (even-numbered position). Furthermore, for the first-order low-frequency components... Performing the same Haar transform yields the low-frequency components of the second stage. With high frequency components All of their dimensions are This two-stage decomposition process decouples the original features in the frequency domain into three sub-bands with different scales and frequencies: (Low frequency at the coarsest scale) (High frequency at the coarse scale) and (The finest high frequency) laid the foundation for subsequent frequency domain specificity enhancement.

[0023] Step S202, Gated Feature Enhancement: For the three frequency sub-bands obtained from the second-level decomposition, independent adaptive gating functions are applied to each to achieve adaptive enhancement of key frequency information and suppression of redundant or noise information. Specifically, for the coarsest-scale low-frequency sub-band... It carries the main structure and stable mode of the traffic, and adopts a Sigmoid gate. Weighted modulation is performed to enhance the discriminative fundamental mode. For the two high-frequency sub-bands... and It typically contains detailed information such as traffic mutations and abnormal interactions, and uses residual gating. Enhancement is performed to highlight significant anomalous fluctuations while preserving all high-frequency details. The enhancement process is described as follows: in, This represents element-wise multiplication (Hadamard product). Gate_low, Gate_high1, and Gate_high2 are all composed of LayerNorm, linear layers, and GELU activation functions, and can learn the importance weights of each frequency component based on the context of the input features.

[0024] Step S203, Inverse wavelet reconstruction: The enhanced second-level low-frequency subband... With high-frequency subband Reconstruction is performed using inverse Haar wavelet transform to obtain the enhanced first-order low-frequency component. The inverse transform is achieved by interleaving the scaled sums and differences to restore the original resolution. Specifically, let the low- and high-frequency sequences to be reconstructed be L and H, and their elements are... and Then the reconstructed sequence elements With The calculation is as follows: Then With the enhanced first-order high-frequency component The feature representation is then reconstructed again using the inverse Haar transform to obtain the final frequency-domain enhanced feature representation. In addition to the padding added in step S201, the first d dimensions are truncated to restore its dimensions to [previous dimensions]. Finally, the output is processed through a hybrid network containing linear layers, layer normalization, and the GELU activation function. A nonlinear transformation is performed, and the residual is connected to the input H(0) of the module to form the final output of the frequency domain wavelet analysis module: This effectively fuses the features that have been filtered and enhanced in the multi-scale frequency domain back into the main feature stream.

[0025] Dynamic head interaction attention module This section is responsible for modeling dynamic positional relevance and interaction along the feature dimension, enhancing the model's discriminative ability. It includes steps S204 to S206 (position-aware embedding, multi-head interaction attention, feature projection and fusion), with the specific mechanism and process as follows: Figure 3 As shown.

[0026] Step S204, Location-aware embedding: Incorporating input features Each feature dimension d is transformed into a high-dimensional token representation. First, an independently learnable matrix P, initialized with a truncated normal distribution, is used. As a location embedding, where =d / H, where H is the number of attention heads. For the feature vector h of each sample in the batch... By expanding its dimensions and performing an affine combination with the positional embedding, the corresponding token sequence T is generated. The j-th token is generated as follows: in, Let be the scalar value of the j-th feature dimension. This is the position encoding vector corresponding to the j-th feature dimension. This operation ensures that even if the feature value is zero, its token representation is not a zero vector, thus providing valid input for subsequent attention calculations. For the entire batch, a token tensor is generated. .

[0027] Step S205, Multi-head Interactive Attention: After applying layer normalization (Pre-Norm) to the token sequence, a query, key, and value vector is generated for each token through linear projection. Specifically, let... =LayerNorm , The projection process of the token tensor after layer normalization (LayerNorm) is as follows: in, , , Generate Q, K, V for the projection weight matrix. Subsequently, The dimension is divided into H heads, that is, the shape is readjusted to (m,d,H, / H), and permuted to the dimension (m,H,d, / H) to separate the attention heads. The calculation of attention weights is achieved through a scaled dot product attention mechanism: Calculated output After dimensional displacement and reshaping, it is restored to , This represents the tensor recovered from the multi-head attention computation output after dimensional permutation and reshaping. To aggregate token sequence information back into the feature vector, inner product pooling is used, with positional embedding P as the pooling kernel: Where ⊙ denotes element-wise multiplication, and the summation is performed along the last dimension to obtain the aggregated feature representation. ∈ .

[0028] Step S206, Feature Projection and Fusion: Projecting and fusing the features after attention aggregation Through an output projection layer Perform the transformation and compare it with the module's input. Residual connections are performed, followed by layer normalization to complete the attention sublayer processing. Next, a feedforward network (FFN) is used. Perform nonlinear transformations and fusions. This represents the intermediate feature representation after processing by the attention sublayer in the dynamic head interaction attention module. This FFN consists of two linear layers, a GELU activation function, and a Dropout layer; its output is then compared with... Perform residual connections and layer normalization to generate the final output of the dynamic head interaction attention module. : in, Finally, H (2) The input is fed into a feature compression and classification module. This module first compresses the feature dimension from d to 128 through a bottleneck structure containing a linear layer, layer normalization, GELU activation function, and Dropout layer, obtaining highly abstract discriminative features F. : in , This represents the feature compression weight matrix in the feature compression and classification module. Finally, it is processed by a linear classifier. The feature F is mapped to the log probabilities (logits) corresponding to C categories to complete the classification decision for intrusion detection: This represents the final predicted output of the student model, which is the log probabilities (logits) corresponding to the C categories.

[0029] Dynamic entropy adaptive topology distillation module This section is responsible for knowledge transfer from a large teacher model to a lightweight student model. Its core is to achieve soft-label distillation through adaptive temperature adjustment while maintaining the topological consistency of the feature space. This includes steps S207 to S209 (entropy adaptive temperature calculation, soft-label knowledge distillation, and topological manifold alignment). The specific mechanism and process are as follows: Figure 4 As shown.

[0030] Step S207, Entropy Adaptive Temperature Calculation: First, during the forward propagation of distillation training, the frozen teacher model generates the output log probability for the input batch data X. With high-dimensional features Where C is the number of categories, This represents the feature dimension of the teacher model. The student model then generates the corresponding... and , This represents the logarithmic probability matrix output by the student model. This represents the high-dimensional discriminative feature matrix extracted by the student model. To dynamically adjust the "smoothness" of knowledge transfer, this method adaptively calculates the distillation temperature T based on the uncertainty (information entropy) of the teacher model's prediction. Specifically, it first calculates the soft classification probability output by the teacher model. =Softmax Then its information entropy is calculated. : in, This represents the predicted probability distribution of the i-th sample. ic This represents the soft classification probability value of the teacher model predicting that the i-th sample belongs to the c-th class. To minimize the value and prevent numerical overflow, the average entropy of this batch of samples was ultimately used to calculate the dynamic temperature T. This design allows for the use of a higher temperature T (calculated above) when the teacher model has low prediction confidence (high entropy) for a certain batch of data. This temperature scaling applies to the output log probabilities of both the teacher and student models, smoothing the soft labels in one step to convey richer information about inter-class relationships. Conversely, a lower temperature is used to focus on the main category knowledge.

[0031] Step S208, Soft-label knowledge distillation: using the obtained smoothed probability distribution Subsequently, the Kullback-Leibler divergence was used as the loss function to minimize the difference between the student model distribution and the teacher model soft label distribution, and its calculation is as follows: in This represents the smoothed probability distribution of the student model after scaling by temperature T. This represents the smoothed probability distribution of the teacher model after scaling by temperature T. This indicates the number of samples in the current training batch. This represents the smoothed probability value of the i-th sample belonging to the c-th class in the teacher model at temperature T. This represents the smoothed probability value of the i-th sample belonging to the c-th class in the student model at temperature T. Multiplied by This is to counteract the gradient scaling effect caused by temperature scaling during gradient backpropagation, ensuring the stability of the loss dimension. This loss enables the student model to learn not only the "correct answer" predicted by the teacher, but also the inter-class similarity knowledge implied in its entire probability vector.

[0032] Step S209, Topological Manifold Alignment: To further ensure that the student model learns the inter-sample structural relationships constructed by the teacher model in the feature space, this invention introduces a topological manifold consistency loss. This loss is achieved by comparing the relationship matrices of the student and teacher models at the sample feature level. First, the feature vectors output by both models are... and Perform L2 normalization separately: in, This represents the feature vector output by the student model. This represents the feature vector output by the teacher model; Subsequently, the cosine similarity matrix (i.e., the relationship graph) between each sample is calculated. and : Among them, , , This represents the cosine similarity matrix between samples in the student model. The sample-to-sample cosine similarity matrix of the teacher model, with matrix elements... This represents the correlation strength between the i-th sample and the j-th sample within a batch in the feature space. Topological consistency loss is defined as the mean squared error between two relation matrices. This loss function forces the student model to maintain a sample neighborhood relationship that is isomorphic to the teacher model's feature space in the compressed feature space, thereby preserving the discriminative information at the manifold structure level that is crucial for intrusion detection.

[0033] Final dynamic entropy adaptive topological distillation loss Standard cross-entropy classification loss Soft label distillation loss With topology consistency loss Weighted sum: Here, α and β are hyperparameters that control the strength of soft-label knowledge transfer and topological alignment, respectively. By jointly optimizing this loss function, the lightweight student model can efficiently inherit the powerful feature representation and discrimination capabilities of the teacher model while significantly reducing the number of parameters, thus achieving high-performance lightweight intrusion detection.

[0034] End-to-end model compression training process module This section implements the complete knowledge distillation process from teacher model training to lightweight student model generation, including steps S210 to S213 (teacher model training, student model initialization, multi-objective distillation training, and lightweight model deployment). The specific mechanism and process are as follows: Figure 5 As shown.

[0035] Step S210, Teacher Model Training: First, in the labeled network traffic dataset... Independently train a high-performance teacher model with complete parameters. Its training objective is to minimize the standard cross-entropy classification loss. The goal is to learn accurate boundary values ​​for intrusion behavior classification. The training process employs stochastic gradient descent or its variants (such as AdamW) for optimization until the model's performance converges on the validation set. After training, the final state dictionary of the teacher model is saved. And its parameters are frozen, and expressed as a formula: in, This represents the number of parameters in the teacher model. This represents a single network traffic feature sample (or a small batch of samples) from the input, typically a feature vector or matrix. This represents a labeled network traffic training dataset. This represents the actual intrusion category label corresponding to sample X; The teacher models produced in this stage will serve as the "knowledge source" for the subsequent knowledge distillation process, providing high-quality soft labels and deep features.

[0036] Step S211, Student Model Initialization: Construct a simplified and lightweight student model. This model retains the core modular architecture of the teacher model (frequency domain wavelet analysis module and dynamic head interaction attention module), but significantly reduces the hidden dimensions and number of parameters in each layer. Specifically, the dimensions of its input projection layer and intermediate feature layer are compressed, for example, from 256 dimensions in the teacher model to 128 dimensions. The student model's parameters... It is randomly initialized. Its parameter count is... |Number of parameters in the teacher model| The ratio | defines the model compression ratio p = | | / |, typically ρ>2. This step completes the blueprint definition and initialization preparation for the target lightweight model.

[0037] Step S212, Multi-objective distillation training: The training loss adopts the aforementioned dynamic entropy adaptive topological distillation loss, which jointly optimizes three objectives: classification task, soft label alignment, and feature topological consistency. in, and The weighting hyperparameters are used to balance the various losses; This represents the input feature matrix of a batch of network traffic sampled from the dataset. This represents the actual intrusion category label vector for the corresponding batch. It is iteratively updated through backpropagation and optimization algorithms. This allows the student model to approximate the decision boundary and feature space structure of the teacher model with a significantly smaller capacity.

[0038] Step S213, Lightweight Model Deployment: After multi-objective distillation training, the student model is comprehensively evaluated on an independent test set. Its performance metrics, such as accuracy, precision, recall, and F1 score, are calculated and compared with the teacher model's performance to verify the "performance-efficiency" trade-off. After the evaluation is successful, the final student model state dictionary is... The complete network structure definition is serialized and saved as a standardized format file. The resulting lightweight model file can be directly integrated into actual network intrusion detection systems or edge computing devices to achieve low-latency, high-energy-efficiency real-time traffic analysis and threat detection, completing the end-to-end process from algorithm development to application deployment.

[0039] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A lightweight intrusion detection method based on frequency domain gating and dynamic head attention distillation, characterized in that, include: Step 1: Construct a teacher model, which includes a frequency domain wavelet analysis module and a dynamic head interaction attention module. The frequency domain wavelet analysis module performs multi-scale decomposition and information enhancement of network traffic features in the frequency domain through a two-level Haar wavelet transform and an adaptive gating mechanism. The dynamic head interaction attention module treats the feature dimension as a token sequence and models the global dependency relationship between features by introducing position-aware embedding and multi-head self-attention mechanism. The frequency domain wavelet analysis module is configured to perform the following steps: Step S201, Two-pole wavelet decomposition: For the input features Zero-padding is performed at the end of the feature dimension d to obtain , This represents the feature matrix after zero-padding of the 0th layer features; subsequently, a two-stage discrete Haar wavelet transform is performed to decouple the original features in the frequency domain into three sub-bands with different scales and frequencies: the coarsest scale low-frequency component. Secondary coarse-scale high-frequency components and the finest-scale high-frequency components ; Step S202, Gated Feature Enhancement: Apply independent adaptive gating functions to the three frequency sub-bands obtained from the second-level decomposition. Step S203, Inverse Wavelet Reconstruction: Through two-stage inverse Haar wavelet transform, the enhanced second-level subband and the enhanced first-level high-frequency subband are reconstructed step by step. The final output of the frequency domain wavelet analysis module is then formed through nonlinear transformation and residual connection. ; The dynamic head interaction attention module is configured to perform the following steps: Step S204, Position-Aware Embedding: The final output of the frequency domain wavelet analysis module... Each feature dimension d is converted into a high-dimensional token representation, generating the corresponding token sequence; Step S205, Multi-head Interactive Attention: Apply layer normalization to the token sequence, generate a query vector Q, a key vector K, and a value vector V for each token through linear projection, and obtain the aggregated feature representation through a scaling dot product attention mechanism. ; Step S206, Feature Projection and Fusion: This involves processing the aggregated feature representation... The attention sub-layer is processed, and a feedforward network (FFN) is used to represent the intermediate features after the attention sub-layer processing. Nonlinear transformations and fusions are performed to generate the final output of the dynamic head interaction attention module. , H (2) The input is fed into the feature compression and classification module for compression, resulting in highly abstract discriminative features F; Step 2: Design a dynamic entropy adaptive topological distillation strategy. Soften teacher knowledge through entropy-aware temperature regulation and introduce topological manifold consistency loss. This allows the lightweight student model to inherit the discriminative features and sample relationship structure of the teacher model while compressing the feature space. In step 2, the dynamic entropy adaptive topology distillation module performs the following steps to implement the distillation strategy, including: Step S207, Entropy Adaptive Temperature Calculation: Based on the uncertainty of the teacher model prediction, adaptively calculate the distillation temperature T; Step S208, Soft-label knowledge distillation: Use KL divergence to minimize the difference between the student model distribution and the teacher model soft-label distribution; Step S209, Topological manifold alignment: Introduce topological manifold consistency loss, which is achieved by comparing the relationship matrix of student and teacher models at the sample feature level; Step 3: Transfer the knowledge of the teacher model to the lightweight student network through end-to-end training.

2. The lightweight intrusion detection method according to claim 1, characterized in that, In step S201, The two-level discrete Haar wavelet transform includes: The first-level discrete Haar wavelet transform steps are as follows: The first-level decomposition splits the feature sequence into low-frequency components along the feature dimension according to the parity index. With high frequency components The calculation process involves linear combination and scaling; let... The kth feature element is The low-frequency coefficients and high-frequency coefficients generated by the first-order decomposition are as follows: in, and Each constitutes a first-order low-frequency component With first-order high-frequency components , Representing feature dimension, This represents the i-th low-frequency coefficient of the first-level wavelet decomposition. This refers to the i-th high-frequency coefficient of the first-level wavelet decomposition. Represents the filled feature matrix The feature element value at the 2i-th index position; Second-level Discrete Haar Wavelet Transform Steps: For the first-level low-frequency components... Performing the same Haar transform yields the low-frequency components of the second stage. With high frequency components All of their dimensions are ; Step S202 specifically includes: For the coarsest-scale low-frequency components Sigmoid gate is used Weighted modulation is performed on the finest-scale high-frequency components. and secondary coarse-scale high-frequency components Gating using residual form Enhance; Step S203 specifically includes: The enhanced second-order low-frequency component With high frequency components Reconstruction is performed using inverse Haar wavelet transform to obtain the enhanced first-order low-frequency component. The inverse transform is achieved by interleaving the scaled sums and differences; this enhances the first-order low-frequency components. With the enhanced first-order high-frequency component The feature representation is then reconstructed again using the inverse Haar transform to obtain the final frequency-domain enhanced feature representation. m represents the batch size; Remove the padding added in step S201, truncate the first d dimensions, and restore its dimensions to normal. ; Using a hybrid network containing linear layers, layer normalization, and the GELU activation function. Perform a nonlinear transformation and compare it with the input features H of the frequency domain wavelet analysis module. (0) Perform residual connections to generate the final output of the frequency domain wavelet analysis module. .

3. The lightweight intrusion detection method according to claim 2, characterized in that, In step S201, the fill length pad = .

4. The lightweight intrusion detection method according to claim 1, characterized in that, In step S201, the input features The steps to obtain it are as follows: In the teacher model, the network traffic feature matrix is ​​input through the projection layer. The mapping is to a feature representation with a uniform hidden dimension d=256, where m is the batch size. Given the feature dimension, the projection layer is composed of a linear transformation, layer normalization, and a GELU activation function connected sequentially, and its mathematical expression is as follows: in, Let be the projection weight matrix. The bias vector is the projected feature. As input features for the frequency domain wavelet analysis module, the network traffic features include IP address, port, and protocol type.

5. The lightweight intrusion detection method according to claim 2, characterized in that, Step S204 specifically includes: initializing the independently learnable matrix P by truncating the normal distribution. As a location embedding, where =d / H, where H is the number of attention heads, and h is the feature vector of each sample in the batch. By expanding its dimensions and performing an affine combination with the positional embedding, the corresponding token sequence T is generated. The j-th token is generated as follows: in, Let j be the scalar value of the j-th feature dimension. This is the position encoding vector corresponding to the j-th feature dimension; for the entire batch, a token tensor is generated. , Indicates hidden dimensions; Step S205 specifically includes: [Let...] =LayerNorm , The projection process of the token tensor after layer normalization is as follows: in, , , Generate Q, K, V for the projection weight matrix. ;Will The dimensions are divided into H heads, meaning the shape is readjusted to m, d, H. / H, and replace the dimensions with m,H,d. / H is used to separate the attention head; the attention weights are calculated using a scaled dot product attention mechanism: Calculated output After dimensional displacement and reshaping, it is restored to , This represents the tensor recovered from the multi-head attention computation output after dimensional permutation and reshaping; the token sequence information is aggregated back into the feature vector, and inner product pooling is used, with positional embedding P as the pooling kernel: Where ⊙ denotes element-wise multiplication, and the summation is performed along the last dimension to obtain the aggregated feature representation. ∈ ; Step S206 specifically includes: Feature representation after attention aggregation By output projection layer Perform transformations and interact with the dynamic head's attention module input. Perform residual connections, then perform layer normalization, and complete the attention sublayer processing: Next, a feedforward network (FFN) is used to... Perform nonlinear transformations and fusions. This represents the intermediate feature representation after processing by the attention sublayer in the dynamic head interaction attention module. This FFN consists of two linear layers, a GELU activation function, and a Dropout layer. Its output is then compared with... Perform residual connections and layer normalization to generate the final output of the dynamic head interaction attention module. : in, ; Finally, H (2) The input is fed into the feature compression and classification module, which, through a bottleneck structure including linear layers, layer normalization, GELU activation function, and Dropout layers, compresses the feature dimension from d to 128, obtaining highly abstract discriminative features F. : in , This represents the feature compression weight matrix in the feature compression and classification module. Then through a linear classifier The discriminative feature F is mapped to the log probabilities corresponding to C categories to complete the classification decision for intrusion detection. This represents the final predicted output of the student model, which is the log probability corresponding to the C categories.

6. The lightweight intrusion detection method according to claim 5, characterized in that, Step S207 specifically includes: During the forward propagation of the distillation training, the frozen teacher model generates the output log probability for the input batch data X. With high-dimensional features Where C is the number of categories, The teacher model provides the feature dimensions; the student model generates the corresponding features. and , This represents the logarithmic probability matrix output by the student model. This represents the high-dimensional discriminative feature matrix extracted by the student model; to dynamically adjust the smoothness of knowledge transfer, the distillation temperature T is adaptively calculated based on the uncertainty of the teacher model's prediction; specifically, the soft classification probability output by the teacher model is first calculated. =Softmax Then its information entropy is calculated. : in, This represents the predicted probability distribution of the i-th sample. This represents the soft classification probability value of the teacher model predicting that the i-th sample belongs to the c-th class. To minimize the value and prevent numerical overflow, the average entropy of this batch of samples was ultimately used to calculate the dynamic temperature T. ； In step S208, the calculation formula is as follows: in This represents the smoothed probability distribution of the student model after scaling by temperature T. This represents the smoothed probability distribution of the teacher model after scaling by temperature T. This indicates the number of samples in the current training batch. This represents the smoothed probability value of the i-th sample belonging to the c-th class in the teacher model at temperature T. This represents the smoothed probability value of the i-th sample belonging to the c-th class in the student model at temperature T. Step S209 specifically includes: processing the feature vectors output by the student and teacher models. and Perform L2 normalization separately: ， in, This represents the feature vector output by the student model. This represents the feature vector output by the teacher model; Calculate the cosine similarity matrix between samples of the student model. Cosine similarity matrix between samples of the teacher model : in, , Matrix elements The topological consistency loss represents the correlation strength between the i-th and j-th samples in the feature space within a batch; the mean squared error between the two relation matrices is defined as: ； Final dynamic entropy adaptive topological distillation loss Standard cross-entropy classification loss Soft label distillation loss With topology consistency loss Weighted sum: Here, α and β are hyperparameters that control the intensity of soft tag knowledge transfer and topological alignment, respectively.

7. The lightweight intrusion detection method according to claim 6, characterized in that, In step 3, an end-to-end model compression training process module is used, and the module is configured as follows: Step S210, Teacher Model Training: On the labeled network traffic dataset High-performance teacher models with complete, independently trained parameters Its training objective is to minimize the standard cross-entropy classification loss. The training process employs stochastic gradient descent or its variants for optimization until the model's performance converges on the validation set; after training, the final state dictionary of the teacher model is saved. And its parameters are frozen, and expressed as a formula: ， in, This represents the number of parameters in the teacher model. This represents a single network traffic feature sample or a small batch of samples from the input, typically a feature vector or matrix. This represents a labeled network traffic training dataset. This represents the actual intrusion category label corresponding to sample X; Step S211, Student Model Initialization: Construct a lightweight student model This model retains the frequency domain wavelet analysis module and dynamic head interaction attention module of the teacher model, but reduces the hidden dimensions and number of parameters in each layer; Step S212, Multi-objective distillation training: The training loss adopts dynamic entropy adaptive topological distillation loss, which jointly optimizes three objectives: classification task, soft label alignment, and feature topological consistency. in, and The weighting hyperparameters are used to balance the various losses; This represents a batch of network traffic input feature matrices sampled from the dataset. This represents the actual intrusion category label vector for the corresponding batch, and the parameter count is iteratively updated through backpropagation and optimization algorithms. This allows the student model to approximate the decision boundary and feature space structure of the teacher model; Step S213, Lightweight Model Deployment: Evaluate the student model on an independent test set, calculate its performance metrics including accuracy, precision, recall, and F1 score, and compare it with the teacher model's performance to verify the performance-efficiency trade-off; after the evaluation meets the standards, the final student model state dictionary is generated. Its complete network structure definition is serialized and saved as a standardized format file.

8. The lightweight intrusion detection method according to claim 7, characterized in that, In step S211, reducing the hidden dimensions and parameter count of each layer includes: compressing the dimensions of the input projection layer and intermediate feature layers, and randomly initializing the parameters of the student model. Its parameter count |Number of parameters in the teacher model| |The compression ratio p = | is defined in the model. | / |,p>

2.

9. A lightweight intrusion detection system based on frequency domain gating and dynamic head attention distillation, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the lightweight intrusion detection method according to any one of claims 1-8 when invoked by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the lightweight intrusion detection method according to any one of claims 1-8 when invoked by a processor.

Images