Knowledge distillation double-teacher model-based Internet of Things malicious traffic detection method and system

Abstract

The invention provides an Internet of Things malicious traffic detection method and system based on a knowledge distillation double-teacher model, and belongs to the technical field of network security. The problems that a single pre-training model cannot capture multi-dimensional features of traffic at the same time and a lightweight student model is difficult to effectively design are solved. The method comprises the following steps: preprocessing original Internet of Things traffic data to generate a standardized data set; inputting the standardized data set into a double-teacher model for training; a hidden layer and a soft label of the double-teacher model are extracted, a lightweight student model is constructed based on the hidden layer and the soft label of the double-teacher model, and the lightweight student model is a combination of a one-dimensional depth separable convolution and a stacked bidirectional gating loop unit; knowledge of the double-teacher model is migrated to the student model through a knowledge distillation loss function; and inputting to-be-detected traffic into the trained student model, and outputting a classification result of normal or malicious traffic. The method is mainly used in intelligent control field.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and in particular relates to an Internet of Things malicious traffic detection method based on a knowledge distillation dual-teacher model. Background Art

[0002] With the rapid development of the Internet of Things (IoT) technology, the number of smart devices has grown exponentially. They are widely used in scenarios such as smart homes, industrial control systems, and smart cities, significantly improving the convenience of life. However, the widespread deployment of IoT devices also brings severe security challenges. Malicious traffic, as a primary vector of network attacks (such as DDoS attacks and data theft), poses a serious threat to resource-constrained IoT devices. These devices typically have kilobytes of memory, battery power, and millisecond-level communication windows, making them unable to withstand the high computational overhead of security. Signature-based intrusion detection systems (IDS), while easy to deploy, rely on predefined rule bases and are difficult to adapt to the ever-evolving new attack patterns (such as zero-day vulnerability exploits). This results in low detection accuracy and insufficient adaptability.

[0003] To overcome the limitations of traditional methods, machine learning-based intrusion detection technologies have become a research hotspot. Deep learning models (such as CNNs, LSTMs, and Transformers) demonstrate great potential in identifying complex attacks by capturing the spatiotemporal characteristics of traffic data. However, these models often face a fundamental conflict between computational resource requirements and device limitations. High-precision models (such as Transformers) often exceed 10MB of parameters, far exceeding the KB-level memory capacity of IoT devices. Furthermore, IoT traffic data itself presents three major challenges: a scarcity of labeled samples, extreme class imbalance (normal traffic far outnumbers malicious traffic), and the coupling of multi-dimensional features (such as protocol type, packet length distribution, and intertwined temporal dependencies), further complicating model training. Statistics show that existing shallow, lightweight models (such as those based on single structures like 1D-DSC or BiGRU) have enabled edge deployment by reducing parameter count (for example, by replacing standard convolutions with depthwise separable convolutions), but detection accuracy remains below 85%, with performance significantly declining when identifying stealthy attacks such as low-rate DDoS attacks. This shortcoming stems from the limited feature extraction capabilities of lightweight models, which cannot fully cover the dynamic spatiotemporal characteristics of IoT traffic.

[0004] In recent years, knowledge distillation technology has been introduced to reconcile the contradiction between model accuracy and lightweightness. This method aims to improve the performance of a lightweight student model by transferring the "dark knowledge" of a high-precision teacher model (such as soft labels and intermediate layer features). However, existing knowledge distillation solutions face two bottlenecks in IoT scenarios: A single pre-trained model (such as a pure CNN or LSTM) cannot simultaneously capture the multi-dimensional characteristics of traffic. For example, CNN models excel at parsing packet header structure (such as protocol type) but ignore long-term dependencies. LSTM models effectively model temporal features but struggle with localized spatial patterns. Static multi-teacher fusion strategies (such as averaging the weights of multiple teacher outputs) fail to dynamically adapt to the varying contributions of features in different traffic scenarios (e.g., in bursty traffic, temporal features outweigh spatial features).

[0005] The transfer of intermediate layer knowledge fails, the heterogeneous structure of the teacher model makes it difficult to align the feature maps, and traditional distillation only transfers the softened labels (Logits) of the output layer, while ignoring the key knowledge of the intermediate layer (such as attention maps or relationship matrices), making it difficult to effectively design a lightweight student model. Summary of the Invention

[0006] In view of this, the present invention aims to propose a method and system for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model, so as to solve the problems that a single pre-trained model (such as pure CNN or pure LSTM) cannot simultaneously capture the multi-dimensional features of traffic and it is difficult to effectively design a lightweight student model.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model, the method comprising: Preprocess the raw IoT traffic data to generate a standardized data set; Inputting the standardized data set into a dual-teacher model for training, the dual-teacher model comprising: a teacher model 1 and a teacher model 2; The teacher model 1 is a spatial feature extraction model based on the residual Inception structure, which includes parallel convolution branches, residual blocks and channel attention modules; The second teacher model is a temporal feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; Extracting the hidden layers and soft labels of the dual-teacher model, and constructing a lightweight student model based on the hidden layers and soft labels of the dual-teacher model. The lightweight student model is a combination of one-dimensional depthwise separable convolution and stacked bidirectional gated recurrent units. Transfer the knowledge of the dual teacher model to the student model through the knowledge distillation loss function; The traffic to be detected is input into the trained student model, and the classification result of normal or malicious traffic is output.

[0008] Furthermore, a preferred method is proposed, in which the spatial feature extraction model based on the residual Inception structure includes three parallel convolution branches: the convolution kernel sizes of branch one are 1×1, 3×3, and 1×1, the convolution kernel sizes of branch two are 1×1, 5×5, and 1×1, and the convolution kernel sizes of branch three are 3×3, 3×3, and 3×3; a jump connection is added to each residual block.

[0009] Furthermore, a preferred method is proposed, wherein the knowledge of the dual-teacher model is transferred to the student model through the knowledge distillation loss function, including: Use Kullback-Leibler loss to calculate the loss between the student model and the teacher model for generating soft labels; Transfer the soft labels of the teacher model to the student model, and use cost-sensitive loss to measure the fitting error of the student model to the hard labels; The hidden layers of the teacher and student models are distilled using a temporal correlation window loss function.

[0010] Furthermore, a preferred method is proposed, in which the time correlation window loss function is:

[0011] in, is the total number of time steps, i.e. the window length, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k.

[0012] Furthermore, a preferred method is proposed, where the knowledge distillation loss function satisfies:

[0013] in, is the weight coefficient of the KL divergence loss term, is the weight coefficient of the KL divergence loss term, γ is the weight coefficient of the time correlation window loss term, is the divergence distillation loss, For cost-sensitive losses, is the temporal correlation window loss.

[0014] Furthermore, a preferred method is proposed in which the soft label generation adopts a temperature regulation mechanism:

[0015] in, is the original logits output of the teacher model for input x, is the temperature parameter.

[0016] Furthermore, a preferred method is proposed, wherein the preprocessing of the original IoT traffic data includes: Calculate the correlation between features and labels through mutual information and screen key features; Perform maximum and minimum normalization on the feature data.

[0017] Based on the same inventive concept, the present invention also proposes an IoT malicious traffic detection system based on a knowledge distillation dual-teacher model, the system comprising: A preprocessing unit, used to preprocess the original IoT traffic data and generate a standardized data set; A training unit is configured to input the standardized data set into a dual-teacher model for training, wherein the dual-teacher model includes: a first teacher model and a second teacher model; the first teacher model is a spatial feature extraction model based on a residual Inception structure, comprising a parallel convolution branch, a residual block, and a channel attention module; the second teacher model is a temporal feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; A lightweight student model construction unit is used to extract the hidden layer and soft labels of the dual-teacher model, and construct a lightweight student model based on the hidden layer and soft labels of the dual-teacher model. The lightweight student model is a combination of a one-dimensional depthwise separable convolution and a stacked bidirectional gated recurrent unit; The knowledge transfer unit is used to transfer the knowledge of the dual teacher model to the student model through the knowledge distillation loss function; The output unit is used to input the traffic to be detected into the trained student model and output the classification results of normal or malicious traffic.

[0018] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model as described in any one of the above items.

[0019] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, it executes the steps of any one of the above-mentioned methods for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The present invention adopts a dual-teacher model structure, using teacher model 1 based on the residual Inception structure to extract spatial features and teacher model 2 based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network to extract temporal features. This strategy enables the dual-teacher model to simultaneously capture spatial and temporal features in IoT traffic data. Teacher 1, based on the residual Inception structure, parses packet header structural features, such as protocol type and packet length distribution, and adapts to encrypted traffic scenarios without the need for payload data. Teacher 2's one-dimensional convolutional neural network and stacked bidirectional long short-term memory network capture long-term attack patterns, complementing the knowledge blind spots of the single teacher model. The two results are spliced ​​together as the final result through a multi-head attention mechanism, solving the problem that traditional single models (such as pure CNN or pure LSTM) cannot process multi-dimensional features simultaneously.

[0021] Thanks to its dual-teacher model architecture, one teacher model is responsible for spatial information, while the other for temporal information. Knowledge from both is transferred to the student model through knowledge distillation. This approach avoids the shortcomings of static multi-teacher fusion, which suffers from fixed weight distribution and lacks dynamic adaptability. As a result, the model can dynamically adjust its focus on various features based on the characteristics of the input traffic, thereby improving the accuracy of malicious traffic detection.

[0022] By combining a one-dimensional depthwise separable convolution with a stacked bidirectional gated recurrent unit as the student model, this paper successfully achieves a lightweight model design. The student model not only inherits the performance of the teacher model but also significantly reduces computational resource consumption while maintaining efficient detection performance, making it suitable for deployment on IoT devices, especially those with limited resources.

[0023] The present invention not only relies on the soft labels of the teacher model output layer for knowledge transfer, but also extracts the hidden layer information of the teacher model for distillation, thereby retaining more intermediate layer knowledge. At the same time, the designed distillation loss module and the time correlation window loss function force the smooth transition of the hidden states of consecutive time steps, enhance the time series correlation of traffic, and the time correlation window loss TCW strengthens the perception of covert attacks and improves the recall rate during attacks. It forms a closed-loop optimization with the divergence loss, which is not only effective for static alignment features, but also optimizes the dynamic features of the time series. Through the knowledge distillation loss function, the knowledge of the teacher model is effectively transferred to the student model, so that the trained student model can achieve a balance between accuracy, detection speed and resource consumption. The trained student model can quickly determine whether the traffic is normal or malicious, which is suitable for the needs of real-time detection in the Internet of Things environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of the method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model of the present invention; Figure 2 This is an architectural diagram of the dual-teacher-student distillation model described in the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely explain the technical solutions in the embodiments of the present invention. It should be noted that the embodiments of the present invention and the features therein can be combined with each other in the absence of conflict, and the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0026] Implementation method 1, see Figure 1 This embodiment describes a method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model, the method comprising: Preprocess the raw IoT traffic data to generate a standardized data set; Inputting the standardized data set into a dual-teacher model for training, the dual-teacher model comprising: a teacher model 1 and a teacher model 2; The teacher model 1 is a spatial feature extraction model based on the residual Inception structure, which includes parallel convolution branches, residual blocks and channel attention modules; The second teacher model is a temporal feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; Extracting the hidden layers and soft labels of the dual-teacher model, and constructing a lightweight student model based on the hidden layers and soft labels of the dual-teacher model. The lightweight student model is a combination of one-dimensional depthwise separable convolution and stacked bidirectional gated recurrent units. Transfer the knowledge of the dual teacher model to the student model through the knowledge distillation loss function; The traffic to be detected is input into the trained student model, and the classification result of normal or malicious traffic is output.

[0027] In this implementation, a dual-teacher model structure is adopted, in which a teacher model 1 based on the residual Inception structure is used to extract spatial features, and a teacher model 2 based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network is used to extract temporal features. This strategy enables the dual-teacher model to simultaneously capture spatial and temporal features in IoT traffic data. Teacher 1, based on the residual Inception structure, parses packet header structural features, such as protocol type and packet length distribution, and adapts to encrypted traffic scenarios without the need for payload data. Teacher 2's one-dimensional convolutional neural network and stacked bidirectional long short-term memory network capture long-term attack patterns, complementing the knowledge blind spots of the single teacher model. The two results are spliced ​​together as the final result through a multi-head attention mechanism, solving the problem that traditional single models (such as pure CNN or pure LSTM) cannot process multi-dimensional features simultaneously.

[0028] Thanks to its dual-teacher model architecture, one teacher model is responsible for spatial information, while the other for temporal information. Knowledge from both is transferred to the student model through knowledge distillation. This approach avoids the shortcomings of static multi-teacher fusion, which suffers from fixed weight distribution and lacks dynamic adaptability. As a result, the model can dynamically adjust its focus on various features based on the characteristics of the input traffic, thereby improving the accuracy of malicious traffic detection.

[0029] By combining a one-dimensional depthwise separable convolution with a stacked bidirectional gated recurrent unit as the student model, this paper successfully achieves a lightweight model design. The student model not only inherits the performance of the teacher model but also significantly reduces computational resource consumption while maintaining efficient detection performance, making it suitable for deployment on IoT devices, especially those with limited resources.

[0030] The method proposed in this embodiment not only relies on the soft labels of the output layer of the teacher model for knowledge transfer, but also extracts the hidden layer information of the teacher model for distillation, thereby retaining more intermediate layer knowledge. At the same time, the designed distillation loss module and the time correlation window loss function force the smooth transition of the hidden states of consecutive time steps, enhance the time series correlation of traffic, and TCW strengthens the perception of covert attacks and improves the recall rate during attacks. It forms a closed-loop optimization with the divergence loss, which is not only effective for static alignment features, but also optimizes the dynamic features of the time series. Through the knowledge distillation loss function, the knowledge of the teacher model is effectively transferred to the student model, so that the trained student model can achieve a balance between accuracy, detection speed and resource consumption. The trained student model can quickly determine whether the traffic is normal or malicious, which is suitable for the needs of real-time detection in the Internet of Things environment.

[0031] Implementation method 2. This implementation method is a further limitation of the IoT malicious traffic detection method based on the knowledge distillation dual-teacher model described in implementation method 1. The spatial feature extraction model based on the residual Inception structure contains three parallel convolution branches: the convolution kernel size of branch one is 1×1, 3×3, and 1×1, the convolution kernel size of branch two is 1×1, 5×5, and 1×1, and the convolution kernel size of branch three is 3×3, 3×3, and 3×3; a jump connection is added to each residual block.

[0032] In this implementation, the three parallel convolution branches use convolution kernels of different sizes (1×1, 3×3, 5×5, etc.), enabling the extraction of spatial features at different scales. This allows the capture of both detailed and global features in traffic data, improving the model's ability to identify complex malicious traffic. By using multiple convolution kernel sizes within the same layer, the model can fuse receptive fields of varying sizes, achieving richer and more diverse feature representations, which helps improve the accuracy of malicious traffic detection.

[0033] By introducing skip connections, the residual block enables the model to better learn the detailed information in the feature map and alleviate the problem of gradient disappearance or explosion in deep networks, thereby enhancing the training effect of the network and helping to improve the training efficiency and accuracy of the model.

[0034] Implementation 3: This implementation further limits the IoT malicious traffic detection method based on the knowledge distillation dual-teacher model described in Implementation 1. The knowledge distillation loss function is used to transfer the knowledge of the dual-teacher model to the student model, including: Use Kullback-Leibler loss to calculate the loss between the student model and the teacher model for generating soft labels; Transfer the soft labels of the teacher model to the student model, and use cost-sensitive loss to measure the fitting error of the student model to the hard labels; The hidden layers of the teacher and student models are distilled using a temporal correlation window loss function.

[0035] In this implementation, through knowledge distillation, the student model can effectively utilize the knowledge of the two teacher models. The student model can learn the important features and patterns in the teacher model while retaining its lightweight, thereby improving the accuracy and generalization ability of detection.

[0036] Using Kullback-Leibler loss to calculate the loss between the student model and the teacher model generates soft labels that provide more detailed information than traditional hard labels (i.e., raw labels). Soft labels convey the label's category information, helping the student model learn richer features, making the model more robust to changes in input data and effectively resisting noise and interference in malicious traffic. By introducing a cost-sensitive loss to measure the student model's hard label fitting error, the model can focus more on categories with higher misclassification costs during the learning process. This is particularly important in malicious traffic detection, as malicious traffic detection often faces the problem of category imbalance. Traditional methods may overlook the detection of low-frequency malicious traffic, while cost-sensitive loss can effectively compensate for this deficiency, thereby improving detection accuracy.

[0037] Using a temporal correlation window loss function to distill the hidden layers of the teacher and student models can enhance the model's ability to learn time series data. In IoT malicious traffic detection, traffic data is typically continuous and exhibits a certain degree of temporal correlation. By accounting for this temporal correlation, the model can better capture temporal patterns in the data and improve its ability to identify malicious traffic in time series.

[0038] Implementation 4: This implementation further limits the method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model described in Implementation 3. The time correlation window loss function is:

[0039] in, is the total number of time steps, i.e. the window length, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k, 1 means it comes from the first teacher model, 2 means it comes from the second teacher model, k is the time step.

[0040] Implementation 5: This implementation further limits the method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model described in Implementation 1. The knowledge distillation loss function satisfies:

[0041] in, is the weight coefficient of the KL divergence loss term, is the weight coefficient of the KL divergence loss term, γ is the weight coefficient of the time correlation window loss term, is the divergence distillation loss, For cost-sensitive losses, Temporal Correlation Window Loss.

[0042] Implementation 6: This implementation further limits the method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model described in Implementation 1. The soft label generation adopts a temperature adjustment mechanism:

[0043] in, is the original logits output of the teacher model for input x, is the temperature parameter used to control the smoothness of the softmax output.

[0044] Implementation 7: This implementation further limits the method for detecting malicious IoT traffic based on the knowledge distillation dual-teacher model described in Implementation 1. The preprocessing of the original IoT traffic data includes: Calculate the correlation between features and labels through mutual information and screen key features; Perform maximum and minimum normalization on the feature data.

[0045] Implementation 8: This implementation describes an IoT malicious traffic detection system based on a knowledge distillation dual-teacher model, the system comprising: A preprocessing unit, used to preprocess the original IoT traffic data and generate a standardized data set; A training unit is configured to input the standardized data set into a dual-teacher model for training, wherein the dual-teacher model includes: a first teacher model and a second teacher model; the first teacher model is a spatial feature extraction model based on a residual Inception structure, comprising a parallel convolution branch, a residual block, and a channel attention module; the second teacher model is a temporal feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; A lightweight student model construction unit is used to extract the hidden layer and soft labels of the dual-teacher model, and construct a lightweight student model based on the hidden layer and soft labels of the dual-teacher model. The lightweight student model is a combination of a one-dimensional depthwise separable convolution and a stacked bidirectional gated recurrent unit; The knowledge transfer unit is used to transfer the knowledge of the dual teacher model to the student model through the knowledge distillation loss function; The output unit is used to input the traffic to be detected into the trained student model and output the classification results of normal or malicious traffic.

[0046] Implementation method nine: A computer device described in this implementation method includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model as described in any one of implementation methods one to seven.

[0047] Implementation method ten, a computer-readable storage medium described in this implementation method, on which a computer program is stored, and when the computer program is run by a processor, the steps of a method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model as described in any one of implementation methods one to seven are executed.

[0048] Implementation method 11, see Figure 2 This embodiment provides a specific example of the method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model described in embodiment 1, and is also used to explain embodiments 2 to 7. Specifically: S1. Perform preprocessing operations on traffic data, including data cleaning, feature extraction, data normalization, etc., and integrate the traffic samples after the above operations into a new data set; Specifically, step S1 includes: S11. Performing data cleaning, feature extraction, and data normalization according to the traffic sample set; S12. Analyze the significance of each feature in the sample set and select the features that need to be retained.

[0049] Preferably, in the data cleaning step, samples containing Nan and Infinity are deleted from the sample set; in the feature extraction step, mutual information is used to calculate the correlation between each feature and the corresponding sample label. The specific calculation formula is as follows:

[0050] in, Represents the joint probability distribution of X and Y, that is, the feature Pick And label Pick The probability of express The marginal probability distribution of The value is The probability of express The marginal probability distribution of label The value is The probability of Representation characteristics The specific value of Representation Label The specific value of .

[0051] In the data normalization step, the maximum and minimum normalization is used to normalize the sample set. This step is conducive to model training and convergence. The specific calculation formula is as follows: =

[0052] in, represents the original value of the feature, Indicates the minimum value of the feature in the data set. Indicates the maximum value of this feature in the dataset.

[0053] S2. Input the new dataset into the multi-teacher model for training, so that it can learn the characteristics of normal traffic and malicious traffic, and generate the intermediate layer hidden state and soft label of the teacher model; The multi-teacher model includes a teacher model 1 and a teacher model 2; The teacher model 1 is a spatial feature extraction model (ResidualInception) based on the residual Inception structure, which includes parallel convolution branches, residual blocks and channel attention modules. Specifically, the spatial feature extraction model based on the residual Inception structure contains three parallel convolution branches: the convolution kernel sizes of branch 1 are 1×1, 3×3, and 1×1, the convolution kernel sizes of branch 2 are 1×1, 5×5, and 1×1, and the convolution kernel sizes of branch 3 are 3×3, 3×3, and 3×3. Each residual block has one more skip connection than a normal residual block, which can learn more spatial features while solving the gradient vanishing problem. The second teacher model is a temporal feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network (1D-CNN-BiLSTM). Specifically, the 1D-CNN layer performs a convolution operation on the differential feature sequence of adjacent data packets to directly capture local temporal features, and the BiLSTM modeling long-term dependencies between streams. The obtained temporal and spatial features are then fused using a multi-head attention layer. The teacher model training steps include: For a given input, each teacher model produces a logical vector Z i =(Z1, Z2, ... Z n ), outputs a temperature-adjusted prediction distribution;

[0054] Use Kullback-Leibler loss to calculate the loss between the student model and the teacher model to generate soft labels;

[0055] in, Represents the teacher model for the category The predicted probability of Represents the student model for category The predicted probability of Indicates the categories, represents the predicted probability of the teacher model, Represents the predicted probability of the student model, N represents the total number of categories, i represents the index variable when summing, and traverses all categories from to N; The Adam optimizer is used for optimization, and the early stopping mechanism is used to stop training when the model no longer improves.

[0056] S3. Based on the intermediate hidden states and soft labels of the teacher model, a lightweight student model is constructed. The lightweight student model is a one-dimensional depthwise separable convolution and a stacked bidirectional GRU structure, specifically: Transfer the soft labels of the teacher model to the student model, and use cost-sensitive loss to measure the fitting error of the student model to the hard labels;

[0057] in, represents the weight coefficient of the i-th category, Indicates the temperature parameter of the student model Next pair The predicted probability of each category, Indicates the teacher model's temperature parameter Next pair The predicted probability of each category, γ is the adjustment factor used to control the loss weight of easy-to-classify samples; Use the time-correlation window loss to distill the intermediate layers of the teacher model and the student model, thereby improving the student model's sensitivity to changes in temporal features;

[0058] in, is the total number of time steps, i.e. the window length, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k, 1 means it comes from the first teacher model, 2 means it comes from the second teacher model, k is the time step; The distillation loss is used to update the weights of the student model to improve the performance of the student model.

[0059]

[0060] in, is the weight coefficient of the divergence loss term, is the weight coefficient of the cost-sensitive loss term, γ is the weight coefficient of the time-correlation window loss term, is the divergence distillation loss, For cost-sensitive losses, is the temporal correlation window loss.

[0061] S4. Input the traffic data to be detected into the trained student model for classification, and output the classification results of normal or malicious traffic.

[0062] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0063] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes. These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure and are not intended to limit its scope of protection. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the disclosed claims.

Claims

1. A method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model, characterized in that: The method comprises: Preprocess the raw IoT traffic data to generate a standardized data set; Inputting the standardized data set into a dual-teacher model for training, the dual-teacher model comprising: a teacher model 1 and a teacher model 2; The teacher model 1 is a spatial feature extraction model based on the residual Inception structure, which includes parallel convolution branches, residual blocks and channel attention modules; The second teacher model is a temporal feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; Extracting the hidden layers and soft labels of the dual-teacher model, and constructing a lightweight student model based on the hidden layers and soft labels of the dual-teacher model. The lightweight student model is a combination of one-dimensional depthwise separable convolution and stacked bidirectional gated recurrent units. Transfer the knowledge of the dual teacher model to the student model through the knowledge distillation loss function; The traffic to be detected is input into the trained student model, and the classification result of normal or malicious traffic is output.

2. The method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model according to claim 1 is characterized in that: The spatial feature extraction model based on the residual Inception structure includes three parallel convolution branches: the convolution kernel sizes of branch one are 1×1, 3×3, and 1×1, the convolution kernel sizes of branch two are 1×1, 5×5, and 1×1, and the convolution kernel sizes of branch three are 3×3, 3×3, and 3×3; each residual block is added with a jump connection.

3. The method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model according to claim 1 is characterized in that: The knowledge distillation loss function is used to transfer the knowledge of the dual teacher model to the student model, including: Use Kullback-Leibler loss to calculate the loss between the student model and the teacher model for generating soft labels; Transfer the soft labels of the teacher model to the student model, and use cost-sensitive loss to measure the fitting error of the student model to the hard labels; The hidden layers of the teacher and student models are distilled using a temporal correlation window loss function.

4. The method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model according to claim 3 is characterized in that: The time correlation window loss function is: in, is the total number of time steps, i.e. the window length, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k.

5. The method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model according to claim 1 is characterized in that: The knowledge distillation loss function satisfies: in, is the weight coefficient of the divergence loss term, is the weight coefficient of the cost-sensitive loss term, γ is the weight coefficient of the time-correlation window loss term, is the divergence distillation loss, For cost-sensitive losses, is the temporal correlation window loss.

6. The method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model according to claim 1 is characterized in that: The soft label generation adopts a temperature regulation mechanism: in, is the original logits output of the teacher model for input x, is the temperature parameter.

7. The method for detecting malicious traffic in the Internet of Things based on the knowledge distillation dual-teacher model according to claim 1 is characterized in that: The preprocessing of the original IoT traffic data includes: Calculate the correlation between features and labels through mutual information and screen key features; Perform maximum and minimum normalization on the feature data.

8. A malicious traffic detection system for the Internet of Things based on a knowledge distillation dual-teacher model, characterized by: The system comprises: A preprocessing unit, used to preprocess the original IoT traffic data and generate a standardized data set; A training unit is configured to input the standardized data set into a dual-teacher model for training, wherein the dual-teacher model includes: a first teacher model and a second teacher model; the first teacher model is a spatial feature extraction model based on a residual Inception structure, comprising a parallel convolution branch, a residual block, and a channel attention module; the second teacher model is a temporal feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; A lightweight student model construction unit is used to extract the hidden layer and soft labels of the dual-teacher model, and construct a lightweight student model based on the hidden layer and soft labels of the dual-teacher model. The lightweight student model is a combination of a one-dimensional depthwise separable convolution and a stacked bidirectional gated recurrent unit; The knowledge transfer unit is used to transfer the knowledge of the dual teacher model to the student model through the knowledge distillation loss function; The output unit is used to input the traffic to be detected into the trained student model and output the classification results of normal or malicious traffic.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a method for detecting malicious traffic in the Internet of Things based on a knowledge distillation dual-teacher model as described in any one of claims 1 to 7.

Images