Knowledge distillation double-teacher model-based internet of things malicious traffic detection method and system

Abstract

The application provides a malicious traffic detection method and system based on a knowledge distillation double-teacher model Internet of Things, and belongs to the technical field of network security. The method solves the problems that a single pre-training model cannot simultaneously capture multi-dimensional features of traffic and cannot effectively design a lightweight student model. The method comprises the following steps: preprocessing original Internet of Things traffic data to generate a standardized data set; inputting the standardized data set into a double-teacher model for training; extracting a hidden layer and a soft label of the double-teacher model, constructing a lightweight student model based on the hidden layer and the soft label of the double-teacher model, wherein the lightweight student model is a combination of one-dimensional deep separable convolution and stacked bidirectional gated recurrent unit; migrating the knowledge of the double-teacher model to the student model through a knowledge distillation loss function; inputting to-be-detected traffic into the trained student model to output a classification result of normal or malicious traffic. The method is mainly used in the field of intelligent control.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network security, and particularly relates to a malicious traffic detection method for Internet of Things based on a knowledge distillation double-teacher model. BACKGROUND

[0002] With the rapid development of Internet of Things (IoT) technology, the number of intelligent devices is growing exponentially, and is widely used in smart home, industrial control system and smart city scenarios, significantly improving the convenience of life. However, the widespread deployment of IoT devices also poses serious security challenges. Malicious traffic, as the main carrier of network attacks (such as DDoS attacks, data theft, etc.), poses a serious threat to resource-constrained IoT terminals. These devices usually have KB-level memory, battery-powered characteristics, and millisecond-level communication windows, and cannot bear the high computational overhead of security. Although signature-based intrusion detection systems (IDS) are easy to deploy, they rely on predefined rule libraries and are difficult to cope with continuously evolving new attack patterns (such as zero-day exploits), resulting in low detection accuracy and insufficient adaptability.

[0003] To overcome the limitations of traditional methods, machine learning-based intrusion detection technology has gradually become a research hotspot. Deep learning models (such as CNN, LSTM and Transformer) capture the spatiotemporal features of traffic data and show high potential in complex attack recognition. However, these models generally have a fundamental contradiction between computational resource requirements and device limitations: high-precision models (such as Transformer) have parameter quantities often exceeding 10 MB, far exceeding the KB-level memory capacity of IoT devices; at the same time, IoT traffic data itself has three major challenges of scarcity of labeled samples, extreme imbalance of classes (normal traffic is much more than malicious traffic), and multi-dimensional feature coupling (such as protocol type, packet length distribution, and time sequence dependence interlaced), further exacerbating the difficulty of model training. Statistics show that existing shallow lightweight models (such as single structures based on 1D-DSC or BiGRU) achieve edge deployment by reducing parameter quantities (for example, using depthwise separable convolution instead of standard convolution), but the detection accuracy is less than 85%, especially in identifying covert attacks (such as low-rate DDoS), the performance decreases significantly. This defect is due to the limited feature extraction capability of lightweight models, which cannot fully cover the dynamic spatiotemporal characteristics of IoT traffic.

[0004] In recent years, knowledge distillation technology has been introduced to reconcile the contradiction between model accuracy and lightweight. This method migrates the "dark knowledge" (such as soft labels and intermediate layer features) of high-precision teacher models to lightweight student models, aiming to improve the performance of the latter. However, existing knowledge distillation schemes face double bottlenecks in the IoT scenario:

[0005] A single pre-training model (such as a pure CNN or a pure LSTM) cannot capture the multidimensional features of traffic at the same time. For example, the CNN model is good at analyzing the packet header structure (such as the protocol type), but ignores the long time sequence dependence; the LSTM model effectively models the time sequence features, but it is difficult to process the spatial local pattern. The static multi-teacher fusion strategy (such as averaging the weighted outputs of multiple teachers) cannot dynamically adapt to the contribution difference of features in different traffic scenarios (such as the time sequence feature is more important than the spatial feature in burst traffic).

[0006] The intermediate layer knowledge transfer fails, the teacher model structure is heterogeneous, the feature map is difficult to align, the traditional distillation only transfers the output layer soft label (Logits), and the key knowledge (such as the attention map or the relationship matrix) of the intermediate layer is ignored, and it is difficult to effectively design a lightweight student model. SUMMARY

[0007] Therefore, the present application aims to provide a knowledge distillation double-teacher model-based Internet of Things malicious traffic detection method and system to solve the problems that a single pre-training model (such as a pure CNN or a pure LSTM) cannot capture the multidimensional features of traffic at the same time and it is difficult to effectively design a lightweight student model.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical scheme: a knowledge distillation double-teacher model-based Internet of Things malicious traffic detection method, the method comprising:

[0009] Pretreating the original Internet of Things traffic data to generate a standardized data set;

[0010] Inputting the standardized data set into a double-teacher model for training, the double-teacher model comprising a teacher model one and a teacher model two;

[0011] The teacher model one is a spatial feature extraction model based on a residual Inception structure, comprising a parallel convolution branch, a residual block and a channel attention module;

[0012] The teacher model two is a time sequence feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network;

[0013] Extracting the hidden layer and the soft label of the double-teacher model, constructing a lightweight student model based on the hidden layer and the soft label of the double-teacher model, the lightweight student model being a combination of a one-dimensional deep separable convolution and a stacked bidirectional gated recurrent unit;

[0014] Transferring the knowledge of the double-teacher model to the student model through a knowledge distillation loss function;

[0015] Inputting the traffic to be detected into the trained student model, and outputting the classification result of normal or malicious traffic.

[0016] Further, it is proposed that a preferred mode, the spatial feature extraction model based on residual Inception structure contains three parallel convolution branches: the convolution kernel size of branch one is 1x1, 3x3, 1x1, the convolution kernel size of branch two is 1x1, 5x5, 1x1, and the convolution kernel size of branch three is 3x3, 3x3, 3x3; each residual block is additionally connected by jump connection.

[0017] Further, it is proposed that a preferred mode, the knowledge of the double-teacher model is transferred to the student model through the knowledge distillation loss function, which includes:

[0018] The Kullback-Leibler loss is used to calculate the loss between the student model and the teacher model, which is used to generate soft labels;

[0019] The soft labels of the teacher model are transferred to the student model, and the cost-sensitive loss is used to measure the fitting error of the student model to the hard labels;

[0020] The hidden layers of the teacher model and the student model are distilled using the time correlation window loss function.

[0021] Further, it is proposed that a preferred mode, the time correlation window loss function is:

[0022]

[0023] Wherein, is the total time step, i.e. the window length, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k.

[0024] Further, it is proposed that a preferred mode, the knowledge distillation loss function satisfies:

[0025]

[0026] Wherein, is the weight coefficient of the KL divergence loss term, is the weight coefficient of the KL divergence loss term, and γ is the weight coefficient of the time correlation window loss term, is the divergence distillation loss, is the cost-sensitive loss, is the time correlation window loss.

[0027] Further, it is proposed that a preferred mode, the soft label generation adopts a temperature adjustment mechanism:

[0028]

[0029] wherein, is the original logits output of the teacher model for input x, is a temperature parameter.

[0030] Further, a preferred mode is also proposed, the preprocessing of the original Internet of Things traffic data comprises:

[0031] The correlation between the features and the labels is calculated by mutual information, and key features are screened out;

[0032] The feature data is normalized by maximum and minimum.

[0033] Based on the same inventive concept, the present application also proposes an Internet of Things malicious traffic detection system based on knowledge distillation double-teacher model, which comprises:

[0034] A preprocessing unit is configured to preprocess original Internet of Things traffic data to generate a standardized data set;

[0035] A training unit is configured to input the standardized data set into a double-teacher model for training, wherein the double-teacher model comprises a teacher model one and a teacher model two; the teacher model one is a spatial feature extraction model based on a residual Inception structure, which comprises a parallel convolution branch, a residual block and a channel attention module; the teacher model two is a time sequence feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network;

[0036] A lightweight student model construction unit is configured to extract hidden layers and soft labels of the double-teacher model, and construct a lightweight student model based on the hidden layers and the soft labels of the double-teacher model, wherein the lightweight student model is a combination of one-dimensional deep separable convolution and stacked bidirectional gated recurrent unit;

[0037] A knowledge transfer unit is configured to transfer the knowledge of the double-teacher model to the student model through a knowledge distillation loss function;

[0038] An output unit is configured to input the traffic to be detected into the trained student model, and output the classification result of normal or malicious traffic.

[0039] Based on the same inventive concept, the present application also proposes a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the Internet of Things malicious traffic detection method based on the knowledge distillation double-teacher model according to any one of the above.

[0040] Based on the same inventive concept, the application further provides a computer readable storage medium, which stores a computer program, and the computer program performs the steps of the knowledge distillation double-teacher model-based Internet of Things malicious traffic detection method according to any one of the above embodiments when executed by a processor.

[0041] Compared with the prior art, the application has the following beneficial effects:

[0042] The application adopts a double-teacher model structure, and uses a teacher model one based on a residual Inception structure to extract spatial features and a teacher model two based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network to extract time sequence features, so that the double-teacher model can capture spatial features and time sequence features in Internet of Things traffic data at the same time, the teacher one based on the residual Inception structure analyzes header structure features such as protocol type and packet length distribution, and is suitable for encrypted traffic scenarios without payload data, the one-dimensional convolutional neural network and the stacked bidirectional long short-term memory network of the teacher two capture long-time attack patterns, complement the knowledge blind area of a single teacher model, and splice the two results through a multi-head attention mechanism as a final result, so that the problem that a traditional single model (such as a pure CNN or a pure LSTM) cannot simultaneously process multi-dimensional features is solved.

[0043] Since the double-teacher model structure is adopted, the teacher model one side is responsible for spatial information, and the other side is responsible for time sequence information, and the knowledge of the two is transferred to the student model through knowledge distillation. This way avoids the fixed weight distribution and the lack of dynamic adaptability in static multi-teacher fusion. Therefore, the model can dynamically adjust the attention to various features according to different features of the input traffic, thereby improving the detection accuracy of malicious traffic.

[0044] By adopting a combination of a one-dimensional deep separable convolution and a stacked bidirectional gated recurrent unit as the student model, the application successfully realizes the design of a lightweight model. The student model not only inherits the performance of the teacher model, but also significantly reduces the consumption of computing resources while maintaining high-efficiency detection performance, so that it is suitable for deployment on Internet of Things devices, especially devices with limited resources.

[0045] The application not only depends on the soft label of the output layer of the teacher model for knowledge transfer, but also distills the hidden layer information of the teacher model, thereby retaining more intermediate layer knowledge. Meanwhile, the designed distillation loss module and the time correlation window loss function force the smooth transition of the hidden state at consecutive time steps, enhance the flow time series correlation, the time correlation window loss TCW strengthens the perception of covert attacks, improves the recall rate in attacks, forms a closed loop optimization with the divergence loss, not only effectively aligns the static features, but also optimizes the dynamic features in time sequence; through the knowledge distillation loss function, the knowledge of the teacher model is effectively transferred to the student model, so that the trained student model can balance between accuracy, detection speed and resource consumption. The trained student model can quickly judge whether the traffic is normal or malicious traffic, which is suitable for the real-time detection demand in the Internet of Things environment. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the illustrative embodiments of the present application and their descriptions, and do not constitute improper limitations on the present application. In the drawings:

[0047] Figure 1 The knowledge distillation-based double-teacher model Internet of Things malicious traffic detection method described in the present application is shown in the flowchart.

[0048] Figure 2 The architecture diagram of the double-teacher-student distillation model described in the present application is shown. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict, and the described embodiments are only part of the embodiments of the present application, not all the embodiments.

[0050] Embodiment one, see Figure 1 This embodiment describes a knowledge distillation-based double-teacher model Internet of Things malicious traffic detection method, which comprises the following steps:

[0051] The original Internet of Things traffic data is preprocessed to generate a standardized data set;

[0052] The standardized data set is input into a double-teacher model for training, and the double-teacher model comprises a teacher model one and a teacher model two;

[0053] The teacher model one is a spatial feature extraction model based on residual Inception structure, which contains parallel convolution branches, residual blocks and channel attention modules;

[0054] the second teacher model is a time sequence feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network;

[0055] hidden layers and soft labels of the double teacher model are extracted, and a lightweight student model is constructed based on the hidden layers and soft labels of the double teacher model, the lightweight student model being a combination of a one-dimensional deep separable convolution and a stacked bidirectional gated recurrent unit;

[0056] knowledge of the double teacher model is transferred to the student model through a knowledge distillation loss function;

[0057] the trained student model is input with to-be-detected traffic, and a classification result of normal or malicious traffic is output.

[0058] In the embodiment, a double teacher model structure is adopted, a teacher model one based on a residual Inception structure is used to extract spatial features, and a teacher model two based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network is used to extract time sequence features, which enables the double teacher model to capture spatial features and time sequence features in the Internet of Things traffic data at the same time, the teacher one based on the residual Inception structure analyzes header structure features such as protocol type and packet length distribution, and is suitable for encrypted traffic scenarios without payload data, the one-dimensional convolutional neural network and the stacked bidirectional long short-term memory network of the teacher two capture long-time attack patterns, complement the knowledge blind area of a single teacher model, and through a multi-head attention mechanism, the two results are spliced together as a final result, solving the problem that a traditional single model (such as a pure CNN or a pure LSTM) cannot process multi-dimensional features at the same time.

[0059] Since the double teacher model structure is adopted, the teacher model one is responsible for spatial information, and the other side is responsible for time sequence information, and the knowledge of the two is transferred to the student model through knowledge distillation. This way avoids the shortcomings of fixed weight distribution and lack of dynamic adaptability in static multi-teacher fusion. Therefore, the model can dynamically adjust the attention to various features according to different features of the input traffic, thereby improving the detection accuracy of malicious traffic.

[0060] By adopting a combination of a one-dimensional deep separable convolution and a stacked bidirectional gated recurrent unit as the student model, the application successfully realizes lightweight model design. The student model not only inherits the performance of the teacher model, but also significantly reduces the consumption of computing resources while maintaining high detection performance, making it suitable for deployment on Internet of Things devices, especially devices with limited resources.

[0061] The method proposed in the embodiment not only relies on the soft labels of the output layer of the teacher model for knowledge transfer, but also extracts the hidden layer information of the teacher model for distillation, thereby retaining more intermediate layer knowledge. Meanwhile, the designed distillation loss module and the time correlation window loss function force the smooth transition of the hidden state at consecutive time steps, enhance the time series correlation of the traffic, the TCW strengthens the perception of covert attacks, improves the recall rate in attacks, and forms a closed loop optimization with the divergence loss, which not only effectively aligns the static features, but also optimizes the dynamic features in time sequence; through the knowledge distillation loss function, the knowledge of the teacher model is effectively transferred to the student model, so that the trained student model can balance between accuracy, detection speed and resource consumption. The trained student model can quickly judge whether the traffic is normal or malicious traffic, which is suitable for the real-time detection demand in the Internet of Things environment.

[0062] Embodiment two, the embodiment is a further limitation of the method for detecting malicious traffic in the Internet of Things based on a double-teacher model of knowledge distillation described in embodiment one, the spatial feature extraction model based on residual Inception structure contains three parallel convolution branches: the convolution kernel size of branch one is 1×1, 3×3, 1×1, the convolution kernel size of branch two is 1×1, 5×5, 1×1, and the convolution kernel size of branch three is 3×3, 3×3, 3×3; each residual block is additionally connected by a skip connection.

[0063] In the embodiment, the three parallel convolution branches use different sizes of convolution kernels (1×1, 3×3, 5×5, etc.), which can extract spatial features at different scales. In this way, both the detailed features and the global features in the traffic data can be captured at the same time, improving the model's ability to recognize complex malicious traffic. By using multiple convolution kernel sizes in the same layer, the model can fuse different sizes of receptive fields, obtain more rich and diverse feature representations, and help improve the detection accuracy of malicious traffic.

[0064] The residual block introduces a skip connection, which enables the model to better learn the detailed information in the feature mapping and alleviates the problem of gradient vanishing or explosion in deep networks, enhancing the training effect of the network and helping to improve the training efficiency and accuracy of the model.

[0065] Embodiment three, the embodiment is a further limitation of the method for detecting malicious traffic in the Internet of Things based on a double-teacher model of knowledge distillation described in embodiment one, the knowledge of the double-teacher model is transferred to the student model through a knowledge distillation loss function, including:

[0066] The Kullback-Leibler loss is used to calculate the loss between the student model and the teacher model for generating soft labels;

[0067] The soft labels of the teacher model are transferred to the student model, and a cost-sensitive loss is used to measure the fitting error of the student model to the hard labels.

[0068] The hidden layers of the teacher model and the student model are distilled using a time correlation window loss function.

[0069] In this embodiment, the student model can effectively utilize the knowledge of the two teacher models through knowledge distillation. The student model can learn important features and patterns from the teacher models while preserving the lightweight nature, thereby improving the accuracy and generalization ability of the detection.

[0070] The Kullback-Leibler loss is used to calculate the loss between the student model and the teacher model, which can generate soft labels, providing more detailed information than traditional hard labels (i.e., original labels). Soft labels convey class information of labels, which helps the student model to learn more rich features, making the model more robust to changes in input data and effectively resisting noise and interference in malicious traffic. By introducing cost-sensitive loss, the hard label fitting error of the student model is measured, which can make the model pay more attention to the classes with higher misclassification cost during the learning process. This is particularly important in malicious traffic detection, as the detection of malicious traffic often faces the problem of class imbalance. Traditional methods may ignore the detection of low-frequency malicious traffic, while cost-sensitive loss can effectively compensate for this deficiency, thereby improving the detection accuracy.

[0071] The hidden layers of the teacher model and the student model are distilled using a time correlation window loss function, which can enhance the learning ability of the model for time series data. In IoT malicious traffic detection, traffic data is usually continuous and has certain time correlation. By considering this time correlation, the model can better capture the time series patterns in the data and improve the recognition ability of malicious traffic in time series.

[0072] Embodiment four, the embodiment is a further limitation of the knowledge distillation double teacher model IoT malicious traffic detection method of embodiment three, the time correlation window loss function is:

[0073]

[0074] wherein, is the total time step, i.e., the window length, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k, 1 represents the first teacher model, and 2 represents the second teacher model, kis the time step.

[0075] Embodiment five, the embodiment is one kind based on the further limitation of the knowledge distillation double teacher model Internet of Things malicious traffic detection method described in embodiment one, the knowledge distillation loss function satisfies:

[0076]

[0077] Wherein, is the weight coefficient of KL divergence loss term, is the weight coefficient of KL divergence loss term, and γ is the weight coefficient of time correlation window loss term, is the divergence distillation loss, is the cost-sensitive loss, time correlation window loss.

[0078] Embodiment six, the embodiment is one kind based on the further limitation of the knowledge distillation double teacher model Internet of Things malicious traffic detection method described in embodiment one, the soft label generation adopts temperature adjustment mechanism:

[0079]

[0080] Wherein, is the original logits output of the teacher model to the input x, is the temperature parameter, used to control the smoothness of the softmax output.

[0081] Embodiment seven, the embodiment is one kind based on the further limitation of the knowledge distillation double teacher model Internet of Things malicious traffic detection method described in embodiment one, the preprocessing of the original Internet of Things traffic data includes:

[0082] The correlation between features and labels is calculated by mutual information, and key features are screened;

[0083] The feature data is normalized by maximum and minimum.

[0084] Embodiment eight, the embodiment described a kind of based on the knowledge distillation double teacher model Internet of Things malicious traffic detection system, the system includes:

[0085] The preprocessing unit is used to preprocess the original Internet of Things traffic data, and generate a standardized data set;

[0086] a training unit configured to input the standardized dataset into a double-teacher model for training, the double-teacher model comprising a teacher model one and a teacher model two; the teacher model one is a spatial feature extraction model based on a residual Inception structure, comprising parallel convolution branches, residual blocks, and a channel attention module; the teacher model two is a time sequence feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network;

[0087] a lightweight student model construction unit configured to extract hidden layers and soft labels of the double-teacher model, and construct a lightweight student model based on the hidden layers and the soft labels of the double-teacher model, the lightweight student model being a combination of a one-dimensional deep separable convolution and a stacked bidirectional gated recurrent unit;

[0088] a knowledge transfer unit configured to transfer knowledge of the double-teacher model to the student model through a knowledge distillation loss function;

[0089] an output unit configured to input a to-be-detected traffic into the trained student model, and output a classification result of normal or malicious traffic.

[0090] Embodiment Nine, a computer device according to the embodiment, comprising a memory and a processor, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the method for detecting malicious traffic of Internet of Things based on knowledge distillation double-teacher model according to any one of the embodiments One to seven.

[0091] Embodiment Ten, a computer readable storage medium according to the embodiment, the computer readable storage medium stores a computer program, when the computer program is run by a processor, the steps of the method for detecting malicious traffic of Internet of Things based on knowledge distillation double-teacher model according to any one of the embodiments One to seven are executed.

[0092] Embodiment Eleven, see Figure 2 This embodiment is to provide a specific embodiment for the method for detecting malicious traffic of Internet of Things based on knowledge distillation double-teacher model according to the embodiment One, and also used to explain the embodiments Two to seven, specifically:

[0093] S1, pre-processing operation is performed on the traffic data, including data cleaning, feature extraction, data normalization, etc., and the traffic samples after the above operations are integrated into a new dataset;

[0094] Specifically, step S1 comprises:

[0095] S11, data cleaning, feature extraction and data normalization are performed according to the traffic sample set;

[0096] S12, analyze the meaning of each feature in the sample set and select the features to be retained.

[0097] Preferably, in the data cleaning step, samples containing Nan and Infinity in the sample set are deleted; in the feature extraction step, mutual information is used to calculate the correlation between each feature and the sample label to which it belongs. The specific calculation formula is as follows:

[0098]

[0099] wherein, denotes the joint probability distribution of X and Y, i.e. the feature takes and the label takes the probability when denotes the marginal probability distribution of X, i.e. the feature takes the value the probability, denotes the marginal probability distribution of Y, i.e. the label takes the value the probability, denotes the specific value of the feature , denotes the specific value of the label .

[0100] In the data normalization step, the sample set is normalized using the maximum and minimum normalization, which is beneficial to the training and convergence of the model. The specific calculation formula is as follows:

[0101] =

[0102] wherein, denotes the original value of the feature, denotes the minimum value of the feature in the data set, denotes the maximum value of the feature in the data set.

[0103] S2, input the new data set into the multi-teacher model for training, so that it learns the features of normal traffic and malicious traffic, and generates the intermediate layer hidden state and soft label of the teacher model;

[0104] The multi-teacher model comprises a teacher model one and a teacher model two;

[0105] The teacher model one is a spatial feature extraction model based on residual Inception structure (Residual Inception), which comprises parallel convolution branches, residual blocks and channel attention modules; specifically, the spatial feature extraction model based on residual Inception structure comprises three parallel convolution branches: the convolution kernel size of branch one is 1×1, 3×3, 1×1, the convolution kernel size of branch two is 1×1, 5×5, 1×1, and the convolution kernel size of branch three is 3×3, 3×3, 3×3; each residual block has one more skip connection than an ordinary residual block, which can learn more spatial features while solving the gradient vanishing problem;

[0106] The teacher model two is a time sequence feature extraction model based on one-dimensional convolutional neural network and stacked bidirectional long short-term memory network (1D-CNN-BiLSTM); specifically, the 1D-CNN layer performs convolution operation on the difference feature sequence of adjacent data packets, directly captures local time sequence features, and the BiLSTM models the inter-flow long-term dependence relationship; the obtained time and space features are fused through a multi-head attention layer to obtain fused features;

[0107] The teacher model training steps include:

[0108] For a given input, each teacher model generates a logical vector Z i = (Z1, Z2, … Z n ), and outputs a temperature-adjusted prediction distribution;

[0109]

[0110] The Kullback-Leibler loss is used to calculate the loss between the student model and the teacher model, which is used to generate soft labels;

[0111]

[0112] wherein, represents the prediction probability of the teacher model for the class , represents the prediction probability of the student model for the class , represents the i-th class, represents the prediction probability of the teacher model, represents the prediction probability of the student model, N represents the total number of classes, and i represents the index variable when summing, which traverses all classes from to N; The Adam optimizer is used for optimization, and the early stopping mechanism is used to stop training when the model no longer improves.

[0113]

[0114] ​​​S3, based on the intermediate layer hidden state and soft label of the teacher model, a lightweight student model is constructed, which is a one-dimensional depth separable convolution and a stacked bidirectional GRU structure, specifically:

[0115] The soft label of the teacher model is migrated to the student model, and the cost-sensitive loss is used to measure the fitting error of the student model to the hard label;

[0116]

[0117] wherein, represents the weight coefficient of the i-th category, represents the prediction probability of the student model for the i-th category under the temperature parameter , represents the prediction probability of the teacher model for the i-th category under the temperature parameter ,γ is an adjustment factor for controlling the loss weight of easy-to-classify samples;

[0118] The intermediate layers of the teacher model and the student model are distilled using the time correlation window loss, thereby improving the sensitivity of the student model to the change of time sequence characteristics;

[0119]

[0120] wherein, is the total time step, i.e. the window length, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k, 1 represents from the first teacher model, and 2 represents from the second teacher model, k is the time step;

[0121] The weights of the student model are updated using the distillation loss, so that the performance of the student model is improved.

[0122]

[0123] wherein, is the weight coefficient of the divergence loss term, is the weight coefficient of the cost-sensitive loss term, and γ is the weight coefficient of the time correlation window loss term, is the divergence distillation loss, is the cost-sensitive loss, is the time correlation window loss.

[0124] ​S4, inputting the to-be-detected traffic data into the trained student model for classification, and outputting a classification result of normal traffic or malicious traffic.

[0125] Those skilled in the art will understand that embodiments of the present disclosure can be provided as methods, systems or computer program products. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0126] The present disclosure is described with reference to the flowcharts and / or block diagrams according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flow Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the flow Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0128] It should be pointed out finally that the above embodiments are only used for illustrating the technical solutions of the present disclosure but not for limiting the protection scope thereof, and although the present disclosure is described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific embodiments of the present disclosure can be changed, modified or replaced equivalently by those skilled in the art after reading the present disclosure, but these changes, modifications or equivalent replacements are all within the protection scope of the disclosed claims.

Claims

1. A knowledge distillation double-teacher model-based Internet of Things malicious traffic detection method, characterized in that, The method comprises: Pretreatment of original Internet of Things traffic data to generate a standardized data set; Input the standardized data set into a double-teacher model for training, the double-teacher model comprising: a teacher model one and a teacher model two; The teacher model one is a spatial feature extraction model based on a residual Inception structure, comprising parallel convolution branches, residual blocks and channel attention modules; The teacher model two is a time sequence feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; Extract the hidden layers and soft labels of the double-teacher model, and construct a lightweight student model based on the hidden layers and soft labels of the double-teacher model, the lightweight student model being a combination of one-dimensional deep separable convolution and stacked bidirectional gated recurrent units; Migrate the knowledge of the double-teacher model to the student model through a knowledge distillation loss function; Input the traffic to be detected into the trained student model to output the classification result of normal or malicious traffic; The knowledge of the double-teacher model is migrated to the student model through a knowledge distillation loss function, comprising: Use Kullback-Leibler loss to calculate the loss between the student model and the teacher model for generating soft labels; Migrate the soft labels of the teacher model to the student model, and use cost-sensitive loss to measure the fitting error of the student model to the hard labels; Use a time correlation window loss function to distill the hidden layers of the teacher model and the student model; The time correlation window loss function is: wherein, is the total number of time steps, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k.

2. The method of claim 1, wherein the method is based on a knowledge distillation double-teacher model. The spatial feature extraction model based on the residual Inception structure comprises three parallel convolution branches: the convolution kernel size of branch one is 1x1, 3x3, 1x1, the convolution kernel size of branch two is 1x1, 5x5, 1x1, and the convolution kernel size of branch three is 3x3, 3x3, 3x3; each residual block is additionally provided with a jump connection.

3. The method of claim 1, wherein the method is based on a knowledge distillation double teacher model. The knowledge distillation loss function satisfies: wherein, is a weight coefficient of the divergence loss term, is a weight coefficient of the cost-sensitive loss term, and γ is a weight coefficient of the temporal correlation window loss term, is a divergence distillation loss, is a cost-sensitive loss, is a temporal correlation window loss.

4. The method of claim 1, wherein the method is based on a knowledge distillation double teacher model. The soft label generation adopts a temperature adjustment mechanism: wherein, is the original logits output of the teacher model for input x, is the temperature parameter.

5. The method of claim 1, wherein the method is based on a knowledge distillation double teacher model for Internet of Things (IoT) malicious traffic detection. The pretreatment of the original Internet of Things traffic data comprises: Calculate the correlation between features and labels through mutual information to screen key features; Perform maximum and minimum normalization on feature data.

6. A knowledge distillation double-teacher model-based Internet of Things malicious traffic detection system, characterized in that, The system comprises: A preprocessing unit for pretreating original Internet of Things traffic data to generate a standardized data set; A training unit for inputting the standardized data set into a double-teacher model for training, the double-teacher model comprising: a teacher model one and a teacher model two; the teacher model one is a spatial feature extraction model based on a residual Inception structure, comprising parallel convolution branches, residual blocks and channel attention modules; the teacher model two is a time sequence feature extraction model based on a one-dimensional convolutional neural network and a stacked bidirectional long short-term memory network; A lightweight student model construction unit for extracting the hidden layers and soft labels of the double-teacher model, and constructing a lightweight student model based on the hidden layers and soft labels of the double-teacher model, the lightweight student model being a combination of one-dimensional deep separable convolution and stacked bidirectional gated recurrent units; A knowledge migration unit for migrating the knowledge of the double-teacher model to the student model through a knowledge distillation loss function; An output unit is configured to input the traffic to be detected into the trained student model, and output a classification result of the normal or malicious traffic. The knowledge of the double-teacher model is migrated to the student model through the knowledge distillation loss function, including: The Kullback-Leibler loss is used to calculate the loss between the student model and the teacher model, for generating soft labels; The soft labels of the teacher model are migrated to the student model, and the cost-sensitive loss is used to measure the fitting error of the student model to the hard labels; The hidden layers of the teacher model and the student model are distilled using a time correlation window loss function; The time correlation window loss function is: wherein, is the total number of time steps, is the feature difference value of the first teacher model at time step k, is the feature difference value of the student model at time step k, is the feature difference value of the second teacher model at time step k.

7. A computer device, characterized by: The computer readable storage medium stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method of detecting malicious traffic of an Internet of Things based on a double-teacher model of knowledge distillation according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method of detecting malicious traffic of an Internet of Things based on a double-teacher model of knowledge distillation according to any one of claims 1-5.

Images