Intrusion detection method based on flow model and ensemble learning

Abstract

The application belongs to the field of intrusion detection, and relates to an intrusion detection method based on a flow model and integrated learning. The method comprises the following steps: S1) performing preprocessing on used data; S2) generating new samples for minority class samples in a data set by using an AE-Flow model; S3) performing clustering undersampling on majority class samples in the data set by using a K-means algorithm; S4) removing majority class samples near a class boundary of data samples obtained in the above steps by using an OSS algorithm; and S5) classifying the obtained data with balanced sample quantities of various classes by using an integrated learning classifier XGBoost. The application can realize high-performance intrusion detection, and compared with other methods, the application achieves better effects in precision, recall and comprehensive index F1-score in multi-classification intrusion detection, and greatly improves F1-score of minority class samples while improving the performance of an intrusion detection system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intrusion detection, and relates to an intrusion detection method based on a flow model and integrated learning. BACKGROUND

[0002] Intrusion detection technology is an important research direction in the field of information security in modern society. In today's network environment, various types of network attacks are emerging in an endless stream, and intrusion detection technology plays an important role in the task of maintaining network security. The task of network intrusion detection is to identify various abnormal flows in the network and determine which type of attack it belongs to, so as to take corresponding defense mechanisms. At present, classical machine learning classifiers such as decision tree, support vector machine, logistic regression and multilayer perception model are widely used in the classification task of network intrusion detection. However, due to the different frequencies of various types of network attacks in network operation, the number of samples of each category collected in the network intrusion record is very different, which makes the related data sets in the field of network intrusion detection have the problem of class imbalance. Because the number of samples of some rare attack categories is too small, the classifier is difficult to learn its general characteristics during training, so it is easy to misjudge the minority class samples as the majority class during classification, resulting in a low recall rate of minority class samples. To solve this problem, the existing processing method generally expands the number of minority class samples in the original data set and reduces the number of majority class samples to obtain a balanced data set, and then trains the classifier to improve the intrusion detection performance of the classifier.

[0003] The traditional method for processing unbalanced data sets is to oversample the minority class samples and undersample the majority class samples. Common undersampling methods are generally random undersampling and improvement based on various clustering algorithms, and oversampling algorithms are generally various improvements around the SMOTE (Synthetic Minority Oversampling Technique) algorithm. However, SMOTE is a simple interpolation sampling method, which is easy to generate redundant samples and noise samples, affecting the classification performance. With the continuous development of deep learning technology, many researchers choose to apply the generative model in deep learning to the field of generating new samples.

[0004] The generative model uses its powerful learning ability to learn the probability distribution of the sample data through training, and generates new samples conforming to the distribution. The generated new samples have randomness and diversity, which can effectively expand the number and sample states of the original data, so that the classifier can better learn the characteristics of different class sample data. At present, the deep generative model, such as generative adversarial network (GAN) and variational auto-encoder (VAE), has been widely applied in network intrusion detection. Another school of generative model, flow-based generative model, has not been widely applied due to its large model structure and long training time. However, since it uses a reversible neural network to establish the probability mapping relationship between the real sample and the generated sample, and the relationship is deterministic and one-to-one, the quality of the generated new sample is often higher than that of GAN and VAE, so it is an effective data enhancement method to apply the flow model to the generation of network intrusion data new samples.

[0005] Since each machine learning classifier applies different algorithms, the classification performance in different application scenarios is different, so researchers choose the ensemble learning method to integrate the advantages of each basic classifier to obtain higher classification performance. For example, the random forest model which combines multiple decision tree classifiers in parallel training mode, and the XGBoost model which combines multiple weak classifiers into a strong classifier through serial training mode, are widely used in intrusion detection systems.

[0006] Therefore, it is of great significance to design an intrusion detection method based on flow model and ensemble learning to improve the performance of the intrusion detection system. SUMMARY

[0007] Therefore, the purpose of the present application is to provide an intrusion detection method based on flow model and ensemble learning. The goal of the generative model is to fit the probability distribution of the real sample using the known probability model. However, the probability distribution of the real sample is usually complex, and the flow model can convert simple distribution to complex distribution through a series of reversible transformations to infinitely fit the distribution of the real sample. Because a single-step reversible transformation cannot achieve very strong nonlinearity, and the entire model needs very strong nonlinearity to complete the mapping from the sample probability space to the hidden variable probability space, the flow model needs many steps of reversible transformation coupled together.

[0008] Because the flow model needs to be coupled by multi-step reversible transformation to achieve strong nonlinearity, complete the mapping from the sample space to the known probability distribution state of the hidden variable space, so the flow model is usually large, and the training time is also long. Network intrusion data is usually high-dimensional data containing multiple attributes, and directly using the flow model to generate data from the original network intrusion data will result in a large model structure and difficulty in training, so we need to improve the flow model. Because each step of transformation of the flow model is reversible, the dimension of the output hidden variable and the original sample is the same. If the low-dimensional features extracted from the original sample are used as the input of the flow model instead of the original sample itself, because the input and output dimensions are consistent before and after the reversible transformation, the task of the entire flow model has changed from the previous mapping from the high-dimensional sample space to the high-dimensional hidden variable space to the current mapping from the low-dimensional feature space to the low-dimensional hidden variable space. Compared before and after, the task of the entire model is much easier after using low-dimensional sample features as the input of the flow model, so the flow model can be simplified. The unsupervised learning model of automatic encoder in deep learning is widely used in feature extraction and data dimension reduction, and can reconstruct the original sample from the feature vector after dimension reduction. Therefore, the present application proposes to combine the flow model with the automatic encoder to realize efficient generation of new sample data, and obtain a new generation model Autoencoder-Flow (AE-Flow).

[0009] In addition to generating new minority class samples through the AE-Flow model to increase the number of minority class samples, the present application also removes redundant samples and noise samples in the majority class samples in the data set. For the majority class samples, the present application uses the K-means algorithm for clustering, determines the sampling proportion from the sparsity of the samples in each cluster after clustering, extracts the sample representative points to achieve the purpose of removing redundant samples. Then the OneSidedSelection (OSS) algorithm is applied to remove the majority class noise samples near the boundary of the minority class samples and the majority class samples which may interfere with the judgment of the minority class samples, and finally realize the goal of balancing the number distribution of each class of samples in the data set.

[0010] Therefore, the present application proposes an intrusion detection method based on the flow model and ensemble learning, which applies the AE-Flow model improved by Flow to generate minority class samples in the network intrusion detection data set, and combines the K-means and OSS undersampling algorithms to achieve the purpose of balancing the data set, and finally trains the XGBoost classifier on the balanced data set to realize high-performance intrusion detection. The method comprises the following steps:

[0011] Step 1) Preprocess the data used;

[0012] Step 2) generating new samples for the minority class samples in the data set using the AE-Flow model;

[0013] Step 3) clustering undersampling for the majority class samples in the data set using the K-means algorithm;

[0014] Step 4) merging the data samples obtained in the above steps, and removing the majority class samples near the class boundary using the OSS algorithm;

[0015] Step 5) applying the integrated learning classifier XGBoost to the obtained data with balanced number of samples in each class for classification.

[0016] Further, the step 1) specifically comprises the following steps:

[0017] Step 11) Network intrusion data usually have character features, which are inconvenient to input into the model for operation, so it is necessary to encode them into numerical values. For example, network data usually have protocol features protocol, which have three attribute values: "TCP", "UDP", and the attribute values are encoded as '0', '1', and '2' by using LabelEncoder label encoding. In this way, the size of the numerical value can be used to distinguish different attribute values, and it is convenient to input into the model for operation.

[0018] Step 12) Each dimension feature of network intrusion data has a different value range. If these value ranges can be scaled to around 0 in proportion and have a variance of 1, the feature attribute values of each data will be greatly reduced, and the operation speed in the model will be greatly improved. The present application selects the Z-score standardization method, and the formula is:

[0019]

[0020] x * is the transformed output value, x is the original data of each dimension attribute value, u and sigma are the mean and variance of the original data of each dimension attribute value.

[0021] Further, the step 2) specifically comprises the following steps:

[0022] Step 21) sending the preprocessed data x into the encoder module E of the autoencoder, and outputting the reduced dimension feature vector z, that is, z=E(x). Wherein E represents a 3-layer convolutional neural network, and the convolution kernel size is 3*3, which is used for feature extraction of high-dimensional network intrusion data to obtain abstract features after dimension reduction, so as to send the data into the flow model for data mapping.

[0023] Step 22) the feature vector z is passed through the flow model F, and the output is the hidden variable h conforming to the Gaussian distribution, that is, h = F(z). Where F represents a series of reversible transformations: F = f1*f2*…*f n Each transformation f i is implemented through a convolutional neural network (linear transformation) and a Relu activation function (nonlinear transformation). The present application uses 24 layers of such reversible transformations to implement the function of the flow model. The flow model F maps the probability space of the sample to the probability space of the hidden variable by performing multiple reversible mathematical transformations on the input data, so that the hidden variable finally becomes a known probability distribution type (generally Gaussian distribution). Since each step of transformation in the flow model is reversible, when the model training is completed, a new sample can be generated by randomly sampling from the hidden variable space and performing multiple inverse transformations. The mapping process from x to z is as follows:

[0024]

[0025] Step 23) a sample h' is randomly sampled from the hidden variable space, and the inverse transformation F -1 of the flow model is performed on h', and the feature vector z' conforming to the original sample distribution is output, that is, z' = F -1 (h'). Where F -1 represents the inverse transformation of the flow model in step 22.

[0026] Step 24) the feature vector z' obtained in step 23 is passed through the decoder module of the autoencoder, and the reconstructed new network intrusion data x' is output, that is, x' = D(z'). Where D represents a transposed convolutional neural network with the same structure as the encoder module E. The transposed convolutional neural network can convert the low-dimensional feature vector z' into a high-dimensional network intrusion sample through deconvolution and upsampling operations. Thus, the task of generating new sample data based on the AE-Flow model is completed.

[0027] Further, the step 3) specifically comprises the following steps:

[0028] Step 31) use the K-means algorithm to cluster the majority class samples in the network intrusion data set. The K-means algorithm divides the samples into K different clusters based on the Euclidean distance between the samples, so that the sample points in the cluster are as compact as possible, and the distance between the clusters is as large as possible. The algorithm minimizes the squared error E through multiple iterations, and the expression is as follows:

[0029]

[0030] Where C i is the cluster obtained by clustering, and u i is the mean vector of the cluster, also known as the center of the cluster, and the expression is as follows:

[0031] Step 32) Determine the proportion of samples to be sampled from each cluster according to the density of the clusters after clustering. If the density of a certain sample cluster after clustering is high, it means that there are many similar data in the samples in this cluster, and a large number of redundant samples need to be removed. Conversely, if the density of a certain sample cluster after clustering is low, it means that there are fewer similar data in the samples in this cluster, and we need to retain the samples in the entire cluster.

[0032] Further, the step 4) specifically includes the following steps:

[0033] Step 41) The OSS algorithm removes noise samples in the majority class. In addition to removing redundant majority class samples, we also need to pay attention to the majority class samples near the boundary between the minority class and the majority class. Because they are more adjacent to the minority class samples, they can easily interfere with the judgment of the minority class during classification, and are considered noise for the minority class, so they need to be removed. The OSS algorithm finds the nearest neighbor sample for each minority class sample near the class boundary, and if its nearest neighbor sample belongs to the majority class, it is deleted. Applying the OSS algorithm can obtain a data set with a clear class boundary, effectively removing majority class samples that can interfere with the judgment of the minority class.

[0034] Further, the step 5) specifically includes the following steps:

[0035] Step 51) Use the classifier to verify the effectiveness of the above balanced data set method. The above steps can obtain a balanced data set with balanced class quantity ratio and clear class boundary. We use the original training set and the new balanced training set to train the classifier XGBoost, and then compare the classification performance of the classifier on the test set. XGBoost is an algorithm that improves weak learners to strong learners. The algorithm first trains a base learner from the initial training set, then adjusts the training sample distribution based on the performance of the base learner, so that the training samples previously misjudged by the base learner are given more attention in the subsequent training, and then the next base learner is trained based on the adjusted sample distribution. Repeat this process until the number of base learners reaches the specified number, and finally combine these base learners to obtain a strong learner.

[0036] Beneficial effects:

[0037] 1. The proposed generation model AE-Flow can simplify the structure of the original flow model, reduce the training time, and generate high-quality samples to effectively expand the number and sample states of the minority class samples in the data set.

[0038] 2. Using the K-means algorithm, the sample representative points of the samples with high similarity in the cluster of different states of the majority class samples can be extracted, and the effect of effectively removing redundant samples can be achieved.

[0039] 3. Using the OSS algorithm, the majority class samples near the boundaries of each category can be deleted to avoid the interference of other category samples.

[0040] 4. The balance processing of the original data set is completed from the perspective of the imbalance of network intrusion data categories, and the intrusion detection system constructed based on this has high performance in various classification detection indicators. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to express the purpose, technical scheme and advantages of the present application more clearly, the present application will be described in more detail below in combination with the drawings, in which:

[0042] Figure 1 is an Autoencoder structure schematic diagram

[0043] Figure 2 is an AE-Flow model training process schematic diagram

[0044] Figure 3 is an AE-Flow model new sample generation process schematic diagram

[0045] Figure 4 is an overall structure diagram of the intrusion detection model based on the autoencoder and flow model DETAILED DESCRIPTION

[0046] The present application provides an intrusion detection method based on space-time features and attention mechanisms, as shown in Figure 1 The Autoencoder maps the input sample X to the feature space through the Encoder to obtain the feature vector Z, and then maps the feature

[0047] vector Z back to the original sample space through the Decoder to obtain the reconstructed sample The loss function is the reconstruction error between the reconstructed sample and the original sample During model training, the Encoder and the Decoder are optimized by minimizing the reconstruction error, so as to learn the dimension-reduced feature representation Z of the input sample X.

[0048] As shown in Figure 2 , the sample x is input into the AE-Flow, first passes through the encoder network to become the dimension-reduced feature vector z, and then z is transformed into the hidden variable h of the known probability distribution of the same dimension through the Flow model, and at the same time, the decoder network is transformed into the reconstructed sample x'. The loss function of the AE part from x to x' is:

[0049] L AE = ||x - x' || 2

[0050] = ||x - D(z) || 2

[0051] = ||x - D(E(x)) || 2

[0052] In the formula, z represents the features of the sample after encoding, D represents the encoder network, and E represents the decoder network.

[0053] The task of the flow model is to constantly update the parameters of a series of reversible transformations in the flow model network according to the input feature vector z through training, so as to maximize the likelihood probability p θ (z) of z. That is:

[0054]

[0055] The loss function of the flow model is:

[0056]

[0057] In the formula, h represents the hidden variable conforming to the Gaussian distribution, and F represents the flow model network.

[0058] The model gradually reduces the loss function through training to maximize logp θ (z). However, the calculation cost of the Jacobian matrix determinant in the formula may be high, and if the Jacobian matrix can be designed to be an upper triangular matrix or a lower triangular matrix through transformation, its determinant can be easily calculated as the product of each item on its diagonal. The input real sample x can be divided into two parts through the design of an additive coupling layer, and then the following transformation is performed:

[0059] y1 = x1,

[0060] y2 = x1 + g(x2)

[0061] In the formula, g represents a single-step reversible transformation.

[0062] This makes the Jacobian matrix a triangular matrix, and the diagonal elements are all 1. The entire transformation is also reversible, and its inverse transformation is:

[0063] x1 = y1

[0064] x2 = y2 - g(y1)

[0065] At this point, the two requirements of the flow model: reversible transformation and easy calculation of the determinant of the corresponding Jacobian matrix are solved through this design.

[0066] The loss function of the entire AE-Flow model is:

[0067]

[0068] The AE and Flow are simultaneously optimized by minimizing the loss function L AE-Flow during the model training.

[0069] As shown in Figure 3 , when the AE-Flow training is completed, a noise sample h' can be randomly sampled from the hidden variable space of the known probability distribution, and after a series of inverse transformations F -1 of the Flow model, a low-dimensional feature z' conforming to the original sample probability distribution is obtained, and the feature is reconstructed by the decoder to become a new sample x' conforming to the original sample probability distribution.

[0070] As shown in Figure 4 , the specific process of the entire network intrusion detection includes the following steps:

[0071] Step 1: Since the network data is high-dimensional linear data, if it is directly used as a neural network model, data preprocessing module needs to be used for data preprocessing of the original data. First, the symbolic features in the data need to be digitized, and then the data needs to be standardized.

[0072] Step 2: According to the network structure characteristics of the AE-Flow model, one-dimensional linear data is converted into matrix data for input. NSL-KDD data set is used for experiment. The original data size has 41 features, and after data preprocessing and deletion of unnecessary features, the data dimension is 40, which is converted into a two-dimensional matrix with a size of 5*8 as the input of the network.

[0073] Step 3: The few-class samples in the original training set are input into the AE-Flow model for training. After the training is completed, a sample can be randomly sampled in the hidden variable space of the known probability distribution, and the new sample conforming to the original few-class sample distribution is generated through the transformation of AE-Flow.

[0074] Step 4: For the majority-class samples in the original data set, the K-means algorithm is applied for clustering, the representative points in the cluster are extracted to remove the redundant samples, and then the OSS algorithm is used to remove the majority-class noise samples near the boundary of the few-class samples.

[0075] Step 5: The original training set and the balanced new training set are used to train the classifier respectively, and then the classification performance of the classifier on the test set is compared.

[0076] The above description is that of current embodiments of the application, and is not intended to limit the scope, applicability or configuration of the application in any way. Many modifications, variations and adaptations will occur to those skilled in the art in the practice of the present application, including variations and substitutions of known equivalents. The scope of the application is not to be limited to what is described in the specification or shown in the figures.

Claims

1. An intrusion detection method based on flow model and ensemble learning, characterized in that, The method comprises the following steps: S1) generating new samples for the minority class samples in the data set using an AE-Flow model, wherein the AE-Flow model comprises: an encoder module E composed of a 3-layer convolutional neural network with a convolution kernel size of 3*3; a flow model F composed of 24 reversible transformations, each transformation comprising a convolutional neural network and a Relu activation function; a decoder module D which is a transposed convolutional neural network symmetrical to the encoder module E; S2) performing cluster undersampling on the majority class samples in the data set using a K-means algorithm; S3) merging the data samples obtained in the above steps and removing the majority class samples near the class boundary using an OSS algorithm; S4) applying an integrated learning classifier XGBoost to the obtained data with balanced class sample quantities for classification. 2.The flow model and ensemble learning based intrusion detection method of claim 1, wherein: In the step S1), the generation of new samples using the AE-Flow model comprises the following steps: In step S1), the sample x is input into the AE-Flow, first passing through the encoder network to become a reduced-dimensional feature vector z, and then z is transformed into a hidden variable h with the same dimension and a known probability distribution through the flow model, and is transformed into a reconstructed sample x' through the decoder network, wherein the loss function of the AE part from x to x' is: L AE =‖x-x′‖ 2 =‖x-D(z)‖ 2 =‖x-D(E(x))‖ 2 In the formula, z represents the features of the sample after encoding, E represents the decoder network which includes a 3-layer convolutional neural network with a convolution kernel size of 3*3, and the function is to extract abstract features from high-dimensional network intrusion data to facilitate data mapping in the flow model, and D represents the encoder network which includes a transposed convolutional neural network with the same structure as the encoder module E, and the transposed convolutional neural network can transform the low-dimensional feature vector z' into a high-dimensional network intrusion sample through inverse convolution and upsampling operation, wherein the loss function of the flow model is: In the formula, h represents the hidden variable conforming to the Gaussian distribution, F represents the encoder network, which represents a series of reversible transformations: F = f1*f2* *f n Each transformation f i is implemented through a convolutional neural network (linear transformation) and a Relu activation function (nonlinear transformation), the flow model F is composed of 24 reversible transformations, and the loss function of the entire AE-Flow model is: The model is trained by minimizing the loss function L AE-Flow to simultaneously optimize the AE and the Flow. When the training is complete, a noise sample can be randomly sampled from the latent variable space of the known probability distribution, and a series of inverse transformations F -1 of the Flow model are applied to the noise sample to obtain low-dimensional features that conform to the probability distribution of the original sample. The features are then reconstructed by the decoder to become new samples that conform to the probability distribution of the original sample. 3.The flow model and ensemble learning based intrusion detection method of claim 1, wherein: In the step S2), the cluster undersampling on the majority class samples in the data set using the K-means algorithm comprises the following steps: In step S2), in addition to expanding the number of minority class samples, we also remove redundant majority class samples to avoid the final bias of the classifier towards the majority class due to too many majority class samples, so we use the K-means algorithm to cluster the majority class samples, the K-means algorithm divides the samples into K different clusters based on the Euclidean distance between the samples, so that the sample points within the cluster are as compact as possible, and the distance between the clusters is as large as possible, and the algorithm minimizes the squared error E through multiple iterations, and the expression is: where C i is the cluster obtained by clustering, u i is the mean vector of the cluster, also called the center of the cluster, and the expression is: According to the density of each cluster after clustering, the proportion of sampling is determined, that is, if the density of a certain sample cluster after clustering is very high, it means that there are many similar data in the samples in this cluster, and a large number of redundant samples need to be removed, otherwise if the density of a certain sample cluster after clustering is very low, it means that there are fewer similar data in the samples in this cluster, and we need to keep the samples in the entire cluster. 4.The flow model and ensemble learning based intrusion detection method of claim 1, wherein: In the step S3), the removal of the majority class samples near the class boundary using the OSS algorithm comprises the following contents: Step S3) In addition to removing redundant majority class samples, we also need to focus on the majority class samples near the boundary between the minority class and the majority class, because they are more adjacent to the minority class samples and are more likely to interfere with the judgment of the minority class during classification. They are noise for the minority class and need to be removed. The OSS algorithm finds the nearest neighbor sample for each minority class sample near the class boundary. If its nearest neighbor sample belongs to the majority class, it is removed. Applying the OSS algorithm can obtain a data set with a clear class boundary, effectively removing majority class samples that interfere with the judgment of the minority class. 5.The flow model and ensemble learning based intrusion detection method of claim 1, wherein: In step S4, applying the integrated learning classifier XGBoost to the obtained data set with balanced class sample quantities for classification includes the following content: Step S4) After the above steps, a balanced data set with balanced class quantities and clear class boundaries can be obtained. We use the original training set and the balanced new training set to train the classifier XGBoost, and then compare the classification performance of the classifier on the test set.

Images