A High-Efficiency Intrusion Detection Method Based on Heterogeneous Datasets

Abstract

The present invention relates to a high-efficiency intrusion detection method based on heterogeneous datasets, belonging to the technical field of network security intrusion detection. This method utilizes the sensitivity of convolutional neural networks to image-formatted data. First, the data is preprocessed. A training dataset is constructed by heterogeneously recombining abnormal and normal data to generate image-formatted data. The generated dataset is then trained using a pre-built convolutional model to achieve efficient data detection. The trained model is then retrained through incremental learning to continuously update the detection model.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security intrusion detection, and in particular relates to a high-efficiency intrusion detection method based on heterogeneous data sets. Background Art

[0002] With the continuous advancement of information technology, various network attack methods and means are constantly emerging. Intrusion detection technology plays a vital role in ensuring network security. Currently, the main challenges faced by intrusion detection technology lie in the huge amount of data, the imbalance between abnormal data and safe data, and how to achieve real-time data detection. Therefore, it is extremely necessary to develop an intrusion detection technology that can effectively address this data imbalance and achieve efficient data detection. Summary of the Invention

[0003] (1) Technical issues to be solved

[0004] The technical problem to be solved by the present invention is: how to design an intrusion detection method that can effectively solve the problem of data imbalance and can achieve high-efficiency detection of data.

[0005] (2) Technical solution

[0006] In order to solve the above technical problems, the present invention provides a high-efficiency intrusion detection method based on heterogeneous data sets, comprising the following steps:

[0007] Step 1, data preprocessing:

[0008] Collect network data, and according to the feature types in the data, digitize and normalize the acquired data features into binary features to generate data in 1×n image format;

[0009] Step 2: Build a heterogeneous dataset:

[0010] Obtain S pieces of normal data from the 1×n image format data generated in step 1, and randomly extract n pieces of data from them to merge into n×n image format data. Repeat the extraction a times to generate a normal data set A with a total amount of data a.

[0011] Obtain Q pieces of abnormal data from the 1×n image format data generated in step 1, and randomly extract k pieces of 1-dimensional data from them, where k is a random number and k< <n;

[0012] Extract data A1, A2, ...Aa from A in sequence, and replace k random 1-dimensional data in the extracted data A1, A2, ...Aa with k 1-dimensional data. Repeat the replacement step to generate an abnormal dataset B in n × n image format with a total data volume of a.

[0013] The normal dataset A and the abnormal dataset B are mixed to generate a new dataset AB, which is used as the training dataset for the subsequent neural network model;

[0014] Step 3: Build a convolutional neural network model and set network parameters:

[0015] Build a convolutional neural network model and set the network parameters of the convolutional neural network model for n×n image format data;

[0016] Step 4: Set the training parameters and train the convolutional neural network model:

[0017] Set the hyperparameters;

[0018] Use dataset AB to train the built convolutional neural network model to ensure that the model performance reaches the predetermined target;

[0019] Use the trained convolutional neural network model to detect and classify unknown data;

[0020] The data classified as abnormal samples are re-divided into n pieces of data in 1×n image format, and the preset intrusion detection model is used to detect these n pieces of data.

[0021] Preferably, step 4 is followed by step 5, in which the convolutional neural network model is updated by using incremental learning: the new data obtained from the cyberspace is processed by steps 1 and 2 to obtain a heterogeneous data set C, and C is mixed with part of the data obtained from data sets A and B to retrain the convolutional neural network model.

[0022] Preferably, step 5 further includes a step of using the updated convolutional neural network model to detect and classify unknown data.

[0023] Preferably, in step 1, the numerical features in the data features are converted into binary features after Min-Max normalization processing, and the symbolic data in the data features are first digitized and then normalized to be converted into binary features.

[0024] Preferably, in step 1, network data is collected from the NSL-KDD dataset.

[0025] Preferably, in step 2, the sum of the dimensions of the features is calculated, and the obtained sum of the dimensions is set to n, thereby obtaining S pieces of normal data in a 1×n image format and Q pieces of abnormal data in a 1×n image format.

[0026] Preferably, in step 3, the structure of the constructed convolutional neural network model includes, in sequence: input data, convolution layer, pooling layer, fully connected layer, and output data, wherein the model loss function selects the cross entropy loss function, and the output is a binary classification, that is, determining whether the data is abnormal data or not.

[0027] Preferably, the hyperparameters are set by selecting appropriate training period epoch, batch size batchsize, and learning rate η.

[0028] Preferably, the training cycle epoch is set to 200, the batch size is set to 16, and the learning rate η is set to 10 -4 .

[0029] The present invention also provides a system for implementing the method.

[0030] (3) Beneficial effects

[0031] The present invention greatly reduces the impact of data imbalance on the basis of ensuring the data volume of the data set, and can generate a large amount of image format data from a small amount of data.

[0032] By combining n data points into an n×n image format, this method leverages the superiority of convolutional neural networks in processing image data, enabling batch detection and meeting the real-time requirements of intrusion detection systems. Compared to image data composed of a single data point, this method generates image data with greater dimensions and richer information, enabling the construction of wider and deeper convolutional neural networks, thus improving the performance of intrusion detection models.

[0033] The present invention uses incremental learning to retrain a trained intrusion detection model using a small amount of new abnormal data, thereby achieving uninterrupted updating of the intrusion detection model and avoiding the tedious process of retraining the intrusion detection model due to updates in attack methods and approaches. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the method of the present invention;

[0035] Figure 2 It is a schematic diagram of generating an image format data set of the present invention;

[0036] Figure 3 It is a schematic diagram of the convolutional neural network structure of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0038] The present invention proposes a high-efficiency intrusion detection method based on heterogeneous data sets. By utilizing the sensitivity of convolutional neural networks to image format data, the data is first preprocessed, and then a training data set is constructed by heterogeneously recombining abnormal data and normal data to generate image format data. The generated data set is then trained with the constructed convolutional model to achieve efficient detection of the data. The trained model is then retrained through incremental learning to achieve continuous updating of the detection model.

[0039] like Figure 1 As shown, the overall process of the present invention includes the following steps:

[0040] Step 1, data preprocessing:

[0041] Data is collected from the latest public datasets or actual network environments. Symbolic and numerical features in the data are uniformly digitized and normalized into binary features, generating a large amount of 1×n data. The conversion method in this example is: numerical features are normalized using Min-Max processing and then converted into binary features. Symbolic data is first digitized and then normalized.

[0042] Step 2: Build a heterogeneous dataset:

[0043] The sum of the feature dimensions is calculated and set to n, thereby obtaining S pieces of normal data in a 1×n image format and Q pieces of abnormal data in a 1×n image format. In the present invention, after the feature conversion, the data dimensions of common data are generally within the range of several hundred, which does not cause the problem of n being too large. In this embodiment, the NSL-KDD dataset is used as an example. The data contains 41 features. After converting the symbolic and numerical features into binary features, the dimension of each piece of data is 454, from which a large amount of normal data and abnormal data is obtained.

[0044] Randomly extract n pieces of data from the obtained S pieces of normal data and merge them into n×n image format data. Repeat this step a times to generate a normal data set A with a total of a. In this example, n is 454, and it will be 454 when it is specified below.

[0045] Calculation basis: For the convenience of calculation, assuming that S is 10000 and n is 100, the maximum amount of data generated when data can be repeatedly extracted is

[0046] When no repeated extraction is performed, the amount of data generated is much greater than in This is the possible situation calculated when selecting 100 pieces of data from 10,000 pieces of data. 100! is the number of possible situations for the random arrangement of the selected 100 pieces of data; when sampling without replacement, 100 pieces of data can be sampled 100 times from 10,000 pieces of data, which far exceeds the possible situation of sampling once. Therefore, the amount of data generated by this method far exceeds the amount required to construct the dataset.

[0047] Randomly select k one-dimensional data from Q pieces of abnormal data. k is a random number and k << n. The specific value of k and the proportion of k = 1 data can be adjusted according to the distribution of abnormal data and normal data in the actual data. In this example, k ≤ 3 is selected, and k = 1, k = 2, and k = 3 data proportions are 80%, 15%, and 5% respectively.

[0048] Calculation basis: For the convenience of calculation, assume that the probability that a piece of data to be detected is normal data is 99.9%, and the probability of abnormal data is 0.1% (the actual situation may be different from this ratio). The probability that 454 pieces of data do not contain abnormal data is about 63.5%, and the probability of containing one piece of abnormal data is about 28.8%. Then the probability of containing 2 or more pieces is about 7.7%. It can be calculated that the proportion of the situation where 454 pieces of data contain one piece of abnormal data in abnormal situations is about 78.9%. Therefore, in this example, the data proportions of k = 1, 2, and 3 are set to 80%, 15%, and 5% in turn.

[0049] Successively extract data A1, A2... Aa from A, and replace k random one-dimensional data in the extracted data A1, A2... Aa with k one-dimensional data. The specific operation process is shown in Figure 2 Repeat this step to generate an abnormal dataset B in the n×n image format with a total data volume of a.

[0050] Mix dataset A and dataset B to generate a new dataset AB, which is used as the training dataset for the subsequent neural network model.

[0051] In common detection methods, the data volume ratio of normal data and abnormal data is very large, at the level of dozens to 1 or even hundreds to 1. In this case, the abnormal data is too few, resulting in unbalanced training data, which affects the recognition accuracy of the model and causes many false negative phenomena. For example, assume that a dataset has 100 pieces of data, including 99 normal data and 1 abnormal data. Then, even if the trained model cannot recognize the abnormal data, the recognition success rate of this model is 99%, and the effect of training the model is poor. Therefore, the present invention constructs an abnormal dataset B with the same total amount of data as the normal data. The dataset generated by this method can greatly reduce the data imbalance problem caused by the large difference in the data volume between normal data and abnormal data, thereby improving the effect of training the model.

[0052] Step 3: Build a convolutional neural network model and set network parameters:

[0053] Build a model based on convolutional neural network, see Figure 3 The input data size is n×n image format data. The structure is as follows: input data, convolution layer, pooling layer, fully connected layer, and output data. The cross-entropy loss function is used as the loss function, and the output is a binary classification, that is, determining whether the data is anomaly or not. In this example, the number of convolutional and pooling layers can be set according to actual conditions. Set the network parameters of the convolutional neural network model for n×n image format data.

[0054] Step 4: Test the data:

[0055] Set the hyperparameters and select appropriate training cycle epoch, batch size, learning rate η, etc. In this example, the training cycle epoch is set to 200, the batch size is set to 16, and the learning rate η is set to 10 -4 , and adjust as appropriate during training. Use datasets AB to train and optimize the built convolutional neural network model so that its model performance reaches the predetermined target.

[0056] Use the trained convolutional neural network model to detect and classify unknown data, and re-divide the data classified as abnormal samples (assuming the number is x) into n 1×n data. Use the existing intrusion detection model (not the model trained this time, but an existing model or a public intrusion detection model, etc.) to detect the n data.

[0057] In this case, assuming that the data in the original 1×n image format is n*n pieces of data (composed of n pieces of data in the original 1×n image format), the traditional detection method (the detected data are all data in the 1×n image format) requires identification n*n times, and the method of the present invention only needs to identify n*x times. Because in real situations, the number of normal data is much larger than the number of abnormal data, so x is much smaller than n, thereby realizing batch detection of data.

[0058] Step 5: Retrain the model using incremental learning:

[0059] The new data obtained from the cyberspace is processed through steps 1 and 2 to obtain a heterogeneous dataset C. C is mixed with part of the data obtained from datasets A and B and the model is retrained. This allows the model to learn new knowledge while retaining a large amount of original learning knowledge, so that the updated convolutional neural network model can be used to detect and classify unknown data in the future.

[0060] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A high-efficiency intrusion detection method based on heterogeneous data sets, characterized in that: The following steps are involved: Step 1, data preprocessing: Collect network data, and according to the feature types in the data, digitize and normalize the acquired data features into binary features to generate data in 1×n image format; Step 2: Build a heterogeneous dataset: Obtain S pieces of normal data from the 1×n image format data generated in step 1, and randomly extract n pieces of data from them to merge into n×n image format data. Repeat the extraction a times to generate a normal data set A with a total amount of data a. Obtain Q pieces of abnormal data from the 1×n image format data generated in step 1, and randomly extract k pieces of 1-dimensional data from them, where k is a random number and k< <n; Extract data A1, A2, ...Aa from A in sequence, and replace k random 1-dimensional data in the extracted data A1, A2, ...Aa with k 1-dimensional data. Repeat the replacement step to generate an abnormal dataset B in n × n image format with a total data volume of a. The normal dataset A and the abnormal dataset B are mixed to generate a new dataset AB, which is used as the training dataset for the subsequent neural network model; Step 3: Build a convolutional neural network model and set network parameters: Build a convolutional neural network model and set the network parameters of the convolutional neural network model for n×n image format data; Step 4: Set the training parameters and train the convolutional neural network model: Set the hyperparameters; Use dataset AB to train the built convolutional neural network model to ensure that the model performance reaches the predetermined target; Use the trained convolutional neural network model to detect and classify unknown data; The data classified as abnormal samples are re-divided into n pieces of data in 1×n image format, and the preset intrusion detection model is used to detect these n pieces of data.

2. The method according to claim 1, wherein Step 4 is followed by step 5, in which the convolutional neural network model is updated using incremental learning: the new data obtained from the cyberspace is processed through steps 1 and 2 to obtain a heterogeneous dataset C, and C is mixed with part of the data obtained from datasets A and B to retrain the convolutional neural network model.

3. The method according to claim 2, wherein After step 5, the method also includes the step of using the updated convolutional neural network model to detect and classify unknown data.

4. The method according to claim 1, wherein In step 1, the numerical features in the data features are converted into binary features after Min-Max normalization processing, and the symbolic data in the data features are first digitized and then normalized to convert into binary features.

5. The method according to claim 1, wherein In step 1, network data is collected from the NSL-KDD dataset.

6. The method according to claim 1, wherein In step 2, the sum of the dimensions of the features is calculated, and the obtained sum of the dimensions is set to n, thereby obtaining S pieces of normal data in the 1×n image format and Q pieces of abnormal data in the 1×n image format.

7. The method according to claim 1, wherein In step 3, the structure of the constructed convolutional neural network model includes: input data, convolution layer, pooling layer, fully connected layer, and output data in sequence. The model loss function selects the cross entropy loss function, and the output is a binary classification, that is, determining whether the data is abnormal data or not.

8. The method according to claim 1, wherein Setting the hyperparameters specifically involves selecting the appropriate training cycle epoch, batch size batchsize, and learning rate η.

9. The method according to claim 8, wherein Set the training cycle epoch to 200, the batch size batchsize to 16, and the learning rate η to 10 -4 .

10. A system for implementing the method according to any one of claims 1 to 9.

Images