Network intrusion detection method based on deep learning
By calculating the importance of network intrusion characteristics and obtaining associated data for fusing, the sample set is constructed, the data imbalance problem in network intrusion detection is solved, the detection accuracy and efficiency are improved, and the authenticity of the data and the stability of the model are ensured.
Patent Information
- Application Number
- CN202410124415.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-29
- Publication Date
- 2024-11-15
AI Technical Summary
The data imbalance problem in the field of network intrusion detection leads to overfitting or underfitting of deep learning models during training, affecting model performance.
By obtaining network intrusion data, extracting and calculating feature importance, determining the target data volume, obtaining associated data for data fusion, building sample sets and training deep learning models, solving data imbalance problem, and improving detection accuracy.
Effectively process unbalanced data, improve the accuracy and efficiency of network intrusion detection, ensure data authenticity, and enhance the stability and robustness of the model.
Smart Images

Figure CN118972084A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security technology, and in particular to a network intrusion detection method based on deep learning. Background Art
[0002] With the popularization of the Internet and the acceleration of the digitalization process, network security issues are becoming increasingly prominent. As an important means of security protection, network intrusion detection can effectively detect and prevent various network attacks and protect the security of information systems. Traditional network intrusion detection methods are mainly based on rule matching and statistical analysis, but these methods often have problems with high rates of missed reports and false positives when facing complex and changeable network attacks.
[0003] In recent years, with the rapid development of artificial intelligence and deep learning technology, network intrusion detection methods based on deep learning have gradually become a research hotspot. Deep learning technology can automatically learn useful feature representations from a large amount of data, and has powerful classification and recognition capabilities. Therefore, it is widely used in image recognition, speech recognition, natural language processing and other fields. In the field of network intrusion detection, deep learning technology can also play its advantages. By automatically extracting features from network traffic data and performing classification and recognition, it can effectively improve the accuracy and efficiency of detection.
[0004] However, data in the field of network intrusion detection often suffers from imbalance, that is, normal traffic data is much larger than abnormal traffic data (intrusion data). This data imbalance can cause overfitting or underfitting of deep learning models during training, thus affecting the performance of the model. Therefore, how to effectively handle imbalanced data is an important issue that needs to be addressed in deep learning-based network intrusion detection methods.
[0005] To address this problem, researchers have proposed many methods, including oversampling, undersampling, data synthesis, etc. Among them, the oversampling method balances the data set by increasing the number of minority class samples, but it is easy to introduce noise and overfitting problems; the undersampling method balances the data set by reducing the number of majority class samples, but some important information may be lost; the data synthesis method balances the data set by generating new minority class samples, but the generated samples may not be realistic and diverse enough. Summary of the invention
[0006] In view of the defects in the prior art, the present invention provides a network intrusion detection method based on deep learning to solve the problem of data imbalance during model training, ensure the authenticity of the data, and improve the accuracy of network intrusion detection.
[0007] The technical solution of the present invention is: a network intrusion detection method based on deep learning, comprising:
[0008] Acquire network intrusion data of the current device, and extract network intrusion features from the network intrusion data;
[0009] Calculating the importance of the network intrusion feature using a preset model, and determining the target data volume of the network intrusion feature according to the importance;
[0010] When the data volume of the network intrusion feature is less than the target data volume, determining that the network intrusion feature is a non-target network intrusion feature, acquiring associated network intrusion data associated with the current network device, extracting an associated network intrusion feature from the associated network intrusion data, and performing data fusion on the associated network intrusion feature and the non-target network intrusion feature to obtain a target network intrusion feature;
[0011] Constructing a sample set based on the target network features; using the sample set to train a network intrusion detection model to obtain a target network intrusion detection model;
[0012] Use the trained target network intrusion detection model to perform network intrusion detection.
[0013] Preferably, extracting network intrusion features from the network intrusion data includes:
[0014] Performing missing value processing on the network intrusion data;
[0015] Normalize the network intrusion data after missing value processing;
[0016] The convolutional neural network model is used to extract network intrusion features from the normalized network intrusion data.
[0017] Preferably, the importance of network intrusion features is calculated using a preset model, including:
[0018] Calculate the first importance of the network intrusion feature using the lightGBM model;
[0019] Calculating the second importance of the network intrusion feature using a random forest model;
[0020] The importance of the network intrusion feature is calculated according to the first importance, the second importance and their corresponding importance weights.
[0021] Preferably, calculating the importance of network intrusion features using a preset model also includes:
[0022] Determining that differences between the importance and the first importance and the second importance are both within a preset range;
[0023] If so, determining the importance as the target importance;
[0024] If not, the target importance is modified using the difference between the first importance and the importance and the difference between the second importance and the second importance.
[0025] Preferably, determining the target data volume of the network intrusion feature according to the importance includes:
[0026] The target data volume of the network intrusion feature is determined according to the target importance.
[0027] Preferably, obtaining associated network intrusion data associated with the current network device includes:
[0028] Determine whether other network devices communicate with the current network device within a preset time;
[0029] The number of communications is calculated, and when the number of communications is greater than a preset number, it is determined that the other network devices and the current network device are associated devices.
[0030] Preferably, the number of communications includes the number of indirect communications and the number of direct communications, and when the first other network device communicates with the second other device and the first other network device communicates with the current network device within a first preset time during which the first other network device communicates with the current network device, the communication between the second other device and the current network device is determined to be the number of indirect communications between the first other device and the current network device;
[0031] The number of communications is determined according to the number of indirect communications and the number of direct communications and their corresponding number weights, wherein the number weight of the direct communication number is determined according to the correlation between the network intrusion features extractable in the first other network device and the non-target network intrusion features, and the number weight of the indirect communication number is determined according to the correlation between the network intrusion features extractable in the first other network device and the non-target network intrusion features, the correlation between the network intrusion features extractable in the second other network device and the non-target network intrusion features, and the hierarchical difference between the first other network device and the second other network device.
[0032] Preferably, using the target network features to train a network intrusion detection model to obtain a target network intrusion detection model includes:
[0033] Determine the number of training times based on the amount of data in the sample set;
[0034] Divide the sample set into multiple training groups according to the number of training times, each training group includes all the data in the sample set, and each training group includes a validation set and a training set;
[0035] The network intrusion detection model is trained and verified using the training group to obtain multiple verification results. When all verification results meet preset requirements, the trained network intrusion detection model is determined to be the target network intrusion detection model.
[0036] The beneficial effects of the present invention are as follows: The embodiment of the present invention provides a network intrusion detection method based on deep learning, which aims to solve the problem of data imbalance during model training and improve the accuracy of network intrusion detection. The method automatically extracts features from network traffic data, calculates the importance of features using a preset model, determines the target data volume based on the importance, then performs data enhancement and fusion processing on non-target features, and finally uses the processed data to train a deep learning model for network intrusion detection. The method can effectively handle the problem of unbalanced data and improve the accuracy and efficiency of detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.
[0038] Figure 1 A flowchart of a network intrusion detection method based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.
[0040] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.
[0041] like Figure 1 As shown, a flowchart of a network intrusion detection method based on deep learning provided by an embodiment of the present invention includes:
[0042] Step 1: Obtain network intrusion data of the current device and extract network intrusion features from the network intrusion data.
[0043] In an embodiment of the present invention, extracting network intrusion features from the network intrusion data includes: performing missing value processing on the network intrusion data; normalizing the network intrusion data after the missing value processing; and extracting network intrusion features from the normalized network intrusion data using a convolutional neural network model.
[0044] Some records in the network intrusion data may lack some features, such as missing attack types, and these missing values need to be processed. Suppose it is decided to use average values to fill in the missing network traffic and packet sizes. If the network traffic information of a record is missing, it is filled with the average network traffic value; similarly, if the packet size is missing, we use the average packet size to fill it. Then, the data is normalized to scale all features to a uniform range. For example, the minimum-maximum normalization method can be used to scale the data to a range of 0 to 1. Finally, a convolutional neural network (CNN) is used to extract network intrusion features. CNN is a powerful deep learning model that is suitable for processing images and sequence data. In an embodiment of the present invention, the network intrusion data can be regarded as sequence data, and each feature can be regarded as a time step or a feature vector.
[0045] By processing missing values and normalizing network intrusion data and using a convolutional neural network model to extract network intrusion features, the model training effect can be improved, the precision and accuracy of network intrusion detection can be improved, and more effective protection for network security can be provided.
[0046] Step 2: Calculate the importance of the network intrusion feature using a preset model, and determine the target data volume of the network intrusion feature according to the importance.
[0047] In an embodiment of the present invention, a preset model is used to calculate the importance of network intrusion features, including: using a lightGBM model to calculate the first importance of the network intrusion features; using a random forest model to calculate the second importance of the network intrusion features; and calculating the importance of the network intrusion features based on the first importance, the second importance and their corresponding importance weights.
[0048] First, the first importance of network intrusion features is calculated using the lightGBM model and the random forest model. lightGBM is a collection of decision trees. The importance of features is evaluated by counting the number of feature segmentations of each decision tree during the LightGBM training process. That is, the more times a feature is used for segmentation, the more important it is to the classification result. Then, the final importance is obtained by aggregating the scores of the two models. Random forest is also a collection of decision trees. The importance score of the feature is obtained by comparing the change in the model prediction error rate before and after random permutation of the feature. The larger the error rate, the more important the feature is to the model. Finally, the importance is calculated based on the weights of the first importance and the second importance.
[0049] In an embodiment of the present invention, the importance of network intrusion features is calculated using a preset model, and also includes: determining that the differences between the importance and the first importance and the second importance are both within a preset range; if so, determining that the importance is a target importance; if not, correcting the target importance using the difference between the first importance and the importance and the difference between the second importance and the second importance.
[0050] Step 3: When the data volume of the network intrusion feature is less than the target data volume, determine that the network intrusion feature is a non-target network intrusion feature, obtain associated network intrusion data associated with the current network device, extract associated network intrusion features from the associated network intrusion data, and perform data fusion on the associated network intrusion features and the non-target network intrusion features to obtain the target network intrusion features.
[0051] In the embodiment of the present invention, determining the target data volume of the network intrusion feature according to the importance includes: determining the target data volume of the network intrusion feature according to the target importance.
[0052] In an embodiment of the present invention, obtaining associated network intrusion data associated with the current network device includes: determining whether other network devices communicate with the current network device within a preset time; calculating the number of communications, and when the number of communications is greater than the preset number, determining that the other network devices and the current network device are associated devices.
[0053] In an embodiment of the present invention, the number of communications includes the number of indirect communications and the number of direct communications. Within a first preset time when the first other network device communicates with the current network device, when the first other network device communicates with the second other device and the first other network device communicates with the current network device, the communication between the second other device and the current network device is determined to be the number of indirect communications between the first other device and the current network device; the number of communications is determined according to the number of indirect communications and the number of direct communications and their corresponding number weights, wherein the number weight of the direct communication number is determined according to the correlation between the network intrusion features extractable from the first other network device and the non-target network intrusion features, and the number weight of the indirect communication number is determined according to the correlation between the network intrusion features extractable from the first other network device and the non-target network intrusion features, the correlation between the network intrusion features extractable from the second other network device and the non-target network intrusion features, and the hierarchical difference between the first other network device and the second other network device.
[0054] Step 4: construct a sample set based on the target network features; use the sample set to train a network intrusion detection model to obtain a target network intrusion detection model.
[0055] In an embodiment of the present invention, the target network features are used to train a network intrusion detection model to obtain a target network intrusion detection model, including: determining the number of training times according to the amount of data in a sample set; dividing the sample set into multiple training groups according to the number of training times, each training group including all the data in the sample set, and each training group including a verification set and a training set; using the training groups to train and verify the network intrusion detection model to obtain multiple verification results, and when all verification results meet preset requirements, determining that the trained network intrusion detection model is the target network intrusion detection model.
[0056] Traditional machine learning methods usually divide the sample set into a training set and a validation set, train the model with the training set, and then use the validation set to evaluate the performance of the model. However, this method has some problems, such as the division of the training set and the validation set may affect the performance of the final model, and the prediction results for small sample data are less reliable.
[0057] In contrast, the method in the embodiment of the present invention introduces the following advantages: by dividing the sample set into multiple training groups, different validation sets and training sets are used for each training, all the data in the data set are effectively utilized, and the utilization rate of the data is improved. : Using different validation sets to verify the model can reduce the fluctuation of model performance caused by different selection of validation sets, and improve the stability and robustness of the model. In multiple validation results, the model can be finely adjusted according to the validation results, so that the model can perform well on a wider range of data sets. For example, suppose there is a sample set, and the sample set includes 1000 samples. The traditional method may divide the data set into 800 samples for training and 200 samples for validation. The method in the embodiment of the present invention divides the data set into 10 training groups, and each training group uses different validation sets and training sets during training, making full use of all samples in the data set and improving the model performance. This method can better evaluate the adaptability of the model to different data subsets and improve the generalization ability and robustness of the model.
[0058] Step 5: Use the trained target network intrusion detection model to perform network intrusion detection.
[0059] In summary, the present invention provides a network intrusion detection method based on deep learning, including: obtaining network intrusion data of the current device, extracting network intrusion features from the network intrusion data; using a preset model to calculate the importance of the network intrusion feature, and determining the target data volume of the network intrusion feature according to the importance; when the data volume of the network intrusion feature is less than the target data volume, determining that the network intrusion feature is a non-target network intrusion feature, obtaining associated network intrusion data associated with the current network device, extracting associated network intrusion features from the associated network intrusion data, fusing the associated network intrusion features with the non-target network intrusion features to obtain target network intrusion features; constructing a sample set based on the target network features; using the sample set to train a network intrusion detection model to obtain a target network intrusion detection model; and using the trained target network intrusion detection model to perform network intrusion detection. The technical solution of the present invention does not undersample the original data, but obtains data from associated devices to achieve data enhancement, thereby ensuring the authenticity of the original data, solving the problem of imbalance during model training, and improving the accuracy of network intrusion detection.
[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A network intrusion detection method based on deep learning, It is characterized in that include: Acquire network intrusion data of the current device, and extract network intrusion features from the network intrusion data; Calculating the importance of the network intrusion feature using a preset model, and determining the target data volume of the network intrusion feature according to the importance; When the data volume of the network intrusion feature is less than the target data volume, determining that the network intrusion feature is a non-target network intrusion feature, acquiring associated network intrusion data associated with the current network device, extracting an associated network intrusion feature from the associated network intrusion data, and performing data fusion on the associated network intrusion feature and the non-target network intrusion feature to obtain a target network intrusion feature; Constructing a sample set based on the target network features; Using the sample set to train a network intrusion detection model to obtain a target network intrusion detection model; Use the trained target network intrusion detection model to perform network intrusion detection.
2. According to the network intrusion detection method based on deep learning in claim 1, It is characterized in that Extracting network intrusion features from the network intrusion data includes: Performing missing value processing on the network intrusion data; Normalize the network intrusion data after missing value processing; The convolutional neural network model is used to extract network intrusion features from the normalized network intrusion data.
3. According to the network intrusion detection method based on deep learning in claim 2, It is characterized in that The importance of network intrusion features is calculated using a preset model, including: Calculate the first importance of the network intrusion feature using the lightGBM model; Calculating the second importance of the network intrusion feature using a random forest model; The importance of the network intrusion feature is calculated according to the first importance, the second importance and their corresponding importance weights.
4. According to the network intrusion detection method based on deep learning in claim 3, It is characterized in that The importance of network intrusion features is calculated using a preset model, which also includes: Determining that differences between the importance and the first importance and the second importance are both within a preset range; If so, determining the importance as the target importance; If not, the target importance is modified using the difference between the first importance and the importance and the difference between the second importance and the second importance.
5. According to the network intrusion detection method based on deep learning in claim 4, It is characterized in that Determining the target data volume of the network intrusion feature according to the importance includes: The target data volume of the network intrusion feature is determined according to the target importance.
6. According to the network intrusion detection method based on deep learning in claim 1, It is characterized in that Acquiring associated network intrusion data associated with the current network device, including: Determine whether other network devices communicate with the current network device within a preset time; The number of communications is calculated, and when the number of communications is greater than a preset number, it is determined that the other network devices and the current network device are associated devices.
7. A network intrusion detection method based on deep learning according to claim 6, It is characterized in that The number of communications includes the number of indirect communications and the number of direct communications, and when the first other network device communicates with the second other device and the first other network device communicates with the current network device within a first preset time during which the first other network device communicates with the current network device, the communication between the second other device and the current network device is determined to be the number of indirect communications between the first other device and the current network device; The number of communications is determined according to the number of indirect communications and the number of direct communications and their corresponding number weights, wherein the number weight of the direct communication number is determined according to the correlation between the network intrusion features extractable in the first other network device and the non-target network intrusion features, and the number weight of the indirect communication number is determined according to the correlation between the network intrusion features extractable in the first other network device and the non-target network intrusion features, the correlation between the network intrusion features extractable in the second other network device and the non-target network intrusion features, and the hierarchical difference between the first other network device and the second other network device.
8. According to the network intrusion detection method based on deep learning in claim 1, It is characterized in that The network intrusion detection model is trained using the target network features to obtain the target network intrusion detection model, including: Determine the number of training times based on the amount of data in the sample set; Divide the sample set into multiple training groups according to the number of training times, each training group includes all the data in the sample set, and each training group includes a validation set and a training set; The network intrusion detection model is trained and verified using the training group to obtain multiple verification results. When all verification results meet preset requirements, the trained network intrusion detection model is determined to be the target network intrusion detection model.
