Network intrusion detection method and system based on Transformer

By converting network intrusion detection data into a sequence of feature vectors and using the Transformer model, the lack of performance problems of traditional systems when processing sequential data is solved, and a more comprehensive capture of network threats and improved detection performance is achieved.

CN117692249BActive Publication Date: 2024-11-05GUANGDONG YIDIAN TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410095345.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-22
Publication Date
2024-11-05
Estimated Expiration
2044-01-22

AI Technical Summary

Technical Problem

Traditional machine learning-based network intrusion detection systems are difficult to capture the time correlation and sequential information in network traffic data, resulting in insufficient performance when processing sequential data.

Method used

The network intrusion detection method based on Transformer is adopted to convert network intrusion detection data into a fixed-length eigenvector sequence and trained through the Transformer model to capture network threats more comprehensively. The specific steps include data acquisition, preprocessing, data division, feature extraction and model training, and use the Transformer model composed of Input encoder, Bert and Classification Head for detection.

Benefits of technology

Through the Transformer model, we can process sequential data more effectively, improve the ability to capture network threats, and improve the performance of network intrusion detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117692249B_ABST
    Figure CN117692249B_ABST
Patent Text Reader

Abstract

The invention discloses a network intrusion detection method and system based on Transformer. The method comprises: constructing a network intrusion detection data set and setting a data set specification; preprocessing the network intrusion detection data set based on the data set specification; performing data division on the preprocessed data to obtain a divided data set; training the divided training set by using contrastive learning to obtain a preliminary training result, inputting the preliminary training result into a machine learning model for training to obtain a feature subset; inputting the feature subset into a Transformer model for training respectively, saving two network intrusion detection models, which are recorded as Model1 and Model2, and recording the accuracy of the model; using the Model1 and Model2 to test the divided test set respectively to obtain the predicted value of the model, outputting the classification result after weighted discrimination of the predicted value, and completing the detection of network intrusion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer network technology security technology, and in particular to a Transformer-based network intrusion detection method and system. Background Art

[0002] Traditional intrusion detection systems usually combine mathematical statistics with machine learning to detect abnormal data. Machine learning can automatically obtain useful information from large data sets, and combined with intrusion detection systems can greatly improve detection efficiency. However, in machine learning-based intrusion detection systems, data is captured in the form of data packets or sequence streams. How to identify subtle and complex patterns in this traffic is crucial to the performance of intrusion detection systems. Network communications are usually sequential in nature, but machine learning-based intrusion detection systems usually have difficulty capturing time correlation and sequential information in the data. Therefore, machine learning-based intrusion detection systems have some shortcomings when processing sequential data such as network traffic data.

[0003] At present, the model based on the Transformer architecture has been proven to be very effective in processing sequential data. The Transformer was originally designed for natural language processing (NLP) tasks, especially in the field of machine translation, and has achieved great success. At the same time, its architecture is also suitable for processing various sequential data, including but not limited to images, charts and languages. Therefore, the present invention proposes a network intrusion detection method based on Transformer to better process sequential data and more comprehensively capture network threats. Summary of the invention

[0004] The present invention aims at the deficiency of traditional network intrusion detection methods based on machine learning in processing sequential data, and provides a network intrusion detection method based on Transformer. The sequential data is converted into a feature vector sequence of fixed length and is passed into the Transformer model for training. By utilizing the advantages of the Transformer model in processing sequential data, network threats can be captured more comprehensively. The method comprises:

[0005] S1. Collect network intrusion detection data streams, construct network intrusion detection data sets, and set data set specifications according to the data sets;

[0006] S2. Preprocessing the network intrusion detection data set based on the data set specification to obtain preprocessed data;

[0007] S3. dividing the preprocessed data to obtain a divided data set;

[0008] S4. Use contrastive learning to train the divided training set to obtain preliminary training results, and input the preliminary training results into the machine learning model for training to obtain a feature subset;

[0009] S5. The feature subset is passed into the Transformer model composed of Input encoder, Bert and Classification Head and the Transformer model composed of Input encoder, GPT2.0 and Classification Head for training respectively, and two network intrusion detection models are saved, denoted as Model1 and Model2, and the accuracy of Model1 and Model2 is recorded;

[0010] S6. Use the Model1 and Model2 to test the divided test sets respectively to obtain the prediction values ​​of the models, perform weighted judgment on the prediction values ​​and output classification results to complete the detection of network intrusion.

[0011] Optionally, in S1, the network intrusion detection data stream is tabular data, and the tabular data consists of numerical fields and classification fields. The tabular data is classified to distinguish between numerical fields and classification fields, and data specifications are set according to the numerical fields and the classification fields.

[0012] Optionally, in S2, the specific method of preprocessing the network intrusion detection data set based on the data set specification to obtain the preprocessed data includes:

[0013] Performing logarithmic transformation on the characteristic value of the numerical field based on the data set specification to obtain the characteristic value after logarithmic transformation, and normalizing the characteristic value after logarithmic transformation to obtain the normalized characteristic value;

[0014] Based on the data set specification, the feature values ​​of the classification field are screened at N frequencies to obtain the screened feature values, and the screened feature values ​​are encoded.

[0015] Optionally, performing N-frequency screening on the feature values ​​of the classification field based on the data set specification to obtain the screened feature values, and encoding the screened feature values ​​specifically includes:

[0016] Based on the data set specification, the characteristic values ​​of the classification fields are statistically sorted according to the frequency of occurrence;

[0017] The eigenvalues ​​with the top 30 frequencies are selected based on statistical sorting, and the selected eigenvalues ​​are One-hot encoded.

[0018] Optionally, in S3, dividing the preprocessed data to obtain the divided data sets specifically includes:

[0019] The preprocessed data is divided into a training data set and a test data set in a ratio of 2:8.

[0020] Optionally, in S4, contrastive learning is used to train the divided training set to obtain preliminary training results, and the preliminary training results are input into a machine learning model for training. The feature subsets obtained specifically include:

[0021] Inputting the training data set into a contrastive learning model for learning, distinguishing different data by categories, and obtaining a training data set after contrastive learning, wherein the categories are divided into normal data and abnormal data;

[0022] The training data set after contrastive learning is trained using an average recursive feature elimination method to obtain feature importance, feature extraction is performed on the feature importance, and an important feature subset is selected from the extracted features according to the average accuracy.

[0023] Optionally, the method of average recursive feature elimination trains the training data set after contrastive learning to obtain feature importance, performs feature extraction on the feature importance, and the method of selecting an important feature subset from the extracted feature tracking according to the average accuracy rate specifically includes:

[0024] Using three tree-based machine learning models to train the training data set after contrastive learning to obtain feature importance, the machine learning models are a random forest model, an extra tree model, and an XGBoost model;

[0025] Add up the feature importances and take the average value. At the same time, form a new feature importance list in descending order and record the average accuracy.

[0026] Delete the last feature in the feature importance list, update the training data set after comparative learning, and form a new feature subset;

[0027] Repeated training is performed according to the new feature subset until the feature subset is empty, and an important feature subset is selected according to the average accuracy.

[0028] The present invention also discloses a network intrusion detection system based on Transformer, the system comprising: a specification setting module, a preprocessing module, a data partitioning module, a feature acquisition module, a model building module and an intrusion detection module;

[0029] The specification setting module is used to collect network intrusion detection data streams, construct network intrusion detection data sets, and set data set specifications according to the data sets;

[0030] The preprocessing module is used to preprocess the network intrusion detection data set based on the data set specification to obtain preprocessed data;

[0031] The data partitioning module is used to partition the preprocessed data to obtain a partitioned data set;

[0032] The feature acquisition module is used to train the divided training set using contrastive learning to obtain preliminary training results, and input the preliminary training results into the machine learning model for training to obtain feature subsets;

[0033] The model building module is used to pass the feature subset into the Transformer model composed of Input encoder, Bert and Classification Head and the Transformer model composed of Input encoder, GPT2.0 and Classification Head for training respectively, save two network intrusion detection models, recorded as Model1 and Model2, and record the accuracy of Model1 and Model2;

[0034] The intrusion detection module is used to use the Model1 and Model2 to test the divided test sets respectively, obtain the prediction values ​​of the models, output the classification results after weighted judgment of the prediction values, and complete the detection of network intrusion.

[0035] Optionally, the preprocessing module further includes a numerical field feature processing submodule and a categorical field feature processing submodule;

[0036] The numerical field feature processing submodule is used to perform logarithmic transformation on the feature value of the numerical field based on the data set specification to obtain the logarithmically transformed feature value, and perform normalization processing on the logarithmically transformed feature value to obtain the normalized feature value;

[0037] The classification field feature processing submodule is used to perform N-frequency screening on the feature values ​​of the classification field based on the data set specification to obtain the screened feature values, and encode the screened feature values.

[0038] The model building module includes a feature splicing submodule, a feature mapping submodule, a data flattening submodule, a data classification submodule and a pre-training submodule;

[0039] The feature splicing submodule is used to splice the numerical field and the categorical field after preprocessing in the Input encoder component to obtain high-dimensional data features;

[0040] The feature mapping submodule is used to map the high-dimensional data features to a low-dimensional continuous feature vector space through an embedding layer, and output a feature vector sequence;

[0041] The data flattening submodule is used to flatten the feature vector sequences output by the Model1 and Model2 in the Classification Head component, respectively, and flatten the multi-dimensional data into one-dimensional data;

[0042] The data classification submodule is used to input the one-dimensional data into a multi-layer perceptron to perform classification of normal data and abnormal data;

[0043] The pre-training module is used to perform comparative learning based on the classification of the normal data and the abnormal data to obtain a training data set after comparative learning, and input the training data set after comparative learning into a machine learning model for training to obtain an initial network intrusion detection model.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] In view of the shortcomings of traditional machine learning-based network intrusion detection methods in processing sequential data, the present invention proposes a Transformer-based network intrusion detection method, which converts sequential data into a fixed-length feature vector sequence and passes it into the Transformer model for training. By utilizing the advantages of the Transformer model in processing sequential data, network threats can be captured more comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 The present invention is a method step diagram of a Transformer-based network intrusion detection method and system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Embodiment 1

[0051] A network intrusion detection method based on Transformer, such as Figure 1 As shown, the method includes:

[0052] S1. In the actual detection process, the network intrusion detection data stream is collected, the network intrusion detection data set is constructed, and the data set specification is set according to the data set; the network intrusion detection data stream is tabular data, and the tabular data consists of numerical fields and classification fields. The characteristic fields of the tabular data are classified to distinguish between numerical fields and classification fields, and the data specification is set according to the numerical fields and the classification fields. Different data set specifications can be set according to different data sets. The specifications distinguish between numerical fields and classification fields, which will be provided for use in the preprocessing stage later.

[0053] The collected flow data is a kind of tabular data, in which the fields of each flow are known and fixed. The data consists of two types of fields: numerical fields, such as the number of packets sent, where the distance between numbers has contextual meaning; categorical fields, such as port numbers, protocol types, etc., whose values ​​are a set of relatively discrete numbers, and there is no guarantee that there is a contextual relationship between numbers. For example, port numbers 25 and 22 are very close, but represent different protocols, while port numbers 443 and 8080 are far apart, but represent the same service protocol.

[0054] S2. Preprocessing the network intrusion detection data set based on the data set specification to obtain preprocessed data; Preprocessing the network intrusion detection data set based on the data set specification to obtain preprocessed data includes:

[0055] Based on the data set specification, the eigenvalues ​​of the numerical fields are logarithmically transformed to obtain the eigenvalues ​​after logarithmic transformation, and the eigenvalues ​​after logarithmic transformation are normalized to obtain the normalized eigenvalues; the values ​​of the numerical features are mapped to between [0, 1], thereby eliminating the adverse effects caused by singular sample data. Logarithmic transformation: Logarithmic transformation can smooth data and reduce the impact of extreme outliers. The eigenvalues ​​of the numerical fields are logarithmically transformed using formula (1). X is the original eigenvalue, min is the minimum value of the corresponding column, because logarithmic operations often need to process zero values. In order to avoid performing logarithmic operations on zero values, 1 is usually added to the data range, and y is the eigenvalue after logarithmic transformation.

[0056] y=log[(x-min)+1] (1)

[0057] Data normalization: The eigenvalues ​​after logarithmic transformation are subjected to Min-Max normalization using formula (2), where X is the original eigenvalue, X min is the minimum value of the corresponding column, X max The original data is linearly transformed to the maximum value of the corresponding column, and the result is mapped to the range of [0,1] to achieve geometric scaling of the original data. The eigenvalue after normalization is:

[0058]

[0059] The categorical fields are one-hot encoded according to the dataset specification.

[0060] N-frequency one-hot encoding: Perform N-frequency one-hot encoding on the classification field according to the data specification. In the stream data, each field has N eigenvalues. If one-hot encoding is performed directly, some fields may be difficult to encode due to the high dimension of the generated feature space. Therefore, in this embodiment, each eigenvalue of the classification field is first counted, sorted according to the frequency of occurrence, and the eigenvalues ​​with the top 32 frequencies are retained for one-hot encoding, and uncommon eigenvalues ​​are discarded, which helps to reduce the data dimension and improve the model performance.

[0061] S3. Dividing the preprocessed data to obtain divided data sets; In S3, dividing the preprocessed data to obtain divided data sets specifically includes: dividing the preprocessed data in a ratio of 2:8 to obtain a training data set and a test data set.

[0062] S4. Using contrastive learning to train the divided training set to obtain preliminary training results, inputting the preliminary training results into the machine learning model for training, and obtaining feature subsets; In S4, using contrastive learning to train the divided training set to obtain preliminary training results, inputting the preliminary training results into the machine learning model for training, and obtaining feature subsets specifically includes:

[0063] The training data set is input into a contrastive learning model for learning, and different data are distinguished in categories to obtain a training data set after contrastive learning; the feature importance is extracted using an average recursive feature elimination method, and an important feature subset is selected from the extracted features based on the average accuracy.

[0064] The method of average recursive feature elimination extracts the feature importance of the feature, and the method of selecting an important feature subset based on the average accuracy from the extracted feature tracking specifically includes:

[0065] The training data set after contrastive learning is trained using three tree-based machine learning models to obtain feature importance. The machine learning models are a random forest model (RF), an additional tree model (ET) and an XGBoost model. Combining contrastive learning and tree models can improve the performance of the model on specific tasks, especially in processing tasks that require complex feature extraction.

[0066] The preprocessed data is handed over to the contrast learning model for learning and processing, different data are distinguished in categories, and the output is passed into three tree-based machine learning models for training, feature importance is obtained, and important feature subsets are selected. The present invention proposes a method for average recursive feature elimination: as shown in Algorithm 1: first, three tree-based machine learning models (random forest (RF), additional tree (ET) and XGBoost) are used for training, and the feature importance obtained by each trainer is added and averaged, and a new feature importance list is formed in order from large to small, and the average accuracy is recorded (the accuracy of the three models is added and averaged), and the feature with the smallest feature importance is deleted, that is, the last feature in the feature importance list; then the data set is updated to form a new feature subset, and repeated training is performed based on the new feature set until the feature set is empty; finally, the feature subset corresponding to the highest average accuracy is selected as the most important feature subset for use in step five, so as to achieve the effect of data dimensionality reduction or improving the generalization ability of the model.

[0067] Algorithm:

[0068] 1: Input: X, y

[0069] 2: Ouput: N

[0070] 3: while features_List:

[0071] 4: Train ET(X, y), RF(X, y), XGBoost(X, y) on Avg_Features_List;

[0072] 5: Avg_Features_List=(et_features_fist+rf_features_fist+xgh_features_fist) / 3

[0073] 6: RecordAccurary:

[0074] 7: del Avg_Features_List[-1];

[0075] 8: features_List=Avg_Features_List;

[0076] 9: end while

[0077] 10: N = arg max Accuracy

[0078] S5. The feature subset is passed into the Transformer model composed of Input encoder, Bert and Classification Head and the Transformer model composed of Input encoder, GPT2.0 and Classification Head for training respectively, and two network intrusion detection models are saved, recorded as Model1 and Model2, and the accuracy of Model1 and Model2 is recorded; the Transformer model is mainly composed of three components: Input encoder, Bert and Classification Head. In the Input encoder component, the preprocessed numerical fields and classification fields are spliced, and the high-dimensional data features are mapped to the low-dimensional continuous feature vector space through the embedding layer (fully connected layer). The feature vector sequence that can be effectively processed by the Bert model is output, and the features of the network traffic sequence data are learned through the Bert model. In the ClassificationHead component, the output sequence of the Bert model will be flattened, and the multi-dimensional data will be flattened into one-dimensional data, and then passed to the multi-layer perceptron (MLP), and then the final classification is performed.

[0079] The hyperparameter optimization method Bayesian Tree Parzen Estimator (BO-TPE) is used to optimize the important hyperparameters of the model. BO-TPE creates two density functions l(x) and g(x) as the generative model of the variable. * The better and worse observations are separated and the two groups of outcomes are modeled by a simple Parzen estimator:

[0080]

[0081] BO-TPE detects the optimal hyperparameter value by maximizing the ratio of these two density functions (l(x) / g(x)), and BO-TPE can effectively optimize all types of hyperparameters.

[0082] S6. Use the Model1 and Model2 to test the divided test sets respectively to obtain the prediction values ​​of the models, perform weighted judgment on the prediction values ​​and output classification results to complete the detection of network intrusion.

[0083] Model1 and Model2 are tested on the test data respectively. The output of Model1 is y1 and the weight is w1. The output of Model2 is y2 and the weight is w2. The weights are assigned according to the accuracy stored in step 5. The weight of the model with high accuracy is assigned to 0.6 and that of the model with low accuracy is assigned to 0.4. The final classification result output is: y=w1*y1+w2*y2.

[0084] Embodiment 2

[0085] A network intrusion detection system based on Transformer, the system comprising: a specification setting module, a preprocessing module, a data partitioning module, a feature acquisition module, a model building module and an intrusion detection module;

[0086] The specification setting module is used to collect network intrusion detection data streams, construct network intrusion detection data sets, and set data set specifications according to the data sets; the network intrusion detection data stream is tabular data, and the tabular data consists of numerical fields and classification fields. The characteristic fields of the tabular data are classified to distinguish between numerical fields and classification fields, and data specifications are set according to the numerical fields and classification fields. Different data set specifications can be set according to different data sets. The specifications distinguish between numerical fields and classification fields, which will be provided for use in the preprocessing stage later.

[0087] The collected flow data is a kind of tabular data, in which the fields of each flow are known and fixed. The data consists of two types of fields: numerical fields, such as the number of packets sent, where the distance between numbers has contextual meaning; categorical fields, such as port numbers, protocol types, etc., whose values ​​are a set of relatively discrete numbers, and there is no guarantee that there is a contextual relationship between numbers. For example, port numbers 25 and 22 are very close, but represent different protocols, while port numbers 443 and 8080 are far apart, but represent the same service protocol.

[0088] The numerical field feature processing submodule is used to perform logarithmic transformation on the feature value of the numerical field based on the data set specification to obtain the logarithmically transformed feature value, normalize the logarithmically transformed feature value to obtain the normalized feature value; map the value of the numerical feature to between [0,1], thereby eliminating the adverse effects caused by singular sample data.

[0089] Logarithmic transformation: Logarithmic transformation can smooth data and reduce the impact of extreme outliers. The eigenvalues ​​of numerical fields are logarithmically transformed using formula (3). X is the original eigenvalue, min is the minimum value of the corresponding column, and because logarithmic operations often need to process zero values, in order to avoid performing logarithmic operations on zero values, 1 is usually added to the data range, and y is the eigenvalue after logarithmic transformation.

[0090] y=log[(x-min)+1] (3)

[0091] Data normalization: The eigenvalues ​​after logarithmic transformation are subjected to Min-Max normalization using formula (4), where X is the original eigenvalue, X min is the minimum value of the corresponding column, X max The original data is linearly transformed to the maximum value of the corresponding column, and the result is mapped to the range of [0,1] to achieve geometric scaling of the original data. The eigenvalue after normalization is:

[0092]

[0093] Based on the data set specification, the feature values ​​of the classification field are screened at N frequencies to obtain the screened feature values, and the screened feature values ​​are encoded.

[0094] The categorical fields are one-hot encoded according to the dataset specification.

[0095] N-frequency one-hot encoding: Perform N-frequency one-hot encoding on the classification field according to the data specification. In the stream data, each field has N eigenvalues. If one-hot encoding is performed directly, some fields may be difficult to encode due to the high dimension of the generated feature space. Therefore, in this embodiment, each eigenvalue of the classification field is first counted, sorted according to the frequency of occurrence, and the eigenvalues ​​with the top 32 frequencies are retained for one-hot encoding, and uncommon eigenvalues ​​are discarded, which helps to reduce the data dimension and improve the model performance.

[0096] The data partitioning module is used to partition the preprocessed data to obtain a partitioned data set; partitioning the preprocessed data to obtain a partitioned data set specifically includes: partitioning the preprocessed data in a ratio of 2:8 to obtain a training data set and a test data set.

[0097] The feature acquisition module is used to use contrastive learning to train the divided training set to obtain preliminary training results, input the preliminary training results into the machine learning model for training, and obtain a feature subset; input the training data set into the contrastive learning model for learning, distinguish different data in categories, and obtain a training data set after contrastive learning; use the average recursive elimination feature method to extract the feature importance, and select important feature subsets from the extracted features according to the average accuracy.

[0098] The method of average recursive feature elimination extracts the feature importance of the feature, and the method of selecting an important feature subset based on the average accuracy from the extracted feature tracking specifically includes:

[0099] The training data set after contrastive learning is trained using three tree-based machine learning models to obtain feature importance. The machine learning models are a random forest model (RF), an additional tree model (ET) and an XGBoost model. Combining contrastive learning and tree models can improve the performance of the model on specific tasks, especially in processing tasks that require complex feature extraction.

[0100] The preprocessed data is handed over to the contrast learning model for learning and processing, different data are distinguished in categories, and the output is passed into three tree-based machine learning models for training, feature importance is obtained, and important feature subsets are selected. The present invention proposes a method for average recursive feature elimination: as shown in Algorithm 1: first, three tree-based machine learning models (random forest (RF), additional tree (ET) and XGBoost) are used for training, and the feature importance obtained by each trainer is added and averaged, and a new feature importance list is formed in order from large to small, and the average accuracy is recorded (the accuracy of the three models is added and averaged), and the feature with the smallest feature importance is deleted, that is, the last feature in the feature importance list; then the data set is updated to form a new feature subset, and repeated training is performed based on the new feature set until the feature set is empty; finally, the feature subset corresponding to the highest average accuracy is selected as the most important feature subset for use in step five, so as to achieve the effect of data dimensionality reduction or improving the generalization ability of the model.

[0101] Algorithm:

[0102] 1: Input: X, y

[0103] 2: Ouput: N

[0104] 3: while features_List:

[0105] 4: Train ET(X, y), RF(X, y), XGBoost(X, y) on Avg_Features_List;

[0106] 5: Avg_Features_List=(et_features_fist+rf_feetures_fist+xgb_features_fist) / 3

[0107] 6: RecordAccurary:

[0108] 7: del Avg_Features_List[-1];

[0109] 8: features_List=Avg_Features_List;

[0110] 9: end while

[0111] 10: N = arg max Accuracy

[0112] The model construction module is used to pass the feature subset into the Transformer model composed of Input encoder, Bert and ClassificationHead and the Transformer model composed of Input encoder, GPT2.0 and Classification Head for training respectively, save two network intrusion detection models, recorded as Model1 and Model2, and record the accuracy of Model1 and Model2; the model construction module includes a feature splicing submodule, a feature mapping submodule, a data flattening submodule, a data classification submodule and a pre-training submodule; the feature splicing submodule is used to splice the numerical field and the classification field after preprocessing in the Inputencoder component to obtain high-dimensional data features; the feature mapping submodule is used to map the high-dimensional data features to a low-dimensional continuous feature vector space through an embedding layer, and output a feature vector sequence; the data flattening submodule is used to output a feature vector sequence in the Classification In the Head component, the feature vector sequences output by the Model1 and Model2 are flattened respectively, and the multi-dimensional data is flattened into one-dimensional data; the data classification submodule is used to pass the one-dimensional data into the multi-layer perceptron to perform classification of normal data and abnormal data; the pre-training module is used to perform comparative learning based on the classification of the normal data and abnormal data to obtain a training data set after comparative learning, and the training data set after comparative learning is input into the machine learning model for training to obtain an initial network intrusion detection model.

[0113] The network intrusion initial detection model (Transformer model) is mainly composed of three components: Input encoder, Bert and Classification Head. In the Input encoder component, the preprocessed numerical fields and classification fields are spliced, and the high-dimensional data features are mapped to the low-dimensional continuous feature vector space through the embedding layer (fully connected layer). The feature vector sequence that can be effectively processed by the Bert model is output, and the characteristics of the network traffic sequence data are learned through the Bert model. In the Classification Head component, the output sequence of the Bert model is flattened, and the multi-dimensional data is flattened into one-dimensional data, and then passed to the multi-layer perceptron (MLP), and then the final classification is performed.

[0114] The hyperparameter optimization method Bayesian Tree Parzen Estimator (BO-TPE) is used to optimize the important hyperparameters of the model. BO-TPE creates two density functions l(x) and g(x) as the generative model of the variable. *The better and worse observations are separated and the two groups of outcomes are modeled using a simple Parzen estimator:

[0115]

[0116] BO-TPE detects the optimal hyperparameter value by maximizing the ratio of these two density functions (l(x) / g(x)), and BO-TPE can effectively optimize all types of hyperparameters.

[0117] The intrusion detection module is used to use the Model1 and Model2 to test the divided test sets respectively, obtain the prediction values ​​of the models, output the classification results after weighted judgment of the prediction values, and complete the detection of network intrusion.

[0118] Use Model1 and Model2 to classify the divided test set as normal and abnormal, and output the classification results by weighted discrimination. Test the test data with Model1 and Model2 respectively. The output of Model1 is y1, and the weight is w1. The output of Model2 is y2, and the weight is w2. According to the accuracy stored in step 5, the weight is assigned. The weight of the model with high accuracy is 0.6, and the weight of the model with low accuracy is 0.4. The final classification result is output as: y = w1*y1+w2*y2.

[0119] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A network intrusion detection method based on Transformer, characterized in that: The method comprises: S1. Collecting network intrusion detection data streams, constructing network intrusion detection data sets, and setting data set specifications according to the data sets; in S1, the network intrusion detection data stream is tabular data, the tabular data consists of numerical fields and classification fields, the tabular data is classified, the numerical fields and classification fields are distinguished, and the data set specifications are set according to the numerical fields and the classification fields; S2. Preprocessing the network intrusion detection data set based on the data set specification to obtain preprocessed data; in S2, the specific method of preprocessing the network intrusion detection data set based on the data set specification to obtain preprocessed data includes: performing logarithmic transformation on the characteristic values ​​of the numerical fields based on the data set specification to obtain the characteristic values ​​after logarithmic transformation, and normalizing the characteristic values ​​after logarithmic transformation to obtain the normalized characteristic values; statistically sorting the characteristic values ​​of the classification fields according to the frequency of occurrence based on the data set specification; filtering out the characteristic values ​​with the top 30 frequencies according to the statistical sorting, and performing One-hot encoding on the filtered characteristic values; S3. Dividing the preprocessed data to obtain a divided data set; In S3, dividing the preprocessed data to obtain a divided data set specifically includes: dividing the preprocessed data in a ratio of 2:8 to obtain a training set and a test set; S4. Use contrastive learning to train the training set obtained after division to obtain preliminary training results, input the preliminary training results into the machine learning model for training, and obtain feature subsets; the machine learning models are three tree-based machine learning models, specifically a random forest model, an additional tree model, and an XGBoost model, and the three tree-based machine learning models are used to train the training data set after contrastive learning to obtain feature importance; In S4, contrastive learning is used to train the training set obtained after the division to obtain preliminary training results, which specifically includes: inputting the training set into a contrastive learning model for learning, distinguishing different data in categories, and obtaining a training data set after contrastive learning, wherein the categories are divided into normal data and abnormal data; S5. The feature subset is passed into the Transformer model composed of Input encoder, Bert and Classification Head and the Transformer model composed of Input encoder, GPT2.0 and Classification Head for training respectively, and two network intrusion detection models, i.e., the two Transformer models, are recorded as Model1 and Model2, and the accuracy of Model1 and Model2 is recorded; Among them, in the Input encoder component, the preprocessed numerical fields and categorical fields are concatenated to obtain high-dimensional data features; the high-dimensional data features are mapped to a low-dimensional continuous feature vector space through an embedding layer, and a feature vector sequence that can be effectively processed by the Bert model is output, and the features of the network traffic sequence data are learned through the Bert model; S6. Use the Model1 and Model2 to test the test sets obtained after division, respectively, to obtain the prediction values ​​of the models, perform weighted judgment on the prediction values, and output classification results to complete the detection of network intrusion.

2. A Transformer-based network intrusion detection system, the system being used to implement the network intrusion detection method according to claim 1, characterized in that: The system includes: a specification setting module, a preprocessing module, a data partitioning module, a feature acquisition module, a model building module and an intrusion detection module; The specification setting module is used to collect network intrusion detection data streams, construct network intrusion detection data sets, and set data set specifications according to the data sets; The preprocessing module is used to preprocess the network intrusion detection data set based on the data set specification to obtain preprocessed data; The data partitioning module is used to partition the preprocessed data to obtain a partitioned data set; The feature acquisition module is used to use contrastive learning to train the training set obtained after division to obtain preliminary training results, and input the preliminary training results into the machine learning model for training to obtain feature subsets; The model building module is used to pass the feature subset into the Transformer model composed of Input encoder, Bert and ClassificationHead and the Transformer model composed of Input encoder, GPT2.0 and Classification Head for training respectively, save two network intrusion detection models, that is, these two Transformer models, recorded as Model1 and Model2, and record the accuracy of Model1 and Model2; The intrusion detection module is used to use the Model1 and Model2 to test the test sets obtained after division, obtain the prediction values ​​of the models, output the classification results after weighted judgment of the prediction values, and complete the detection of network intrusion.

3. The Transformer-based network intrusion detection system according to claim 2, characterized in that: The preprocessing module also includes a numerical field feature processing submodule and a categorical field feature processing submodule; The numerical field feature processing submodule is used to perform logarithmic transformation on the feature value of the numerical field based on the data set specification to obtain the logarithmic transformed feature value, and perform normalization processing on the logarithmic transformed feature value to obtain the normalized feature value; The classification field feature processing submodule is used to statistically sort the feature values ​​of the classification field according to the frequency of occurrence based on the data set specification; filter out the feature values ​​with the top 30 frequencies according to the statistical sorting, and perform One-hot encoding on the filtered feature values.

4. The Transformer-based network intrusion detection system according to claim 3, characterized in that: The model building module includes a feature splicing submodule, a feature mapping submodule, a data flattening submodule, a data classification submodule and a pre-training submodule; The feature splicing submodule is used to splice the numerical field and the categorical field after preprocessing in the Input encoder component to obtain high-dimensional data features; The feature mapping submodule is used to map the high-dimensional data features to a low-dimensional continuous feature vector space through an embedding layer, and output a feature vector sequence; The data flattening submodule is used to flatten the feature vector sequences output by the Model1 and Model2 in the Classification Head component, respectively, and flatten the multi-dimensional data into one-dimensional data; The data classification submodule is used to input the one-dimensional data into a multi-layer perceptron to perform classification of normal data and abnormal data; The pre-training submodule is used to perform comparative learning based on the classification of the normal data and the abnormal data to obtain a training data set after comparative learning, and input the training data set after comparative learning into a machine learning model for training to obtain an initial network intrusion detection model.