A network abnormal traffic detection method and system based on ensemble learning

Abstract

The application relates to the technical field of network abnormal flow detection, solves the technical problems that a traditional network abnormal flow detection method has high-dimensional feature redundancy, unbalanced data samples and weak model generalization capability, and particularly relates to a network abnormal flow detection method and system based on integrated learning. First, the method of fusing extreme gradient boosting is used to realize efficient screening and dimension reduction of network flow features; second, a stacked generalization model is constructed, and the Optuna framework is introduced to efficiently find the best hyperparameters of the model, so that high-precision identification of network threats is realized. The threat identification accuracy of the application on the UNSW-NB15 and CIC-IDS2017 data sets reaches 83.26% and 99.96% respectively, the performance is significantly improved compared with traditional algorithms, accurate abnormal flow detection can be carried out, and a reliable reference is provided for security decision-making in a complex network environment.

Description

Technical Field

[0001] The present invention relates to the technical field of network abnormal traffic detection, and in particular to a network abnormal traffic detection method and system based on ensemble learning. Background Art

[0002] Traditional network security defense methods include firewalls, intrusion detection systems, and vulnerability scanning. Firewalls isolate different networks and protect them by blocking data flows. This method is used to defend against conventional external attacks in simple network architectures, but it is helpless against abnormal internal network traffic and hidden threats in encrypted information. Intrusion detection or protection systems intercept attacks by analyzing abnormal network traffic behavior in alarm logs, but this method cannot identify unknown attacks and relies on rule library updates, making it unable to respond to unknown attacks in a timely manner. Vulnerability scanning of hosts and network devices using tool software faces problems such as delayed vulnerability signature library updates, blind spots in traffic detection, and time-consuming global scans.

[0003] Network anomaly traffic detection aims to identify patterns in network traffic that deviate from expected behavior, thereby ensuring network security. With the increasing complexity of networks and the proliferation of attack vectors, traditional signature- or rule-based detection methods are no longer sufficient to address new and advanced threats. Therefore, researchers are actively exploring the use of machine learning and deep learning technologies to improve the performance and efficiency of network anomaly detection.

[0004] While some progress has been made in the field of anomaly network traffic detection, with the escalating threat landscape in recent years, existing methods still have significant room for improvement. Traditional feature extraction methods are unable to effectively handle high-dimensional features and redundant information, resulting in a large number of features that contribute little to anomaly detection. Features can also be highly correlated, increasing computational complexity and potentially leading to model overfitting and low evaluation accuracy. Furthermore, current traffic detection methods have poor generalization capabilities, achieving good performance only for specific datasets. They lack the ability to effectively capture the underlying patterns in the data and are unable to adapt to dynamic network changes. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a network abnormal traffic detection method and system based on ensemble learning, which solves the technical problems of traditional network abnormal traffic detection methods such as high-dimensional feature redundancy, data sample imbalance, and weak model generalization ability.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a method for detecting abnormal network traffic based on ensemble learning, the method comprising the following steps:

[0007] By integrating extreme gradient boosting feature importance analysis, Pearson correlation analysis, and mutual information score, we can filter and reduce the dimensionality of network traffic data to form the final traffic feature set.

[0008] Using extreme gradient boosting, categorical gradient boosting, and lightweight gradient boosting machines as base learners, and a random forest algorithm as a meta-learner, and introducing the Optuna optimization framework for quickly finding optimal hyperparameters in high-dimensional hyperparameter spaces, this paper constructs a stacked generalization model for identifying abnormal traffic data in network security threat situation identification scenarios.

[0009] A stacked generalization model is used to detect abnormal traffic data in the traffic feature set, identify threats, and determine whether the traffic data is abnormal.

[0010] By integrating the feature importance analysis of Extreme Gradient Boosting (XGBoost), Pearson correlation analysis, and the XPM feature selection method of Mutual Information Score, network traffic data is screened and reduced in multiple dimensions to eliminate redundant features and retain key threat-related features.

[0011] Furthermore, based on the improved stacking generalization model (OPStacking), with the extreme gradient boosting algorithm (XGBoost), the category gradient boosting algorithm (CatBoost) and the lightweight gradient boosting machine (LightGBM) as base learners, and random forest as the meta-learner, combined with the Optuna framework, the model hyperparameters are automatically optimized to construct a high-precision and highly generalized threat identification model; through the coordinated operation of the traffic data preprocessing module, the feature selection module and the threat identification module, efficient detection and classification of abnormal traffic in complex network environments are achieved.

[0012] Furthermore, the specific process of forming the final traffic feature set includes the following steps:

[0013] For network traffic data F={f1,f2,...,f n Perform data preprocessing to form multiple traffic feature subsets F1. Data preprocessing includes invalid feature removal, feature one-hot encoding, feature standardization, and label digitization.

[0014] Based on the XGBoost feature screening model, the traffic data in the traffic feature subset F1 is quantified by feature importance, sorted in descending order, and traffic features with high feature importance scores are retained according to preset standards to form the preliminary screened traffic feature subset F2;

[0015] Perform Pearson correlation analysis on the traffic data in the traffic feature subset F2, set the Pearson traffic feature correlation threshold θ2, and calculate the mutual information score of the traffic features in the traffic feature subset F2;

[0016] If there is a traffic feature that exceeds the Pearson traffic feature correlation threshold θ2, the mutual information scores are compared and the traffic features with low scores are eliminated to form the final traffic feature set F3;

[0017] If no traffic feature exceeds the set Pearson traffic feature correlation threshold θ2, the traffic feature set F3 is directly formed. For traffic feature f j have:

[0018]

[0019] In the formula, |r jk | is the Pearson correlation coefficient r jk The absolute value of .

[0020] Furthermore, the traffic feature subset F2 formed for preliminary screening is:

[0021] Filter out the traffic feature subset F1 whose feature importance is higher than the XGBoost traffic feature importance threshold θ1, and the expression is:

[0022] F2={f j ∈F|S XGB (f j )>θ1}

[0023] Where, f j is the jth traffic feature in the traffic feature subset F1; S XGB (f j ) is the flow characteristic f j XGBoost importance score for .

[0024] Furthermore, the traffic data in the traffic feature subset F2 is subjected to Pearson correlation analysis, which is:

[0025] For any two traffic features f in the traffic feature subset F2 j and f k , if its Pearson correlation coefficient r jk If the absolute value of exceeds the Pearson traffic feature correlation threshold θ2, the traffic feature with a higher mutual information score is retained, and the expression is:

[0026]

[0027] Where, I(f j ,Y) is the flow characteristic fj Mutual information score with the target variable Y; I(f k ,Y) is the flow characteristic f k Mutual information score with the target variable Y.

[0028] Furthermore, the final flow feature set F3 is formed as follows:

[0029] For the filtered traffic feature subset F2, retain the traffic features whose mutual information scores are higher than the traffic feature mutual information score threshold θ3, and the expression is:

[0030] F3={f j ∈F2|I(f j ; Y)>θ3}

[0031] Where, I(f j ,Y) is the flow characteristic f j Mutual information score with the target variable Y.

[0032] Furthermore, the stacked generalization model is used to detect abnormal traffic data in the traffic feature set, identify threats, and thus determine whether the traffic data is abnormal. The specific process includes the following steps:

[0033] Let the traffic feature set in the input threat identification module be The optimization goal is to maximize the comprehensive evaluation index, namely:

[0034] S=ω1Accuacy+ω2Precision+ω3Recall+ω4F1

[0035] Where x i Represents the network traffic data feature vector; y i is the actual threat label of the i-th sample; Accuracy, Precision, Recall, and F1 represent accuracy, precision, recall, and F1 score, respectively; ω1, ω2, ω3, and ω4 are indicator weights, and ω1+ω2+ω3+ω4=1;

[0036] Input traffic feature set F3 to base learner L j , the base learner L1 iteratively constructs multiple decision trees to minimize the loss function The expression is:

[0037]

[0038] Where l(·) is the logarithmic loss calculation method; y is the actual threat label; To predict threat labels; is the predicted threat label of the i-th sample; n is the total number of samples, that is, the number of samples included in the traffic feature set F3;

[0039] The parameters of the tree model are obtained by calculating the gradient of the loss function to obtain the predicted output Right now:

[0040]

[0041] Where η is the learning rate; f k is a single regression tree; k∈K is the number of iterations;

[0042] The base learner L2 is trained using a symmetric decision tree and its prediction output is obtained after K iterations. Right now:

[0043]

[0044] In the formula, a(x i ,T) is the probability prediction value output after sample i falls into the leaf node of decision tree T;

[0045] For the base learner L3, its predicted output for:

[0046]

[0047] Where ε is the learning rate of the base learner L3; h k (x i ) represents the predicted value of the k-th decision tree for sample i;

[0048] The prediction results of the base learner are used as new features and compared with the original network traffic data feature vector x i Merge to form a new network traffic feature vector

[0049] Use the new network traffic feature vector z i and the corresponding threat label y i Train the meta-learner M, whose loss function is:

[0050]

[0051] Where, l'(·) is calculated using the logarithmic loss method;

[0052] Building surrogate models using Gaussian processes Approximate the objective function S, based on the existing experimental results set To estimate the S value under different hyperparameter combinations, the new hyperparameter combination θ is calculated by the expected improvement acquisition function EI(θ) new , the expression is:

[0053]

[0054] Where S best is the optimal optimization objective; θ (k) is the hyperparameter combination of the kth experiment; S (k) is the objective function value of the kth experiment; t is the number of experiments included in the experimental result set;

[0055] The hyperparameter combination θ obtained from the experiment new and the optimal optimization objective S best Save to the result set;

[0056] Repeat the experiment until the preset number of trials is reached to obtain the final optimal hyperparameter combination θ * , the best hyperparameter combination θ * The corresponding stacked generalization model is the optimal training model, which is used to predict the probability that sample i belongs to category c.

[0057] After calculating the probability distribution of each category, the final predicted category for:

[0058]

[0059] Where argmax is the calculated probability The function of the maximum c value is used to determine the most likely threat type of traffic sample i.

[0060] Furthermore, the stacked generalization model is used to predict the probability that sample i belongs to category c. The expression is:

[0061]

[0062] Where, f M,c (z i θ M ) is the prediction score of the meta-learner M for sample i in category c, with a total of C categories; f M,k (z i θ M ) is the prediction score of the meta-learner M for sample i in category k; exp(·) is the natural exponential function.

[0063] A system for implementing the above-mentioned network abnormal traffic detection method comprises:

[0064] Traffic data preprocessing module preprocesses network traffic data, including data cleaning, feature screening, encoding conversion, and standardization operations;

[0065] The feature selection module analyzes the importance, linear correlation, and nonlinear dependency of traffic features based on feature importance analysis, Pearson correlation analysis, and mutual information score, and selects the most valuable features to form a traffic feature set;

[0066] The threat identification module integrates the Optuna optimization framework and the Stacking ensemble learning algorithm to detect abnormal traffic data in the traffic feature set, identify threats, and determine whether the traffic is abnormal.

[0067] By means of the above technical solution, the present invention provides a method and system for detecting abnormal network traffic based on ensemble learning, which has at least the following beneficial effects:

[0068] 1. To address the problem of dimensional explosion of network security situation factor features, this paper proposes an XPM network traffic data feature selection method that integrates extreme gradient boosting feature importance analysis, Pearson correlation analysis, and mutual information score. This method efficiently screens important features and realizes automatic dimensionality reduction of high-dimensional data.

[0069] 2. In response to the problems of low efficiency, poor accuracy and high requirements for training data quality in traditional network traffic anomaly identification methods, this paper proposes an improved Stacking threat identification method that integrates the Stacking integration algorithm and Optuna hyperparameter optimization. This method has high identification efficiency, good accuracy, strong generalization, and is applicable to multiple sample data.

[0070] 3. The proposed network abnormal traffic detection method has a threat identification accuracy of 83.26% and 99.96% on the UNSW-NB15 and CIC-IDS2017 datasets, respectively. This significantly improves performance compared to traditional algorithms and enables accurate abnormal traffic detection, providing a reliable reference for security decision-making in complex network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0072] Figure 1 This is a flow chart of the network abnormal traffic detection method of the present invention;

[0073] Figure 2 Flowchart of the network traffic feature selection method of the present invention;

[0074] Figure 3 This is a network structure diagram of the stacked generalization model in the present invention;

[0075] Figure 4This is a curve diagram showing the convergence of the hyperparameter optimization of the present invention on UNSW-NB15;

[0076] Figure 5 This is a curve diagram showing the convergence of hyperparameter optimization of the present invention at CIC-IDS2017;

[0077] Figure 6 A radar chart showing the performance of the stacked generalization model and various machine learning algorithms on UNSW-NB15.

[0078] Figure 7 This is a radar chart showing the performance of the stacked generalization model and various machine learning algorithms in the CIC-IDS2017 test.

[0079] Figure 8 This is a bar chart showing the ranking of the influence of the ablation modules in the present invention. DETAILED DESCRIPTION

[0080] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0081] As the types and scales of network threats become increasingly complex, traditional methods for detecting abnormal network traffic are facing severe challenges in terms of high-dimensional feature redundancy, unbalanced data samples, and weak model generalization capabilities. In response to the above problems, this embodiment proposes a method for detecting abnormal network traffic based on ensemble learning, which achieves efficient screening and dimensionality reduction of network traffic features by integrating feature importance analysis of extreme gradient boosting, Pearson correlation analysis, and mutual information scoring. In addition, the extreme gradient boosting algorithm, the category gradient boosting algorithm, and the lightweight gradient boosting machine are used as base learners, and random forest is used as a meta-learner to construct a stacked generalization model, and the Optuna framework is introduced to efficiently find the optimal hyperparameters of the model, thereby achieving high-precision identification of network threats. Figure 1 As shown, the method includes the following steps:

[0082] This embodiment proposes a network traffic feature selection method (XPM) that integrates XGBoost feature importance analysis, Pearson correlation analysis, and MI mutual information score. The method aims to extract the most valuable traffic features for abnormal traffic identification through multi-dimensional traffic feature screening and dimensionality reduction, thereby improving the efficiency and accuracy of abnormal traffic data identification in network traffic data. Specifically, by integrating extreme gradient boosting feature importance analysis, Pearson correlation analysis, and mutual information score, traffic data in network traffic data is screened and reduced to form a final set of traffic features.

[0083] The key to detecting abnormal network traffic is extracting the most valuable features from complex, high-dimensional network data. Traditional traffic feature extraction methods, when faced with high-dimensional features, suffer from high feature redundancy, computational complexity, and a failure to consider the correlation and dependencies between features.

[0084] In this embodiment, the XGBoost algorithm is an ensemble learning algorithm based on gradient boosting trees. It iteratively optimizes the loss function by constructing multiple decision trees, and excels at efficiently modeling complex, high-dimensional traffic data. The contribution of each threat feature to the model performance is quantified by the gain (Gain) when each decision tree node is split. The gain represents the degree of optimization of the loss function after the feature split. The final feature importance score is obtained by weighted summation of the total gain. The calculation formula is:

[0085]

[0086] Where g i and h i are the first-order and second-order derivatives of the loss function, respectively; L and R are the left and right child nodes after splitting; P is the parent node; λ is the regularization coefficient.

[0087] The Pearson correlation coefficient is used to analyze the linear correlation between two traffic features. Its value range is [-1, 1]. The closer the coefficient is to the range boundary, the stronger the correlation between the features. By setting the coefficient threshold, we can determine whether the traffic features are highly redundant, thereby screening out feature pairs with high correlation. The calculation formula is:

[0088]

[0089] Where x i and y i is the eigenvalue; and is the characteristic mean.

[0090] MI mutual information is used to evaluate the nonlinear statistical dependency between traffic features and target variables, and is good at capturing complex dependencies. By calculating the mutual information score between traffic data features and threat labels, combined with the Pearson correlation coefficient r xy Select the most valuable features. The calculation formula of mutual information score I(X;Y) is:

[0091]

[0092] Where p(x,y) is the joint probability distribution; p(x) and p(y) are the marginal probability distributions.

[0093] The specific process of forming the final flow feature set is as follows: Figure 2 As shown, the following steps are included:

[0094] For network traffic data F={f1,f2,...,f n}Perform data preprocessing to form multiple traffic feature subsets F1. Data preprocessing includes invalid feature elimination, feature one-hot encoding, feature standardization, and label digitization to ensure that the traffic data can be effectively used by the feature extraction module.

[0095] Based on the XGBoost feature screening model, the traffic data in the traffic feature subset F1 is quantified in feature importance, sorted in descending order, and traffic features with high feature importance scores are retained according to the preset criteria to form a preliminary screened traffic feature subset F2; that is, the traffic feature subset F1 whose feature importance is higher than the XGBoost traffic feature importance threshold θ1 is screened out, and the expression is:

[0096] F2={f j ∈F|S XGB (f j )>θ1}

[0097] Where, f j is the jth traffic feature in the traffic feature subset F1; S XGB (f j ) is the flow characteristic f j XGBoost importance score for .

[0098] The traffic data in the traffic feature subset F2 are subjected to Pearson correlation analysis, a correlation coefficient matrix heat map is constructed, and the Pearson traffic feature correlation threshold θ2 is set. At the same time, the mutual information score of the traffic features in the traffic feature subset F2 is calculated; that is, for any two traffic features f in the traffic feature subset F2 j and f k , if its Pearson correlation coefficient r jk If the absolute value of exceeds the Pearson traffic feature correlation threshold θ2, the traffic feature with a higher mutual information score is retained, and the expression is:

[0099]

[0100] Where, I(f j ,Y) is the flow characteristic f j Mutual information score with the target variable Y; I(f k ,Y) is the flow characteristic f k Mutual information score with the target variable Y.

[0101] If there is a traffic feature that exceeds the Pearson traffic feature correlation threshold θ2, the mutual information scores are compared and the traffic features with low scores are eliminated to form the final traffic feature set F3. That is, for the filtered traffic feature subset F2, the traffic features with mutual information scores higher than the traffic feature mutual information score threshold θ3 are retained. The expression is:

[0102] F3={f j ∈F2|I(f j ; Y)>θ3}

[0103] If no traffic feature exceeds the set Pearson traffic feature correlation threshold θ2, the traffic feature set F3 is directly formed, that is, for the traffic feature f j have:

[0104]

[0105] In the formula, |r jk | is the Pearson correlation coefficient r jk The absolute value of .

[0106] This embodiment uses phased feature screening to identify potential linear and nonlinear relationships in traffic features, retaining highly contributing features while avoiding information overlap and noise interference. The traffic feature set extracted using this method is highly efficient and robust.

[0107] In the existing technology, Stacking is an integrated learning strategy. Its principle is to construct training data for multiple base learners and input the output of the base learners as new features into the meta-learner for further learning. The purpose is to improve the overall performance of the model by integrating the advantages of multiple base learners. In the scenario of network security threat situation identification, the Stacking method uses multiple different base learners to learn threat features in network traffic. However, the traditional Stacking method usually relies on experience in setting the hyperparameters of the base and meta-learners, making it difficult to achieve the optimal configuration, resulting in limited efficiency and accuracy in threat identification. In addition, when faced with complex network traffic data with high-dimensional features, the model's generalization ability is insufficient and overfitting is prone to occur.

[0108] To address the above issues, this embodiment proposes an improved stacking generalization model based on Optuna hyperparameter optimization: OPStacking. This embodiment uses the extreme gradient boosting algorithm, the category gradient boosting algorithm, and the lightweight gradient boosting machine as base learners, and the random forest algorithm as the meta-learner. It also introduces the Optuna optimization framework for quickly finding the optimal hyperparameters in the high-dimensional hyperparameter space, and constructs a stacking generalization model for abnormal traffic data identification in the network security threat situation identification scenario, thereby achieving efficient and accurate network threat identification. Its model structure is as follows: Figure 3 In this embodiment, XGBoost, CatBoost and LightGBM are selected as base learners, and Random Forest is used as the meta-learner.

[0109] XGBoost boasts powerful parallel computing and modeling capabilities, effectively capturing the complex nonlinear relationships between abnormal traffic and threats when processing large-scale network traffic data. CatBoost, a machine learning algorithm based on symmetric decision trees, excels at classification problems, automatically handles categorical features, and is robust to overfitting. LightGBM, a distributed gradient boosting framework, utilizes optimization methods such as the histogram algorithm to rapidly process feature data. Through gradient unilateral sampling and mutually exclusive feature bundling, it further improves the efficiency and performance of threat identification based on network traffic, saving significant memory resources and computational time. Random Forest, a classic ensemble learning algorithm, uses bootstrap resampling to generate differentiated training sets for each tree. It selects only the best threat features from random subsets during node splitting and automatically adjusts class weights to mitigate the impact of data imbalance, effectively leveraging the prediction data output by the base learner. Optuna, a hyperparameter optimization framework based on Sequential Model Optimization (SMBO), is an improvement on Bayesian optimization. It features intelligent sampling strategies and efficient search algorithms, enabling it to quickly find the most optimal model parameters in a high-dimensional hyperparameter space.

[0110] This embodiment uses a stacked generalization model to detect abnormal traffic data in the traffic feature set, identify threats, and thus determine whether the traffic data is abnormal. Among them, the base learners L1, L2, and L3 are XGBoost, CatBoost, and LightGBM respectively, and the meta learner M is RandomForest. j , whose hyperparameter set is The set of hyperparameters of the meta-learner M is θ M , the hyperparameter space optimized by Optuna is recorded as The specific process of the corresponding OPStacking threat identification method is as follows:

[0111] Let the traffic feature set in the input threat identification module be The optimization goal is to maximize the comprehensive evaluation index, namely:

[0112] S=ω1Accuracy+ω2Precision+ω3Recall+ω4F1

[0113] Where x i Represents the network traffic data feature vector; y i is the actual threat label of the i-th sample; Accuracy, Precision, Recall, and F1 represent accuracy, precision, recall, and F1 score, respectively; ω1, ω2, ω3, and ω4 are indicator weights, and ω1+ω2+ω3+ω4=1.

[0114] Input traffic feature set F3 to base learner L j , the base learner L1 iteratively constructs multiple decision trees to minimize the loss function The expression is:

[0115]

[0116] Where l(·) is the logarithmic loss calculation method; y is the actual threat label; To predict threat labels; is the predicted threat label of the i-th sample; n is the total number of samples, that is, the number of samples contained in the traffic feature set F3.

[0117] The parameters of the tree model are obtained by calculating the gradient of the loss function to obtain the predicted output Right now:

[0118]

[0119] Where η is the learning rate; f k is a single regression tree; k∈K is the number of iterations.

[0120] The base learner L2 is trained using a symmetric decision tree, and its prediction output is obtained after K iterations, namely:

[0121]

[0122] In the formula, a(x i ,T) is the probability prediction value output after sample i falls into the leaf node of decision tree T. Similarly, for the base learner L3, its prediction output is:

[0123]

[0124] Where ε is the learning rate of the base learner L3; h k (x i ) represents the predicted value of sample i by the kth decision tree.

[0125] The prediction results of the base learner are used as new features and compared with the original network traffic data feature vector x i Merge to form a new network traffic feature vector

[0126] Use the new network traffic feature vector z i and the corresponding threat label y i Train the meta-learner M, whose loss function is:

[0127]

[0128] Where l'(·) is calculated using the logarithmic loss method.

[0129] Building surrogate models using Gaussian processes Approximate the objective function S, based on the existing experimental results set To estimate the S value under different hyperparameter combinations, the new hyperparameter combination θ is calculated by the expected improvement acquisition function EI(θ) new , the expression is:

[0130]

[0131] Where S best is the optimal optimization objective; θ (k) is the hyperparameter combination of the kth experiment; S (k) is the objective function value of the kth experiment; the superscript t is the number of experiments included in the experimental result set.

[0132] The hyperparameter combination θ obtained from the experiment new and the optimal optimization objective S best Save to the result set.

[0133] Repeat the experiment until the preset number of trials is reached to obtain the final optimal hyperparameter combination θ * ,θ * The corresponding OPStacking model is the optimal training model, which is used to predict the probability that sample i belongs to category c. The expression is:

[0134]

[0135] Where, f M,c (z i θ M) is the prediction score of the meta-learner M for sample i in category c, with a total of C categories; f M,k (z i θ M ) is the prediction score of the meta-learner M for sample i in category k; exp(·) is the natural exponential function;

[0136] After calculating the probability distribution of each category, the final predicted category for:

[0137]

[0138] Where argmax is the calculated probability The function of the maximum c value is used to determine the most likely threat type of traffic sample i.

[0139] This example integrates extreme gradient boosting, feature importance analysis, Pearson correlation analysis, and mutual information scoring to achieve efficient screening and dimensionality reduction of network traffic features. It also uses the extreme gradient boosting algorithm, the categorical gradient boosting algorithm, and the lightweight gradient boosting machine as base learners, and a random forest as a meta-learner to construct a stacked generalization model. The Optuna framework is introduced to efficiently find the optimal hyperparameters for the model, thereby achieving high-precision identification of network threats.

[0140] This embodiment also proposes a network abnormal traffic detection system based on ensemble learning, which consists of three parts: a traffic data preprocessing module, a feature selection module, and a threat identification module. In view of the problem that existing network abnormal traffic detection methods are difficult to capture abnormal traffic features efficiently and accurately, this system can effectively screen high-dimensional network traffic features and effectively identify traffic samples of network systems. Figure 1 As shown, the functions of each module are as follows:

[0141] The traffic data preprocessing module preprocesses network traffic data, including data cleaning, feature screening, encoding conversion, standardization and other operations.

[0142] The feature selection module analyzes the importance, linear correlation, and nonlinear dependency of traffic features based on feature importance analysis, Pearson correlation analysis, and mutual information score, and selects the most valuable features to form a traffic feature set.

[0143] The threat identification module integrates the Optuna optimization framework and the Stacking ensemble learning algorithm to detect abnormal traffic data in the traffic feature set, identify threats, and determine whether the traffic is abnormal.

[0144] This example, through experiments on two datasets, found that the XPM feature selection method successfully reduced the 48-dimensional traffic features of the UNSW-NB15 dataset to 32, and the 78 features of the CIC-IDS2017 dataset to 40, greatly reducing the redundancy of high-dimensional network traffic data features and significantly saving the computational cost of the abnormal traffic identification model. The OPStacking threat identification method achieved a threat identification accuracy of 83.26% and 99.96% on the UNSW-NB15 and CIC-IDS2017 datasets, respectively. Other performance indicators were also significantly improved compared to traditional algorithms, enabling accurate abnormal traffic detection. The specific experimental process and results analysis are as follows:

[0145] 1.1 Experimental environment configuration and dataset

[0146] This experiment is based on the Windows 11 operating system, Python 3.12 as the base language interpreter, and PyCharm 2023.2.1 Professional Edition as the integrated development platform. The training and evaluation of the machine learning model are primarily based on the Scikit-Learn 1.5.2 framework, with CUDA 118 as auxiliary computing engine to save time and cost for model training. Specific environment information is shown in Table 1.

[0147] Table 1 Experimental environment configuration

[0148]

[0149] The experimental datasets used are the UNSW-NB15 dataset and the CIC-IDS2017 dataset. The UNSW-NB15 dataset is user network traffic data created by the University of New South Wales in 2015 using software such as IXIA PerfectStorm and Metasploit to simulate a real network environment.

[0150] This experiment used the officially provided "UNSW_NB15_training-set.csv" and "UNSW_NB15_testing-set.csv" files, which contain 48 network traffic features, 10 types of sample labels, and 257,673 samples. The number of samples for each label is shown in Table 2. After preprocessing, these files were merged and repartitioned into training and test sets. This dataset overcomes the limitations of traditional datasets such as KDD99, which suffer from outdated threat categories and insufficiently complex network topologies, and has been widely used in network anomaly detection.

[0151] Table 2 UNSW-NB15 experimental dataset information

[0152]

[0153] The CIC-IDS2017 dataset is a network traffic dataset created by the Canadian Institute for Cybersecurity in 2017 using various tools to simulate a real-world network environment. This dataset collects traffic from five days, Monday through Friday, and includes over 8 million network connection records. This dataset, which captures rich features through network traffic collection, features labels focused on threats such as web attacks, penetration attacks, and denial of service, effectively validating the generalization capabilities of models. This experiment used the official data file "Wednesday-workingHours.pcap_ISCX.csv," specifically for machine learning model training. It contains 78 network features, four types of labels, and 692,703 samples. Detailed information is shown in Table 3.

[0154] Table 3 Wednesday-workingHours experimental dataset information

[0155]

[0156] 1.2 Performance Evaluation Metrics

[0157] This example uses accuracy A, precision P, recall R, and the F1 score to evaluate the performance of the XPM-OPStacking model on two datasets. Accuracy represents the proportion of correct threat predictions; precision represents the proportion of true positive examples among samples predicted as positive; recall represents the proportion of true positive examples detected; and the F1 score represents the harmonic mean of precision and recall. These four metrics enable a comprehensive, objective, and integrated evaluation of the XPM-OPStacking model's anomaly identification performance based on network traffic data.

[0158] 1.3 Data Preprocessing and Feature Engineering

[0159] (1) Data preprocessing

[0160] For the UNSW-NB15 dataset, we first merged the original datasets into a single dataset, removed the id and label columns that are not used in training, and removed the following features: srcip, sport, dstip, dsport, stime, ltime, ct_flw_http_mthd, is_ftp_login, ct_ftp_cmd, proto, and service. The first six features represent the source IP address, source port number, destination IP address, destination port number, sample start time, and sample end time. These features are not valuable for training the XPM-OPStacking model. The last five features contain a large number of missing values, null values, and non-numeric features. One-hot encoding them would result in dimensionality explosion, so they were discarded. We then one-hot-encoded the non-numeric state column into a numerical feature, merged the encoded data with the original features, and converted the attack_cat column into a numerical value. The resulting dataset has 47 features. Finally, the features are normalized and the training and test sets are divided into 8:2 ratios. For the CIC-IDS2017 dataset, due to the presence of outliers and missing values, it is processed by replacing infinite values ​​with missing values, filling missing values ​​with the median, and dividing the training set and test set into a ratio of 8:2.

[0161] (2) Feature Engineering

[0162] The XPM method was used to perform feature selection on both datasets. For the UNSW-NB15 dataset, the XGBoost algorithm was used to screen feature importance. This involved removing features that had little contribution to the prediction results and those that might introduce noise. Limiting the number of features significantly reduced the computational burden and accelerated model training. After considering these considerations, features with an importance score of 0 were removed. The correlation and mutual information scores of the remaining features were calculated. The closer the correlation coefficient, the higher the correlation between the two features. A higher mutual information score indicates a greater dependency between the feature and the label.

[0163] Eliminate the following features with a correlation coefficient greater than 0.95 and a lower mutual information score in each pair of features: dloss, sloss, state_FIN, ct_dst_ltm, ct_srv_src, spkts, dpkts, ct_dst_src_ltm. The features that are finally retained are: dur, sbytes, dbytes, rate, sttl, dttl, sload, dload, sinpkt, dinpkt, sjit, djit, swin, stcpb, dtc pb, tcprtt, synack, ackdat, smean, dmean, trans_depth, response_body_len, ct_state_ttl, ct_src_dport_ltm, ct_dst_sport_ltm, ct_src_ltm, ct_srv_dst, is_sm_ips_ports, state_CON, state_ECO, state_INT, state_RST, a total of 32 features.

[0164] Similarly, for the CIC-IDS2017 dataset, the XGBoost algorithm is used to screen the feature importance and remove the ten features with a score of 0, namely Fwd Avg Packets / Bulk, CWE Flag Count, ECE Flag Count, FwdAvg Bytes / Bulk, BwdAvg Bulk Rate, RST Flag Count, Bwd PSH Flags, Fwd URG Flags, Bwd URG Flags, BwdAvg Bytes / Bulk, Bwd Avg Packets / Bulk, and FwdAvg Bulk Rate. From the remaining features, the features with the absolute value of the correlation coefficient greater than 0.95 and the lower mutual information score in each pair of features are removed. The features removed are SubflowFwdPackets, Fwd Packets / s, Packet Length Std, act_data_pkt_fwd, Fwd IAT Total, BwdHeader Length, Subflow Bwd Packets, Fwd Packet Length Mean, and Flow IAT Std, TotalBackward Packets, Fwd Header Length, Idle Max, Fwd IAT Std, Fwd Packet LengthStd, Average Packet Size, Bwd Packet Length Max, Total Length ofFwd Packets, SYNFlag Count, Idle Min, Total Fwd Packets, Fwd IAT Max, Idle Mean, Subflow BwdBytes, Avg Bwd Segment Size, Fwd Header Length.1, Bwd Packet Length Std, retaining a total of 40 features from the official dataset.

[0165] Through the above feature engineering, the 48 features of the UNSW-NB15 dataset were successfully reduced to 32, and the 78 features of the CIC-IDS2017 dataset were reduced to 40. This greatly reduced the redundancy of high-dimensional network traffic data features and significantly saved the computational cost of the abnormal traffic identification model.

[0166] 1.4 Analysis of experimental results

[0167] (1) Threat identification results

[0168] The feature-extracted data is input into the OPStacking threat identification module for training. The Optuna framework is used to optimize the hyperparameters of the stacking model. The CATBoost algorithm comes with regularization methods for ordered improvement and adversarial verification. To prevent overfitting, the default sample sampling ratio and feature sampling ratio are used. Keeping the meta-learner parameters relatively simple is conducive to the overall stability and generalizability of the model architecture, so the range of parameter optimization is targeted at the base learner of the model. The hyperparameter configurations obtained by parameter optimization are shown in Table 4. The number of optimization rounds for UNSW-NB15 and CIC-IDS2017 is set to 30. The optimized hyperparameters are used to train the Stacking model. The hyperparameter optimization convergence curves on the two datasets are shown in Table 4. Figure 4 , as shown in 5.

[0169] Table 4 Hyperparameter configuration optimized by Optuna

[0170]

[0171]

[0172] It can be seen that the optimization process on the UNSW-NB15 dataset is not monotonically increasing. The fluctuations in the blue line indicate that different experimental parameters can cause significant fluctuations in the target value during the search for the optimal solution. This is due to the complexity of the optimization problem and the presence of multiple local optima. The red line eventually stabilizes, indicating that the algorithm gradually finds a relatively stable optimal solution as the number of trials increases. The convergence curve on the CIC-IDS2017 dataset fluctuates significantly in the first few experiments, but after 10 optimization attempts, it reaches a high level and stabilizes, indicating that the optimization process quickly finds an optimal hyperparameter combination. The model's convergence curve demonstrates the effectiveness of the Optuna optimization framework in finding the optimal hyperparameters for this threat identification model.

[0173] To verify the effectiveness of the OPStacking threat identification model in threat identification, we compared it with various machine learning algorithms. The training and validation datasets for all algorithms were preprocessed in the same way. The identification results are shown in Table 5.

[0174] Table 5 Comparison of threat identification performance of different algorithms

[0175]

[0176]

[0177] The performance of the model on the two datasets is presented in the form of a radar chart, such as Figure 6 and Figure 7As shown in the performance comparison, the OPStacking method outperforms all other algorithms combined in identifying threat types on both datasets. On the UNSW-NB15 dataset, it achieved an accuracy of 83.26%, surpassing the logistic regression algorithm by 10 percentage points, significantly improving upon all other algorithms and outperforming traditional machine learning algorithms on all other performance metrics. On the CIC-IDS2017 dataset, the OPStacking model achieved an accuracy of 99.96%, with all other performance metrics exceeding 99%, demonstrating its strong generalization capabilities. This is due to the XPM feature selection method, which effectively removes high-dimensional, redundant features while retaining key threat-related features, reducing noise and overfitting risks. XGBoost, CatBoost, and LightGBM, used as base learners, effectively complement the model's ability to capture complex linear and nonlinear traffic patterns. The Optuna framework automatically searches for optimal hyperparameter combinations, enabling this threat identification model to achieve optimal performance balance in complex network traffic data.

[0178] (2) Ablation experiment

[0179] To verify the impact of each module in the constructed threat identification model on its overall performance, an ablation experiment was designed. Using the Stacking model as the baseline model, the XGBoost, CatBoost, and LightGBM components in the base learner layer, as well as the meta-learner layer, were removed to verify the necessity of the base learner integration. The meta-learner layer's Random Forest algorithm was replaced with logistic regression, gradient boosted decision tree (GBDT), and support vector machine (SVM) to verify the necessity of the meta-learner. The results of the ablation experiment are shown in Table 6. The experimental dataset used was the CIC-IDS2017 dataset to demonstrate the model's performance when dealing with complex network features.

[0180] Table 6 Comparison of ablation experiment results

[0181]

[0182] The experimental results show that changing specific modules in the model can lead to varying degrees of performance degradation in terms of accuracy, precision, recall, and F1 score. Since the F1 score can be used as a single metric to demonstrate the balance between accuracy and completeness of an ablation module, the magnitude of the F1 score reduction is used to quantify the impact of each module.

[0183] Removing the XGBoost component had the greatest impact on the model, reducing the F1 score by 0.15%, indicating that this base learner component is a key component of the model. Removing the base learner resulted in a 0.01%-0.15% decrease in F1 score, demonstrating that the heterogeneity of the three base learners is complementary in learning network traffic features. Random Forest, as a meta-learner, had a 0.05% higher impact than GBDT, 0.07% higher than Logistic Regression, and 0.06% higher than SVM, indicating that the Random Forest model is more suitable for feature recombination and capturing high-order features, and that nonlinear meta-learners are more suitable for this model. Figure 8 The absolute value of the reduction in the F1 score in each ablation experiment is used as the impact score, showing the ranking of the impact of the ablation module on the complete model. The results of the ablation experiments indicate that the ensemble learning model constructed in this paper provides a clear direction for future optimization: focusing on maintaining the XGBoost base learner and the ForestRandom meta-learner, focusing on their strengths in handling nonlinear features, while secondary optimization of other base learners.

[0184] In summary, with the rapid development of networks and artificial intelligence, network environments are becoming increasingly complex and severe. Rapid and effective detection of network traffic is of great value and significance. Traditional methods for detecting abnormal network traffic suffer from high-dimensional feature redundancy and poor model generalization. Therefore, this embodiment proposes an XPM-OPStacking integrated machine learning model for detecting abnormal network traffic. XPM extracts high-value key traffic features, and OPStacking accurately identifies network threats, thereby detecting abnormal traffic. Experimental results show that the XPM-OPStacking method outperforms traditional machine learning models in various evaluation indicators and has strong generalization capabilities.

[0185] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to in detail. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For relevant parts, please refer to the partial description of the method embodiments.

[0187] The above embodiments have been described in detail, and the principles and embodiments of the present application have been described by applying specific examples. The above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific embodiments and application scope will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A network abnormal traffic detection method based on ensemble learning, characterized in that: The method comprises the following steps: By integrating extreme gradient boosting feature importance analysis, Pearson correlation analysis, and mutual information score, we can filter and reduce the dimensionality of network traffic data to form the final traffic feature set. This includes the following steps: Network traffic data Perform data preprocessing to form multiple traffic feature subsets ,Data preprocessing includes invalid feature removal, feature one-hot encoding, feature standardization, and label digitization; Based on the XGBoost feature screening model, the traffic feature subset The traffic data in the dataset are quantified for feature importance, sorted in descending order, and traffic features with high feature importance scores are retained according to the preset criteria to form a preliminary screened traffic feature subset. ; Subset the traffic characteristics Perform Pearson correlation analysis on the traffic data in the dataset and set the Pearson traffic feature correlation threshold. , and calculate the traffic feature subset Mutual information score of mid-flow features; If there is a traffic feature that exceeds the Pearson traffic feature correlation threshold , then compare the mutual information scores, eliminate the traffic features with low scores, and form the final traffic feature set ; If no traffic feature exceeds the set Pearson traffic feature correlation threshold , then the traffic feature set is directly formed ; Using extreme gradient boosting, categorical gradient boosting, and lightweight gradient boosting machines as base learners, and a random forest algorithm as a meta-learner, and introducing the Optuna optimization framework for quickly finding optimal hyperparameters in high-dimensional hyperparameter spaces, this paper constructs a stacked generalization model for identifying abnormal traffic data in network security threat situation identification scenarios. A stacked generalization model is used to detect abnormal traffic data in the traffic feature set, identify threats, and determine whether the traffic data is abnormal.

2. The method for detecting abnormal network traffic according to claim 1, wherein: The flow feature subset formed by the preliminary screening , which is: Filter out features whose importance is higher than the XGBoost traffic feature importance threshold Traffic feature subset , the expression is: ； Where, is the traffic feature subset Middle Traffic characteristics; Traffic characteristics XGBoost importance score for .

3. The method for detecting abnormal network traffic according to claim 1, wherein: The traffic feature subset The traffic data in the Pearson correlation analysis is: For the traffic feature subset Any two flow characteristics and , if its Pearson correlation coefficient The absolute value of exceeds the Pearson flow feature correlation threshold , then retain the traffic features with higher mutual information scores, the expression is: ； Where, Traffic characteristics Mutual information score with the target variable Y; Traffic characteristics Mutual information score with the target variable Y.

4. The method for detecting abnormal network traffic according to claim 1, wherein: The final flow feature set is formed , which is: For the filtered traffic feature subset , retain the mutual information score higher than the traffic feature mutual information score threshold The flow characteristics of , expressed as: ； Where, Traffic characteristics Mutual information score with the target variable Y.

5. The method for detecting abnormal network traffic according to claim 1, wherein: The stacked generalization model is used to detect abnormal traffic data in the traffic feature set, identify threats, and thus determine whether the traffic data is abnormal. The specific process includes the following steps: Let the traffic feature set in the input threat identification module be The optimization goal is to maximize the comprehensive evaluation index, namely: ； Where, Represents the network traffic data feature vector; It is The actual threat label of each sample; 、 、 、 They represent accuracy, precision, recall and F1 score respectively; 、 、 、 are the indicator weights, and ; Input traffic feature set To base learner , base learner Iteratively build multiple decision trees to minimize the loss function , the expression is: ； Where, The logarithmic loss calculation method is adopted; For actual threat labels; To predict threat labels; It is The predicted threat labels of samples; is the total number of samples, that is, the traffic feature set The number of samples included in ; The parameters of the tree model are obtained by calculating the gradient of the loss function to obtain the predicted output ,Right now: ； Where, is the learning rate; is a single regression tree; , is the number of iterations; Base Learner Using symmetric decision tree for training, After iterations, the predicted output is obtained ,Right now: ； Where, For the Samples fall into the decision tree The probability prediction value output after the leaf node; For the base learner , its predicted output for: ； Where, Base learner The learning rate; Indicates that the kth decision tree The predicted value of samples; The prediction results of the base learner are used as new features and compared with the original network traffic data feature vector Merge to form a new network traffic feature vector , ; Using new network traffic feature vectors and the corresponding threat labels Training the meta-learner , its loss function is: ； Where, The logarithmic loss calculation method is adopted; Building surrogate models using Gaussian processes For the objective function Make an approximation based on the existing experimental results set To estimate the Value, through the expected improvement acquisition function Calculate new hyperparameter combinations , the expression is: ； Where, is the best optimization goal; is the hyperparameter combination of the kth experiment; is the objective function value of the kth experiment; is the number of experiments included in the experimental result set; Combining the hyperparameters obtained from the experiment and the optimal optimization objective Save to the result set; Repeat the experiment until the preset number of trials is reached to obtain the final optimal hyperparameter combination , the best hyperparameter combination The corresponding stacked generalization model is the optimal training model. samples belong to the category Probability ; After calculating the probability distribution of each category, the final predicted category for: ； Where, Is to calculate the probability Maximum The function of the value is used to determine the The most likely threat type of the sample.

6. The method for detecting abnormal network traffic according to claim 5, wherein: The stacked generalization model is used to predict the samples belong to the category Probability The expression is: ； Where, It is a meta-learner For the first samples in the category The prediction scores on categories; It is a meta-learner For the first samples in the category The prediction score on ; is a natural exponential function.

7. A system for implementing the method for detecting abnormal network traffic according to any one of claims 1 to 6, characterized in that: include: Traffic data preprocessing module preprocesses network traffic data, including data cleaning, feature screening, encoding conversion, and standardization operations; The feature selection module analyzes the importance, linear correlation, and nonlinear dependency of traffic features based on feature importance analysis, Pearson correlation analysis, and mutual information score, and selects the most valuable features to form a traffic feature set; The threat identification module integrates the Optuna optimization framework and the Stacking ensemble learning algorithm to detect abnormal traffic data in the traffic feature set, identify threats, and determine whether the traffic is abnormal.

Images