Network traffic anomaly detection method and device based on feature grouping and auto-encoder integration
Through the integration of feature grouping and autoencoder, the reconstruction error calculation and weighted fusion of network traffic characteristics is used by the autoencoder, which solves the problem of low abnormal detection performance in the prior art and realizes more efficient network traffic abnormal detection .
Patent Information
- Application Number
- CN202410962396.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-08
AI Technical Summary
The existing network traffic anomaly detection technology based on autoencoder failed to effectively utilize forward propagation and backpropagation properties in data preprocessing and model architecture, and did not pay attention to the fusion of learning results of multiple autoencoders, resulting in relatively good abnormal detection performance. Low.
The network traffic characteristics are processed through feature packets, and the autoencoder that pre-learned normal traffic rules are used to perform reconstruction error calculations, and weighted fusion is performed to improve abnormal detection performance.
Without changing the conventional autoencoder structure, the performance of network traffic anomaly detection is improved, and the accuracy and efficiency of detection is improved.
Smart Images

Figure CN118921196A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network security and network management, and in particular to a method and device for detecting network traffic anomalies based on feature grouping and autoencoder integration. Background Art
[0002] At present, the research on network traffic anomaly detection technology based on autoencoders mainly focuses on data preprocessing and machine learning model architecture. For example, in data preprocessing, the research focuses on sample balance, feature principal component analysis, outlier processing, feature selection, etc. In the machine learning model architecture, it involves variational autoencoders, deep autoencoders, sparse autoencoders, memory-enhanced deep autoencoders, integrated autoencoders, etc. These studies do not involve grouping the original features from the forward propagation attributes (from the source end) and the reverse propagation attributes (from the destination end) during data preprocessing, nor do they provide a corresponding relationship between the reconstruction error and the anomaly detection score. In the encoder integration research, the main consideration is the joint learning of multiple autoencoders, and little attention is paid to the fusion of the learning results of multiple autoencoders.
[0003] In view of this, how to provide a network traffic anomaly detection method to improve the anomaly detection performance without changing the conventional autoencoder structure has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] Embodiments of the present application provide a network traffic anomaly detection method based on feature grouping and autoencoder integration, a network traffic anomaly detection device based on feature grouping and autoencoder integration, an electronic device and a computer storage medium, which are used to solve the problem of poor performance of current network traffic anomaly detection.
[0005] In a first aspect of an embodiment of the present application, a network traffic anomaly detection method based on feature grouping and autoencoder integration is provided, comprising:
[0006] Based on the characteristic grouping data processing principle, the characteristics of the network traffic to be detected are characteristically grouped to obtain characteristic group data, wherein the characteristic grouping data processing principle is a characteristic grouping principle determined based on characteristic attributes corresponding to the network traffic;
[0007] Analyzing the feature group data based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected, wherein the autoencoder is an autoencoder that has previously learned the inherent rules of normal traffic grouping features;
[0008] Based on the autoencoder model, the traffic feature to be detected is reconstructed to obtain a reconstructed network traffic feature, and a reconstruction error between the network traffic feature to be detected and the reconstructed network traffic feature is calculated;
[0009] Based on the reconstruction error and the optimal threshold of the reconstruction error, a flow feature detection score under at least one of the autoencoder models is calculated, and weighted fusion is performed to obtain a scoring result, wherein the scoring result includes a first scoring result corresponding to the normal flow data of the flow feature to be detected, and a second scoring result corresponding to the abnormal flow data of the flow feature to be detected;
[0010] Based on the first scoring result and the second scoring result, an abnormal situation corresponding to the network traffic feature to be detected is determined.
[0011] In a second aspect of an embodiment of the present application, a network traffic anomaly detection device based on feature grouping and autoencoder integration is provided, characterized in that it includes:
[0012] The grouping module is configured to perform feature grouping on the network traffic features to be detected based on a feature grouping data processing principle to obtain feature group data, wherein the feature grouping data processing principle is a feature grouping principle determined based on feature attributes corresponding to the network traffic;
[0013] An analysis module is configured to analyze the feature group data based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected, wherein the autoencoder is an autoencoder that has previously learned the inherent rules of normal traffic grouping features;
[0014] A reconstruction module is configured to reconstruct the traffic feature to be detected based on the autoencoder model to obtain a reconstructed network traffic feature, and calculate a reconstruction error between the network traffic feature to be detected and the reconstructed network traffic feature;
[0015] A scoring module is configured to calculate a traffic feature detection score under at least one of the autoencoder models based on the reconstruction error and the optimal threshold of the reconstruction error, and perform weighted fusion to obtain a scoring result, wherein the scoring result includes a first scoring result corresponding to the normal traffic data of the traffic feature to be detected, and a second scoring result corresponding to the abnormal traffic data of the traffic feature to be detected;
[0016] The determination module is configured to determine the abnormal situation corresponding to the network traffic feature to be detected based on the first scoring result and the second scoring result.
[0017] According to a third aspect of an embodiment of the present application, a computing device is provided, including:
[0018] Memory and processor;
[0019] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned network traffic anomaly detection method based on feature grouping and autoencoder integration are implemented.
[0020] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned network traffic anomaly detection method based on feature grouping and autoencoder integration are implemented.
[0021] The present application provides a method for detecting anomalies in network traffic based on feature grouping and autoencoder integration. First, based on the feature grouping data processing principle, the network traffic features to be detected are feature grouped to obtain feature group data, wherein the feature grouping data processing principle is a feature grouping principle determined based on the feature attributes corresponding to the network traffic; then, based on the learned autoencoder, the feature group data is analyzed to determine at least one autoencoder model corresponding to the network traffic features to be detected, wherein the autoencoder is an autoencoder that has previously learned the inherent laws of normal traffic grouping features; secondly, based on the autoencoder model, the traffic features to be detected are reconstructed to obtain reconstructed network traffic features, and the reconstruction error between the network traffic features to be detected and the reconstructed network traffic features is calculated; thirdly, based on the reconstruction error and the optimal threshold of the reconstruction error, a traffic feature detection score under at least one of the autoencoder models is calculated, and weighted fusion is performed to obtain a scoring result, wherein the scoring result includes a first scoring result corresponding to the normal traffic data of the traffic features to be detected, and a second scoring result corresponding to the abnormal traffic data of the traffic features to be detected; finally, based on the first scoring result and the second scoring result, the abnormal situation corresponding to the network traffic features to be detected is determined.
[0022] The network traffic anomaly detection method based on feature grouping and autoencoder integration provided in the embodiment of the present application is applied. Without changing the conventional autoencoder structure, the data to be detected is analyzed by at least one autoencoder, and the respective encoder models are calculated to reconstruct the data to be detected to obtain the reconstruction error, and the data to be detected is scored based on the reconstruction error, thereby improving the network traffic anomaly detection performance.
[0023] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0025] Figure 1 A schematic diagram of the structure of a network traffic anomaly detection system based on feature grouping and autoencoder integration provided in an embodiment of the present application;
[0026] Figure 2 A flowchart of a network traffic anomaly detection method based on feature grouping and autoencoder integration provided in an embodiment of the present application;
[0027] Figure 3 A schematic diagram of the relationship between the normal traffic single group feature reconstruction error and the optimal threshold value in a network traffic anomaly detection method based on feature grouping and autoencoder integration provided in an embodiment of the present application;
[0028] Figure 4 A schematic diagram of the structure of a network traffic anomaly detection device based on feature grouping and autoencoder integration provided in an embodiment of the present application;
[0029] Figure 5 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0031] See also Figure 1 , Figure 1 A schematic diagram of the structure of a network traffic anomaly detection system based on feature grouping and autoencoder integration provided in an embodiment of the present application. Figure 1 As shown, the network traffic anomaly detection system includes a feature grouping module 101, a feature learning module 102, an AUC and determination threshold calculation module 103 and an application module 104.
[0032] Specifically, the feature grouping module 101 is used to select different features to construct grouping data according to the network traffic feature attributes and variance size.
[0033] The feature learning module 102 is used to learn the grouping features of normal traffic based on multiple autoencoders, and obtain the autoencoder model weights and normal traffic reconstruction error distribution under each grouping feature. That is, first determine the training data; then based on the feature grouping module 101, feature group the training data; finally, train multiple autoencoders according to normal traffic, so as to obtain the trained autoencoder model and normal traffic error distribution.
[0034] The AUC and optimal threshold calculation module 103 is used to analyze the learned autoencoder through the verification data, calculate the AUC value of each encoder in combination with the ROC curve, and determine the optimal threshold of each encoder based on the maximum Youden index principle. That is, first determine the verification data; then based on the feature grouping module 101, feature group the verification data; finally, calculate the AUC value and optimal threshold of each encoder model based on the verification data, so as to obtain the AUC value of each encoder model and the optimal threshold of each encoder model abnormality detection.
[0035] The application module 104 is used to select the autoencoder for anomaly detection application through the AUC size of each encoder and calculate the weights of each encoder when weighted fusion is performed. The second is used to calculate the anomaly detection score of the measured data under the selected autoencoder through the normal traffic error distribution and the optimal threshold, and then obtain the final score through weighted fusion to achieve anomaly detection. That is, first determine the data to be tested; then based on the feature grouping module 101, feature group the data to be tested; finally, based on the trained autoencoder model, normal traffic error distribution, autoencoder model AUC, and autoencoder model optimal threshold, realize unknown traffic anomaly detection and obtain anomaly detection results.
[0036] The network traffic anomaly detection system based on feature grouping and autoencoder integration provided in the embodiment of the present application analyzes the data to be detected through at least one autoencoder without changing the conventional autoencoder structure, calculates the reconstruction error of the data to be detected by each encoder model, and scores the data to be detected based on the reconstruction error, thereby improving the network traffic anomaly detection performance.
[0037] See also Figure 2 , Figure 2 A flowchart of a network traffic anomaly detection method based on feature grouping and autoencoder integration provided in an embodiment of the present application. Figure 2 As shown, the specific steps include the following steps.
[0038] Step S202: based on the characteristic grouping data processing principle, characteristic grouping is performed on the characteristics of the network traffic to be detected to obtain characteristic group data, wherein the characteristic grouping data processing principle is a characteristic grouping principle determined based on characteristic attributes corresponding to the network traffic.
[0039] Step S204: Analyze the feature group data based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected, wherein the autoencoder is an autoencoder that has previously learned the inherent rules of normal traffic grouping features.
[0040] Step S206: Based on the autoencoder model, reconstruct the traffic feature to be detected to obtain a reconstructed network traffic feature, and calculate the reconstruction error between the network traffic feature to be detected and the reconstructed network traffic feature.
[0041] Step S208: Based on the reconstruction error and the optimal threshold of the reconstruction error, calculate the flow feature detection score under at least one of the autoencoder models, and perform weighted fusion to obtain a scoring result, wherein the scoring result includes a first scoring result corresponding to the normal flow data of the flow feature to be detected, and a second scoring result corresponding to the abnormal flow data of the flow feature to be detected.
[0042] Step S210: Based on the first scoring result and the second scoring result, determine the abnormal situation corresponding to the network traffic feature to be detected.
[0043] It should be noted that the feature grouping data processing principle here is a feature grouping principle determined based on the feature attributes corresponding to the network traffic, wherein the feature grouping processing principle includes three steps: data preprocessing, feature classification and feature selection.
[0044] In the embodiment of the present application, based on the principle of feature grouping data processing, the network traffic features to be detected are feature grouped to obtain feature group data, including:
[0045] Based on the target normalization method, the network traffic characteristics to be detected are normalized to obtain the initial network traffic characteristics;
[0046] Determine a characteristic attribute corresponding to the initial network traffic characteristic, wherein the characteristic attribute includes a first characteristic attribute, a second characteristic attribute, and a third characteristic attribute;
[0047] Based on the first characteristic attribute and the third characteristic attribute, determining the initial network traffic feature as a first type of characteristic group data, and calculating a first variance value corresponding to each characteristic data in the first type of characteristic group data to generate a first variance set;
[0048] Based on the second characteristic attribute and the third characteristic attribute, determining the initial network traffic feature as a second type of characteristic group data, and calculating a second variance value corresponding to each characteristic data in the second type of characteristic group data to generate a second variance set;
[0049] Based on a preset grouping principle, the first variance set and the second variance set obtain feature group data, wherein the preset grouping principle carries the number corresponding to each type of feature data grouping, the number corresponding to the feature data in each group, and the variance condition satisfied by each variance in the feature group data, and the variance condition is the number corresponding to the feature data in each type of feature data grouping.
[0050] In practical applications, based on the target normalization method, when normalizing the network traffic features to be detected, the non-numerical features in the traffic features are firstly labeled and encoded, and then the features are normalized using the MinMax method. According to the first calculation formula, the features are normalized using the MinMax method. The first calculation formula is as follows:
[0051]
[0052] Among them, x ij is the value corresponding to the feature of the jth column of the i-th data; i is the value corresponding to the order of data arrangement; j is the value corresponding to the order of data feature arrangement.
[0053] After completing the preprocessing of the network traffic features to be detected, the embodiment of the present application will perform feature classification on the obtained initial network traffic features. It should be noted that the embodiment of the present application performs feature classification on the initial network traffic features based on the feature attributes corresponding to the initial network traffic features, wherein the feature attributes here include a first feature attribute, a second feature attribute and a third feature attribute.
[0054] The first characteristic attribute here is a characteristic attribute determined based on the forward propagation attribute of the network traffic characteristic. For example, the network traffic characteristic from the source end may be the source port, the total size of the forward data packet, and the like.
[0055] The second characteristic attribute here is a characteristic attribute determined based on the reverse propagation attribute of the network traffic characteristic. For example, the network traffic characteristic from the destination end may be the destination port, the total size of the reverse data packet, and the like.
[0056] The third characteristic attribute here is a property shared by both the forward propagation and reverse traditions, such as the transmission protocol.
[0057] When performing feature classification on the network traffic to be detected, the embodiment of the present application combines the first feature attribute and the third feature attribute into a first type of feature group data, referred to as a first type of feature; and combines the second feature attribute and the third feature attribute into a second type of feature group data, referred to as a second type of feature.
[0058] After the network traffic to be detected is divided into the first type of feature group data and the second type of feature group data, in order to detect the traffic features more accurately, the embodiment of the present application also performs feature selection.
[0059] Specifically, when performing feature selection, firstly, the first variance value corresponding to each feature data in the first type of feature group data is calculated to generate a first variance set; and the second variance value corresponding to each feature data in the second type of feature group data is calculated to generate a second variance set; and then feature selection is performed based on a preset grouping principle.
[0060] It should be noted that the preset grouping principle here carries the number corresponding to each type of feature data grouping, the number corresponding to the feature data in each group, and the variance condition satisfied by each variance in the feature group data, and the variance condition is the number corresponding to the feature data in each type of feature data grouping.
[0061] In practical applications, the number of groups required for each type of feature (m) and the number of features required for each group (N i , i = 1, 2, 3, ..., m), each group selects the first N with the largest variance i features as a group. Then set different m and N i , so we can get two categories of 2m groups with different features.
[0062] In the embodiment of the present application, the feature group data is analyzed based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected, including:
[0063] Taking the feature group data as verification data, based on the feature grouping data processing principle, feature grouping the verification data to obtain first feature group data, wherein the first feature group data carries label information corresponding to the verification data;
[0064] Reconstructing the first feature group data based on the learned autoencoder to obtain first reconstructed data, and calculating a first reconstruction error between the first feature group data and the first reconstructed data;
[0065] Calculate the true positive rate and false positive rate corresponding to the autoencoder model under different detection thresholds according to the first reconstruction error and the label information corresponding to the verification data;
[0066] Determine the AUC value corresponding to each of the autoencoders based on the ROC curve generated by the true positive rate and the false positive rate;
[0067] An autoencoder model that meets a preset AUC threshold is determined as at least one autoencoder model corresponding to the network traffic feature to be detected.
[0068] In practical applications, first, data containing normal traffic and abnormal traffic are selected as verification data, and the verification data is feature grouped according to the data processing principle of the feature grouping module; then, based on the learned multiple autoencoders, the verification data after feature grouping are reconstructed respectively, and the reconstruction error between the reconstructed data and the original verification data is calculated; secondly, according to the reconstruction error of the verification data and the true label information of the verification data, the true positive rate and false positive rate of the model under different detection thresholds are calculated; finally, the ROC curve is plotted according to the true positive rate and false positive rate under different detection thresholds, and the AUC value of each encoder is obtained, and the autoencoder model with an AUC value greater than the preset AUC value is selected to perform anomaly detection on the detection data.
[0069] In the embodiment of the present application, multiple autoencoders are used to learn the inherent rules of normal traffic grouping characteristics, including:
[0070] Taking normal traffic characteristics as training data, based on the characteristic grouping data processing principle, characteristic grouping is performed on the training data to generate at least one set of second characteristic group data;
[0071] Constructing at least one autoencoder, and inputting the second feature group data into at least one of the autoencoders for learning, to obtain at least one group of second reconstructed data;
[0072] According to the second feature group data and the second reconstructed data, the reconstruction error corresponding to the training data is calculated, and based on the back propagation of the reconstruction error, an optimizer is used to adjust the parameters corresponding to at least one of the autoencoders to obtain an autoencoder model; in actual applications, after error back propagation, the optimizer is used to adjust the parameters of each encoder.
[0073] Output the trained autoencoder model weights and the corresponding normal traffic grouping feature reconstruction error distribution.
[0074] In addition, after determining the AUC value of each encoder, the embodiment of the present application also calculates the optimal threshold corresponding to each encoder. Specifically, based on the Youden index maximization principle, the optimal threshold of the reconstruction error corresponding to at least one of the autoencoders is determined.
[0075] In this embodiment of the application, based on the second calculation formula, the optimal threshold of the reconstruction error corresponding to each encoder is calculated, and the second calculation formula is:
[0076] After determining the reconstruction errors of the respective encoders and the optimal threshold of the reconstruction error, the flow feature detection score under at least one of the autoencoder models is calculated, and weighted fusion is performed to obtain a scoring result.
[0077] In the embodiment of the present application, the flow feature detection score under at least one of the autoencoder models is calculated based on the reconstruction error and the optimal threshold of the reconstruction error, including:
[0078] Counting the reconstruction error distribution of the normal traffic group feature obtained by at least one autoencoder for reconstructing the normal traffic group feature, and determining the optimal threshold of the reconstruction error corresponding to the normal traffic group feature reconstruction error distribution;
[0079] In response to the reconstruction error being less than the reconstruction error optimal threshold, calculating, based on a third calculation formula, a flow feature detection score under at least one of the autoencoder models;
[0080] In response to the reconstruction error being greater than the optimal reconstruction error threshold, a traffic feature detection score under at least one of the autoencoder models is calculated based on a fourth calculation formula.
[0081] In practical applications, we first count the distribution of reconstruction errors of normal traffic group features in their respective encoders during the feature learning phase. The schematic diagram of the reconstruction error of a single group feature of normal traffic is shown in Figure 3 As shown, Figure 3 A schematic diagram of the relationship between the reconstruction error of a single group feature of normal traffic and the optimal threshold in a network traffic anomaly detection method based on feature grouping and autoencoder integration provided in an embodiment of the present application; then the relative relationship between the reconstruction error of the detection data and the optimal threshold is analyzed. If the reconstruction error is less than the optimal threshold, the traffic feature detection score under at least one of the autoencoder models is calculated based on the third calculation formula. The third calculation formula here is:
[0082]
[0083] Among them, P normal and P anormal N is the score of normal and abnormal test results respectively; normal is the number of samples whose normal flow error in the feature learning stage is less than the optimal threshold; M1 is the number of samples whose normal flow reconstruction error in the feature learning stage is between the detection data reconstruction error and the optimal threshold.
[0084] If the reconstruction error is greater than the optimal threshold, the flow feature detection score under at least one of the autoencoder models is calculated based on the fourth calculation formula, where the fourth calculation formula is:
[0085]
[0086] Among them, P normal and P anormal N is the score of normal and abnormal test results respectively; normalis the number of samples whose normal flow error in the feature learning stage is greater than the optimal threshold; M2 is the number of samples whose normal flow reconstruction error in the feature learning stage is between the detection data reconstruction error and the optimal threshold.
[0087] Finally, P normal and P anormal Combine them to get the scores of the detection data by each encoder, namely:
[0088] P AE-i =[P normal-i ,P anormal-i ],i=1,2,...,N
[0089] Where P AE-i Score the detection data under the i-th autoencoder.
[0090] The scores of the detection data obtained by each encoder are weighted and fused. Specifically, the weighted fusion method includes: calculating the corresponding weight coefficient of each autoencoder when participating in the score fusion according to the AUC value corresponding to each autoencoder; and weighted fusion of the score results based on the weight coefficient.
[0091] In practical applications, first, according to the AUC value, the weights of each encoder participating in the score fusion are calculated, where the fifth calculation formula is as follows:
[0092]
[0093] Among them, AUC i For AE i AUC value, N is the number of autoencoders.
[0094] Then, the scores obtained by multiple autoencoders are weighted and fused to obtain the final score P of the detection data. final , where the sixth calculation formula is as follows:
[0095]
[0096] Among them, N' is the number of autoencoders selected to participate in the scoring fusion.
[0097] The scoring results obtained in the embodiment of the present application include a first scoring result corresponding to the normal traffic data of the traffic feature to be detected, and a second scoring result corresponding to the abnormal traffic data of the traffic feature to be detected.
[0098] Therefore, based on the first scoring result and the second scoring result, determining the abnormal situation corresponding to the to-be-detected network traffic feature includes:
[0099] In response to the first scoring result being greater than the second scoring result, it is determined that the traffic feature to be detected is normal traffic data, that is,
[0100] In response to the first scoring result being less than the second scoring result, it is determined that the traffic feature to be detected is abnormal traffic data, that is,
[0101] Based on the above method, the present application also provides an exemplary embodiment, assuming that the test conditions are set as follows:
[0102] (1) The experimental hardware is: 11th Gen Intel(R) Core(TM) i9-11900, 2.50GHz CPU, 64.0GB RAM; the software environment is: Windows 10 operating system, TensorFlow-CPU 2.13.0 deep learning framework, Python 3.8.18.
[0103] (2) Experiments were conducted using the UNSW-NB15 and CICIDS2017 datasets, see Table 1 for details;
[0104] Table 1 Experimental dataset
[0105]
[0106] (3) The autoencoder structure of the baseline method and this method is the same, and a single autoencoder is interconnected by multiple layers of fully connected neural networks;
[0107] (4) The evaluation indicators are F1 value and accuracy;
[0108] (5) The first category of features of UNSW-NB15 includes ‘dur’, ‘proto’, ‘service’, ‘state’, ‘spkts’, ‘sbytes’, ‘sttl’, etc., with a total of 30 features, and the second category of features includes ‘dur’, ‘proto’, ‘service’, ‘state’, ‘dpkts’, ‘dbytes’, dttl’, etc., with a total of 30 features. The first category of features of CICIDS2017 includes ‘Src Port’, ‘Protocol’, ‘Flow Duration’, ‘Total Fwd Packet’, ‘Total Length of Fwd Packet’, etc., with a total of 55 features, and the second category of features includes ‘DstPort’, ‘Protocol’, ‘Flow Duration’, ‘Total Bwd packets’, ‘Total Length of Bwd Packet’, etc., with a total of 53 features.
[0109] (6) The number of feature groups in the UNSW-NB15 and CICIDS2017 datasets is m = 3. The number of high-variance features selected for the UNSW-NB15 dataset is 30, 25, and 20, respectively, and the number of high-variance features selected for the CICIDS2017 dataset is 40, 30, and 20, respectively. The baseline method processes the raw feature data based on a single autoencoder.
[0110] (7) When anomaly detection is applied, an autoencoder corresponding to the maximum AUC value is selected from each of the first and second category features for weighted fusion.
[0111] (8) The rest of the experimental parameter settings are shown in Table 2.
[0112] Table 2 Experimental parameters
[0113] name Value Batch size 64 dropout 0.02 Learning Rate 0.0001 Reconstruction error type Mae Epoch 50 Optimizer Adam()
[0114] Table 3 shows the anomaly detection results under different data sets. The results show that compared with the baseline method, the F1 and accuracy of anomaly detection of this method are better. For example, for the UNSW-NB15 data set, the F1 value and accuracy of this method are 81.53% and 79.23% respectively (the autoencoder structure is [38, 20, 8, 20, 38]), which are 8.00% and 12.04% higher than the baseline method, respectively. For the CICIDS2017 data set, the F1 value and accuracy of this method are 76.38% and 85.13% respectively, which are 10.75% and 10.52% higher than the baseline method (the autoencoder structure is [64, 32, 18, 32, 64]). It can be seen that the anomaly detection method of the present invention is feasible and effective, and feature grouping and multi-autoencoder fusion can improve the performance of network traffic anomaly detection.
[0115] Table 3 Anomaly detection results
[0116]
[0117] By applying the network traffic anomaly detection method based on feature grouping and autoencoder integration provided in the embodiment of the present application, it is possible to improve the network traffic anomaly detection performance without changing the conventional autoencoder structure.
[0118] Corresponding to the above method embodiment, this specification also provides an embodiment of a network traffic anomaly detection device based on feature grouping and autoencoder integration, Figure 4 A schematic diagram of the structure of a network traffic anomaly detection device based on feature grouping and autoencoder integration provided in an embodiment of the present application. Figure 4 As shown, the device comprises:
[0119] The grouping module 402 is configured to perform feature grouping on the network traffic features to be detected based on a feature grouping data processing principle to obtain feature group data, wherein the feature grouping data processing principle is a feature grouping principle determined based on feature attributes corresponding to the network traffic;
[0120] The analysis module 404 is configured to analyze the feature group data based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected, wherein the autoencoder is an autoencoder that has learned the inherent rules of normal traffic grouping features in advance;
[0121] The reconstruction module 406 is configured to reconstruct the traffic feature to be detected based on the autoencoder model to obtain a reconstructed network traffic feature, and calculate a reconstruction error between the network traffic feature to be detected and the reconstructed network traffic feature;
[0122] The scoring module 408 is configured to calculate the traffic feature detection score under at least one of the autoencoder models based on the reconstruction error and the optimal threshold of the reconstruction error, and perform weighted fusion to obtain a scoring result, wherein the scoring result includes a first scoring result corresponding to the normal traffic data of the traffic feature to be detected, and a second scoring result corresponding to the abnormal traffic data of the traffic feature to be detected;
[0123] The determination module 410 is configured to determine the abnormal situation corresponding to the to-be-detected network traffic feature based on the first scoring result and the second scoring result.
[0124] In an optional embodiment, the grouping module 402 is further configured to:
[0125] Based on the target normalization method, the network traffic characteristics to be detected are normalized to obtain the initial network traffic characteristics;
[0126] Determine a characteristic attribute corresponding to the initial network traffic characteristic, wherein the characteristic attribute includes a first characteristic attribute, a second characteristic attribute, and a third characteristic attribute;
[0127] Based on the first characteristic attribute and the third characteristic attribute, determining the initial network traffic feature as a first type of characteristic group data, and calculating a first variance value corresponding to each characteristic data in the first type of characteristic group data to generate a first variance set;
[0128] Based on the second characteristic attribute and the third characteristic attribute, determining the initial network traffic feature as a second type of characteristic group data, and calculating a second variance value corresponding to each characteristic data in the second type of characteristic group data to generate a second variance set;
[0129] Based on a preset grouping principle, the first variance set and the second variance set obtain feature group data, wherein the preset grouping principle carries the number corresponding to each type of feature data grouping, the number corresponding to the feature data in each group, and the variance condition satisfied by each variance in the feature group data, and the variance condition is the number corresponding to the feature data in each type of feature data grouping.
[0130] In an optional embodiment, the analysis module 404 is further configured to:
[0131] Taking the feature group data as verification data, based on the feature grouping data processing principle, feature grouping the verification data to obtain first feature group data, wherein the first feature group data carries label information corresponding to the verification data;
[0132] Reconstructing the first feature group data based on the learned autoencoder to obtain first reconstructed data, and calculating a first reconstruction error between the first feature group data and the first reconstructed data;
[0133] Calculate the true positive rate and false positive rate corresponding to the autoencoder model under different detection thresholds according to the first reconstruction error and the label information corresponding to the verification data;
[0134] Determine the AUC value corresponding to each of the autoencoders based on the ROC curve generated by the true positive rate and the false positive rate;
[0135] An autoencoder model that meets a preset AUC threshold is determined as at least one autoencoder model corresponding to the network traffic feature to be detected.
[0136] In an optional embodiment, the analysis module 404 is further configured to:
[0137] Taking normal traffic characteristics as training data, based on the characteristic grouping data processing principle, characteristic grouping is performed on the training data to generate at least one set of second characteristic group data;
[0138] Constructing at least one autoencoder, and inputting the second feature group data into at least one of the autoencoders for learning, to obtain at least one group of second reconstructed data;
[0139] Calculate the reconstruction error corresponding to the training data according to the second feature group data and the second reconstruction data, and use an optimizer to adjust the parameters corresponding to at least one of the autoencoders after back-propagation of the reconstruction error to obtain an autoencoder model;
[0140] Output the trained autoencoder model weights and the corresponding normal traffic grouping feature reconstruction error distribution.
[0141] In an optional embodiment, the scoring module 408 is further configured to:
[0142] Counting the reconstruction error distribution of the normal traffic group feature obtained by at least one autoencoder for reconstructing the normal traffic group feature, and determining the optimal threshold of the reconstruction error corresponding to the normal traffic group feature reconstruction error distribution;
[0143] In response to the reconstruction error being less than the reconstruction error optimal threshold, calculating, based on a third calculation formula, a flow feature detection score under at least one of the autoencoder models;
[0144] In response to the reconstruction error being greater than the optimal reconstruction error threshold, a traffic feature detection score under at least one of the autoencoder models is calculated based on a fourth calculation formula.
[0145] In an optional embodiment, the scoring module 408 is further configured to:
[0146] Based on the Youden index maximization principle, an optimal threshold of reconstruction error corresponding to at least one of the autoencoders is determined.
[0147] In an optional embodiment, the scoring module 408 is further configured to:
[0148] According to the AUC value corresponding to each of the autoencoders, the corresponding weight coefficient when each of the autoencoders participates in the score fusion is calculated;
[0149] Based on the weight coefficient, the scoring results are weighted and fused.
[0150] In an optional embodiment, the determining module 410 is further configured to:
[0151] In response to the first scoring result being greater than the second scoring result, determining that the traffic feature to be detected is normal traffic data;
[0152] In response to the first scoring result being less than the second scoring result, it is determined that the traffic feature to be detected is abnormal traffic data.
[0153] By using the network traffic anomaly detection device based on feature grouping and autoencoder integration provided in the embodiment of the present application, it is possible to improve the network traffic anomaly detection performance by introducing feature grouping and encoder integration without changing the conventional autoencoder structure.
[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the network traffic anomaly detection device based on feature grouping and autoencoder integration, since it is basically similar to the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration, the description is relatively simple, and the relevant parts can be referred to the partial description of the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration.
[0155] Figure 5 This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0156] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0157] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0158] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.
[0159] The processor 520 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned network traffic anomaly detection method based on feature grouping and autoencoder integration.
[0160] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration, the description is relatively simple, and the relevant parts can be referred to the partial description of the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration.
[0161] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned network traffic anomaly detection method based on feature grouping and autoencoder integration.
[0162] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration, the description is relatively simple, and the relevant parts can be referred to the partial description of the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration.
[0163] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned network traffic anomaly detection method based on feature grouping and autoencoder integration.
[0164] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration, the description is relatively simple, and the relevant parts can be referred to the partial description of the network traffic anomaly detection method embodiment based on feature grouping and autoencoder integration.
[0165] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0166] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0167] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.
[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0169] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A network traffic anomaly detection method based on feature grouping and autoencoder integration, characterized in that: include: Based on the characteristic grouping data processing principle, the characteristics of the network traffic to be detected are characteristically grouped to obtain characteristic group data, wherein the characteristic grouping data processing principle is a characteristic grouping principle determined based on characteristic attributes corresponding to the network traffic; Analyzing the feature group data based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected, wherein the autoencoder is an autoencoder that has previously learned the inherent rules of normal traffic grouping features; Based on the autoencoder model, the traffic feature to be detected is reconstructed to obtain a reconstructed network traffic feature, and a reconstruction error between the network traffic feature to be detected and the reconstructed network traffic feature is calculated; Based on the reconstruction error and the optimal threshold of the reconstruction error, a flow feature detection score under at least one of the autoencoder models is calculated, and weighted fusion is performed to obtain a scoring result, wherein the scoring result includes a first scoring result corresponding to the normal flow data of the flow feature to be detected, and a second scoring result corresponding to the abnormal flow data of the flow feature to be detected; Based on the first scoring result and the second scoring result, an abnormal situation corresponding to the network traffic feature to be detected is determined.
2. The method according to claim 1, characterized in that The characteristic grouping data processing principle based on the characteristic grouping is to perform characteristic grouping on the network traffic characteristics to be detected to obtain characteristic group data, including: Based on the target normalization method, the network traffic characteristics to be detected are normalized to obtain the initial network traffic characteristics; Determine a characteristic attribute corresponding to the initial network traffic characteristic, wherein the characteristic attribute includes a first characteristic attribute, a second characteristic attribute, and a third characteristic attribute; Based on the first characteristic attribute and the third characteristic attribute, determining the initial network traffic feature as a first type of characteristic group data, and calculating a first variance value corresponding to each characteristic data in the first type of characteristic group data to generate a first variance set; Based on the second characteristic attribute and the third characteristic attribute, determining the initial network traffic feature as a second type of characteristic group data, and calculating a second variance value corresponding to each characteristic data in the second type of characteristic group data to generate a second variance set; Based on a preset grouping principle, the first variance set and the second variance set obtain feature group data, wherein the preset grouping principle carries the number corresponding to each type of feature data grouping, the number corresponding to the feature data in each group, and the variance condition satisfied by each variance in the feature group data, and the variance condition is the number corresponding to the feature data in each type of feature data grouping.
3. The method according to claim 1, characterized in that The analyzing the feature group data based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected includes: Taking the feature group data as verification data, based on the feature grouping data processing principle, feature grouping the verification data to obtain first feature group data, wherein the first feature group data carries label information corresponding to the verification data; Reconstructing the first feature group data based on the learned autoencoder to obtain first reconstructed data, and calculating a first reconstruction error between the first feature group data and the first reconstructed data; Calculate the true positive rate and false positive rate corresponding to the autoencoder model under different detection thresholds according to the first reconstruction error and the label information corresponding to the verification data; Determine the AUC value corresponding to each of the autoencoders based on the ROC curve generated by the true positive rate and the false positive rate; An autoencoder model that meets a preset AUC threshold is determined as at least one autoencoder model corresponding to the network traffic feature to be detected.
4. The method according to claim 3, characterized in that The method for the autoencoder to pre-learn the inherent rules of normal traffic grouping features includes: Taking normal traffic characteristics as training data, based on the characteristic grouping data processing principle, characteristic grouping is performed on the training data to generate at least one set of second characteristic group data; Constructing at least one autoencoder, and inputting the second feature group data into at least one of the autoencoders for learning, to obtain at least one group of second reconstructed data; Calculate the reconstruction error corresponding to the training data according to the second feature group data and the second reconstruction data, and use an optimizer to adjust the parameters corresponding to at least one of the autoencoders after back-propagation of the reconstruction error to obtain an autoencoder model; Output the trained autoencoder model weights and the corresponding normal traffic grouping feature reconstruction error distribution.
5. The method according to claim 1, characterized in that The calculating, based on the reconstruction error and the optimal threshold of the reconstruction error, a flow feature detection score under at least one of the autoencoder models, comprises: Counting the reconstruction error distribution of the normal traffic group feature obtained by at least one autoencoder for reconstructing the normal traffic group feature, and determining the optimal threshold of the reconstruction error corresponding to the normal traffic group feature reconstruction error distribution; In response to the reconstruction error being less than the reconstruction error optimal threshold, calculating, based on a third calculation formula, a flow feature detection score under at least one of the autoencoder models; In response to the reconstruction error being greater than the optimal reconstruction error threshold, a traffic feature detection score under at least one of the autoencoder models is calculated based on a fourth calculation formula.
6. The method according to claim 1, characterized in that The method for calculating the optimal threshold of the reconstruction error includes: Based on the Youden index maximization principle, an optimal threshold of reconstruction error corresponding to at least one of the autoencoders is determined.
7. The method according to claim 1, characterized in that The weighted fusion method comprises: According to the AUC value corresponding to each of the autoencoders, the corresponding weight coefficient when each of the autoencoders participates in the score fusion is calculated; Based on the weight coefficient, the scoring results are weighted and fused.
8. The method according to claim 1, characterized in that The determining, based on the first scoring result and the second scoring result, an abnormal situation corresponding to the to-be-detected network traffic feature includes: In response to the first scoring result being greater than the second scoring result, determining that the traffic feature to be detected is normal traffic data; In response to the first scoring result being less than the second scoring result, it is determined that the traffic feature to be detected is abnormal traffic data.
9. A network traffic anomaly detection device based on feature grouping and autoencoder integration, characterized in that: include: The grouping module is configured to perform feature grouping on the network traffic features to be detected based on a feature grouping data processing principle to obtain feature group data, wherein the feature grouping data processing principle is a feature grouping principle determined based on feature attributes corresponding to the network traffic; An analysis module is configured to analyze the feature group data based on the learned autoencoder to determine at least one autoencoder model corresponding to the network traffic feature to be detected, wherein the autoencoder is an autoencoder that has previously learned the inherent rules of normal traffic grouping features; A reconstruction module is configured to reconstruct the traffic feature to be detected based on the autoencoder model to obtain a reconstructed network traffic feature, and calculate a reconstruction error between the network traffic feature to be detected and the reconstructed network traffic feature; A scoring module is configured to calculate a traffic feature detection score under at least one of the autoencoder models based on the reconstruction error and the optimal threshold of the reconstruction error, and perform weighted fusion to obtain a scoring result, wherein the scoring result includes a first scoring result corresponding to the normal traffic data of the traffic feature to be detected, and a second scoring result corresponding to the abnormal traffic data of the traffic feature to be detected; The determination module is configured to determine the abnormal situation corresponding to the network traffic feature to be detected based on the first scoring result and the second scoring result.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Cited By
Network traffic anomaly detection method based on deep learning
CN120811797A
A deep learning-based network traffic anomaly detection method
CN120811797B
Industrial control abnormal flow detection method and system based on coupling variational auto-encoder
CN121644205A
Classification method and system for detecting abnormal network traffic
CN122027277A
