A method for detecting network traffic anomalies based on temporal representation of reconstruction error

Abstract

This invention belongs to the technical field of network security, specifically relating to a network traffic anomaly detection method based on the temporal representation of reconstruction error. This network traffic detection method employs a convolutional autoencoder module and an anomaly detection module from deep learning to achieve anomaly detection in network traffic. First, a damped incremental statistical algorithm is used to extract statistical features of the traffic; then, the convolutional autoencoder module is used to obtain the reconstruction error of the network traffic. Since damped incremental statistics can extract flow-level contextual information, the reconstruction error can be used to represent the temporal relationship between traffic flows; subsequently, the reconstruction error is divided through a source IP aggregation mechanism to obtain a fine-grained representation of the temporal information; finally, the anomaly detection module completes the anomaly detection of the traffic. Experimental results show that this invention achieves good results on real-world datasets. Therefore, this invention can be used for anomaly detection in network traffic to maintain network security.

Description

Technical Field

[0001] This invention belongs to the technical field of network security, specifically relating to a method for detecting network traffic anomalies based on the temporal representation of reconstruction error. Background Technology

[0002] The rapid development of the internet and communications sectors has led to an explosive growth in network scale and communication data volume, resulting in the emergence of various network applications that have changed people's lives, learning, and work, playing a positive role in my country's social development. At the same time, many new types of cyberattacks have begun to emerge, posing a significant threat to network security. To address the increasingly prominent network security issues, research into network traffic anomaly detection has become particularly important. Network traffic anomaly detection refers to the analysis and modeling of traffic data to identify traffic data that does not conform to normal behavior.

[0003] Traditional network traffic anomaly detection identifies malicious traffic through pre-configured rules, which requires maintaining a rule database defined by experts. Although this method effectively improves detection accuracy, it cannot effectively identify zero-day attacks and multi-stage attacks. To solve the above problems, machine learning methods are widely used in the field of anomaly detection. Since the number of normal samples far exceeds the number of abnormal samples in anomaly detection tasks, labeling samples requires a lot of manpower. Unsupervised learning methods have received widespread attention from researchers in the field of anomaly detection. Unsupervised learning methods for network traffic anomaly detection are trained on normal samples, so they are also called "zero-day" learning. These methods need to obtain a detection criterion to distinguish between normal and abnormal samples. Unsupervised shallow learning methods for anomaly detection can be classified as: (1) density-based methods; (2) clustering-based methods; (3) isolation-based methods, etc. These methods usually assume that the samples are independent and identically distributed, model the data in a relatively simple way, cannot capture the correlation between data, and cannot well deal with high-dimensional nonlinear data in the real world.

[0004] Currently, deep learning has received widespread attention in fields such as computer vision, natural language processing, and anomaly detection due to its ability to extract high-quality representations and fit complex functions. Unsupervised deep learning can detect unknown attacks in the field of anomaly detection and has high detection accuracy. Unsupervised deep learning methods can be classified according to anomaly detection criteria as: methods based on reconstruction error, methods based on prediction, and methods based on correlation differences. Among them, detection methods based on reconstruction error have the following problems: (1) the selection of the threshold requires multiple attempts and the threshold is fixed after selection, which cannot adapt to the dynamic changes of the network; (2) the detection results are heavily dependent on the model's ability to model the distribution of normal samples; (3) packet-level detection focuses on the RMSE of a single time point and is difficult to model the temporal correlation between samples. Based on the above analysis, detection methods based on reconstruction error still have the problems of difficulty in selecting the threshold and the inability of packet-level detection to model temporal correlation. Summary of the Invention

[0005] To address the shortcomings of existing network traffic anomaly detection methods, this invention proposes a temporal representation of RMSE to address the aforementioned problem (1). This transforms the threshold selection problem into a hyperparameter setting problem for the Isolation Forest algorithm, whose hyperparameters are relatively fixed, thus avoiding manual threshold setting. Regarding the aforementioned problem (2), this invention uses a Convolutional Autoencoder (CAE) to model normal samples. Inspired by the U-Net architecture, feature maps from the encoding stage are concatenated to feature maps from the decoding stage, thereby better combining shallow and deep features. To address the aforementioned problem (3), a source IP address aggregation mechanism is proposed to divide the original RMSE into fine-grained RMSEs. By setting a time window to obtain the temporal features of fine-grained RMSEs, the contextual information of a single time point RMSE can be fully utilized, which also helps in tracing the source of attacking IP addresses.

[0006] Based on the above network structure redesign, this invention adopts a reconstruction error-based method, that is, the reconstruction error of the sample is used as the anomaly score, and proposes a network traffic anomaly detection method based on the temporal representation of reconstruction error. This method can fully extract the features of the data flow and classify them according to the extracted features. Furthermore, in response to the imbalance in the distribution of data samples, an algorithm is designed using the distribution changes of data samples, which further improves the accuracy of the model in identifying minority class samples.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a network traffic anomaly detection method based on reconstruction error time series representation, which is mainly implemented through a network traffic anomaly detection model. The network traffic anomaly detection model includes a data preprocessing module, a time series feature extraction module, and an anomaly detection module. The network traffic anomaly detection method includes the following steps:

[0008] Step 1, Data Preprocessing: In the data preprocessing module, the tshark tool is used to capture network packets to obtain a pcap file, and the pcap file is parsed to obtain a set of meta-information. The damped incremental statistical algorithm is used to extract features from the set of meta-information to obtain a set of statistical features. The set of statistical features is normalized using z-values ​​to obtain 100-dimensional statistical features.

[0009] Step 2: Construction and Training of the Network Traffic Anomaly Detection Model: Model construction, where the temporal feature extraction module consists of a convolutional autoencoder module and an RMSE aggregation module. The convolutional autoencoder module is built using PyTorch, and the RMSE aggregation module is implemented using Python. The anomaly detection module is built using Scikit-learn. Model training: CAE and IF training rounds are set for the convolutional autoencoder module and the anomaly detection module, respectively. The 100-dimensional statistical features obtained in Step 1 are input into the convolutional autoencoder module for training to obtain the reconstruction error of the 100-dimensional statistical features. This reconstruction error is input into the RMSE aggregation module, where the source IP aggregation mechanism is used to obtain a fine-grained division of the reconstruction error. A time window is set on the fine-grained reconstruction error. The temporal features are obtained; in the anomaly detection module, the temporal features are used to train the isolated forest algorithm.

[0010] Step 3: Performance evaluation of the network traffic anomaly detection model: The macro average of precision, recall, and F1 score is used as the evaluation index of the model. If the macro average of the F1 score is greater than 95%, proceed to step 4; otherwise, proceed to step 2 to continue model training.

[0011] Step 4: Network Traffic Detection: Collect network traffic from a real network environment and input it into the data preprocessing module to execute Step 1. The resulting 100-dimensional statistical features are then input into the convolutional autoencoder module to obtain the reconstruction error of the 100-dimensional statistical features. This reconstruction error is then input into the RMSE aggregation module, where the source IP aggregation mechanism is used to obtain a fine-grained division of the reconstruction error. A time window is then set on the fine-grained reconstruction error. The temporal features are obtained; in the anomaly detection module, the isolated forest algorithm is used to detect the temporal features.

[0012] As a further implementation of the above technical solution, the specific steps of data preprocessing in step 1 are as follows:

[0013] Step 1.1, Packet parsing: Use the tshark tool to capture network packets and obtain a pcap file. Then, use the tshark tool to parse the pcap file to obtain a set of metadata of the packets, which will be used as input for step 1.2.

[0014] Step 1.2, Statistical Feature Extraction: A set of statistical features is calculated from the component information using the damped incremental statistical algorithm, which serves as the input for Step 1.3;

[0015] Step 1.3, z-value normalization: The statistical features obtained in step 1.2 are normalized by z-value to obtain 100-dimensional statistical features.

[0016] As a further implementation of the above technical solution, the specific steps for constructing and training the network traffic anomaly detection model in step 2 are as follows:

[0017] Step 2.1, Data Sample Loading: Using the deep learning framework PyTorch 2.0, inherit the Dataset class, override the __getitem__ method in the Dataset class, and reshape the 100-dimensional statistical features into... The matrix form is used as the input for step 2.2;

[0018] Step 2.2, Convolutional Autoencoder Module Construction and Training: A convolutional autoencoder module is built using the deep learning framework PyTorch 2.0. The module adopts the U-Net architecture, concatenating the feature maps from the encoding stage to the feature maps from the decoding stage, with a size of... The matrix blocks are input into the convolutional autoencoder module to obtain the reconstruction error of the samples. The reconstruction error of the samples and their corresponding source IP addresses are used as inputs in step 2.3; where the training objective of the convolutional autoencoder module is... The calculation formula is as follows:

[0019] ;

[0020] In the formula: RMSE represents the reconstruction error of the sample. This is represented as the output of the convolutional autoencoder module. Represented as a matrix block of the input convolutional autoencoder module, This is represented as the training dataset. Represented as Parameters;

[0021] The formula for calculating RMSE reconstruction error is as follows:

[0022] ;

[0023] In the formula: This is represented as the input sample. They are respectively represented as samples and samples The One portion, Let the input sample be the first... One dimension, This is expressed as the dimension of the input sample;

[0024] Step 2.3, RMSE aggregation module reconstruction error division: using the source IP address aggregation mechanism, the reconstruction error of the samples is divided. Divide the sequence to obtain a set of subsequences. , where subsequence To characterize the fine-grained temporal properties of the reconstruction error, the resulting subsequences satisfy the following conditions:

[0025] ;

[0026] ;

[0027] In the formula: This is represented by the number of source IP addresses. Represented as a sequence of reconstruction errors, i.e. , Indicates the number of network traffic packets;

[0028] Step 2.4, Construction and Training of the Isolation Forest Algorithm: In the subsequence Set a time window W above the subsequence obtained in step 2.3. We obtain a set of dimensions as The temporal features of this set of features are used for training the Isolation Forest algorithm.

[0029] As a further implementation of the above technical solution, the specific steps for detecting network traffic in step 4 are as follows:

[0030] Step 4.1, Data Preprocessing: For the network traffic to be detected, firstly, it is parsed in Step 1.1 to obtain a set of metadata; then, feature extraction is performed in Step 1.2; finally, z-value normalization is performed in Step 1.3 to obtain... 2D statistical characteristics;

[0031] Step 4.2, Temporal Feature Extraction: First, the 100-dimensional statistical features obtained in Step 4.1 are reshaped using Step 2.1 to obtain a feature of size [missing information]. The matrix block is processed by the convolutional autoencoder module trained in step 2.2 to obtain its corresponding reconstruction error. Then, the fine-grained division of the reconstruction error can be obtained through step 2.3, and the subsequence obtained after division is used as the input of step 4.3.

[0032] Step 4.3, Anomaly Detection: Set a time window W on the subsequence, and obtain a set of dimensions from the subsequence of Step 4.2. The time-series features are input into the Isolation Forest algorithm trained in step 2.4. The Isolation Forest algorithm provides anomaly scores for the time-series features. Samples within the range are considered normal network traffic, and the outlier score is... Network traffic within the specified range is considered abnormal network traffic, and its corresponding source IP address is marked as the attacker's IP address.

[0033] As a further implementation of the above technical solution, in step 3, the evaluation index is calculated as follows:

[0034] ;

[0035] in, Indicates the first Precision, recall, and F1 score for each class, where N represents the number of test samples, 0 represents normal network traffic, and 1 represents abnormal network traffic;

[0036] The calculation methods for each indicator are as follows:

[0037] ;

[0038] ;

[0039] ;

[0040] Wherein, TP (true positive) represents the number of abnormal traffic instances detected as abnormal; FP (false positive) represents the number of normal traffic instances detected as abnormal; FN (false negative) represents the number of abnormal traffic instances detected as normal; and TN (true negative) represents the number of normal traffic instances detected as normal.

[0041] As a further implementation of the above technical solution, in step 1, the component information includes the packet length, packet arrival timestamp, source IP address, destination IP address, source MAC address, destination MAC address, source port number, and destination port number of the data packet.

[0042] Compared with existing technologies, the network anomaly traffic detection method based on reconstruction error temporal representation proposed in this invention has the following four main advantages:

[0043] (1) This invention employs a convolutional autoencoder (CAE) module and an isolation forest (IF) module from deep learning to achieve anomaly detection in network traffic. First, a damped incremental statistical algorithm is used to extract statistical features of the traffic. Then, the CAE module is used to obtain the root mean square error (RMSE) of the network traffic. Since the damped incremental statistical algorithm can extract flow-level contextual information, the RMSE can be used to represent the temporal relationship between traffic flows. Next, the RMSE is partitioned using a source IP aggregation mechanism to obtain a fine-grained representation of the temporal information. Finally, the anomaly detection module completes the anomaly detection of the traffic. Experimental results show that this invention achieves good results on real-world datasets. Therefore, this invention can be used for anomaly detection in network traffic to maintain cyberspace security.

[0044] (2) This invention proposes a temporal representation of reconstruction error by observing the reconstruction error of network traffic. Therefore, this invention can capture the temporal relationship between network traffic and avoid threshold selection during the reconstruction process.

[0045] (3) The present invention uses a convolutional autoencoder module to reconstruct traffic samples. The convolution operation can capture the correlation between sample features and stitch the feature map of the encoding stage to the feature map of the decoding stage, which further improves the network traffic anomaly detection model's ability to model the distribution of normal samples.

[0046] (4) This invention uses a source IP aggregation mechanism in the RMSE aggregation module to obtain a fine-grained division of the reconstruction error. A time window is then used on the aggregated reconstruction error to obtain a temporal subsequence for anomaly detection. This method can overcome the shortcomings of convolutional autoencoders in modeling the temporal correlation of network traffic, and can better detect traffic. Attached Figure Description

[0047] Figure 1 This is a flowchart of network traffic anomaly detection in this invention;

[0048] Figure 2 This is an architecture diagram of the network traffic anomaly detection model in this invention;

[0049] Figure 3 This is a structural diagram of the convolutional autoencoder module in this invention;

[0050] Figure 4This is a graph showing the reconstruction error results without using the RMSE aggregation module in this invention;

[0051] Figure 5 This is a graph showing the reconstruction error results using the RMSE aggregation module in this invention.

[0052] Figure 6 This is a diagram showing the detection results of abnormal traffic in a real-world dataset based on the present invention. Specific implementation methods

[0053] To better understand this invention, the following description is in conjunction with the appendix. Figures 1 to 6 We have selected the optimal implementation scheme to further clarify the content of the present invention, but the content of the present invention is not limited to the following embodiments.

[0054] As attached Figure 1 and 2 As shown, the network traffic anomaly detection method is mainly implemented through the network traffic anomaly detection model designed in this invention. The network traffic anomaly detection model includes three parts: a data preprocessing module, a temporal feature extraction module, and an anomaly detection module. The data preprocessing module mainly parses network data packets and obtains a set of packet metadata. Then, it extracts the statistical features of the traffic from this set of metadata and performs normalization processing to obtain 100-dimensional statistical features. The temporal feature extraction module consists of a convolutional autoencoder module and an RMSE aggregation module. The convolutional autoencoder module reconstructs the 100-dimensional statistical features of the network traffic preprocessing to obtain the traffic reconstruction error. Then, the RMSE aggregation module uses the source IP aggregation mechanism to obtain a fine-grained division of RMSE. Finally, the anomaly detection module uses the isolated forest algorithm to complete the anomaly detection of the traffic.

[0055] As shown in Table 1, the specific implementation steps for network traffic anomaly detection are as follows:

[0056] Step 1, Data Preprocessing: In the data preprocessing module, the tshark tool is used to capture network packets to obtain a pcap file, and the pcap file is parsed to obtain a set of meta-information. The damped incremental statistical algorithm is used to extract features from the set of meta-information to obtain a set of statistical features. The set of statistical features is normalized using z-values ​​to obtain 100-dimensional statistical features.

[0057] Step 2: Construction and Training of the Network Traffic Anomaly Detection Model: Model Construction. The network traffic anomaly detection model consists of two parts: a temporal feature extraction module and an anomaly detection module. The temporal feature extraction module comprises a convolutional autoencoder module and an RMSE aggregation module. The convolutional autoencoder module is built using PyTorch, and the RMSE aggregation module is implemented using Python. The anomaly detection module is built using Scikit-learn. Model Training. CAE and IF training rounds are set for both the convolutional autoencoder module and the anomaly detection module. The 100-dimensional statistical features obtained in Step 1 are input into the convolutional autoencoder module for training to obtain the reconstruction error of the 100-dimensional statistical features. This reconstruction error is then input into the RMSE aggregation module, where the source IP aggregation mechanism is used to obtain a fine-grained division of the reconstruction error. A time window is then set on the fine-grained reconstruction error. The temporal features are obtained; in the anomaly detection module, the temporal features are used to train the isolated forest algorithm.

[0058] Step 3: Performance evaluation of the network traffic anomaly detection model: The macro average of precision, recall, and F1 score is used as the evaluation index of the model. If the macro average of the F1 score is greater than 95%, proceed to step 4; otherwise, proceed to step 2 to continue model training.

[0059] Step 4: Network Traffic Detection: Collect network traffic from a real network environment and input it into the data preprocessing module to execute Step 1. The resulting 100-dimensional statistical features are then input into the convolutional autoencoder module to obtain the reconstruction error of the 100-dimensional statistical features. This reconstruction error is then input into the RMSE aggregation module, where the source IP aggregation mechanism is used to obtain a fine-grained division of the reconstruction error. A time window is then set on the fine-grained reconstruction error. The temporal features are obtained; in the anomaly detection module, the isolated forest algorithm is used to detect the temporal features.

[0060] Table 1 shows the algorithms for the training and execution phases of the network traffic anomaly detection model.

[0061]

[0062] In the above embodiments, the specific processing methods of steps 1 to 4 are as follows:

[0063] (I) In step 1, the specific steps are as follows:

[0064] Step 1.1, Packet parsing: Use the tshark tool to capture network packets and obtain a pcap file. Then, use the tshark tool to parse the pcap file to obtain a set of metadata of the packets. This set of metadata includes the packet length, packet arrival timestamp, source IP address, destination IP address, source MAC address, destination MAC address, source port number, and destination port number, which serve as the input for step 1.2.

[0065] Step 1.2, Statistical Feature Extraction: Calculate a set of statistical features shown in Table 2, and use a set of constraints shown in Table 3 to obtain a set of statistical features for model input, which are used for model training and flow detection. Table 4 gives the damping increment statistical algorithm used for feature extraction, which is used as the input for Step 1.3.

[0066] Step 1.3, z-value normalization: The statistical features obtained in step 1.2 are normalized by z-value to obtain 100-dimensional statistical features.

[0067] Table 2 shows the calculation expressions for feature extraction.

[0068]

[0069] Table 3 shows the statistical characteristics used for model input.

[0070]

[0071] Table 4 shows the damping increment statistical algorithm used for network traffic feature extraction.

[0072]

[0073] (II) In step 2, the temporal feature extraction module consists of a convolutional autoencoder module and an RMSE aggregation module. The specific steps for its construction and training are as follows:

[0074] Step 2.1, Data Sample Loading: Using the deep learning framework PyTorch 2.0, inherit the Dataset class, override the __getitem__ method in the Dataset class, and reshape the 100-dimensional statistical features into... The matrix form is used as the input for step 2.2;

[0075] Step 2.2, Construction and training of the convolutional autoencoder module: as attached. Figure 3 As shown in Table 5, a convolutional autoencoder module was built using the deep learning framework PyTorch 2.0. The convolutional autoencoder module adopts the U-Net architecture, concatenating the feature maps from the encoding stage to the feature maps from the decoding stage. The size of the convolutional autoencoder module is shown in Table 5. The matrix blocks are input into the convolutional autoencoder module to obtain the reconstruction error of the samples. The reconstruction error of the samples and their corresponding source IP addresses are used as inputs in step 2.3; where the training objective of the convolutional autoencoder module is... The calculation formula is as follows:

[0076] ;

[0077] In the formula: RMSE represents the reconstruction error of the sample. This is represented as the output of the convolutional autoencoder module. Represented as a matrix block of the input convolutional autoencoder module, This is represented as the training dataset. Represented as Parameters;

[0078] The formula for calculating RMSE reconstruction error is as follows:

[0079] ;

[0080] In the formula: This is represented as the input sample. They are respectively represented as samples and samples The One portion, Let the input sample be the first... One dimension, This is expressed as the dimension of the input sample;

[0081] Table 5 shows the hyperparameter settings for the convolutional autoencoder module.

[0082]

[0083] Step 2.3, RMSE aggregation module reconstruction error division: as attached. Figure 4 and 5 As shown, the source IP address aggregation mechanism is used to analyze the reconstruction error of the samples. Divide the sequence to obtain a set of subsequences. , where subsequence To characterize the fine-grained temporal properties of the reconstruction error, the resulting subsequences satisfy the following conditions:

[0084] ;

[0085] ;

[0086] In the formula: This is represented by the number of source IP addresses. Represented as a sequence of reconstruction errors, i.e. , Indicates the number of network traffic packets;

[0087] Step 2.4, Construction and Training of the Isolation Forest Algorithm: In the subsequence Set a time window W above the subsequence obtained in step 2.3. We obtain a set of dimensions as The temporal features of this set of features are used for training the Isolation Forest algorithm.

[0088] (III) In step 3, the evaluation indicators are calculated as follows:

[0089] ;

[0090] in, Indicates the first Precision, recall, and F1 score for each class, where N represents the number of test samples, 0 represents normal network traffic, and 1 represents abnormal network traffic;

[0091] The calculation methods for each indicator are as follows:

[0092] ;

[0093] ;

[0094] ;

[0095] Wherein, TP (true positive) represents the number of abnormal traffic instances detected as abnormal; FP (false positive) represents the number of normal traffic instances detected as abnormal; FN (false negative) represents the number of abnormal traffic instances detected as normal; and TN (true negative) represents the number of normal traffic instances detected as normal.

[0096] (iv) The specific steps in step 4 are as follows:

[0097] Step 4.1, Data Preprocessing: For the network traffic to be detected, firstly, it is parsed in Step 1.1 to obtain a set of metadata; then, feature extraction is performed in Step 1.2; finally, z-value normalization is performed in Step 1.3 to obtain... 2D statistical characteristics;

[0098] Step 4.2, Temporal Feature Extraction: First, the 100-dimensional statistical features obtained in Step 4.1 are reshaped using Step 2.1 to obtain a feature of size [missing information]. The matrix block is processed by the convolutional autoencoder module trained in step 2.2 to obtain its corresponding reconstruction error. Then, the fine-grained division of the reconstruction error can be obtained through step 2.3, and the subsequence obtained after division is used as the input of step 4.3.

[0099] Step 4.3, Anomaly Detection: Set a time window W on the subsequence, and obtain a set of dimensions from the subsequence of Step 4.2. The time-series features are input into the Isolation Forest algorithm trained in step 2.4. The Isolation Forest algorithm provides anomaly scores for the time-series features. Samples within the range are considered normal network traffic, and the outlier score is... Network traffic within the specified range is considered abnormal network traffic, and its corresponding source IP address is marked as the attacker's IP address.

[0100] To verify the effectiveness of the network traffic anomaly detection model designed in this invention, we conducted numerous experiments to evaluate it. Table 6 shows a comparison of the evaluation results of this invention with other detection models. Table 7 shows the ablation experiments performed on this model. (Appendix) Figure 6 The detection results of this invention on a real-world dataset are shown, where 192.168.100.5 represents the attacker's IP address. The proposed network traffic anomaly detection model represents RMSE as a time series, obtains fine-grained time-series division of RMSE through a source IP address aggregation mechanism, and finally uses the Isolation Forest algorithm to complete the accurate classification of time-series features. The model's performance is evaluated using real-world traffic datasets Mirai, OS Scan, and SSDP Flood, demonstrating that the algorithm proposed in this patent achieves good results in network traffic anomaly detection.

[0101] Table 6 shows a comparison of the experimental results of the abnormal flow detection method of the present invention with related methods.

[0102]

[0103] Table 7 shows the ablation experiment results of the abnormal flow detection method of the present invention.

[0104] The foregoing has shown and described the main features and advantages of the present invention. It will be apparent to those skilled in the art that the specific embodiments of the present invention are not limited to the details of the exemplary embodiments described above. Furthermore, without departing from the spirit or essential characteristics of the present invention, the inventive concept and design ideas of the present invention can be implemented in other specific forms, and these should be equivalently included within the protection scope disclosed in the technical solutions of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included within the present invention.

[0105] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A network traffic anomaly detection method based on reconstruction error time-series representation, which is mainly implemented through a network traffic anomaly detection model, the network traffic anomaly detection model including a data preprocessing module, characterized in that: The network traffic anomaly detection model also includes a time-series feature extraction module and an anomaly detection module. The network traffic anomaly detection method includes the following steps: Step 1, Data Preprocessing: In the data preprocessing module, the tshark tool is used to capture network packets to obtain a pcap file, and the pcap file is parsed to obtain a set of meta-information. The damped incremental statistical algorithm is used to extract features from the set of meta-information to obtain a set of statistical features. The set of statistical features is normalized using z-values ​​to obtain 100-dimensional statistical features. Step 2: Construction and Training of the Network Traffic Anomaly Detection Model: Model construction, where the temporal feature extraction module consists of a convolutional autoencoder module and an RMSE aggregation module. The convolutional autoencoder module is built using PyTorch, and the RMSE aggregation module is implemented using Python. The anomaly detection module is built using Scikit-learn. Model training: CAE and IF training rounds are set for the convolutional autoencoder module and the anomaly detection module, respectively. The 100-dimensional statistical features obtained in Step 1 are input into the convolutional autoencoder module for training to obtain the reconstruction error of the 100-dimensional statistical features. This reconstruction error is input into the RMSE aggregation module, where the source IP aggregation mechanism is used to obtain a fine-grained division of the reconstruction error. A time window is set on the fine-grained reconstruction error. The temporal features are obtained; in the anomaly detection module, the temporal features are used to train the isolated forest algorithm. Step 3: Performance evaluation of the network traffic anomaly detection model: The macro average of precision, recall, and F1 score is used as the evaluation index of the model. If the macro average of the F1 score is greater than 95%, proceed to step 4; otherwise, proceed to step 2 to continue model training. Step 4: Network Traffic Detection: Collect network traffic from a real network environment and input it into the data preprocessing module to execute Step 1. The resulting 100-dimensional statistical features are then input into the convolutional autoencoder module to obtain the reconstruction error of the 100-dimensional statistical features. This reconstruction error is then input into the RMSE aggregation module, where the source IP aggregation mechanism is used to obtain a fine-grained division of the reconstruction error. A time window is then set on the fine-grained reconstruction error. The temporal features are obtained; in the anomaly detection module, the isolated forest algorithm is used to detect the temporal features.

2. The network traffic anomaly detection method based on reconstruction error time-series representation according to claim 1, characterized in that: In step 1, the specific steps of data preprocessing are as follows: Step 1.1, Packet parsing: Use the tshark tool to capture network packets and obtain a pcap file. Then, use the tshark tool to parse the pcap file to obtain a set of metadata of the packets, which will be used as input for step 1.

2. Step 1.2, Statistical Feature Extraction: A set of statistical features is calculated from the component information using the damped incremental statistical algorithm, which serves as the input for Step 1.3; Step 1.3, z-value normalization: The statistical features obtained in step 1.2 are normalized by z-value to obtain 100-dimensional statistical features.

3. The network traffic anomaly detection method based on reconstruction error time-series representation according to claim 2, characterized in that: In step 2, the specific steps for constructing and training the network traffic anomaly detection model are as follows: Step 2.1, Data Sample Loading: Using the deep learning framework PyTorch 2.0, inherit the Dataset class, override the __getitem__ method in the Dataset class, and reshape the 100-dimensional statistical features into... The matrix form is used as the input for step 2.2; Step 2.2, Convolutional Autoencoder Module Construction and Training: A convolutional autoencoder module is built using the deep learning framework PyTorch 2.

0. The module adopts the U-Net architecture, concatenating the feature maps from the encoding stage to the feature maps from the decoding stage, with a size of... The matrix blocks are input into the convolutional autoencoder module to obtain the reconstruction error of the samples. The reconstruction error of the samples and their corresponding source IP addresses are used as inputs in step 2.3; where the training objective of the convolutional autoencoder module is... The calculation formula is as follows: ； In the formula: RMSE represents the reconstruction error of the sample. This is represented as the output of the convolutional autoencoder module. Represented as a matrix block of the input convolutional autoencoder module, This is represented as the training dataset. Represented as Parameters; The formula for calculating RMSE reconstruction error is as follows: ； In the formula: This is represented as the input sample. They are respectively represented as samples and samples The One portion, Let the input sample be the first... One dimension, This is expressed as the dimension of the input sample; Step 2.3, RMSE aggregation module reconstruction error division: using the source IP address aggregation mechanism, the reconstruction error of the samples is divided. Divide the sequence to obtain a set of subsequences. , where subsequence To characterize the fine-grained temporal properties of the reconstruction error, the resulting subsequences satisfy the following conditions: ； ； In the formula: This is represented by the number of source IP addresses. Represented as a sequence of reconstruction errors, i.e. , Indicates the number of network traffic packets; Step 2.4, Construction and Training of the Isolation Forest Algorithm: In the subsequence Set a time window W above the subsequence obtained in step 2.

3. We obtain a set of dimensions as The temporal features of this set of features are used for training the Isolation Forest algorithm.

4. The network traffic anomaly detection method based on reconstruction error time-series representation according to claim 3, characterized in that: In step 4, the specific steps for detecting network traffic are as follows: Step 4.1, Data Preprocessing: For the network traffic to be detected, firstly, it is parsed in Step 1.1 to obtain a set of metadata; then, feature extraction is performed in Step 1.2; finally, z-value normalization is performed in Step 1.3 to obtain... 2D statistical characteristics; Step 4.2, Temporal Feature Extraction: First, the 100-dimensional statistical features obtained in Step 4.1 are reshaped using Step 2.1 to obtain a feature of size [missing information]. The matrix block is processed by the convolutional autoencoder module trained in step 2.2 to obtain its corresponding reconstruction error. Then, the fine-grained division of the reconstruction error can be obtained through step 2.3, and the subsequence obtained after division is used as the input of step 4.

3. Step 4.3, Anomaly Detection: Set a time window W on the subsequence, and obtain a set of dimensions from the subsequence of Step 4.

2. The time-series features are input into the Isolation Forest algorithm trained in step 2.

4. The Isolation Forest algorithm provides anomaly scores for the time-series features. Samples within the range are considered normal network traffic, and the outlier score is... Network traffic within the specified range is considered abnormal network traffic, and its corresponding source IP address is marked as the attacker's IP address.

5. A network traffic anomaly detection method based on reconstruction error time-series representation according to any one of claims 1 to 4, characterized in that: In step 3, the evaluation indicators are calculated as follows: ； in, Indicates the first Precision, recall, and F1 score for each class, where N represents the number of test samples, 0 represents normal network traffic, and 1 represents abnormal network traffic; The calculation methods for each indicator are as follows: ； ； ； Wherein, TP (true positive) represents the number of abnormal traffic instances detected as abnormal; FP (false positive) represents the number of normal traffic instances detected as abnormal; FN (false negative) represents the number of abnormal traffic instances detected as normal; and TN (true negative) represents the number of normal traffic instances detected as normal.

6. The network traffic anomaly detection method based on reconstruction error time-series representation according to claim 5, characterized in that: In step 1, the data packet information includes the packet length, packet arrival timestamp, source IP address, destination IP address, source MAC address, destination MAC address, source port number, and destination port number.

Images