An automated encrypted traffic analysis method across multiple protocols and protocol combinations
By employing an automated encrypted traffic analysis method that spans multiple protocols and protocol combinations, the low robustness of manual feature extraction and model adjustment in network traffic analysis is addressed, thereby achieving automated network traffic analysis and improved accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 国家电网有限公司客户服务中心
- Filing Date
- 2022-12-12
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies rely on manual feature extraction and model tuning in network traffic analysis, resulting in low robustness and complexity. Furthermore, each task requires redesigning features and models, and there is a lack of automated cross-protocol analysis methods.
It employs an automated encrypted traffic analysis method that spans multiple protocols and protocol combinations. It generates fixed-length feature vectors through preprocessing, feature extraction, and binary representation, and combines multiple machine learning models for classification, supporting custom field expansion.
It enables automated analysis of network traffic, reduces reliance on manual feature extraction, is applicable to various protocols and protocol combinations, and improves the robustness and accuracy of the analysis.
Smart Images
Figure CN116232642B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer networks and encrypted network traffic technology, and in particular to an automated encrypted traffic analysis method across multiple protocols and protocol combinations. Background Technology
[0002] Many traffic classification tasks in cybersecurity rely on machine learning. While current research has focused heavily on machine learning models applied to these tasks and their performance, in practice, these tasks heavily depend on pipelines involving manual feature extraction and model selection and tuning. The appropriate combination of feature extraction, models, and model parameters is typically an iterative process. Indeed, the effectiveness of applying machine learning to network traffic analysis often depends on the proper selection and representation of the model's features, but this part of the process still requires significant manual work and places stringent demands on skilled technicians. Feature extraction is a complex process, often requiring technicians to utilize extensive domain knowledge to design relevant features that can be used for measurement or derivation and produce accurate models. Even with highly specialized domain knowledge, feature exploration and extraction remain largely unrobust and error-prone, as feature selection and representation significantly impact model accuracy. This manual extraction approach may overlook features with underlying or complex relationships (e.g., non-linear relationships between features). Furthermore, due to constantly changing interaction patterns and extraction conditions, corresponding rendering models and handcrafted functionalities have become obsolete with current technological requirements. Furthermore, every new network detection or classification task requires a new round of technical tools: designing new features, selecting appropriate models, and manually adjusting new parameters.
[0003] Whether network packets can be characterized using a single standard, and how to automate methods applicable to common traffic analysis tasks, are technical problems that this invention urgently needs to solve. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and to propose an automated encrypted traffic analysis method that supports multiple protocols and combinations of protocols. It supports multiple network protocols and combinations of multiple network protocols and supports custom field extension methods to achieve automated encrypted network traffic analysis.
[0005] This invention is achieved using the following technical solution:
[0006] An automated method for analyzing encrypted traffic across multiple protocols and protocol combinations, comprising the following steps:
[0007] Step 1: Preprocess the network traffic of the encrypted application to be classified to obtain the data packet of each network flow in the network traffic. The specific operation includes: dividing the network traffic into network flows based on the five-tuple information including source IP address, destination IP address, source port, destination port and protocol, to obtain multiple tagged network traffic data packets, which are used as the input for Step 2.
[0008] Step 2: Obtain the semantic representation of the current network traffic data packets and perform feature extraction on the current network traffic data packets. This specifically includes the following processing:
[0009] Step 2.1: Extract all IP protocol header information from the network data packets, including but not limited to the IP protocol number, IP protocol options field, source IP protocol address, and destination IP protocol address;
[0010] Step 2.2: Extract all characteristic information of the TCP protocol packet header and the UDP protocol packet header, including but not limited to the source port number and the destination port number;
[0011] Step 2.3: Extract the part information of the ICMP protocol data packet header;
[0012] Step 3: Obtain the binary representation of network packet features and construct a fixed-length network packet feature vector containing 1088 features. This includes the following steps:
[0013] Step 3.1: Map all 60 bytes of features in the IP protocol packet header to 480 IP protocol packet header features. Each IP protocol packet header feature corresponds to the position information within each byte of the IP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 480 features in the IP protocol packet header.
[0014] Step 3.2: Map all 60 bytes of features in the TCP protocol packet header to 480 TCP protocol packet header features. Each TCP header feature corresponds to the position information within each byte of the TCP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 480 features in the TCP protocol packet header.
[0015] Step 3.3: Map all 8 bytes of UDP header features into 64 UDP header features. Each UDP header feature corresponds to the position information within each byte of the UDP header. Each byte is represented by 8 bits, and the 0 and 1 values at each position correspond to the 64 features of the UDP header.
[0016] Step 3.4: Map all 8 bytes of ICMP protocol packet header features into 64 ICMP packet header features. Each feature corresponds to the position information within each byte of the ICMP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 64 features of the ICMP protocol packet header.
[0017] Step 3.5: Concatenate all the above features to obtain a fixed-length network data packet feature vector containing 1088 features. If the data packet does not contain the corresponding protocol data packet header, then all corresponding positions are assigned a value of -1.
[0018] Step 4: Export the network packet features obtained in Step 3 into a CSV file;
[0019] Step 5: Find the optimal classification model. This involves training a model for each encrypted network traffic classification problem. These models are derived from 6 different basic model classes, which are 50 classes in total, derived from tree-based machine learning methods, deep neural network encoders, and distance-based classification methods. The best-performing classification model is selected based on average accuracy and F1 score.
[0020] Step 6: Classify encrypted network traffic. Process the encrypted network traffic to be classified according to the methods in Steps 3-5, and then input it as a feature into the optimal classification model obtained in Step 6 to complete the classification of encrypted network traffic.
[0021] The advantages of the automated encrypted traffic analysis method across multiple protocols and protocol combinations of the present invention are as follows:
[0022] (1) The proposed tool generates network traffic data packet features in a unified data packet representation, which is applicable to representation learning and training of multiple models;
[0023] (2) The method of applying machine learning to network traffic analysis is standardized, which reduces the current reliance of machine learning on manual feature extraction in network traffic analysis.
[0024] (3) Supports multiple network protocols and combinations of multiple network protocols, and supports custom field extension methods. Attached Figure Description
[0025] Figure 1 This is a schematic diagram of an automated encrypted traffic analysis method for multiple protocols and protocol combinations according to the present invention.
[0026] Figure 2 This is a schematic diagram illustrating the characteristic structure of network traffic data packets;
[0027] Figure 3This is a data flow diagram illustrating an automated encrypted traffic analysis method across multiple protocols and protocol combinations according to the present invention. Detailed Implementation
[0028] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings and embodiments.
[0029] like Figure 1 The diagram shown is a flowchart of an automated encrypted traffic analysis method for multiple protocols and protocol combinations according to the present invention. The process includes the following steps:
[0030] Step 1: Preprocess the network traffic of the encrypted application to be classified to obtain the data packet of each network flow in the network traffic. The specific operation includes: dividing the network traffic into network flows based on the five-tuple information including source IP address, destination IP address, source port, destination port and protocol, to obtain multiple tagged network traffic data packets, which are used as the input for Step 2.
[0031] Step 2: Obtain the semantic representation of the current network traffic data packets and perform feature extraction on the current network traffic data packets. This specifically includes the following processing:
[0032] Step 2.1: Extract all IP header information from the network data packets, including but not limited to IP version number, IP options field, source IP address, destination IP address, etc.
[0033] Step 2.2: Extract all feature information from the TCP and UDP headers, including but not limited to source port number, destination port number, etc.
[0034] Step 2.3: Extract all information from the ICMP protocol packet header;
[0035] like Figure 2 The diagram shown is a schematic representation of the characteristic structure of network traffic data packets.
[0036] Step 3: Obtain the binary representation of the network packet features and construct a fixed-length feature vector containing 1088 features. This includes the following steps:
[0037] Step 3.1: Map all 60 bytes of features in the IP protocol packet header to 480 IP header features. Each IP header feature corresponds to the position information within each byte of the IP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 480 features in the IP header.
[0038] Step 3.2: Map all 60 bytes of features in the TCP protocol packet header to 480 TCP packet header features. Each TCP header feature corresponds to the position information within each byte of the TCP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 480 features in the TCP header.
[0039] Step 3.3: Map all 8 bytes of UDP header features into 64 UDP header features. Each UDP header feature corresponds to the position information within each byte of the UDP header. Each byte is represented by 8 bits, and the 0 and 1 values at each position correspond to the 64 features of the UDP header.
[0040] Step 3.4: Map all 8 bytes of ICMP protocol packet header features into 64 ICMP packet header features. Each feature corresponds to the position information within each byte of the ICMP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 64 features of the ICMP packet header.
[0041] Step 3.5: Concatenate all the above features to obtain a fixed-length feature vector containing 1088 features. If the data packet does not contain the corresponding protocol header (please provide an example of the underlined part), then all corresponding positions are assigned a value of -1. For example, the TCP protocol does not include ICMP and UDP protocols, so the 64 UDP feature vectors corresponding to UDP and the 64 ICMP feature vectors corresponding to ICMP are all assigned a value of -1.
[0042] Step 4, Feature Export: Export the network packet features obtained in Step 3 (or Step 4 if needed) into a CSV file to facilitate reading by various machine learning algorithms;
[0043] Step 5, Finding the Optimal Classification Model: Models are trained for each encrypted network traffic classification problem. These models originate from six different basic model classes, derived from tree-based machine learning methods, deep neural network encoder-based methods, and variations of distance-based classification methods, totaling 50 models. The best-performing classification model is selected based on metrics such as average accuracy and F1 score.
[0044] Step 6: Classify encrypted network traffic. Process the encrypted network traffic to be classified according to the methods in Steps 3-5, and then input it as a feature into the optimal classification model obtained in Step 6 to complete the classification of encrypted network traffic.
[0045] When business requirements necessitate expanding the load characteristic fields, the following processing is included after step 3 of this invention:
[0046] The extended field representation of the network packet characteristics is obtained. The load characteristic field needs to be extended according to business requirements. If N load characteristics need to be extended, the first N characteristic values (0 or 1) of the load are automatically concatenated into N characteristics and appended to the characteristics obtained in step 3.
Claims
1. An automated encrypted traffic analysis method across multiple protocols and protocol combinations, characterized in that, The method includes the following steps: Step 1: Preprocess the network traffic of the encrypted application to be classified to obtain the data packet of each network flow in the network traffic. The specific operation includes: dividing the network traffic into network flows based on the five-tuple information including source IP address, destination IP address, source port, destination port and protocol, to obtain multiple tagged network traffic data packets, which are used as the input for Step 2. Step 2: Obtain the semantic representation of the current network traffic data packets and perform feature extraction on the current network traffic data packets. This specifically includes the following processing: Step 2.1: Extract all IP protocol header information from the network data packets, including but not limited to the IP protocol number, IP protocol options field, source IP protocol address, and destination IP protocol address; Step 2.2: Extract all characteristic information of the TCP protocol packet header and UDP protocol packet header, including but not limited to the source port number and destination port number; Step 2.3: Extract all information from the ICMP protocol packet header; Step 3: Obtain the binary representation of network packet features and construct a fixed-length network packet feature vector containing 1088 features. This includes the following steps: Step 3.1: Map all 60 bytes of features in the IP protocol packet header to 480 IP protocol packet header features. Each IP protocol packet header feature corresponds to the position information within each byte of the IP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 480 features in the IP protocol packet header. Step 3.2: Map all 60 bytes of features in the TCP protocol packet header to 480 TCP protocol packet header features. Each TCP header feature corresponds to the position information within each byte of the TCP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 480 features in the TCP protocol packet header. Step 3.3: Map all 8 bytes of UDP header features into 64 UDP header features. Each UDP header feature corresponds to the position information within each byte of the UDP header. Each byte is represented by 8 bits, and the 0 and 1 values at each position correspond to the 64 features of the UDP header. Step 3.4: Map all 8 bytes of ICMP protocol packet header features into 64 ICMP packet header features. Each feature corresponds to the position information within each byte of the ICMP protocol packet header. Each byte is represented by 8 bits of binary, and the 0 and 1 values at each position correspond to the 64 features of the ICMP protocol packet header. Step 3.5: Concatenate all the above features to obtain a fixed-length network data packet feature vector containing 1088 features. If the data packet does not contain the corresponding protocol data packet header, then all corresponding positions are assigned a value of -1. When business requirements necessitate expanding the load feature field, and based on the business requirements, the load feature field needs to be expanded by N load features, then the first N feature values of 0 or 1 of the load are automatically concatenated into N features, which are then appended to the fixed-length network data packet feature vector containing 1088 features obtained in this step. Step 4: Export the network packet features obtained in Step 3 into a CSV file; Step 5: Find the optimal classification model. This involves training a model for each encrypted network traffic classification problem. These models are derived from six different basic model classes, which are 50 classes of models derived from tree-based machine learning methods, deep neural network encoders, and distance-based classification methods. The best-performing classification model is selected based on the average accuracy and F1 score. The 50 models are variant models derived from the above six basic model classes. Step 6: Classify encrypted network traffic. Process the encrypted network traffic to be classified according to the methods in Steps 3-5, and then input it as a feature into the optimal classification model obtained in Step 5 to complete the classification of encrypted network traffic.