A method and system for automatic identification and extraction of network traffic features

Abstract

This invention provides a method for automatically identifying and extracting network traffic features, comprising: starting process packet capture, stopping process packet capture, and automatically saving the captured packets as pcap files to a specified directory; automatically decompressing and reading one or more pcap data packets, constructing data using 5-tuples as dictionary keys to temporarily store variables, constructing child node dictionary key-value pairs, and writing the data to a shelve container for persistent storage; reading the shelve data, classifying and processing the data according to traffic features, comparing and analyzing, merging features, and extracting features; forming a data matrix using pandas technology, and finally generating a feature library CSV file and a log file. The beneficial effects of this invention are: it enables process packet capture, stopping packet capture, automatic identification and extraction of traffic analysis features, and automatic output of key statistical indicators of features to a CSV file, thereby improving product development progress, reducing labor costs, and increasing work efficiency.

Description

Technical Field

[0001] This invention relates to the field of identification and extraction methods, and in particular to a method and system for automatic identification and extraction of network traffic features. Background Technology

[0002] With the rapid development of the internet and network technology, current R&D personnel developing gateway products often need to use traffic feature libraries for intelligent access restriction functions. Manually collecting data from thousands of data packets, analyzing network traffic characteristics, and extracting key unique features for output to text is a significant increase in manpower and time, involving repetitive and tedious work with very low efficiency. Therefore, a method is needed that can automatically read pcap packet file data after packet capture, collect traffic session data packets, construct a data model format for storage, train data recognition matching degree, automatically classify and analyze traffic, extract unique features, statistically analyze various key indicators, and output them to a feature library CSV file. Pcap files are a commonly used data packet storage format; they can be understood as a file format, but the data is stored in a specific format, so parsing the data must also follow a specific format. CSV files, or Comma-Separated Values ​​files, sometimes also called character-separated values, store tabular data (numbers and text) in plain text format.

[0003] Current methods for extracting network traffic features generally suffer from the following drawbacks:

[0004] 1. The lack of a corresponding network application traffic feature library means that the parental control function of gateway products cannot identify traffic feature restrictions, making it difficult to meet market demands;

[0005] 2. Manual data collection is too costly and prone to errors.

[0006] 3. Manual traffic flow feature analysis involves a lot of repetitive work and is too frequent. It lacks automated traffic flow feature extraction and statistical indicator output to text files. Summary of the Invention

[0007] To address the problems in existing technologies, this invention provides a method and system for automatic identification and extraction of network traffic features. It can capture and stop packet capture processes, read single or multiple pcap data packets, automatically identify and extract traffic analysis features, and automatically output key statistical indicators to a CSV file. This supports various gateway products for intelligent site access restriction functions, thereby improving product development progress, reducing labor costs, and increasing work efficiency. It solves the problems in existing technologies such as the lack of a corresponding network application traffic feature library, high cost and error-prone manual data collection, and low efficiency of manual traffic feature analysis.

[0008] The method for automatic identification and extraction of network traffic features according to the present invention includes the following steps:

[0009] Step 1: Send a command to capture packets to call the dumpcap command to construct a command line to start the process packet capture. After capturing packets for a certain period of time, send a command to terminate the process packet capture to stop the process packet capture. The captured packets are automatically saved as a pcap file to the specified directory.

[0010] Step 2: After sending the command to generate the feature library, automatically decompress and read one or more pcap data packets, construct data using 5-tuples as dictionary keys to temporarily store variables, then train the data model, filter out non-critical data, construct child node dictionary key-value pairs, and write the data to the shelve container for persistent storage.

[0011] Step 3: Read the shelve data, classify and process the data according to traffic characteristics, and call the general module to process the carrier data byte stream. Classify and compare the data according to different dimensions such as protocol, port, packet length, carrier data hexadecimal header or footer, and HTTP request header. After feature merging, use regular expressions to extract unique features. The method of reading the packet_datas data of the shelve container is as follows: iteratively read the size key value of the master node to count the total packet length, and at the same time, disassemble the key of the master node. The child node reads the key corresponding to the master node and appends the data packet size, IP, packet length per session, and port of the master node to the key value of the child node.

[0012] Step 4: Based on the traffic-classified data, perform statistical analysis on various indicators and add them to the sub-key value list of the child nodes. Then, iterate through the child nodes to read their key values ​​and form a data matrix using pandas technology. Finally, generate a feature library CSV file and a log file.

[0013] The present invention is further improved by including the following steps in step 2:

[0014] Step 201: Send the command to generate the feature library document;

[0015] Step 202: Automatically decompress the captured pcap packet file, read the complete TCP or UDP packets of the session, calculate the TCP or UDP carrier data length, and construct the source and destination 5-tuples to store in variables;

[0016] Step 203: Filter out non-IP packet types and incomplete session traffic data, and read the original carrier data;

[0017] Step 204: Append the packet length, timestamp, and session characteristics of each session to the list of specified 5-tuple keys, construct the 5-tuple master node data model, and at the same time construct the child node dictionary key-value pair, append the session characteristics, timestamp, IP, port, and packet length of each session to the corresponding key-value pair list of the child nodes, and write the data to the shelve container for persistent storage.

[0018] The present invention is further improved by including the following step in step 3:

[0019] Step 301: Read the packet_datas data of the shelve container using the shelve's open method;

[0020] Step 302: Process data according to traffic characteristics;

[0021] Step 303: Call the encapsulated get_sig method to perform network byte feature comparison and analysis on the session carrier data;

[0022] Step 304: Perform feature merging again, using regular expressions to describe and extract unique features.

[0023] The present invention is further improved in that, in step 1, before starting to execute process packet capture, the net_if_stats method of the third-party library psutil is initialized to check whether the network card in the current environment is enabled. The service process name of the passed parameter is written to the ini configuration file by the set method of the configparser library. At the same time, the network card device name and service process name are passed in and the dumpcap command is called to construct the command line execution process packet capture.

[0024] The present invention is further improved by calling the tasklist command or the Process method of the psutil library in step 1 to terminate the process and capture packets.

[0025] The present invention is further improved by using the Reader method of the third-party library dpkt to decompress the captured pcap file data packets in step 202.

[0026] In a further improvement to this invention, in step 204, the packet size, timestamp, session characteristics, packet length of each session, and carrier data prefix are protocol + source IP + source port + destination IP + destination port + packet size. The carrier data is then converted to hexadecimal using binascii.b2a_hex to construct a quintuple master node data model.

[0027] The present invention is further improved in that, in step 4, the key values ​​of the child nodes are read iteratively, including the protocol, traffic matching degree, traffic packet length, IP range, port range and session characteristics.

[0028] The present invention also provides a system for the above-mentioned method of automatic identification and extraction of network traffic features, comprising:

[0029] The command-line instruction interface management module is used to send command-line instructions, call the corresponding interface subsystem to execute them, and return data.

[0030] The packet capture management subsystem module is used for process packet capture, terminating process packet capture, viewing network card device names, and viewing application service process names; it is also used to write the application service process name as a parameter into the ini configuration file when sending process packet capture commands.

[0031] The traffic feature generation subsystem module is used to automatically parse the ini configuration file, read the service process name, search for the pcap packet file, and read TCP or UDP data after the user sends a traffic feature generation command. Through data filtering and data model training, it constructs a dictionary key-value pair for master and child nodes and writes it to the shelve persistent storage. Then, it reads data from the shelve's packet_datas file for traffic classification. It calls a general module to process the packet carrier byte stream, converting it to hexadecimal and classifying it into multiple features based on different dimensions such as protocol, port, packet length, hexadecimal header or tail of carrier data, and HTTP request header. It then compares and analyzes to extract unique features, uses regular expressions to describe the features, appends them to the child node key-value storage, and finally, it statistically analyzes various indicators and outputs a feature library CSV file and log file records.

[0032] The beneficial effects of this invention are as follows: This invention provides a method and system for automatic identification and extraction of network traffic features. It can capture and stop packet capture, and read single or multiple pcap data packets. It can automatically identify and extract traffic analysis features and automatically output key statistical indicators to a CSV file. It can support various gateway products for intelligent site access restriction functions, thereby improving product development progress, reducing labor costs, and increasing work efficiency. Furthermore, this invention uses a lower-level dumpcap command, giving it a performance advantage in capturing large data packets. This invention uses the dpkt Reader method to decompress captured pcap file data packets, resulting in very fast reading of large data packets and no lag or slow decompression of large data packets, improving the stability of the tool. This invention uses PSU... The til library method can efficiently acquire data such as network interface cards (NICs) and application service processes in the current environment using iterators. It employs a core technology of unsupervised training data models, classifies and stores data according to network traffic characteristics, and uses the binascii.b2a_hex method to convert TCP or UDP carrier network byte streams to hexadecimal. This enables forward and reverse comparison analysis of multiple feature bytes, followed by feature merging and extraction. Unique features are described using regular expressions, and finally, various indicators are statistically analyzed. Pandas technology is used to map the results of traffic feature analysis to corresponding data and integrate them into a data matrix. The to_csv method is then used to generate a feature library CSV file and a log file. This solves the problems of existing technologies, such as the lack of corresponding network application traffic feature libraries, high cost and error-prone manual data collection, and low efficiency of manual traffic feature analysis. Attached Figure Description

[0033] Figure 1 This is a flowchart of a method for automatic identification and extraction of network traffic features according to the present invention;

[0034] Figure 2 This is a flowchart of a method for automatic identification and extraction of network traffic features according to the present invention;

[0035] Figure 3 This is a flowchart of a method for automatic identification and extraction of network traffic features according to the present invention;

[0036] Figure 4 This is a system block diagram illustrating a method for automatically identifying and extracting network traffic features according to the present invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0038] Please see Figure 1-4 The method for automatic identification and extraction of network traffic features according to the present invention includes the following steps:

[0039] Step 1: Sending Capture Commands Prerequisites: First, obtain the application process name from the INI configuration file and find the corresponding process ID. Use the `connections` method of the `psutil.Process` interface to obtain the source IP and destination port of the network connection associated with the application process ID as filtering conditions. Call the `dumpcap` command to construct a command line to start capturing packets from the process. After capturing packets for a certain period, send a command to terminate the process packet capture. Automatically save the captured packets as a pcap file to the specified directory. Before starting process packet capture, the `net_if_stats` method of the third-party library `psutil` is used to check if the network card is enabled. The passed parameter, the application service process name, is written to the INI configuration file using the `set` method of the `configparser` library. Simultaneously, the passed parameters, the network card device name and the service process name, are used to construct a command line to execute process packet capture using the `dumpcap` command. Finally, the `tasklist` command or the `Process` method of the `psutil` library is called to terminate process packet capture.

[0040] Step 2: After sending the command to generate the feature library file, automatically decompress and read one or more pcap data packets, construct data using 5-tuples as dictionary keys to temporarily store variables, then train the data model, filter out non-critical data, construct child node dictionary key-value pairs, and write the data to the shelve container for persistent storage.

[0041] Step 3: Read the shelve data, classify and process the data according to traffic characteristics, call the general module to process the carrier data byte stream, classify and compare different dimensions such as protocol, port, packet length, carrier data hexadecimal header or footer, and HTTP request header, and then merge the features and use regular expressions to extract unique features.

[0042] Step 4: Based on the traffic-classified data, perform statistical analysis on various indicators and add them to the sub-key value list of the child nodes. Then, iterate through the child nodes to read their key values ​​and form a data matrix using pandas technology. Finally, generate a feature library CSV file and a log file. The key values ​​read from the child nodes in the iterative process include the protocol, traffic matching degree, traffic packet length, IP range, port range, and session characteristics.

[0043] In this embodiment, the main program entry point uses the getopt method of a third-party library to encapsulate commands, providing command help and instructions, and allowing users to view all network interface device names, application service process names, process packet capture, stop packet capture, and generate data files for traffic feature analysis. This satisfies the user's ability to set parameters and call the interface for execution. Prerequisites: The user starts process packet capture using the tool, operates the application service, and then sends a stop packet capture command. The program automatically saves the pcap packet file to the specified directory. Sending a traffic feature generation command automatically helps the user decompress the pcap packet file. After filtering the data, a data model is constructed for training. The data is organized into traffic classification and temporary storage. TCP or UDP packet carrier bytes are converted to hexadecimal and ASCII codes. Forward and reverse byte comparisons are performed. Unique traffic features are extracted from multiple repeated features and described using regular expressions. Simultaneously, multiple indicators are statistically analyzed to automatically generate a traffic feature data CSV file. The generated feature library CSV file facilitates comprehensive analysis of key points of traffic features in daily work, while also fulfilling the feature library function of a gateway product to identify traffic features.

[0044] Please see Figure 2 Step 2 further includes the following steps:

[0045] Step 201: Send the command to generate the feature library.

[0046] Step 202: Automatically decompress the captured pcap packet file, read the complete TCP or UDP packets of the session, calculate the TCP or UDP carrier data length, and construct the source and destination 5-tuples to store in variables; among them, the Reader method of the third-party library dpkt is used to decompress the captured pcap file packets.

[0047] Step 203: Filter out non-IP packet types and incomplete session traffic data, and read the original carrier data.

[0048] Step 204: Append the packet length, timestamp, and session characteristics of each session to a list of specified 5-tuple keys to construct the 5-tuple master node data model. Simultaneously, construct the child node dictionary key-value pairs, appending session characteristics, timestamp, IP address, port, and packet length of each session to the corresponding key-value lists of the child nodes, and write the data to the shelve container for persistent storage. Among them, the packet size, timestamp, session characteristics, packet length of each session, and carrier data prefix are protocol + source IP + source port + destination IP + destination port + packet size. The carrier data is then converted to hexadecimal using binascii.b2a_hex to construct the 5-tuple master node data model.

[0049] Please see Figure 3 Step 3 further includes the following steps:

[0050] Step 301: Read the packet_datas data of the shelve container using the open method of shelve; the method for reading the packet_datas data of the shelve container is to iteratively read the size key value of the master node to count the total packet length, and at the same time disassemble the key of the master node. The child node reads the key corresponding to the master node and appends the packet size, IP, packet length per session, and port of the master node to the key value of the child node.

[0051] Step 302: Process data according to traffic characteristics.

[0052] Step 303: Call the encapsulated get_sig method to perform network byte feature comparison analysis on the session carrier data.

[0053] Step 304: Perform feature merging again, using regular expressions to describe and extract unique features.

[0054] In this embodiment, the automatic network traffic feature identification and extraction tool is developed using the Python programming language. Its functions include process packet capture, stopping packet capture, viewing network interface card (NIC) device names, viewing application service process names, and automatically generating a feature database CSV file. Specifically, for process packet capture and stopping packet capture, the `Popen` method of the third-party library `subprocess` is used to construct a `dumpcap` command line method for capturing packets. The iterator from the `psutil` library is used to find the process ID to be terminated. After obtaining the process ID, the `Process` method of the `psutil` library is used to terminate the process, or the `popen` method of the `os` library is used to call the `taskkill` command to terminate the process. To view NIC device and application process names, the `process_iter` method of the `psutil` library is used to iterate, appending to a list and outputting all current application service process names. The `net_if_addrs` method of the `psutil` library is used to iterate and read all NIC device names returned by the current environment. Traffic features are automatically generated. The data construction module uses the `dpkt` library's `Reader` method to decompress the pcap data packet file, obtain the TCP or UDP data of the IP packets, and construct the packet's five-tuple and key indicators to define a dictionary key-value storage format. The `shelve` library's `open` method is used to persistently store the constructed data model in the `packet_datas` file. The feature analysis module reads the `packet_datas` data model, associates the key-value pairs of the main and child nodes, performs data processing and traffic classification, integrates key indicator data and appends it to the child node key-value pairs, calls the `get_sig` function to convert the traffic carrier bytes to hexadecimal using the `binascii.b2a_hex` method, performs forward and reverse multi-feature comparison analysis, and uses regular expressions to extract unique feature descriptions. Finally, pandas technology is used to construct a data matrix, and the `to_csv` method is called to write the feature data to a CSV file.

[0055] Please see Figure 4 The present invention also provides a system for the above-mentioned method of automatic identification and extraction of network traffic features, comprising:

[0056] The command-line instruction interface management module is used to send command-line instructions, call the corresponding interface subsystem to execute them, and return data.

[0057] The packet capture management subsystem module is used for process packet capture, terminating process packet capture, viewing network card device names, and viewing application service process names; it is also used to write the application service process name as a parameter into the ini configuration file when sending process packet capture commands.

[0058] The traffic feature generation subsystem module is used to automatically parse the ini configuration file, read the service process name, search for the pcap packet file, and read TCP or UDP data after the user sends a traffic feature generation command. Through data filtering and data model training, it constructs a dictionary key-value pair for master and child nodes and writes it to the shelve persistent storage. Then, it reads data from the shelve's packet_datas file for traffic classification. It calls a general module to process the packet carrier byte stream, converting it to hexadecimal and classifying it into multiple features based on different dimensions such as protocol, port, packet length, hexadecimal header or tail of carrier data, and HTTP request header. It then compares and analyzes to extract unique features, uses regular expressions to describe the features, appends them to the child node key-value storage, and finally, it statistically analyzes various indicators and outputs a feature library CSV file and log file records.

[0059] In this embodiment, the directory structure of the network traffic feature automatic identification and extraction tool is as follows:

[0060] Dipflowfeature

[0061] |--db ==> contains the ini configuration file;

[0062] --logs ==> stores log records;

[0063] --resource ==> Stores template files and resource files;

[0064] The report is automatically generated and stored as a feature library file, data model, and pcap message file saved from packet capture.

[0065] Dipflowfeature.exe is used as the main entry point for the program.

[0066] II. The command line provides menu-driven instructions, which are used as follows:

[0067] (1) Enter Dipflowfeature or Dipflowfeature -h in the DOS window command line to print the help content;

[0068] Parameter description:

[0069] -h|--help: Help;

[0070] --run : network interface device name: application process name [parameters are required; it is recommended to enter the application service name for packet capture (use the -p parameter to view the process name)];

[0071] -s|--stop: Stop packet capture;

[0072] -v|--view: View all network interface device names;

[0073] -p|--process: View the application service process name;

[0074] -c|--create: Generate feature analysis data from a single pcap file [Applicable tools and prerequisites: generating features after packet capture (packet capture must be stopped before generating features)];

[0075] -m|--more: Generate feature analysis data from a single or multiple pcap files [Applicable to feature data generated from pcap data packets that already exist in the report directory];

[0076] (2) Process packet capture: Dipflowfeature --run wan:QQ;

[0077] (3) Stop packet capture: Dipflowfeature -s or Dipflowfeature --stop;

[0078] (4) Check the network card device: Dipflowfeature -v;

[0079] (5) View the application service process name: Dipflowfeature -p;

[0080] (6) Generate a single feature library file: Dipflowfeature -c;

[0081] (7) Generation of feature libraries for single or multiple pcap files: Dipflowfeature -m.

[0082] As can be seen from the above, the beneficial effects of the present invention are as follows: The present invention provides a method and system for automatic identification and extraction of network traffic features, which can realize process packet capture, stop packet capture, and read single or multiple pcap data packets. It can automatically identify and extract traffic analysis features and automatically output key indicators of statistical features to a CSV file. It can support various gateway products for intelligent restriction of access sites, thereby improving product development progress, reducing labor costs, and improving work efficiency. Moreover, the present invention uses the lower-level dumpcap command, which has a performance advantage in capturing large-capacity data packets. The present invention uses the dpkt Reader method to decompress the captured pcap file data packets, which has very fast performance in reading large-capacity data and does not have a slow or lag-free decompression of large-capacity data packets, thus improving the performance and stability of the tool. The present invention uses p The sutil library method can efficiently acquire data such as network interface cards (NICs), application service processes, etc., in the current environment using iterators. It employs a core technology of unsupervised training data models, classifies and stores data according to network traffic characteristics, and uses the binascii.b2a_hex method to convert TCP or UDP carrier network byte streams to hexadecimal. This enables forward and reverse comparison analysis of multiple feature bytes, followed by feature merging and extraction. Unique features are described using regular expressions, and finally, various indicators are statistically analyzed. Pandas technology is used to map the results of traffic feature analysis to corresponding data and integrate them into a data matrix. The to_csv method is called to generate a feature library CSV file and a log file. This solves the problems of existing technologies, such as the lack of corresponding network application traffic feature libraries, high cost and error-prone manual data collection, and low efficiency of manual traffic feature analysis.

[0083] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A method for automatic identification extraction of network traffic features, characterized in that, Comprising the following steps, Step 1: send capture packet instruction call dumpcap instruction to construct command line to start process packet capture, send termination process packet capture instruction to stop process packet capture after capturing data packet for a certain time, and automatically save the captured packet to a specified directory in the form of a pcap file; Step 2: after sending the generate feature library instruction, automatically decompress and read the pcap data packet single or multiple files, construct data as a dictionary key temporarily stored in a variable in the form of a five tuple, and then perform data model training to filter out non-key data, while constructing a sub-node dictionary key value and writing it to a shelve container for persistent data storage; Step 3: read the shelve data, perform data processing according to traffic characteristics classification, call the general module processing carrier data byte stream, compare and judge according to different dimensions such as protocol, port, packet length, carrier data 16 hexadecimal header or tail, http request header, and then perform feature merging and use regular expression description to extract unique features; wherein the method for reading packet_datas data of the shelve container is to loop and iterate to read the size key value of the main node to count the total packet length, and at the same time, to disassemble the key of the main node, read the corresponding key of the main node from the sub-node, and append the packet size, IP, and per-session packet length of the main node to the key value of the sub-node; Step 4: according to the classified traffic data, perform each index statistics and append to the sub key value list of the sub node, and then loop and iterate to read the key value of the sub node, form a data matrix through pandas technology, and finally generate a feature library CSV file and a log file.

2. The method of claim 1, wherein, In the step 2, further comprising the following steps, Step 201: send generate feature library instruction; Step 202: automatically decompress the captured pcap data packet file, read the complete TCP or UDP data packet of the session, calculate the TCP or UDP carrier data length, and construct the five tuple of the source and destination to store in a variable; Step 203: filter out non-IP packet types and incomplete session traffic data, and read the carrier raw data; Step 204: append the packet length, timestamp, and session characteristics of each session to the specified five tuple key list, construct a five tuple main node data model, and at the same time, construct a sub-node dictionary key value, append the session characteristics, timestamp, IP, port, and per-session packet length to the corresponding key value list of the sub node, and write it to the shelve container for persistent data storage.

3. The method of automatic network traffic feature extraction of claim 2, wherein, In the step 3, further comprising the following steps, Step 301: read the packet_datas data of the shelve container through the open method of the shelve; Step 302: perform data processing according to traffic characteristics classification; Step 303: call the encapsulated get_sig method to compare and analyze the session carrier data for network byte characteristics; Step 304: perform feature merging and use regular expression description to extract unique features.

4. The method of automatic network traffic feature extraction of claim 3, wherein: In the step 1, the net_if_stats method of the third-party library psutil is used to check whether the current environment network card is enabled before starting the process packet capture, the application service process name is applied to the ini configuration file by the set method of the configparser library, the service process name is written into the ini configuration file, and the network card device name and the service process name are applied to the dumpcap instruction to construct the command line to execute the process packet capture.

5. The method of automatic network traffic feature identification extraction of claim 4, wherein: In the step 1, the tasklist command or the Process method of the psutil library is called to terminate the process packet capture.

6. The method of automatic network traffic feature extraction of claim 5, wherein: In the step 202, the Reader method of the third-party library dpkt is referenced to decompress the captured pcap file data packet.

7. The method of automatic network traffic feature identification extraction of claim 6, wherein: In the step 204, the size, timestamp, session characteristics, packet length of each session and carrier data prefix of the data packet are protocol+source IP+source port+destination IP+destination port+packet size, followed by the carrier data, which is converted into hexadecimal by binascii.b2a_hex to construct the five-tuple master node data model.

8. The method of automatic network traffic feature extraction of claim 7, wherein: In the step 4, the key values of the child node are iteratively read, including the protocol, traffic matching degree, traffic packet length, IP range, port range and session characteristics.

9. A system for implementing the method of automatic extraction of network traffic signatures according to any one of claims 1-8, characterized in that, Comprise: The command line instruction interface management module is used to send command line instructions, call the corresponding interface subsystem to execute, and return data. The packet capture management subsystem module is used for process packet capture, termination of process packet capture, viewing of network card device name and application service process name. The application service process name is written into the ini configuration file when the process packet capture instruction is sent. The traffic feature generation subsystem module is used to automatically parse the ini configuration file to read the service process name to find the pcap data packet file to read TCP or UDP data after the user sends the traffic feature generation instruction, filter the data, train the data model, construct the master node and child node dictionary key values and write them into the shelve persistent storage, then read the packet_datas file data of the shelve to process the traffic classification, call the general module to convert the carrier byte stream of the data packet into hexadecimal, classify multiple features from different dimensions such as protocol, port, packet length, carrier data hexadecimal header or tail, http request header, compare and judge to extract unique features, use regular expressions to describe the features and append them to the child node key value storage, finally, count each index and output the feature library CSV file and log file record.

Images