A distributed network protocol analysis and data processing system

Abstract

This invention discloses a distributed network protocol analysis and data processing system. It employs a traffic distribution and application decoupling model to achieve efficient traffic splitting of high-speed online network traffic and massive offline sampling files, as well as targeted data distribution to distributed protocol message analysis and statistical calculation program modules. This achieves both improved performance in parsing basic network traffic metadata and increased flexibility and scalability for custom functions. The invention adopts a data processing framework that separates raw data splitting from deep protocol parsing, avoiding the need for all raw traffic to undergo complete parsing from the link layer to the application layer. Unparseable application layer protocol data or invalid parsing operations are pre-judged at the transport layer, reducing the time overhead of the parsing program in single-packet parsing.

Description

Technical Field

[0001] This invention belongs to the field of network data processing technology, specifically relating to a distributed network protocol analysis and data processing system. Background Technology

[0002] In network traffic monitoring, security protection, and application management, correctly parsing the protocol messages carried in network data packets and fully calculating network traffic statistical characteristics are fundamental technologies for detecting network attacks, discovering illegal network applications, and monitoring network operation status. For hosts or LANs used by individuals and small and medium-sized enterprises, there are currently many commercial and open-source software and systems for network protocol analysis and traffic monitoring available both domestically and internationally. Typical software and systems include Wireshark, Microsoft's Network Monitor, and open-source software such as Nethor and NetworkMiner. Wireshark can read pcap files and online data streams from network cards as input, presenting the protocol parsing results as a list of data packets as the main view. All analysis functions are based on data packets, and the parsing results of each protocol field in a single data packet and the original binary sequence of the single data packet are dynamically correlated with the main view, allowing for comparison and viewing of the original binary data and its parsing results. Based on data packets, Wireshark provides flow-oriented packet aggregation analysis functions, such as TCP, UDP, SIP, HTTP, SCTP, and packet reassembly for multimedia sessions. Due to its most comprehensive set of protocol decoders, this software has become the preferred tool in the field of protocol analysis. Protocol analysis software that follows a "packet-data stream-application layer message" analysis logic includes Nethor, WireEdit, and NetAnalyzer. While their protocol parsing capabilities are not as strong as Wireshark's, they offer more in-depth analytical features, typically in terms of visualization and ease of use during user interaction. NetworkMiner is unique in that its data organization logic and analysis targets are all IP hosts. Unlike the aforementioned software, it does not allow viewing raw binary data; instead, it directly provides analysis results specific to IP hosts.

[0003] However, for large-scale network security protection and monitoring, high-speed network traffic makes the protocol parsing performance of network data packets a bottleneck restricting system application efficiency. The aforementioned software and systems are unsuitable for high-speed network traffic or massive data analysis scenarios, as protocol parsing speed is far slower than network data reception speed. Furthermore, with the increasing complexity and proliferation of network attack methods and network application types, the existence of known protocol variants and proprietary protocols with undisclosed packet specifications necessitates the flexible and dynamic embedding of specific protocol packet analysis into existing protocol parsing processes in a plug-in manner. In other words, network protocol analysis and data processing software and systems need to support user-defined functional extensions while minimizing the learning curve for users. This flexible extensibility is relatively lacking in the aforementioned analysis software and systems. Even though Wireshark supports parsing extensions for custom protocols, users still need to convert the parser into Lua scripts, increasing the difficulty of use to some extent. In addition, Wireshark does not yet support extended functions such as custom payload data analysis, custom calculation of traffic statistics characteristics, and custom concatenation of multiple analysis processes. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a distributed network protocol analysis and data processing system. It employs a traffic distribution and application decoupling model to achieve efficient traffic splitting of high-speed online network traffic and massive offline sampling files, as well as targeted data distribution to distributed protocol message analysis and statistical calculation program modules. This achieves both improved performance in parsing basic network traffic metadata and increased flexibility and scalability for custom functions. This invention adopts a data processing framework that separates raw data splitting from deep protocol parsing, avoiding the need for all raw traffic to undergo complete parsing from the link layer to the application layer. Unparseable application layer protocol data or invalid parsing operations are pre-judged at the transport layer, reducing the time overhead of the parsing program in single-packet parsing.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows: A distributed network protocol analysis and data processing system includes a service scheduling module, a data access module, a stream parsing module, a custom protocol parsing module group, a high-dimensional traffic statistics feature calculation module, a TCP / UDP / SCTP grouping module, a custom protocol message analysis module, a message middleware and data cache module, and a data persistent storage module. The service scheduling module is responsible for registering the basic services and data processing modules, and starts the protocol parsing and analysis processing module as needed through the configuration file; The data access module is responsible for reading real-time traffic on the online network card or reading data packets from offline pcap / pcapng files and passing the data to the stream parsing module; at the same time, the data access module writes the amount of data read, file information, and customizable additional information into the message pipeline and data cache module. The stream parsing module extracts the link layer address, network layer source and destination IP addresses, transport layer source and destination TCP / UDP / SCTP port numbers, and traffic statistics parameters, and writes the parsing results into the data cache module. When further analysis and processing of the stream is required, the subsequent processing module is notified through a message pipeline to continue the data analysis task. The custom protocol parsing module group adds the names of the protocol fields to be parsed through configuration files. Each protocol starts a dedicated protocol parsing process to extract the content of the specified protocol fields. The stream parsing module distributes data processing messages according to the protocol rules. The custom protocol parsing process will only receive network data streams containing the protocol message and will not notify network data streams that do not contain the protocol message. The high-dimensional traffic statistical feature calculation module realizes statistical analysis of traffic; the high-dimensional traffic statistical feature calculation module concurrently processes the IP streams generated by the stream parsing module through a distributed deployment method; The TCP / UDP / SCTP grouping module realizes the grouping restoration of the payload data of the three transport layer protocols TCP / UDP / SCTP, and sequentially concatenates the transport layer payloads of multiple data packets in a stream into the original message, and writes the result into the persistent data storage module. The custom protocol message analysis module provides an interface template for receiving message notifications from the stream parsing module and outputting analysis result data. It can obtain the data parameters of the IP traffic and protocol messages to be processed simply by receiving messages from the message pipeline. The output results are also notified to the data persistence storage module through the message pipeline to complete the result writing to disk and database. The message middleware and data cache module adopts self-developed message pipeline, commercial products, or open source products. The middleware type only needs to be specified in the configuration file. This module shields the underlying implementation from other modules and provides a unified message service to the outside world through the message_send and message_receive function interfaces. All message content is encapsulated in a unified format using message_body. The data persistence storage module supports multiple data storage types. This module shields the specific storage type from other modules, requiring only the storage method to be specified in the configuration file. Each data analysis and processing module provides unified storage operations to the outside world through the data_persist function interface.

[0006] Preferably, the extracted flow statistics parameters include flow length, flow velocity, flow duration, and average packet length.

[0007] Preferably, the open-source products include RocketMQ, RabbitMQ, and Kafka.

[0008] Preferably, the multi-type data storage includes writing to a Redis database, an Elasticsearch index, or a file to disk.

[0009] The beneficial effects of this invention are as follows: 1. This invention adopts a data processing framework that separates raw data splitting and deep protocol parsing, avoiding the need for all raw traffic to undergo complete parsing from the link layer to the application layer protocol. It pre-judges unparseable application layer protocol data or invalid parsing operations at the transport layer, reducing the time overhead of the parsing program in single packet parsing. 2. This invention provides high application-oriented scalability through message middleware and high-speed data caching. Adding new protocol parsing or data analysis functions can be achieved without modifying or recompiling the original program code. Simply add a message interface to the new function code and update the flow splitting rules in the stream parsing module configuration file to realize the new function expansion.

[0010] 3. The distributed deployment processing architecture of this invention enables dynamic adjustment of the overall system processing performance on demand. In contrast, existing single-machine deployment protocol parsing and data processing software and systems can only improve computing performance by upgrading the host hardware configuration. Therefore, this invention is more suitable for distributed computing scenarios with high-speed network traffic under big data conditions. Furthermore, since each data processing module is deployed independently and distributed, the code of each module is not coupled into a single program. The failure, downtime, upgrades, or resource consumption of a single module will not interfere with the normal operation of other modules, nor will it cause the fault propagation phenomenon that may occur when the code is tightly coupled, thus enhancing the overall stability of the system. Attached Figure Description

[0011] Figure 1 This is a system block diagram of the present invention; Figure 2 It is a flow statistical characteristic graph. Detailed Implementation

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0013] To address the problem of existing network protocol parsing and data processing software and systems being unable to simultaneously achieve high performance and functional flexibility, this invention adopts a traffic distribution and application decoupling model. This enables efficient traffic splitting between high-speed online network traffic and massive offline sampling files, as well as targeted data distribution to distributed protocol message analysis and statistical calculation program modules. This achieves the goal of both improving the parsing performance of basic network traffic metadata (such as IP 5-tuples) and increasing the flexibility and scalability of custom functions.

[0014] This invention proposes a distributed network protocol analysis and data processing method, apparatus, and system that can improve protocol parsing performance while ensuring flexibility in function expansion and traffic processing, and reduce user costs and the difficulty of secondary customization development.

[0015] The overall analysis and processing framework of this invention is as follows: Figure 1 As shown, the system comprises several main components, including a service scheduling module, a message middleware and data caching module, a persistent data storage module, a data access module, a stream parsing module, a high-dimensional traffic statistics feature calculation module, a TCP / UDP / SCTP grouping module, a custom protocol parsing module group, and a custom protocol message analysis module. The data access module and stream parsing module constitute the basic service modules of this invention, representing the necessary processing flow after online traffic and offline file input. The high-dimensional traffic statistics feature calculation module, the custom protocol parsing module group, and the custom protocol message analysis module constitute a distributed set of protocol parsing and data processing modules. This means that the basic service modules do not need to complete the full parsing of protocol messages; they only need to parse the necessary fields of the link layer, network layer, and transport layer protocols, thereby improving parsing speed. For tasks requiring detailed analysis and processing of specific protocol messages, subsequent modules can be linked via message notification, thus decoupling data and functionality. When the data volume is large, multiple hosts or a single machine with multiple processes can be deployed to concurrently process the data parsing messages generated by the stream parsing module, achieving a smooth scaling effect for parsing performance.

[0016] The service scheduling module is responsible for registering the basic services and data processing modules. The protocol parsing and analysis processing modules can be started as needed through the configuration file.

[0017] The data access module is responsible for reading real-time traffic from online network cards or data packets from offline pcap / pcapng files, and then passing the data to the stream parsing module. Simultaneously, this module writes the read data volume, file information, and customizable additional information to the message pipeline and data cache module for subsequent processing by subsequent modules.

[0018] The stream parsing module completes the parsing of link layer addresses, network layer source and destination IP addresses, transport layer source and destination TCP / UDP / SCTP port numbers, and extracts simple traffic statistics parameters such as flow length, flow rate, flow duration, and average packet length. The parsing results are written to a high-speed data cache. If further analysis and processing of the flow is required, a message pipeline notifies subsequent processing modules to continue the data analysis task. Because the stream parsing module only parses a small number of protocol fields below the transport layer, the protocol specifications are simple, and the number of protocol types is limited, eliminating the complex parsing requirements of application layer protocols, thus accelerating the parsing performance at level 1.

[0019] The custom protocol parsing module group allows users to add the names of protocol fields to be parsed via configuration files. Each protocol will launch a dedicated protocol parsing process to extract the content of the specified protocol fields. It's important to note that the stream parsing module distributes data processing messages according to protocol rules. The custom protocol parsing process will only receive network data streams containing packets of that protocol; it will not notify users of network data streams that do not contain packets of that protocol. Therefore, the actual amount of data that the custom protocol parsing process needs to parse is much smaller than the original input data, thus achieving a performance acceleration for the second level of parsing specific protocol data.

[0020] The high-dimensional traffic statistics feature calculation module enables in-depth statistical analysis of traffic and serves as a foundational dataset for building and understanding traffic behavior patterns. Due to the large amount of statistical computation, this module can be deployed in a distributed manner to concurrently process the IP streams generated by the flow parsing module.

[0021] The TCP / UDP / SCTP grouping module implements grouping reconstruction of the payload data of these three transport layer protocols. It sequentially concatenates the transport layer payloads of multiple data packets in a stream into the original message, and writes the result to the persistent data storage module so that the subsequent analysis module can perform further data mining.

[0022] The custom protocol message analysis module provides an interface template for receiving message notifications from the flow parsing module and outputting analysis results. It can obtain the data parameters of the IP traffic and protocol messages to be processed simply by receiving messages from the message pipeline. The output results are also written to the disk and stored in the database by notifying the data persistence storage module through the message pipeline.

[0023] The message middleware and data caching module can use either a self-developed message pipeline or mature commercial or open-source products such as RocketMQ, RabbitMQ, and Kafka. Simply specify the middleware type in the configuration file. This module shields other modules from the underlying implementation details, providing a unified message service through the `message_send` and `message_receive` function interfaces. All message content is encapsulated in a unified format using `message_body`.

[0024] The data persistence storage module supports multiple data storage types, such as writing to Redis databases, Elasticsearch indexes, or files to disk. This module shields other modules from the specific storage type; they only need to specify the storage method in the configuration file. All data analysis and processing modules provide unified storage operations to the outside world through the data_persist function interface.

[0025] Example: The present invention is as follows Figure 1 The system components shown can be deployed on the same host, with each module running as an independent process simultaneously; or they can be deployed in a distributed manner on multiple hosts.

[0026] First, ensure that external hardware and software resources such as databases, file systems, network cards, or offline files are ready, and that the corresponding connection addresses and account information are correctly configured in the system configuration file.

[0027] The second step is to correctly configure the data sources to be read, as well as the protocol parsing modules and data processing modules to be enabled and their parallel parameters in the system configuration file. After the service scheduling module starts, it starts each module according to the configuration file parameters, and each module enters a standby state waiting for messages.

[0028] The third step involves the data access module periodically scanning offline files or receiving real-time online traffic from the network card. After parsing the data source and additional information, it passes the raw traffic to the flow parsing module. The flow parsing module extracts the IP 5-tuple and simple traffic statistics parameters from the data packets and assigns a unique flow identifier to each IP 5-tuple for subsequent analysis and tracing. Each data packet in the raw traffic is grouped into a data cache according to its flow identifier, and the flow identifier and data cache path are encapsulated in a message body, which is then used by a message broker to notify the corresponding modules for processing.

[0029] The fourth step involves the distributed data processing modules, such as the protocol parsing module, statistical feature calculation module, and group reporting module, receiving data notifications according to predetermined rules, parsing the data cache path from the message body, reading the data, and completing the corresponding parsing and processing. The analysis results, along with the stream identifier, are written to the persistent data storage module, which then periodically writes them to the database and file system in batches.

[0030] After the system starts, the data processing flow repeats in the third and fourth steps. When the offline file is processed or the network card has no data to receive, each module of the system is in standby mode until new data arrives and automatically enters a new round of processing.

[0031] The performance of this invention was tested on a server with an Intel Xeon® Gold 6336Y CPU@2.4GHz, 64GB of memory, and CentOS 7.6 operating system. The input size was 289.79 MB, containing a pcap file with 1 million packets. Wireshark, a well-known industry software, took 12.76 seconds to parse the stream, achieving a parsing speed of 181.69 Mbps. The stream parsing module of this invention, under single-core non-parallel conditions, took 5.46 seconds to parse, achieving a parsing speed of 424.6 Mbps, representing an approximately 2.3-fold improvement in single-core computing performance. When the server is a multi-core high-performance server, multi-core parallel computing mode can be enabled, and the parsing performance can increase linearly with the number of CPU cores. The computational performance for parsing complex protocol fields and extracting specified protocol fields is comparable. Furthermore, Wireshark's flow statistical features are approximately 10-dimensional, while the high-dimensional traffic statistical feature calculation module of this invention can generate 90-dimensional flow statistical features, more comprehensively depicting the statistical behavior patterns of IP flows. Figure 2 The characteristic dimensions of the traffic statistics are displayed.

Claims

1. A distributed network protocol analysis and data processing system, characterized in that, It includes a service scheduling module, a data access module, a stream parsing module, a custom protocol parsing module group, a high-dimensional traffic statistics feature calculation module, a TCP / UDP / SCTP grouping module, a custom protocol message analysis module, a message middleware and data cache module, and a data persistent storage module; The service scheduling module is responsible for registering the basic services and data processing modules, and starts the protocol parsing and analysis processing module as needed through the configuration file; The data access module is responsible for reading real-time traffic on the online network card or reading data packets from offline pcap / pcapng files and passing the data to the stream parsing module; at the same time, the data access module writes the amount of data read, file information, and customizable additional information into the message pipeline and data cache module. The stream parsing module extracts the link layer address, network layer source and destination IP addresses, transport layer source and destination TCP / UDP / SCTP port numbers, and traffic statistics parameters, and writes the parsing results into the data cache module. When further analysis and processing of the stream is required, the subsequent processing module is notified through a message pipeline to continue the data analysis task. The custom protocol parsing module group adds the names of the protocol fields to be parsed through configuration files. Each protocol starts a dedicated protocol parsing process to extract the content of the specified protocol fields. The stream parsing module distributes data processing messages according to the protocol rules. The custom protocol parsing process will only receive network data streams containing the protocol message and will not notify network data streams that do not contain the protocol message. The high-dimensional traffic statistical feature calculation module realizes statistical analysis of traffic; the high-dimensional traffic statistical feature calculation module concurrently processes the IP streams generated by the stream parsing module through a distributed deployment method; The TCP / UDP / SCTP grouping module realizes the grouping restoration of the payload data of the three transport layer protocols TCP / UDP / SCTP, and sequentially concatenates the transport layer payloads of multiple data packets in a stream into the original message, and writes the result into the persistent data storage module. The custom protocol message analysis module provides an interface template for receiving message notifications from the stream parsing module and outputting analysis result data. It can obtain the data parameters of the IP traffic and protocol messages to be processed simply by receiving messages from the message pipeline. The output results are also notified to the data persistence storage module through the message pipeline to complete the result writing to disk and database. The message middleware and data cache module adopts self-developed message pipeline, commercial products, or open source products. The middleware type only needs to be specified in the configuration file. This module shields the underlying implementation from other modules and provides a unified message service to the outside world through the message_send and message_receive function interfaces. All message content is encapsulated in a unified format using message_body. The data persistence storage module supports multiple data storage types. This module shields the specific storage type from other modules, requiring only the storage method to be specified in the configuration file. Each data analysis and processing module provides unified storage operations to the outside world through the data_persist function interface.

2. The distributed network protocol analysis and data processing system according to claim 1, characterized in that, The extracted flow statistics parameters include flow length, flow velocity, flow duration, and average packet length.

3. The distributed network protocol analysis and data processing system according to claim 1, characterized in that, The open-source products include RocketMQ, RabbitMQ, and Kafka.

4. The distributed network protocol analysis and data processing system according to claim 1, characterized in that, The various data storage types include writing to a Redis database, an Elasticsearch index, or files to disk.

Images