Rapid processing and monitoring system based on network flow data

Abstract

The invention discloses a rapid processing and monitoring system based on network flow data, and the system comprises a network flow data obtaining module which is used for transmitting mirror image network data to a designated network port, binding the designated network port through a DPDK, and obtaining the network flow data through multiple threads and multiple queues; the session flow table module is used for acquiring quintuple information of an Ethernet message, creating a session flow table based on the quintuple information and storing the network flow data into the session flow table; and the traffic application identification module is used for acquiring a data message of the network traffic data, identifying based on a protocol identification engine, completing data acquisition and data cleaning by adopting DPDK multi-core thread binding, and realizing high-coverage and high-accuracy service identification and intelligent analysis of network application traffic in combination with a rule engine.

Description

Network Traffic Data Rapid Processing and Monitoring System Technical Field

[0001] This application relates to the field of network traffic data processing technology, specifically to a monitoring system based on rapid processing of network traffic data. Background Technology

[0002] With the rapid development of internet and mobile internet technologies, the rapid construction of infrastructure, and the immense enrichment of network applications, the demands on internet applications are also increasing. The rapid growth of network traffic, especially the explosive growth of short video traffic from platforms like Douyin, Kuaishou, and Xigua Video in China, and YouTube, Netflix, and Twitter internationally, has led to a massive increase in "entertainment" data on the network. This has had a significant impact on the effective extraction and storage of network data. How to quickly identify network traffic distribution, quickly extract effective data to identify network applications, how to quickly store traffic data, how to monitor network traffic data, and how operations and maintenance personnel can manage and maintain the network are all pain points faced by operators and enterprises. However, how to intelligently maintain the network and identify network application traffic has revealed many problems.

[0003] The specific defects and problems of the existing technology are as follows: 1. Single application identification: Relying solely on network port application identification is no longer sufficient for modern network application identification. If some applications deliberately change their ports during network transmission, the application will inevitably fail to be identified. 2. Low efficiency of network data identification: Extracting effective data from network application payloads using regular expression rules will inevitably affect the application identification speed with massive amounts of data, and may even cause network congestion or network interruption in severe cases. 3. Low degree of automation: Operation and maintenance personnel use existing tools (such as Wireshark) to capture and analyze applications, which cannot meet the needs of real-time analysis. 4. Lack of contextual association: It is impossible to analyze and view historical information, access patterns, and other contextual traffic characteristics. Summary of the Invention

[0004] This invention provides a network traffic data rapid processing and monitoring system to solve the technical problems described in the background section. The system includes: a network traffic data acquisition module, used to send mirrored network data to a designated network port, bind the designated network port through DPDK, and acquire network traffic data through multi-threading and multi-queueing; a session flow table module, used to acquire the five-tuple information of Ethernet packets, create a session flow table based on the five-tuple information, and store the network traffic data in the session flow table; and a traffic application identification module, used to acquire the data packets of the network traffic data and identify them based on a protocol identification engine.

[0005] Preferably, the network traffic data acquisition module specifically includes: a data acquisition initialization module, used to perform initialization operations on the environment abstraction layer of DPDK; and a data acquisition module, used to poll the network card flag bit of the specified network port through DPDK, and determine whether network traffic data exists based on the network card flag bit, and acquire the network traffic data.

[0006] Preferably, the acquisition initialization module is specifically used for: initializing and starting the kernel, which includes loading the user-mode polling driver PMD, setting the RSS key, configuring big page memory, and binding the DPDK network card to receive packets; obtaining big page memory, CPU logical cores, PCI device addresses, and network card queues through EAL; allocating a memory buffer pool independently for each CPU logical core according to the big page mechanism to accelerate memory allocation efficiency, and allocating a circular buffer for each DMA queue to store data messages; allocating a thread on each CPU logical core for association and binding, and allocating a memory buffer to receive network data packets, realizing a 1:1:1 ratio of threads, queues, CPU logical cores, and memory pools, while enabling promiscuous mode for the corresponding network card and starting the thread to receive packets.

[0007] Preferably, the data acquisition module is specifically used for: actively polling the network card's flag bit through the DPDK query engine to determine if there is network traffic data; when network traffic data arrives, performing batch data reading to fill the buffer; when the data in the circular queue reaches the set threshold, the CPU performs a write operation to collect and process all the data in the circular queue; if the preset threshold is not reached, it continues to wait; and enqueuing the collected data into the circular queue of the parsing thread. Preferably, the session flow table module is specifically used for: performing memory mapping according to the TCP / IP protocol structure and quickly extracting the five-tuple data of the packet; calculating the hash value according to the five-tuple information, creating a session flow table, and maintaining all flow data node information through the hash session flow table; initializing flow session node data, each flow session node data includes: five-tuple, timestamp, session direction, data packet statistics for each application type, the number of bytes sent uplink and downlink in the application data packet, as well as storage and probe protocol bitmap information and target host; when a new network data packet is received from the network card, firstly performing hash value calculation, then searching the hash session flow table according to the hash index, if it does not exist, then creating a new flow node and adding it to the session flow table, if it exists, then updating the session traffic statistics; when the flow session ends or the aging time expires, clearing the aged flow node of the session, releasing all information saved by the flow node, and reclaiming the flow node cache.

[0008] Preferably, the traffic application identification module is specifically used for: loading and initializing a feature rule base when the system starts. The feature rule base is in the format of protocol ID: protocol type: protocol rule, where the protocol ID is a value, the protocol type is TCP / UDP / ICMP for Ethernet packets, or application layer protocols such as HTTP and FTP, or the type of website or specific application software accessed by the user; the protocol rule is a regular expression or string pattern that defines the feature. The feature rule base maps the ID type and protocol type to a one-to-one correspondence according to a certain data structure and saves it in the cache; when the system starts, it initializes the hyperscan engine to compile block mode, stream mode, and vector mode, compiles multiple expressions of protocol ID: protocol rule type according to the feature rule base and caches them in the engine database. Each mode can be associated with a unique protocol ID and protocol rule; and parses the network plaintext payload data to obtain valid payload feature data. The optimal hyperscan engine mode is selected for matching based on the business scenario. A single data input undergoes parallel processing and unordered scanning of all rules. The engine matches the specific location of the protocol rule within the payload data, as well as the corresponding protocol rule. Simultaneously, it automatically identifies and returns the matched rule features and the corresponding protocol ID. The protocol ID is indexed into the protocol type using a pre-cached feature library and saved to the session node. For network encrypted data, during the TLS handshake phase, the client sends an SNI extension in the ClientHello message to specify the domain name to be accessed. The obtained domain name information is then used by the hyperscan engine for pattern matching to identify the protocol type, which is then saved to the session node. Customized business identification is performed according to business needs. The destination IP and destination port of the customized business are added to the rule feature library. When network traffic packets enter the traffic application identification module, protocol identification is prioritized based on the destination IP and destination port.

[0009] Preferably, the system further includes a data storage module, which specifically includes: a stream session caching module for caching data in memory pre-allocated to the system according to a time period; and a database storage module for writing stream sessions into the database according to an agreed format.

[0010] Preferably, the stream session cache module is specifically used for: obtaining a pre-allocated stream session cache of the flow table in large page memory when the system starts, configuring the time interval for initializing the system's real-time statistics, and writing cached data into the stream cache according to the time interval of the real-time statistics. The cached data is used to provide the system platform with real-time display of bandwidth traffic, host analysis, and application analysis.

[0011] Preferably, the database storage module is specifically used for: completing Kafka-related configurations and initiating the connection between Kafka and the server when the system starts; reading the stream table session cache data according to the agreement when the stream session ends or times out; performing ordered byte reading and escaping of the stream table session data into JSON data; uploading it to the service platform through the Kafka message queue; and writing the stream session data into the MangoDB database in real time.

[0012] Preferably, the system further includes an intelligent traffic analysis and monitoring module, which is specifically used for: obtaining real-time data traffic from the stream session cache via a socket connection according to an agreed data format; starting a socket communication service in the stream session cache module and determining the data type of the real-time data traffic, wherein the data type includes host analysis, application analysis, or filtering condition query; sending back corresponding data based on the data type; performing multi-dimensional statistical analysis on the real-time data traffic, generating and outputting statistical reports, wherein the multi-dimensional statistical analysis specifically includes link statistical analysis, user statistical analysis, domain name statistical analysis, and latency statistical analysis.

[0013] Compared with existing technologies, the present invention has the following beneficial effects: The present invention discloses a network traffic data rapid processing and monitoring system, which includes: a network traffic data acquisition module, used to send mirrored network data to a designated network port, bind the designated network port through DPDK, and acquire network traffic data through multi-threaded multi-queue; a session flow table module, used to acquire the five-tuple information of Ethernet packets, create a session flow table based on the five-tuple information, and store the network traffic data in the session flow table; and a traffic application identification module, used to acquire the data packets of the network traffic data, identify them based on a protocol identification engine, complete data collection and data cleaning by using DPDK multi-core thread binding, and combine it with a rule engine to achieve high coverage and high accuracy of service identification and intelligent analysis of network application traffic. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 shows a schematic diagram of the structure of a network traffic data fast processing and monitoring system proposed in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0017] This invention aims to solve the problems existing in the prior art and provide a network traffic data rapid processing and monitoring system. It utilizes DPDK (Data Plane Development Kit) multi-core thread binding to complete data acquisition and cleaning, combined with a rule engine, to achieve high coverage and high accuracy in service identification and intelligent analysis of network application traffic. It is particularly suitable for complex network environments with large data traffic from telecom operators. The innovations of this invention are as follows: 1. DPDK multi-core thread data acquisition and processing: It adopts DPDK polling mode zero-copy data technology; it uses large-page memory binding to allow the program to exclusively occupy memory, preventing memory switching and improving memory data capture; it binds application threads to different CPU cores, fully utilizing CPU cores and reducing the performance overhead of repeated scheduling; it uses DPDK's unique RTE_buf data structure to store data, requiring only one memory allocation, reducing memory allocation overhead and improving the speed of data reception, storage, and acquisition.

[0018] 2. Hyperscan Rule Engine: Utilizing multi-threaded Hyperscan rule engine matching technology, each thread has an independent state context, avoiding resource contention and improving data concurrency processing.

[0019] 3. Intelligent Network Analysis: Utilizing a visualized network structure, it provides a clear view of the network layout, helping network administrators understand the interrelationships of network traffic and the distribution of user traffic. Through real-time graphical display, it can quickly locate network problems and performance bottlenecks, providing decision support for network design and helping to optimize network configuration.

[0020] Referring to Figure 1, the embodiment of the present invention proposes a monitoring system based on rapid processing of network traffic data, specifically: a network traffic data acquisition module, used to send mirrored network data to a designated network port, bind the designated network port through DPDK, and acquire network traffic data through multiple threads and multiple queues.

[0021] In a preferred embodiment of this application, the network traffic data acquisition module specifically includes: a data acquisition initialization module 100, used to perform initialization operations on the environment abstraction layer of DPDK; and a data acquisition module, used to poll the network card flag bit of the specified network port through DPDK, and determine whether network traffic data exists based on the network card flag bit, and acquire the network traffic data.

[0022] In a preferred embodiment of this application, the acquisition initialization module is specifically used for: initializing and starting the kernel, which includes loading the user-mode polling driver PMD, setting the RSS key, configuring big page memory, and binding the DPDK network card to receive packets; obtaining big page memory, CPU logical cores, PCI device addresses, and network card queues through EAL; allocating a memory buffer pool independently for each CPU logical core according to the big page mechanism to accelerate memory allocation efficiency, and allocating a circular buffer for each DMA queue to store data messages; allocating a thread on each CPU logical core for association and binding, and allocating a memory buffer to receive network data packets, realizing a 1:1:1 ratio of threads, queues, CPU logical cores, and memory pools, while simultaneously enabling promiscuous mode for the corresponding network card and starting the thread to receive packets.

[0023] In a preferred embodiment of this application, the data acquisition module is specifically used to: actively poll the network card's flag bit through the DPDK query engine to determine whether there is network traffic data; when network traffic data arrives, perform batch data reading to fill the buffer; when the data in the circular queue reaches the set threshold, the CPU performs a write operation to collect and process all the data in the circular queue; if the preset threshold is not reached, continue to wait; and enqueue the collected data into the circular queue of the parsing thread.

[0024] Specifically, the network traffic data acquisition module is divided into two parts: packet acquisition initialization and data processing. Acquisition initialization mainly involves the initialization of the DPDK Environment Abstraction Layer (EAL); data processing uses the DPDK polling engine to actively poll the network interface card's flags to indicate whether data has arrived. When data arrives, DPDK reads the data in batches and processes the packets.

[0025] Specifically, the processing steps of the acquisition initialization module are as follows: S111: First, complete the initialization and startup of the kernel, including loading the user-mode polling driver PMD, setting the RSS key, configuring the large page memory, binding the DPDK network card to receive packets, etc.

[0026] S112: Obtain hardware information such as large page memory, CPU logical cores, PCI device addresses, and network card queues through EAL.

[0027] S113: Each CPU logical core independently allocates a memory buffer pool according to the big page mechanism to accelerate memory allocation efficiency, and allocates a circular buffer for each DMA queue to store data messages.

[0028] S114: The application allocates one thread on each CPU logical core, associates and binds them, and allocates a memory buffer to receive network data packets, realizing a 1:1:1 ratio of threads, queues, CPU logical cores, and memory pools. At the same time, the corresponding network card is enabled in promiscuous mode, and the thread is started to receive packets.

[0029] Specifically, the data acquisition module's processing flow is as follows: S121: The DPDK query engine actively polls the network card's flag bit to determine if there is data. When data arrives, it performs batch data reading to fill the buffer.

[0030] S122: When the data in the circular queue reaches the set threshold, the CPU performs a write operation to collect and process all the data in the circular queue. If the threshold is not reached, the CPU continues to wait.

[0031] S123: Enqueue the received data into the circular queue of the parsing thread. This effectively avoids context switching caused by frequent CPU interrupts and improves packet acquisition efficiency.

[0032] The session flow table module 200 is used to obtain the five-tuple information of Ethernet packets, create a session flow table based on the five-tuple information, and store the network traffic data in the session flow table.

[0033] In a preferred embodiment of this application, the session flow table module is specifically used for: performing memory mapping according to the TCP / IP protocol structure and quickly extracting the five-tuple data of the packet; calculating the hash value according to the five-tuple information, creating a session flow table, and maintaining all flow data node information through the hash session flow table; initializing flow session node data, each flow session node data includes: five-tuple, timestamp, session direction, data packet statistics for each application type, the number of bytes sent uplink and downlink for application data packets, and storage and probe protocol bitmap information and target host; when a new network data packet is received from the network card, firstly performing hash value calculation, then searching the hash session flow table according to the hash index; if it does not exist, creating a new flow node and adding it to the session flow table; if it exists, updating the session traffic statistics; when the flow session ends or the aging time expires, clearing the aged flow node, releasing all information saved by the flow node, and reclaiming the flow node cache.

[0034] Specifically, Ethernet packets are disassembled and unpacked to obtain the packet's five-tuple information. The hash value is calculated using the five-tuple information, a hash session flow table is created, and different flow information is identified through the hash value index.

[0035] Specifically, the processing flow of the session flow table module is as follows: S210: Memory mapping is performed according to the TCP / IP protocol structure to reduce copying and quickly extract the five-tuple data of the packet (source address, destination address, source port, destination port, protocol type).

[0036] S220: Calculate the hash value based on the quintuple information, create a session flow table, and the system maintains all flow data node information through the hash session flow table.

[0037] S230: Initialize the stream session node data. Each stream session node data includes: a 5-tuple, a timestamp, the session direction, packet statistics for each application type, the number of bytes sent uplink and downlink in the application data packets, as well as storage and probe protocol bitmap information and target host information.

[0038] S240: When a new network data packet is received from the network card, the hash value is first calculated, and then the hash session flow table is searched according to the hash index. If it does not exist, a new flow node is created and added to the session flow table. If it exists, the session traffic statistics are updated.

[0039] S250: While creating and updating the session flow table, the system will also start the flow session timeout aging mechanism. When the flow session ends or the aging time expires, the system will promptly clear the flow nodes that have been aged out and release the flow nodes to save all data.

[0040] The traffic application identification module 300 is used to acquire the data packets of the network traffic data and identify them based on the protocol identification engine.

[0041] In a preferred embodiment of this application, the traffic application identification module is specifically used for: loading and initializing a feature rule base when the system starts. The feature rule base is in the format of protocol ID: protocol type: protocol rule, where the protocol ID is a numerical value, the protocol type is TCP / UDP / ICMP for Ethernet packets, or application layer protocols such as HTTP and FTP, or the type of website or specific application software accessed by the user; the protocol rule is a regular expression or string pattern that defines the feature. The feature rule base maps the ID type and protocol type to a one-to-one correspondence according to a certain data structure and saves it in the cache; when the system starts, the hyperscan engine is initialized to compile block mode, stream mode, and vector mode, and multiple expressions of protocol ID: protocol rule type are compiled according to the feature rule base and cached in the engine database. Each mode can be associated with a unique protocol ID and protocol rule; and the network plaintext payload data is parsed to obtain the effective features of the payload. For data, the hyperscan engine mode is selected for matching based on the business scenario. A single data input undergoes parallel processing and unordered scanning of all rules. The engine matches the specific location of the protocol rule within the payload data, as well as the corresponding protocol rule. Simultaneously, it automatically identifies and returns the matched rule features and the corresponding protocol ID. The protocol ID is indexed into the protocol type using a pre-cached feature library and saved to the session node. For network encrypted data, during the TLS handshake phase, the client sends an SNI extension in the ClientHello message to specify the domain name to be accessed. The obtained domain name information is then used by the hyperscan engine for pattern matching to identify the protocol type, which is then saved to the session node. Customized business identification is performed according to business needs. The destination IP and destination port of the customized business are added to the rule feature library. When network traffic packets enter the traffic application identification module, protocol identification is prioritized based on the destination IP and destination port.

[0042] Specifically, the processing flow of the traffic application identification module is as follows: S310: When the system starts, it loads and initializes the feature rule base, maps the protocol type according to the feature rules, and the feature rule base needs to be updated regularly to improve the application protocol identification rate. Feature base engine initialization: The Hyperscan multi-mode rule matching engine is initialized in each business processing thread, and Hyperscan's unique block mode, stream mode and vector mode are used for fast matching. This process greatly improves the efficiency of network traffic service identification.

[0043] S320: When a data packet enters the traffic application identification module, it first uses a known protocol port for identification, such as DNS (port 53), based on the port. This can quickly identify the protocol type of the traffic, improve identification efficiency, and save the identified protocol type to the session node.

[0044] S330: Parses plaintext payload data, obtains valid payload feature data, and uses the hyperscan engine for multi-mode matching. If a matching rule feature is found, the corresponding protocol type is returned and saved to the session node. Each session only matches the first 10 packet payload data. If no protocol type is matched in the first 10 packet payload data, subsequent packets from the same session will not be matched again, and the session will be uniformly merged into the TCP or UDP protocol application type. Once the packet payload has successfully completed protocol type identification, subsequent packets from the same session will not be matched again, thus ensuring fast protocol identification and improving identification efficiency.

[0045] S340: For encrypted network data, the SNI (Server Name Indication) in the TLS handshake protocol is extracted. During the TLS handshake phase, the client sends an SNI extension in the ClientHello message to indicate the domain name to be accessed. The obtained domain name information is used by the hyperscan engine to perform pattern matching to identify the protocol type. After the protocol identification is completed, subsequent packets are no longer matched according to rules, and the application type is merged into the TLS application type.

[0046] S350: Customized service identification can be performed according to business needs. The destination IP and destination port of the customized service can be added to the rule feature library. When network traffic packets enter the traffic application identification module, the destination IP and destination port are identified first. If the identification is successful, the protocol identification steps S320, S330 and S340 will not be performed. Otherwise, the application identification steps above will continue.

[0047] Specifically, when the system starts, it loads and initializes the feature rule base, with the format being Protocol ID: Protocol Type: Protocol Rule (e.g., 1:HTTP:"HTTP 1.1"). The Protocol ID is a numerical value; the Protocol Type is TCP / UDP / ICMP for Ethernet packets, or application layer protocols such as HTTP and FTP, or the website or specific application software type accessed by the user (e.g., WeChat, TikTok); and the Protocol Rule is a regular expression or string pattern that defines the feature. The feature rule base maps ID types and protocol types to a one-to-one correspondence according to a certain data structure and saves it in the cache. When the system starts, the hyperscan engine is initialized and compiled in various modes such as block mode, stream mode, and vector mode. According to the feature rule base, multiple expressions of protocol ID: protocol rule type (such as: 1: protocol rule 1, 2: protocol rule 2, etc.) are compiled and cached in the engine database. Each mode can be associated with a unique protocol ID and protocol rule. The network plaintext payload data is parsed to obtain the effective feature data of the payload. The optimal hyperscan engine mode is selected for matching according to the business scenario. The entire rule is scanned in parallel in one data input. The engine will match the specific location of the protocol rule in the payload data and which protocol rule is matched. At the same time, it will automatically identify and return the matched rule features and the corresponding protocol ID. The protocol ID is indexed by the pre-cached feature base rules to find the protocol type and save it to the session node. For encrypted network data, during the TLS handshake phase, the client sends an SNI extension in the ClientHello message to specify the domain name to be accessed. The obtained domain name information is then used by the hyperscan engine to perform pattern matching to identify the protocol type, and the protocol type is saved to the session node. Customized service identification is performed according to business needs, and the destination IP and destination port of the customized service are added to the rule feature library. When network traffic packets enter the traffic application identification module, the destination IP and destination port are prioritized for protocol identification.

[0048] In a preferred embodiment of this application, the system further includes a data storage module 400, which specifically includes: a stream session caching module for caching data in memory pre-allocated to the system according to a time period; and a database storage module for writing stream sessions into the database according to an agreed format.

[0049] In a preferred embodiment of this application, the stream session cache module is specifically used for: obtaining a stream session cache that pre-allocates and initializes the stream table in large page memory when the system starts, configuring the time interval for initializing real-time statistics of the system, and writing cached data into the stream cache according to the time interval for real-time statistics. The cached data is used to provide the system platform with real-time display of bandwidth traffic, host analysis, and application analysis.

[0050] In a preferred embodiment of this application, the database storage module is specifically used for: completing Kafka-related configurations and initiating the connection between Kafka and the server when the system starts; reading the stream table session cache data according to the agreement when the stream session ends or times out; performing ordered byte reading and escaping of the stream table session data into JSON data; uploading it to the service platform through the Kafka message queue; and writing the stream session data into the MangoDB database in real time.

[0051] Specifically, data storage is mainly divided into streaming session caching and database storage. Streaming session caching is memory pre-allocated by the system to cache data periodically according to time periods; database storage is the writing of streaming sessions into the database according to an agreed format.

[0052] Specifically, the data storage module's processing flow is as follows: S410: Upon system startup, the system obtains pre-allocated large page memory to initialize the flow table's flow session cache, and configures the time interval for initializing real-time system statistics, which can be selected as 3 seconds, 10 seconds, or 30 seconds. After completing the session information processing in steps S200 and S300, the system writes the data into the flow cache according to the real-time statistical time interval. This cached data mainly provides the system platform with real-time display of bandwidth traffic, host analysis, and application analysis. The system platform queries the flow session cache every 3, 10, or 30 seconds according to the configured timeout, calculates statistical data according to the agreed data structure, and calculates and reports statistics to the system platform for display based on the timestamps stored in the flow sessions. The real-time cached data also provides an aging mechanism. When the flow session cached data times out, the session is cleared and the cache is released to prevent expired session flow data from occupying memory for a long time, thereby reducing memory allocation and improving memory utilization.

[0053] S420: Upon system startup, Kafka-related configurations are completed, and the connection between Kafka and the server is initiated, ensuring a successful connection. When a streaming session ends or times out, the session cache data in the streaming table is read according to the agreement. The streaming table session data is read in an ordered byte order and converted into JSON data, which is then uploaded to the service platform via the Kafka message queue. The streaming session data is written to the MongoDB database in real time. At this time, the logs in the database contain full-traffic session collection information, including a 5-tuple, timestamp, session direction, application type, packet statistics for each application type, the number of bytes sent uplink and downlink in the application data packets, and the domain name information accessed by the session, in order to trace the network status of historical data.

[0054] In a preferred embodiment of this application, the system further includes an intelligent traffic analysis and monitoring module 500. This intelligent traffic analysis and monitoring module is specifically used for: obtaining real-time data traffic from a stream session cache via a socket connection according to an agreed-upon data format; starting a socket communication service in the stream session cache module and determining the data type of the real-time data traffic, including whether the data type is host analysis, application analysis, or a filtering condition query; sending back corresponding data based on the data type; performing multi-dimensional statistical analysis on the real-time data traffic, generating and outputting statistical reports, specifically including link statistical analysis, user statistical analysis, domain name statistical analysis, and latency statistical analysis.

[0055] Specifically, the workflow of the intelligent traffic analysis and monitoring module is as follows: Intelligent traffic analysis and monitoring mainly focuses on real-time traffic data analysis and historical data statistical analysis. It provides multi-dimensional analysis and display, including real-time traffic analysis and monitoring, and historical traffic source tracing analysis. It identifies nodes with the highest bandwidth consumption, preventing network traffic overload, ensuring the stability of critical nodes, and providing a valid basis for bandwidth allocation. Through efficient data retrieval, it enables rapid mining and tracking of traffic data, allowing users to view and retrieve data for any time period at any time. It provides hierarchical analysis from different dimensions and time intervals, including interfaces, IPs, services, applications, sessions, and QoS, performing layer-by-layer data correlation and drill-down to help network operation and maintenance personnel grasp the network's operational status in real time.

[0056] S510: Real-time data traffic acquisition and analysis. Data is retrieved from the streaming session cache via a socket connection according to a pre-defined data format. Real-time data traffic types are categorized into host analysis and application analysis types. Both types are acquired based on a timeline, collected at 3, 10, or 30-second intervals as configured during system initialization. A socket communication service is started in the streaming session cache module to process data according to query conditions, determining whether the data type is host analysis, application analysis, or a filtered query.

[0057] The host analysis type is based on the data returned by the host analysis module, and the statistics of data returned at different levels are displayed in a hierarchical manner. The first level is the total data statistics for users, showing the number of all users, the number of uplink bytes, the number of downlink bytes, the number of uplink packets, the number of downlink packets, and the uplink and downlink bandwidth. The second level is the data statistics for all applications under each user, showing the total data volume of all applications, the number of uplink bytes, the number of downlink bytes, the number of uplink packets, the number of downlink packets, and the uplink and downlink bandwidth. The third level is the session statistics display, specifically the statistics for all sessions under each application, including source IP, destination IP, source port, destination port, number of uplink bytes, number of downlink bytes, number of uplink packets, number of downlink packets, and the uplink and downlink bandwidth.

[0058] Application analytics data is returned according to the application analytics module, and is also displayed in layers according to different levels. The first level is the total data statistics for all applications, showing the number of all applications, the number of uplink bytes, the number of downlink bytes, the number of uplink packets, the number of downlink packets, and the uplink and downlink bandwidth. The second level is the data statistics for all users under each application, showing the total data volume for all users, the number of uplink bytes, the number of downlink bytes, the number of uplink packets, the number of downlink packets, and the uplink and downlink bandwidth. The third level is the specific session statistics display, specifically the statistics for all sessions under each user, including source IP, destination IP, source port, destination port, number of uplink bytes, number of downlink bytes, number of uplink packets, number of downlink packets, and the uplink and downlink bandwidth.

[0059] Conditional filtering returns data by analyzing different filtering conditions. The filtering conditions are sorted by user type, port type, and application type and displayed in a layered manner according to TOPN. Among them, user type is divided into source IP, destination IP or IP mask; port type is divided into source port and destination port; application type is the application identification type based on traffic analysis.

[0060] The different query conditions described above, after statistical analysis of the streaming session cache and real-time data transmission through fast sorting, include data source, destination, transmission protocol, data traffic, bandwidth, and data transmission timestamps. This information provides administrators with a comprehensive view of network traffic, enabling them to quickly understand the network's traffic status. Within the interface, network administrators can see real-time updates to network traffic data, as well as data change trends displayed through visual charts, including traffic curves and protocol distribution maps, which intuitively reflect the dynamic changes in network traffic. Through real-time monitoring and visualization, network administrators can quickly locate abnormal traffic or potential problems in the network.

[0061] S520: Historical data statistical traceability and analysis primarily provides network operations and maintenance personnel or administrators with a powerful traffic monitoring and management tool through multi-dimensional statistical analysis of traffic data. It employs various flexible query and statistical conditions, allowing users to select single or multiple conditions for precise traffic querying and analysis. This multi-dimensional analytical capability enables operations and maintenance personnel and administrators to comprehensively grasp the distribution characteristics and usage trends of current network traffic from different perspectives (such as time range, user IP, application protocol, etc.). The generated statistical results can be presented intuitively in the form of charts or reports, providing data support for network planning, performance optimization, and security policy formulation. Historical log statistics not only improve the precision of network management but also provide a scientific basis for network operations and maintenance decisions.

[0062] Link Statistical Analysis: Users can select different links, application protocol types, and time dimensions (30 minutes, 1 hour, 24 hours, or custom time periods) as query conditions to issue database query tasks. The system quickly retrieves data from the database, first based on time, then based on the application protocol type and link, and finally aggregates statistical data and returns the output for display. It provides comprehensive statistics and analysis of traffic data on links, with links as the basic query unit, and displays the upstream and downstream traffic and bandwidth of different applications on the link in charts.

[0063] User (Source IP) Statistical Analysis: Users can configure specific source IP addresses or all user types, and use time-based strategies (30 minutes, 1 hour, 24 hours, or custom time periods) as query conditions to issue database query tasks. The system quickly retrieves all users from the database based on the time dimension and returns the query results, visually displaying each user's network access behavior details in tabular or chart format. This provides administrators with refined monitoring and analysis capabilities for user traffic behavior. Administrators can also view all application protocols accessed by a specific user and their detailed information.

[0064] Domain Statistical Analysis: Users can specify a time period as the query condition and issue a database query task. The system quickly retrieves all domains visited by users in the database according to the time dimension. Through query aggregation analysis, administrators can obtain information such as the number of times a user's domain has been accessed, the destination address, the time period of access, and the amount of traffic, providing administrators with a comprehensive insight into domain access behavior.

[0065] Latency Statistical Analysis: Latency statistics provide comprehensive support for network performance evaluation through the collection and analysis of multi-dimensional latency data. It mainly includes multiple dimensions such as session latency, user latency, application latency, latency quality, and domain name service latency, helping administrators accurately pinpoint network performance bottlenecks and optimize network service quality. Users can submit database query tasks using the time dimension as a query condition for each latency module, and the system quickly retrieves data from the database. Session latency analyzes the latency data between user five-tuple sessions, where client latency and server latency reflect the response time of the client and server. Based on the latency length, administrators can assess session quality and locate the root cause of network problems. User latency is based on the user's source IP address, analyzing the user's network quality, including the user's maximum latency, minimum latency, and average latency, accurately reflecting the fluctuation of user access network latency. Application latency is analyzed based on specific application protocol types, comprehensively reflecting the application service latency quality through the application's maximum latency, minimum latency, and average latency. Latency quality analyzes client latency and service latency, displaying the distribution of latency data in a trend distribution chart. Latency is divided into multiple intervals (1-50, 50-100, 100-150, 150-200, >200, failure, in milliseconds), and the latency percentage of each interval is calculated. A distribution chart visually indicates whether network performance is concentrated in the low-latency interval; lower latency indicates better network quality, and higher latency indicates poorer network quality. When latency statistics show failures, it indicates abnormal latency and severe packet loss. Domain name service latency is a statistic for accessing specific domain names, including average latency, maximum latency, and minimum latency. Domain name server response performance is reflected in the domain name latency statistics, helping administrators assess the network quality of domain name services. When high latency is identified, operations personnel and network administrators need to optimize access paths.

[0066] Report Output: After completing the above statistical analysis, users can output the statistical information in report form as needed. The system provides rich report templates and custom templates, allowing users to design the layout and style of the report page according to business needs, including headers, summaries, rows, columns, colors, etc., and select appropriate charts to display data, such as bar charts, pie charts, and line charts. Then, relevant content is extracted from historical statistical analysis information to populate the report template, completing the report generation. In addition, the report content provides data analysis explanations and network improvement suggestions to help users better understand and optimize the network. After generating the report, users can specify the report content to be exported and submitted to the system platform at a specified time (hourly / daily / weekly / monthly) in various report formats (HTML / EXCEL / PDF, etc.). The system also provides automated publishing and retention functions; users can send exported reports to designated recipients via email, and retained reports can be viewed and downloaded at any time through the system page.

[0067] By applying the above technical solutions, this invention discloses a network traffic data rapid processing and monitoring system. The system includes: a network traffic data acquisition module, used to send mirrored network data to a designated network port, bind the designated network port using DPDK, and acquire network traffic data through multi-threaded multi-queue processing; a session flow table module, used to acquire the five-tuple information of Ethernet packets, create a session flow table based on the five-tuple information, and store the network traffic data in the session flow table; and a traffic application identification module, used to acquire the data packets of the network traffic data, identify them based on a protocol identification engine, and complete data collection and cleaning by using DPDK multi-core thread binding, combined with a rule engine, to achieve high coverage and high accuracy in service identification and intelligent analysis of network application traffic.

[0068] In summary, this invention primarily utilizes network mirroring data to rapidly process and monitor network traffic data. The invention mainly comprises four parts: data acquisition and processing, application identification, data storage, and intelligent monitoring and analysis of network traffic data.

[0069] The main process is as follows: 1. DPDK uses multi-threading and multi-queues to collect data, decode and analyze it, and create hash session flow tables; 2. Data extraction utilizes a rule feature library to quickly identify network application types; 3. Format flow table cache data and store it in a database (MongoDB) according to its categories; 4. Intelligent analysis platform provides real-time traffic statistics, monitoring, source tracing, and reporting.

[0070] The key points of the processing steps are as follows: (1) Send the mirror network data to the specified network port, and use DPDK to bind the specified network port to obtain network data traffic through multiple threads and multiple queues.

[0071] (2) Disassemble and unpack the Ethernet message according to the TCP / IP protocol structure, and obtain the five-tuple data of the message (source address, destination address, source port, destination port, protocol type) by memory mapping.

[0072] (3) Perform hash operation on the obtained quintuple data, create a unique flow table session information, ensure that each session has one and only one data, and save it to the session flow table.

[0073] (4) First, identify the application based on the customized business, then identify the specific protocol application based on the known protocol port, then extract the application payload data for rule base multi-mode matching, identify the specific application and uplink and downlink traffic, and save the obtained information to the session list.

[0074] (5) Data storage: Data flow information, application information, upstream and downstream traffic information, and domain name information accessed by users are stored in the database (MongoDB) in a regular and fast manner according to the agreed format.

[0075] (6) The intelligent analysis platform performs real-time statistical analysis of current session information based on time, and displays network traffic analysis information in multiple dimensions such as source address, destination address, source port, destination port, application type, time, and TOPN of uplink and downlink traffic, and monitors the network operation status in real time.

[0076] (7) Based on historical stored data, network traffic can be traced from multiple dimensions such as user (IP address), application type, link information, session latency, etc., and corresponding reports can be exported through statistical record analysis.

[0077] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0078] Based on the above description, the present invention has the following advantages: 1. High-efficiency data acquisition: The system adopts advanced DPDK polling mode zero-copy data technology; uses big page memory binding to allow the program to exclusively occupy memory, preventing memory switching and improving memory data capture; application threads are bound to different CPU cores to make full use of CPU cores and reduce the performance consumption of repeated scheduling; and uses DPDK's unique RTE_buf data structure to store data, requiring only one memory allocation, reducing memory allocation overhead and improving the speed of data reception, storage, and data acquisition.

[0079] 2. Application identification capability: It has a rich feature library, adopts multi-mode engine protocol identification, supports multi-level nested identification and matching, and user-defined resource class identification. The system can quickly identify the application type of network traffic.

[0080] 3. Real-time analysis capability: It can automatically monitor the current distribution of information resources and retrieve and save relevant historical data as needed; it has full-domain comparison and recognition technology, which can quickly and accurately locate the subdivided resource types; it can display the utilization rate of resources in real time without manual intervention.

[0081] 4. Resource monitoring capability: It can monitor the status of network and business resources, and classify, establish, monitor and analyze different resources, and display them in real time.

[0082] 5. Multi-intelligent learning capabilities: It has the ability to identify, verify, and classify the usage status trends of information resources, and can continuously and automatically learn resource flow types and their business categories.

[0083] 6. Intelligent report statistics: Supports report statistics based on user, business, and resource analysis; can export and report specified report content to the management platform in various report formats (HTML / EXCEL / PDF, etc.) at specified times (hourly / daily / weekly / monthly), or send it to specified recipients via email.

[0084] Those skilled in the art will understand that the modules in the device can be distributed within the device of the implementation scenario as described, or they can be located in one or more devices different from this implementation scenario, with corresponding changes. The modules of the above-mentioned implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0085] The serial numbers of the present invention mentioned above are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenarios.

[0086] The above-disclosed examples are only a few specific implementation scenarios of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A monitoring system for rapid processing of network traffic data, characterized in that, The system includes: a network traffic data acquisition module, used to send mirrored network data to a designated network port, bind the designated network port through DPDK, and acquire network traffic data through multi-threading and multi-queueing; a session flow table module, used to acquire the five-tuple information of Ethernet packets, create a session flow table based on the five-tuple information, and store the network traffic data in the session flow table; and a traffic application identification module, used to acquire the data packets of the network traffic data and identify them based on a protocol identification engine.

2. The network traffic data rapid processing and monitoring system as described in claim 1, characterized in that, The network traffic data acquisition module specifically includes: a data acquisition initialization module, used to perform initialization operations on the environment abstraction layer of DPDK; and a data acquisition module, used to poll the network card flag bit of the specified network port through DPDK, and determine whether network traffic data exists based on the network card flag bit, and acquire the network traffic data.

3. The network traffic data rapid processing and monitoring system as described in claim 2, characterized in that, The acquisition initialization module is specifically used for: initializing and starting the kernel, which includes loading the user-mode polling driver PMD, setting the RSS key, configuring big page memory, and binding the DPDK network card to receive packets; obtaining big page memory, CPU logical cores, PCI device addresses, and network card queues through EAL; allocating an independent memory buffer pool for each CPU logical core according to the big page mechanism to accelerate memory allocation efficiency, and allocating a circular buffer for each DMA queue to store data messages; the application allocates a thread on each CPU logical core, associates and binds them, and allocates a memory buffer to receive network data packets, achieving a 1:1:1 ratio of threads, queues, CPU logical cores, and memory pools, while simultaneously enabling promiscuous mode for the corresponding network card and starting the thread to receive packets.

4. The network traffic data rapid processing and monitoring system as described in claim 1, characterized in that, The data acquisition module is specifically used to: actively poll the network card's flag bit through the DPDK query engine to determine if there is network traffic data; when network traffic data arrives, perform batch data reading to fill the buffer; when the data in the circular queue reaches the set threshold, the CPU performs a write operation to collect and process all the data in the circular queue; if the preset threshold is not reached, continue to wait; and enqueue the collected data into the circular queue of the parsing thread.

5. The network traffic data rapid processing and monitoring system as described in claim 1, characterized in that, The session flow table module is specifically used for: performing memory mapping according to the TCP / IP protocol structure and quickly extracting the five-tuple data of the packets; calculating hash values ​​according to the five-tuple information and creating a session flow table, through which the system maintains all flow data node information; initializing flow session node data, each flow session node data includes: a five-tuple, a timestamp, session direction, data packet statistics for each application type, the number of bytes sent uplink and downlink in the application data packet, as well as storage and probe protocol bitmap information and the target host; when a new network data packet is received from the network card, a hash value is first calculated, and then the hash session flow table is searched according to the hash index. If it does not exist, a new flow node is created and added to the session flow table; if it exists, the session traffic statistics are updated; when the flow session ends or the aging time expires, the aged flow node is cleared, all information saved by the flow node is released, and the flow node cache is reclaimed.

6. The network traffic data rapid processing and monitoring system as described in claim 5, characterized in that, The traffic application identification module is specifically used for: loading and initializing a feature rule base when the system starts. The feature rule base is in the format of protocol ID: protocol type: protocol rule, where the protocol ID is a value, the protocol type is TCP / UDP / ICMP for Ethernet packets, or application layer protocols such as HTTP and FTP, or the type of website or specific application software accessed by the user; the protocol rule is a regular expression or string pattern that defines the feature. The feature rule base maps the ID type and protocol type to a one-to-one correspondence according to a certain data structure and saves it in the cache; when the system starts, it initializes the hyperscan engine to compile block mode, stream mode, and vector mode, and compiles multiple expressions of protocol ID: protocol rule type according to the feature rule base and caches them in the engine database. Each pattern can be associated with a unique... Protocol IDs and protocol rules: For plaintext network payload data, the system parses the data to obtain valid payload feature data. Based on the business scenario, it selects the optimal hyperscan engine mode for matching. A single data input is processed in parallel with a full-rule unordered scan. The engine matches the specific location of the protocol rule in the payload data and the corresponding protocol rule. It also automatically identifies and returns the matched rule features and the corresponding protocol ID. The protocol ID is indexed using a pre-cached feature library to determine the protocol type and is saved to the session node. For encrypted network data, during the TLS handshake phase, the client sends an SNI extension in the ClientHello message to specify the domain name to be accessed. The obtained domain name information is then used by the hyperscan engine for pattern matching to identify the protocol type, and the protocol type is saved to the session node. Customized business identification is performed according to business needs. The destination IP and destination port of the customized business are added to the rule feature library. When network traffic packets enter the traffic application identification module, the destination IP and destination port are identified first for protocol identification.

7. The network traffic data rapid processing and monitoring system as described in claim 1, characterized in that, The system also includes a data storage module, which specifically includes: a stream session caching module, used to cache data in memory pre-allocated to the system according to a time period; and a database storage module, used to write stream sessions into the database according to a pre-defined format.

8. The network traffic data rapid processing and monitoring system as described in claim 7, characterized in that, The streaming session cache module is specifically used for: obtaining pre-allocating and initializing the streaming session cache of the streaming table in large page memory when the system starts, configuring the time interval for initializing the real-time statistics of the system, and writing the cached data into the streaming cache according to the time interval of the real-time statistics. The cached data is used to provide the system platform with real-time display of bandwidth traffic, host analysis and application analysis.

9. The network traffic data rapid processing and monitoring system as described in claim 7, characterized in that, The database storage module is specifically used for: completing Kafka-related configurations and initiating the connection between Kafka and the server when the system starts; reading the stream table session cache data according to the agreement when the stream session ends or times out; performing ordered byte reading and escaping of the stream table session data into JSON data; uploading it to the service platform through the Kafka message queue; and writing the stream session data into the MangoDB database in real time.

10. The network traffic data rapid processing and monitoring system as described in claim 1, characterized in that, The system also includes an intelligent traffic analysis and monitoring module, which is specifically used to: obtain real-time data traffic from the stream session cache through a socket connection according to an agreed data format; start a socket communication service in the stream session cache module and determine the data type of the real-time data traffic, including whether the data type is host analysis, application analysis, or filtering condition query; The corresponding data is returned based on the data type described; The real-time data traffic is subjected to multi-dimensional statistical analysis, and statistical reports are generated and output. The multi-dimensional statistical analysis specifically includes link statistical analysis, user statistical analysis, domain name statistical analysis, and latency statistical analysis.

Images