Real-time traffic identification method based on dpdk flow-domain name association

Abstract

The application relates to the technical field of network flow monitoring and data analysis, in particular to a real-time flow identification method based on FLOW-domain name association of DPDK, which comprises the following steps: obtaining full flow data packets through target network service flow; analyzing five-tuple information of each data packet in the user state full flow data packet, and configuring RSS symmetric hash rules of the acquisition network card; establishing a corresponding binding relationship among RSS hardware queues, user state packet processing threads and isolated physical cores; establishing a dedicated local flow table for each user state packet processing thread; performing progressive three-level domain name extraction operations to obtain domain name information corresponding to the Flow flow; accurately binding the Flow flow and the corresponding domain name information, effectively checking the binding relationship, and generating flow-domain name association flow statistical data after the checking, so that the application has the advantages of solving the performance bottleneck and high packet loss problem of kernel state packet capture, solving the defects of low domain name extraction coverage and easy mismatch of the traditional DPDK scheme.

Description

Technical Field

[0001] This application relates to the technical field of network traffic monitoring and data parsing, and in particular to a real-time traffic identification method based on DPDK-FLOW domain name association. Background Technology

[0002] With the rapid development of services such as video, live streaming, cloud computing, and real-time interaction, network bandwidth continues to upgrade, with 100G or even 400G bandwidth becoming the mainstream configuration for data centers and backbone networks. In high-bandwidth scenarios, accurate parsing of HTTP / HTTPS protocol traffic and statistical analysis of traffic volume by domain name are core requirements for network operation and maintenance, traffic scheduling, and security monitoring. Operators need to analyze the bandwidth usage of each domain name to optimize network resource allocation, while network security vendors need to detect malicious access behavior through abnormal domain traffic.

[0003] The current mainstream network traffic identification technologies are mainly divided into two categories. One category is kernel-based packet capture solutions, including cBPF+libpcap, tcpdump, eBPF / XDP, etc. These solutions have serious performance bottlenecks in high-bandwidth scenarios, cannot achieve line-speed processing of bandwidths above 100G, and have problems such as random packet loss and excessive CPU utilization, which cannot meet the requirements of real-time parsing.

[0004] Another category is high-performance user-space packet capture solutions, including commercial PF_RINGZC and open-source DPDK-related solutions. While PF_RINGZC can achieve line-rate processing, it has strict commercial licensing restrictions and extremely high deployment costs. General-purpose DPDK solutions are mostly developed based on the C language, which has problems such as memory safety risks, inter-core competition in multi-core scheduling, and insufficient flow table management capabilities. At the same time, at the flow-domain association level, they generally have defects such as low domain name extraction coverage and easy association mismatch. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this application provides a real-time traffic identification method based on DPDK-FLOW domain association.

[0006] This application provides a real-time traffic identification method based on DPDK and FLOW-domain association, which adopts the following technical solution:

[0007] A real-time traffic identification method based on DPDK for FLOW-domain association includes the following steps:

[0008] Acquire the service traffic of the target network, and generate a mirrored traffic copy by replicating the service traffic through port mirroring or network splitter; connect the mirrored traffic copy to the traffic acquisition and analysis device, set up a DPDK user-space runtime environment for the traffic acquisition and analysis device, and directly capture the mirrored traffic copy in user space through the DPDK user-space runtime environment to obtain the full traffic data packet;

[0009] Enable the hardware RSS multi-queue of the network card in the traffic acquisition and analysis device, parse the five-tuple information of each data packet in the full user-space traffic data packet, configure the RSS symmetric hash rule of the network card, and distribute the bidirectional data packets corresponding to the same five-tuple to the same RSS hardware queue based on the five-tuple information.

[0010] Obtain the number of isolated physical cores in the traffic acquisition and analysis device that is consistent with the number of RSS hardware queues, bind a unique user-space packet processing thread to each isolated physical core, and establish the corresponding binding relationship between the RSS hardware queues, user-space packet processing threads, and isolated physical cores.

[0011] A dedicated local flow table is established for each user-space packet processing thread. Based on the parsed 5-tuple information of each data packet received by the user-space packet processing thread, a corresponding flow is constructed for each unique 5-tuple information. An independent TCP state machine is maintained for each flow to track the full life cycle state of the flow, and the flow state information and corresponding data packet statistics are updated to the corresponding local flow table.

[0012] Based on the full lifecycle time-series characteristics of the Flow in the local flow table, a progressive three-level domain name extraction operation is performed on the full user-space traffic data packets corresponding to the Flow to obtain the domain name information corresponding to the Flow.

[0013] The flow is precisely bound to the corresponding domain name information, and the binding relationship is validated. Once the validation is successful, traffic statistics associated with the flow and domain name are generated.

[0014] In one embodiment: when setting up the DPDK user-space runtime environment, the physical NUMA node to which the network card in the traffic acquisition and analysis device belongs is identified and marked as the target NUMA node; the specified physical core in the target NUMA node is isolated from the system kernel scheduler to generate an isolated physical core;

[0015] Within the target NUMA node, a large page memory is exclusively allocated, and the large page memory is mapped to the address space of the DPDK user-space runtime environment to store the full traffic data packets and local flow tables;

[0016] Disable the swap partition and hyper-threading function of the traffic acquisition and analysis device, configure the hardware interrupt affinity of the acquisition network card, and bind it to the physical core of a non-target NUMA node.

[0017] In one embodiment: when the hardware RSS multi-queue of the acquisition network card is enabled, the total number of RSS hardware queues of the acquisition network card is set to be consistent with the number of isolated physical cores;

[0018] Configure the hash factor of the RSS symmetric hash algorithm to ensure that the hash values ​​of uplink and downlink bidirectional data packets corresponding to the same quintuple are consistent after being calculated by the hash algorithm, thus ensuring that all bidirectional data packets corresponding to the same flow are distributed to the same RSS hardware queue.

[0019] In one embodiment: when establishing the corresponding binding relationship, the running permission of each user-mode packet processing thread is permanently bound to the corresponding isolated physical core, and the system kernel is prohibited from scheduling other processes to occupy the isolated physical core; each user-mode packet processing thread reads the full traffic data packets in the corresponding bound RSS hardware queue only through polling mode, and does not perform cross-core data access and sharing throughout the process.

[0020] In one embodiment: when establishing a dedicated local flow table for each user-space packet processing thread, it is only visible to the user-space packet processing thread to which it belongs and has no cross-core access permissions; the local flow table adopts a cuckoo hash table structure, and each hash table entry corresponds to a unique flow; each hash table entry is set with a dedicated storage field to store the five-tuple information of the corresponding flow, TCP state machine information, full lifecycle timing characteristic information, domain name association information, and cumulative traffic statistics information.

[0021] In one embodiment: when tracking the entire lifecycle state of a flow through a TCP state machine, a dual-threshold adaptive flow recycling mechanism is set for the local flow table, specifically including:

[0022] Based on the TCP state machine, the flow is divided into active flow flows that are in the data transmission phase and inactive flow flows that have been closed or have not transmitted data packets for a long time.

[0023] Set corresponding timeout thresholds for active and inactive flow streams respectively; monitor the memory usage of the local flow table in real time, and adaptively adjust the timeout threshold of inactive flow streams according to the changes in memory usage to realize the reclamation and memory release of inactive flow streams.

[0024] In one embodiment, the steps of the progressive third-level domain name extraction operation specifically include:

[0025] Plaintext domain name extraction is performed in the first data packet window after the TCP connection of the Flow is established. By performing application layer parsing on the full traffic data packets corresponding to the Flow, plaintext domain name is extracted from the Host header field for HTTP protocol and plaintext domain name is extracted from the SNI extension field of the TLSClientHello message for HTTPS protocol. If a valid domain name with a compliant format is extracted, the subsequent extraction operation is terminated.

[0026] After plaintext domain name extraction fails and the Flow enters the data transmission stage, the first data packet window performs IP-domain dynamic mapping matching. Based on the target IP address of the full traffic data packet corresponding to the Flow, the real-time maintained IP-domain dynamic mapping table is queried to match the valid domain name records within the preset time window before and after the Flow creation time. If a unique valid domain name is matched, the subsequent extraction operation is terminated.

[0027] After the IP-domain dynamic mapping match fails and the Flow lifecycle ends, DNS full-link backtracking verification is performed. Based on the Flow's five-tuple information, creation time and end time, a backtracking time window and matching rules are constructed. Based on the backtracking time window and matching rules, the DNS request-response full-link log within the same time period is backtracked to complete the fallback extraction of domain name information.

[0028] In one embodiment: when tracing back the entire DNS request-response log, a two-level exact match is performed, specifically including:

[0029] Candidate DNS session records are selected within the backtracking time window where the source IP of the DNS request message is exactly the same as the source IP of the Flow, and the IP address set returned by the DNS response message contains the target IP of the Flow.

[0030] From the candidate DNS session records, select the unique DNS session record with the shortest interval between the DNS response timestamp and the creation time of the Flow, and extract the requested domain name in the DNS session record as the fallback domain name corresponding to the Flow.

[0031] In one embodiment: when performing precise binding of Flow and domain name information, for Flow with TCP long connection reuse, independent sub-flow units are divided within the Flow; each sub-flow unit corresponds to an independent HTTP / HTTPS request-response session within the Flow, and is uniquely identified by the five-tuple information of the Flow, the TCP sequence number range, and the request time window; each sub-flow unit independently maintains corresponding traffic statistics information and domain name association information, and performs precise binding and validity verification between the sub-flow unit and the corresponding domain name information.

[0032] In one embodiment: when validating the binding relationship, for HTTP protocol traffic, the compliance of the bound domain name is verified by the feature fields of the response message; for HTTPS protocol traffic, the compliance of the bound domain name is verified by the legitimate domain name field of the server certificate; if the verification fails, the domain name extraction and binding operation is retried; if the verification passes, the traffic statistics of the corresponding Flow are locked.

[0033] In summary, this application has the following beneficial effects:

[0034] 1. Based on DPDK, a pure user-space processing environment is built, bypassing the kernel protocol stack, to achieve line-speed packet loss-free processing with bandwidth of 100G and above, solving the performance bottleneck and high packet loss problem of kernel-space packet capture.

[0035] 2. The entire process uses open-source components, with no commercial license restrictions, which greatly reduces deployment costs and avoids the pain points of commercial costs associated with PF_RINGZC;

[0036] 3. A lock-free process is achieved through a multi-core one-to-one binding architecture, which solves the problem of inter-core contention, and the adaptive stream recycling mechanism avoids stream table memory overflow;

[0037] 4. Design a progressive three-level domain name extraction system to increase the domain name extraction coverage to over 99%, achieve precise Flow-domain association, and solve the shortcomings of low domain name extraction coverage and easy mismatch in the traditional DPDK solution. Attached Figure Description

[0038] Figure 1 This is a flowchart of the real-time traffic identification method based on DPDK-domain association in this embodiment. Detailed Implementation

[0039] The present application will be further described in detail below with reference to the accompanying drawings.

[0040] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. In some cases, to avoid obscuring various aspects of this application due to unnecessary description, well-known methods, processes, systems, components, and / or circuits already described at a higher level will not be elaborated upon. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but conforms to the broadest scope consistent with the scope of protection claimed in this application.

[0041] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0042] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0043] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples.

[0044] A real-time traffic identification method based on DPDK for FLOW-domain association, such as Figure 1 As shown, it includes the following steps:

[0045] S100: Obtain the service traffic of the target network, and generate a mirrored traffic copy by replicating the service traffic through port mirroring or network splitter; connect the mirrored traffic copy to the traffic acquisition and analysis device, set up the DPDK user-space runtime environment for the traffic acquisition and analysis device, and directly capture the mirrored traffic copy in user space through the DPDK user-space runtime environment to obtain the full traffic data packet.

[0046] It should be noted that in the above steps, when setting up the DPDK user-space runtime environment, the physical NUMA node to which the network card in the traffic acquisition and analysis device belongs is identified and marked as the target NUMA node; then the specified physical core within the target NUMA node is isolated from the system kernel scheduler to generate an isolated physical core.

[0047] Furthermore, large page memory is exclusively allocated within the target NUMA node and mapped to the address space of the DPDK user-space runtime environment for storing full traffic data packets and local flow tables;

[0048] Simultaneously, the swap partition and hyper-threading functions of the traffic acquisition and analysis device are disabled, and the hardware interrupt affinity of the acquisition network card is configured and bound to the physical core of a non-target NUMA node. Through interrupt affinity configuration, all hardware interrupt signals generated by the acquisition network card are forcibly bound to the physical core of the NUMA node that does not handle traffic packet processing and is not to which the acquisition network card belongs. Only these non-service cores are allowed to handle the network card's hardware interrupts, while the physical cores within the target NUMA node responsible for line-rate traffic processing do not respond to or handle any hardware interrupts from the acquisition network card. This completely eliminates the interference of network card hardware interrupts on the traffic packet processing core, ensuring line-rate packet loss-free processing, low latency jitter, and operational stability in 100G / 400G high-bandwidth scenarios.

[0049] With the above configuration, the core hardware of the traffic acquisition and analysis device includes a network card for acquisition, a NUMA architecture computing module, a large page memory module, a network access module, and a storage module. Each hardware module is specifically designed for high-bandwidth traffic processing. The device is equipped with a Linux-like operating system with deep kernel-level optimization and configuration, and a dedicated DPDK user-space high-performance operating environment is built on this system. The hardware and software work together to achieve line-speed capture and processing of high-bandwidth traffic.

[0050] The acquisition network card supports 100G / 400G high bandwidth, hardware RSS multi-queue function and UIO / VFIO driver, and can be configured with RSS symmetric hash rules; the NUMA architecture computing module is equipped with multiple physical cores, supports core isolation and disables hyper-threading function, and can allocate dedicated isolated physical cores for traffic processing; the large page memory module is a high-capacity memory hardware that supports exclusive allocation within the target NUMA node, used to store user-space full traffic data packets and local flow tables; the network access module adapts to the physical network ports of port mirroring / network splitters, and supports policy-based replication of mirrored traffic copies of specified protocols such as HTTP / HTTPS; the storage module includes large page memory for temporary storage and high-performance solid-state / mechanical hard drives for persistent data storage.

[0051] This step establishes a dedicated user-space runtime foundation for the entire DPDK traffic identification solution from three dimensions: hardware resource isolation, memory management, and system interference shielding. This foundation provides end-to-end determinism, eliminates kernel interference, and eliminates processing jitter. It fundamentally addresses the core pain points of existing technologies, such as cross-NUMA access latency, kernel resource preemption, memory copy overhead, and interrupt jitter, which lead to packet loss under high bandwidth, insufficient processing performance, and operational instability. Simultaneously, it provides indispensable hardware and system-level pre-support for the subsequent implementation of multi-core lock-free scheduling, lock-free access to local flow tables, and line-rate traffic processing. This constructs a dedicated runtime space completely isolated from the system kernel, free from cross-node performance loss, and free from any random interference.

[0052] S200: Enable the hardware RSS multi-queue of the network card in the traffic acquisition and analysis device, parse the five-tuple information of each data packet in the full user-space traffic data packet, configure the RSS symmetric hash rule of the network card, and distribute the bidirectional data packets corresponding to the same five-tuple to the same RSS hardware queue based on the five-tuple information.

[0053] It should be noted that in the above steps, when enabling the hardware RSS multi-queue of the acquisition network card, the total number of RSS hardware queues of the acquisition network card should be set to be consistent with the number of isolated physical cores; configure the hash factor of the RSS symmetric hash algorithm so that the hash values ​​of the uplink and downlink bidirectional data packets corresponding to the same 5-tuple are consistent after being calculated by the hash algorithm, and ensure that all bidirectional data packets corresponding to the same flow are distributed to the same RSS hardware queue.

[0054] Through the above settings, the cross-core processing requirement of the same Flow stream is completely eliminated from the hardware distribution level, providing underlying hardware support for the subsequent multi-core lock-free scheduling architecture with one-to-one binding of thread-core-RSS queue. This fundamentally solves the core technical pain points in existing technologies, such as global flow table lock contention, inter-core resource contention leading to high bandwidth processing latency jitter, and inability to process at line speed with zero packet loss. Now, the entire process is lock-free and parallelized, solving the inter-core contention problem.

[0055] S300: Obtain the number of isolated physical cores in the traffic acquisition and analysis device that are consistent with the number of RSS hardware queues, bind a unique user-mode packet processing thread to each isolated physical core, and establish a corresponding binding relationship between the RSS hardware queue, the user-mode packet processing thread, and the isolated physical core.

[0056] It should be noted that in the above steps, when establishing the corresponding binding relationship, the running permission of each user-mode packet processing thread is permanently bound to the corresponding isolated physical core, prohibiting the system kernel from scheduling other processes to occupy the isolated physical core; each user-mode packet processing thread only reads the full traffic data packets in the corresponding bound RSS hardware queue through polling mode, and does not perform cross-core data access and sharing throughout the process.

[0057] In this step, the user-space packet processing thread reads data packets from the corresponding bound RSS hardware queue only through polling mode, directly addressing the core performance defects of existing kernel-space packet capture schemes and general DPDK schemes.

[0058] On the one hand, it replaces the traditional interrupt-driven mode, eliminating the performance crashes caused by interrupt storms. Existing kernel-based packet capture solutions all use the interrupt-driven mode, where the network card triggers a hardware interrupt every time it receives a data packet, requiring the CPU to frequently switch contexts to handle the interrupt. In 100G high-bandwidth, small-packet scenarios, tens of millions of data packets per second can trigger an interrupt storm, causing CPU utilization to soar to 100% and processing performance to drop drastically, making line-speed processing completely impossible. This design uses a polling mode, allowing the packet processing thread to continuously and actively read data packets from the network card queue, with no hardware interrupts triggered throughout the process. The processing latency is stable at the microsecond level, fully adapting to the line-speed processing requirements in high-packet-rate scenarios.

[0059] On the other hand, strictly limiting the processing boundaries of threads lays the foundation for lock-free processing. This application restricts each thread to reading only its own bound RSS queue, ensuring that all bidirectional data packets of the same flow are processed only by the same thread on the same core. This avoids conflicts in multi-threaded parallel processing from the underlying rules and provides the prerequisite for subsequent local lock-free flow table management.

[0060] Thirdly, by combining the DPDK user-space environment, zero-copy reading is achieved to maximize processing performance. The polling mode directly reads the network card queue data mapped to user-space memory via DPDK, without kernel-space intervention or memory copying from kernel to user space, achieving zero-loss reading of data packets and further ensuring line-speed processing capabilities at 100G bandwidth.

[0061] S400: Establish a dedicated local flow table for each user-space packet processing thread; construct a corresponding flow for each unique 5-tuple based on the parsed 5-tuple information of each data packet received by the user-space packet processing thread; maintain an independent TCP state machine for each flow to track the full lifecycle state of the flow; and update the flow state information and corresponding data packet statistics information to the corresponding local flow table.

[0062] It should be noted that in the above steps, the dedicated local flow table refers to a flow data storage and query structure that is separately established for each user-space packet processing thread bound to an isolated physical core and is only visible to that thread itself. It is the core data carrier for realizing lock-free parallel processing throughout the entire process and accurately tracking the entire life cycle of the TCP flow.

[0063] The 5-tuple information includes the source IP, destination IP, source port, destination port, and transport layer protocol number. As a unique primary key, a compliant and unique 5-tuple corresponds to only one Flow table entry, ensuring that it is completely bound to a single Flow, with no duplication or conflict.

[0064] In one embodiment, when a dedicated local flow table is established for each user-space packet processing thread, it is only visible to the user-space packet processing thread to which it belongs and has no cross-core access permissions. The local flow table adopts a cuckoo hash table structure, with each hash table entry corresponding to a unique flow. Each hash table entry is set with a dedicated storage field to store the five-tuple information of the corresponding flow, TCP state machine information, full lifecycle timing characteristic information, domain name association information, and cumulative traffic statistics information.

[0065] In another embodiment, when tracking the entire lifecycle state of a flow using a TCP state machine, a dual-threshold adaptive flow recycling mechanism is set for the local flow table to address the issues of local flow table memory overflow and system resource exhaustion in high-concurrency flow scenarios. This flow recycling mechanism specifically includes:

[0066] Based on the TCP state machine, the flow is divided into active flow flows that are in the data transmission phase and inactive flow flows that have been closed or have not transmitted data packets for a long time.

[0067] Set corresponding timeout thresholds for active and inactive flow streams respectively; monitor the memory usage of the local flow table in real time, and adaptively adjust the timeout threshold of inactive flow streams according to the changes in memory usage to realize the reclamation and memory release of inactive flow streams.

[0068] For example, an active timeout threshold is set for active Flows, with a value ranging from 60s to 600s, and a preferred value of 300s; a silent timeout threshold is set for inactive Flows, with a value ranging from 5s to 30s, and a preferred value of 15s.

[0069] The system monitors the memory usage of the local flow table in real time. When the memory usage exceeds the first preset threshold, it adaptively shortens the silent timeout threshold to accelerate the recycling and memory release of inactive flow. The first preset threshold ranges from 60% to 80%, with 70% being the preferred value.

[0070] When the memory usage rate is lower than the second preset threshold, the silent timeout threshold is restored. The value of the second preset threshold ranges from 20% to 40%, with a preferred value of 30%.

[0071] S500, based on the full lifecycle time sequence characteristics of the Flow in the local flow table, performs a progressive three-level domain name extraction operation on the full user-space traffic data packets corresponding to the Flow to obtain the domain name information corresponding to the Flow.

[0072] It should be noted that the specific steps in the progressive third-level domain name extraction operation mentioned above include:

[0073] First-level domain name extraction: Plaintext domain name extraction is performed in the first data packet window after the TCP connection of the Flow is established. This is done by performing application layer parsing on the full traffic data packets corresponding to the Flow. For HTTP protocols, plaintext domain names are extracted from the Host header field. For HTTPS protocols, plaintext domain names are extracted from the SNI extension field of the TLSClientHello message. If a valid domain name with a compliant format is extracted, the subsequent extraction operation is terminated.

[0074] TCP is a connection-oriented transport layer protocol. Establishing a TCP connection means that before two devices, such as a client and a server, can communicate through TCP, they must first complete a three-way handshake process to establish a connection.

[0075] Specifically, the core message exchanges in the three-way handshake are as follows: the client sends a SYN message to request the establishment of a connection, along with an initial sequence number; the server replies with a SYN+ACK message to confirm receipt of the request, and also sends its own initial sequence number; the client sends an ACK message to confirm receipt of the server's response, and the connection is officially established.

[0076] The first data packet refers to the first packet carrying business data sent by the client after the three-way handshake is completed, such as an HTTP GET / POST request or an HTTPS ClientHello message. It serves as the true starting point of the business request. The first data packet window starts at the time the TCP connection is established and uses a preset time threshold as the window duration.

[0077] In addition, to address the issue of incomplete packets caused by IP fragmentation and TCP out-of-order delivery, a simple packet reassembly cache can be designed specifically for the handshake phase. This cache only contains fragmented packets from the handshake phase of the flow, and parsing is performed only after the packets are reassembled. The lifetime of the reassembly cache does not exceed 5 seconds, and it is automatically cleaned up after the timeout to avoid memory consumption.

[0078] Level 2: After plaintext domain name extraction fails and the Flow enters the data transmission stage, the first data packet window performs IP-domain dynamic mapping matching. Based on the target IP address of the full traffic data packet corresponding to the Flow, the real-time maintained IP-domain dynamic mapping table is queried to match valid domain name records within the preset time window before and after the Flow creation time. If a unique valid domain name is matched, the subsequent extraction operation is terminated.

[0079] The IP-domain dynamic mapping table is maintained in real time by a dedicated DNS resolution thread created based on the DPDK user-space runtime environment. This dedicated DNS resolution thread is bound to an independent isolated physical core. The dedicated DNS resolution thread resolves the DNS response messages in the full traffic data packets in real time, extracts the domain name, resolved IP address, TTL time to live, and resolution timestamp from the messages, and constructs the IP-domain dynamic mapping table.

[0080] Meanwhile, the constructed IP-domain dynamic mapping entries are only retained within the TTL validity period and are automatically cleaned up after expiration. This ensures the timeliness of the mapping relationship and solves the problems of poor timeliness of IP-domain mapping and matching errors in single IP multi-domain scenarios.

[0081] In the above steps, the first data packet window of the Flow stream entering the data transmission phase refers to the processing period corresponding to the first valid data packet received after the Flow stream completes the TCP three-way handshake and the TCP state machine enters the ESTABLISHED state for data transmission.

[0082] During the matching process, based on the target IP address in the Flow table entry, the corresponding domain name record within M seconds before and after the Flow creation time and within the TTL validity period is searched in the IP-domain dynamic mapping table. If a unique domain name record is matched, the domain name is written to the main domain name field of the Flow table entry; if multiple candidate domain name records are matched, all candidate domain names are written to the candidate domain name list of the flow table entry, triggering third-level domain name extraction; if no matching record is found, third-level domain name extraction is triggered directly.

[0083] It should be noted that a Flow table entry is the smallest storage and management unit of a local flow table. It is a dedicated data structure that is bound one-to-one with a single unique Flow, used to track the Flow throughout its entire lifecycle, and to fully store its full features and business information. It is the core data carrier for the entire traffic identification, domain name association, and traffic statistics process.

[0084] Third-level domain name extraction: After the IP-domain dynamic mapping match fails and the Flow lifecycle ends, DNS full-link backtracking verification is performed. Based on the Flow's five-tuple information, creation time and end time, a backtracking time window and matching rules are constructed. Based on the backtracking time window and matching rules, the DNS request-response full-link log within the same time period is backtracked to complete the fallback extraction of domain name information.

[0085] Among them, the DNS resolution thread, which is independently bound to a dedicated isolated physical core, captures and resolves DNS response messages in the full traffic data packets in real time and maintains the IP-domain dynamic mapping table. At the same time, it also captures DNS request messages corresponding to the same session and combines the DNS request messages and DNS response messages that have completed bidirectional pair matching into a complete DNS request-response session record.

[0086] Then, an independent hash table entry is built for each DNS request-response session record, and a dedicated storage field is assigned to store the client source IP, DNS request source port, requested domain name, set of IP addresses returned by the resolution, request timestamp, response timestamp, and TTL lifetime corresponding to the session. A full-link log of DNS request-response is generated and persistently stored to form a traceable full DNS session database.

[0087] When tracing back the entire DNS request-response log, two levels of exact matching are performed, specifically including:

[0088] Initial screening: Select candidate DNS session records within the backtracking time window where the source IP of the DNS request message is completely consistent with the source IP of the Flow, and the IP address set returned by the DNS response message contains the target IP of the Flow.

[0089] Fine screening: Select the only DNS session record with the shortest interval between the DNS response timestamp and the creation time of the Flow from the candidate DNS session records, and extract the requested domain name in the DNS session record as the fallback domain name corresponding to the Flow.

[0090] It should be noted that the source and destination IPs of the Flow are obtained based on five-tuple information. When setting the backtracking time window, a preset duration is used to trace back from the Flow creation time as the window start time, and the Flow end time is used as the window end time. The preset duration ranges from 1s to 30s, with a preferred value of 10s. This ensures that the normal communication sequence of initiating a TCP connection after the client's DNS resolution is completed is covered, while excluding interference from invalid historical resolution records. This eliminates interference from multiple domains with the same IP address, identifies a unique corresponding domain, and completes the fallback extraction.

[0091] Based on this, if no valid DNS session record matching the matching rules is found after backtracking, the flow is marked as an unknown domain flow, and its five-tuple information, full lifecycle time sequence and traffic statistics are recorded simultaneously, and then it enters the subsequent manual verification process.

[0092] S600: Precisely bind the Flow to the corresponding domain name information, verify the validity of the binding relationship, and generate traffic statistics data associated with the Flow and domain name after the verification is passed.

[0093] It should be noted that when performing precise binding of Flow and domain information in the above steps, for Flows with TCP long connection reuse, independent sub-flow units are divided within the Flow. Each sub-flow unit corresponds to an independent HTTP / HTTPS request-response session within the Flow, and is uniquely identified by the five-tuple information of the Flow, the TCP sequence number range, and the request time window. Each sub-flow unit independently maintains corresponding traffic statistics and domain association information, and performs precise binding and validity verification between the sub-flow unit and the corresponding domain information.

[0094] The aforementioned request time window and backtracking time window are equivalent, and both are obtained in the same way. The TCP sequence number range is obtained based on the Flow TCP state machine and is an associated numerical range consisting of all TCP sequence numbers generated in the corresponding bidirectional TCP session throughout the entire lifecycle of a single Flow.

[0095] In addition, when validating the binding relationship, for HTTP protocol traffic, the compliance of the bound domain name is verified by the characteristic fields of the response message; for HTTPS protocol traffic, the compliance of the bound domain name is verified by the legal domain name field of the server certificate. If the verification fails, the domain name extraction and binding operation is retried; if the verification passes, the traffic statistics of the corresponding Flow are locked.

[0096] The above settings essentially solve the problem of mismatched Flow-domain association in scenarios where TCP long connections reuse multiple domains, significantly improving the accuracy of domain name binding; enabling fine-grained traffic statistics at the request level of a single domain within a long connection, avoiding confusion of traffic data from multiple domains, and meeting the requirements for fine-grained traffic management; and without disrupting the original Flow's full lifecycle management logic, only fine-grained splitting is performed in the binding verification stage, without affecting the main line speed processing performance, and adapting to 100G high bandwidth scenarios.

[0097] The embodiments described in this specific implementation are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A real-time traffic identification method based on DPDK-FLOW domain association, characterized in that, Includes the following steps: Acquire the service traffic of the target network, and generate a mirrored traffic copy by replicating the service traffic through port mirroring or network splitter; Connect the mirrored traffic copy to the traffic acquisition and analysis device, set up the DPDK user-space runtime environment for the traffic acquisition and analysis device, and directly capture the mirrored traffic copy in user space through the DPDK user-space runtime environment to obtain the full traffic data packet. Enable the hardware RSS multi-queue of the network card in the traffic acquisition and analysis device, parse the five-tuple information of each data packet in the full user-space traffic data packet, configure the RSS symmetric hash rule of the network card, and distribute the bidirectional data packets corresponding to the same five-tuple to the same RSS hardware queue based on the five-tuple information. Obtain the number of isolated physical cores in the traffic acquisition and analysis device that is consistent with the number of RSS hardware queues, bind a unique user-space packet processing thread to each isolated physical core, and establish the corresponding binding relationship between the RSS hardware queues, user-space packet processing threads, and isolated physical cores. A dedicated local flow table is established for each user-space packet processing thread. Based on the parsed 5-tuple information of each data packet received by the user-space packet processing thread, a corresponding flow is constructed for each unique 5-tuple information. An independent TCP state machine is maintained for each flow to track the full life cycle state of the flow, and the flow state information and corresponding data packet statistics are updated to the corresponding local flow table. Based on the full lifecycle time-series characteristics of the Flow in the local flow table, a progressive three-level domain name extraction operation is performed on the full user-space traffic data packets corresponding to the Flow to obtain the domain name information corresponding to the Flow. The flow is precisely bound to the corresponding domain name information, and the binding relationship is validated. Once the validation is successful, traffic statistics associated with the flow and domain name are generated. The specific steps of the progressive third-level domain name extraction operation include: Plaintext domain name extraction is performed in the first data packet window after the TCP connection of the Flow is established. By performing application layer parsing on the full traffic data packets corresponding to the Flow, plaintext domain name is extracted from the Host header field for HTTP protocol and plaintext domain name is extracted from the SNI extension field of the TLSClientHello message for HTTPS protocol. If a valid domain name with a compliant format is extracted, the subsequent extraction operation is terminated. After plaintext domain name extraction fails and the Flow enters the data transmission stage, the first data packet window performs IP-domain dynamic mapping matching. Based on the target IP address of the full traffic data packet corresponding to the Flow, the real-time maintained IP-domain dynamic mapping table is queried to match the valid domain name records within the preset time window before and after the Flow creation time. If a unique valid domain name is matched, the subsequent extraction operation is terminated. After the IP-domain dynamic mapping match fails and the Flow lifecycle ends, DNS full-link backtracking verification is performed. Based on the Flow's five-tuple information, creation time and end time, a backtracking time window and matching rules are constructed. Based on the backtracking time window and matching rules, the DNS request-response full-link log within the same time period is backtracked to complete the fallback extraction of domain name information.

2. The real-time traffic identification method based on DPDK-domain association according to claim 1, characterized in that: When setting up the DPDK user-space runtime environment, identify the physical NUMA node to which the network card in the traffic acquisition and analysis device belongs, and mark it as the target NUMA node; isolate the specified physical core in the target NUMA node from the system kernel scheduler, and generate an isolated physical core; Within the target NUMA node, a large page memory is exclusively allocated, and the large page memory is mapped to the address space of the DPDK user-space runtime environment to store the full traffic data packets and local flow tables; Disable the swap partition and hyper-threading function of the traffic acquisition and analysis device, configure the hardware interrupt affinity of the acquisition network card, and bind it to the physical core of a non-target NUMA node.

3. The real-time traffic identification method based on DPDK-domain association according to claim 1, characterized in that: When enabling hardware RSS multi-queues on the data acquisition network card, set the total number of RSS hardware queues on the data acquisition network card to be consistent with the number of isolated physical cores; Configure the hash factor of the RSS symmetric hash algorithm to ensure that the hash values ​​of uplink and downlink bidirectional data packets corresponding to the same quintuple are consistent after being calculated by the hash algorithm, thus ensuring that all bidirectional data packets corresponding to the same flow are distributed to the same RSS hardware queue.

4. The real-time traffic identification method based on DPDK-domain association according to claim 1, characterized in that: establishing... When the corresponding binding relationship is established, the running permission of each user-mode packet processing thread is permanently bound to the corresponding isolated physical core, prohibiting the system kernel from scheduling other processes to occupy the isolated physical core; each user-mode packet processing thread reads the full traffic data packets in the corresponding bound RSS hardware queue only through polling mode, and does not perform cross-core data access and sharing throughout the process.

5. The real-time traffic identification method based on DPDK-domain association according to claim 1, characterized in that: When a dedicated local flow table is created for each user-mode packet processing thread, it is only visible to the user-mode packet processing thread to which it belongs, and there is no cross-core access permission. The local flow table adopts a cuckoo hash table structure, with each hash table entry corresponding to a unique flow. Each hash table entry has a dedicated storage field that stores the five-tuple information of the corresponding flow, TCP state machine information, full lifecycle time sequence characteristics, domain name association information, and cumulative traffic statistics.

6. The real-time traffic identification method based on DPDK-domain association according to claim 5, characterized in that: When tracking the entire lifecycle state of a flow using a TCP state machine, a dual-threshold adaptive flow recycling mechanism is set for the local flow table, specifically including: Based on the TCP state machine, the flow is divided into active flow flows that are in the data transmission phase and inactive flow flows that have been closed or have not transmitted data packets for a long time. Set corresponding timeout thresholds for active and inactive flow streams respectively; monitor the memory usage of the local flow table in real time, and adaptively adjust the timeout threshold of inactive flow streams according to the changes in memory usage to realize the reclamation and memory release of inactive flow streams.

7. The real-time traffic identification method based on DPDK-domain association according to claim 1, characterized in that: When tracing back the entire DNS request-response log, two levels of exact matching are performed, specifically including: Candidate DNS session records are selected within the backtracking time window where the source IP of the DNS request message is exactly the same as the source IP of the Flow, and the IP address set returned by the DNS response message contains the target IP of the Flow. From the candidate DNS session records, select the unique DNS session record with the shortest interval between the DNS response timestamp and the creation time of the Flow, and extract the requested domain name in the DNS session record as the fallback domain name corresponding to the Flow.

8. The real-time traffic identification method based on DPDK-domain association according to claim 7, characterized in that: When performing precise binding of flow and domain name information, for flow with TCP long connection reuse, independent sub-flow units are divided within the flow. Each sub-flow unit corresponds to an independent HTTP / HTTPS request-response session within the flow, and is uniquely identified by the five-tuple information of the flow to which it belongs, the TCP sequence number range, and the request time window. Each sub-flow unit independently maintains corresponding traffic statistics and domain name association information, and performs precise binding and validity verification between the sub-flow unit and the corresponding domain name information.

9. The real-time traffic identification method based on DPDK-domain association according to claim 8, characterized in that: When validating the binding relationship, for HTTP protocol traffic, the compliance of the bound domain name is verified by the characteristic fields of the response message; for HTTPS protocol traffic, the compliance of the bound domain name is verified by the legal domain name field of the server certificate. If the verification fails, the domain name extraction and binding operation is retried; if the verification passes, the traffic statistics of the corresponding Flow are locked.

Images