A high-performance flow reorganization detection forensics method, system, device and medium based on VPP

Abstract

The application discloses a high-performance flow reorganization detection forensics method and system based on VPP, equipment and medium, mainly relates to the flow reorganization detection forensics technical field, in order to solve the problem that the existing scheme has the problem of poor multi-core expansibility caused by global hash table lock competition, frequent memory fragmentation caused by dynamic memory allocation and large performance overhead of threat flow forensics function. Including: maintaining the message buffer chain table of each TCP session involved in the CPU core to temporarily store the message that fails to reorganize; after the TCP flow data continuously reorganizes the message, the reorganized message is combined into a data segment; through the threat feature knowledge base and the threat matching engine, it is determined whether the data segment exists the preset threat; when the preset threat exists, the forensics message buffer chain table is taken out and stored into a PCAP file; when no threat is detected after a preset time period, the messages in the forensics message buffer chain table are released, and the latest message in the message structure space of the preset time period is copied to the forensics message buffer chain table.

Description

Technical Field

[0001] This application relates to the field of traffic reassembly detection technology, and in particular to a high-performance traffic reassembly detection and evidence collection method, system, equipment and medium based on VPP. Background Technology

[0002] As cyberattack methods become increasingly complex, traditional threat detection equipment often faces bottlenecks such as insufficient detection performance, delayed forensic processes, and low session management efficiency, making it difficult to meet the needs of real-time processing of massive traffic and fine-grained threat analysis.

[0003] Existing solutions mostly adopt kernel-mode netfilter framework or DPDK user-mode collection + Snort architecture, which have performance bottlenecks. Especially in high-speed network environments, when faced with complex session matching, TCP reassembly and threat alarm file generation, they are prone to packet loss, misjudgment or inability to trace the source, which seriously affects the timeliness and completeness of network security incident response.

[0004] Traditional TCP reassembly schemes (such as nf_conntrack in the Linux kernel) have two major bottlenecks:

[0005] (1) Global hash table lock contention leads to poor multi-core scalability;

[0006] (2) Frequent dynamic memory allocation leads to memory fragmentation;

[0007] In addition, existing solutions in the industry for threat traffic forensics typically provide packet tracing functionality through a separate tracing component, which incurs significant performance overhead. Summary of the Invention

[0008] This application provides a high-performance traffic reassembly detection and forensics method, system, device and medium based on VPP to solve the problems of poor multi-core scalability caused by global hash table lock contention, memory fragmentation caused by frequent dynamic memory allocation, and high performance overhead of threat traffic forensics function in existing solutions.

[0009] Firstly, this application provides a high-performance traffic reassembly detection and forensics method based on VPP, the method comprising:

[0010] Add a session management handoff node, a packet forensics cache node, a TCP reassembly node, and a threat detection and alarm node under the ip4-sv-reassembly-feature node in the VPP framework;

[0011] The session management handoff node creates a local session table for each CPU core, a global session table for all CPU cores, and a global handoff hash table.

[0012] The CPU core checks if a message exists in the local handoff buffer queue. If the message exists, it uses the hash value corresponding to the message to determine if it exists in the global handoff hash table. If it does not exist in the global handoff hash table, it adds the hash value of the message to the global handoff hash table, inserts the message structure into the local session table, and updates the global session table. If the message exists in the global handoff hash table, it processes the message.

[0013] The CPU core retrieves the packet from the dpdk-input node, determines whether the packet's structure exists in the global session table, and adds the packet's structure to the local session table and updates the global session table if it does not exist in the global session table. If it exists in the global session table but not in the local session table, the packet is added to the handoff buffer queue of the CPU core corresponding to the packet in the global session table. If it exists in the local session table, the packet is processed.

[0014] By using the message evidence caching node, message structure space is allocated, messages from the CPU core are allocated to the message structure space, and messages within a preset time period are added to the evidence message buffer list.

[0015] Through the TCP reassembly node, a cache list of packets that have failed to be reassembled is maintained for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments.

[0016] The threat detection and alerting nodes use a built-in threat signature knowledge base and threat matching engine to determine whether a data segment contains a preset threat.

[0017] When a pre-defined threat is present, the evidence collection message buffer list is retrieved and stored as a PCAP file;

[0018] If no threat is detected after a preset time period, the messages in the forensic message buffer list are released, and the latest message in the preset time period in the message structure space is copied to the forensic message buffer list.

[0019] In one implementation of this application, the CPU core retrieves packets from the dpdk-input node, specifically including:

[0020] Add an intelligent scheduling module to the existing dpdk-input node;

[0021] By adding an intelligent scheduling module, one CPU core is selected from all CPU cores as the distribution core, and the other CPU cores are the working cores; the number of packets received by the working core each time is configured.

[0022] The distribution core retrieves all packets from the network interface hardware queue of the dpdk-input node;

[0023] The intelligent scheduling module creates a software queue for each working core and configures the queue length and queue scheduling threshold for each software queue.

[0024] The distribution core distributes all packets evenly to each software queue;

[0025] During the distribution process, when the queue utilization rate of the software queue exceeds the queue scheduling threshold, the message that should have been distributed to the software queue exceeding the queue scheduling threshold is inserted into the preset shared queue.

[0026] When insertion into the pre-defined shared queue fails, the message is discarded.

[0027] The working core retrieves packets from the software queue corresponding to the dpdk-input node. When the actual number of packets received each time for a preset number of consecutive times is less than the configured number of packets received each time, the control working core retrieves packets from the preset shared queue.

[0028] In one implementation of this application, the existence of the message in the handoff global hash table is determined by the hash value corresponding to the message, specifically including:

[0029] Obtain message information; construct clib_bihash_40_8 matching information based on message information;

[0030] Check if the clib_bihash_40_8 matching information exists in the handoff global hash table.

[0031] In one implementation of this application, it is determined whether the message structure exists in the global session table. If it does not exist in the global session table, the message structure is added to the local session table and updated in the global session table. If it exists in the global session table but not in the local session table, the message is added to the handoff buffer queue of the CPU core corresponding to the message in the global session table. Specifically, this includes:

[0032] Determine if the message structure exists in the global session table. If it does not exist in the global session table, mark the message as a forward message, add the message structure to the local session table, and update the global session table.

[0033] When the packet exists in the global session table but not in the local session table, obtain the packet direction; when the packet direction is a reverse packet, obtain the CPU core corresponding to the reverse packet from the global session table; when the corresponding CPU core exists, add the packet to the handoff buffer queue of the corresponding CPU core.

[0034] When a reverse message is first obtained but no corresponding CPU core exists, the CPU core of the forward message corresponding to the reverse message is obtained; the message is added to the handoff buffer queue of the CPU core of the forward message; when the message direction is a forward message, the CPU core of the forward message is obtained from the global session table; the message is added to the handoff buffer queue of the CPU core of the forward message.

[0035] In one implementation of this application, message structure space is allocated through a message forensics cache node, and messages from the CPU core are allocated to the message structure space, specifically including:

[0036] By using the message forensics cache node, initialize the local memory buffer pool of the CPU core and create a global message buffer pool;

[0037] Allocate message structure space in the local memory buffer pool. If allocation in the local memory buffer pool fails, allocate message structure space corresponding to the current CPU core from the global message buffer pool.

[0038] Allocate the message corresponding to the current CPU core to the message structure space corresponding to the current CPU core.

[0039] In one implementation of this application, a TCP reassembly node maintains a buffer list of unreassembled packets for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments, specifically including:

[0040] The TCP reassembly node maintains a message buffer linked list for each direction of each TCP session involved in the CPU core;

[0041] Based on the seq and ack fields in the TCP stream header, determine the corresponding packet buffer list;

[0042] When TCP stream data overlaps, the latest TCP stream data is used to overwrite the old TCP stream data;

[0043] As TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments according to a preset data segment size.

[0044] In one implementation of this application, the threat detection and alarm node uses a built-in threat feature knowledge base and threat matching engine to determine whether a preset threat exists in the data segment, specifically including:

[0045] The threat matching engine performs multi-modal string matching between the data segment and the data in the threat feature knowledge base.

[0046] When a match is successful, the threat corresponding to the matching data in the threat feature knowledge base is determined to be the preset threat;

[0047] If a match fails, it is determined that there is no pre-defined threat.

[0048] Secondly, this application provides a high-performance traffic reassembly detection and forensics system based on VPP, the system comprising:

[0049] The VPP framework adds a session management handoff node, a packet forensics cache node, a TCP reassembly node, and a threat detection and alarm node under the ip4-sv-reassembly-feature node.

[0050] The session management handoff node is used to create a local session table for each CPU core, a global session table for all CPU cores, and a global handoff hash table.

[0051] The CPU core is used to determine whether a message exists in the local handoff buffer queue. If the message exists, it uses the hash value corresponding to the message to determine whether it exists in the global handoff hash table. If it does not exist in the global handoff hash table, it adds the hash value of the message to the global handoff hash table, inserts the message structure into the local session table, and updates the global session table. If it exists in the global handoff hash table, the message is processed. The CPU core retrieves the message from the dpdk-input node, determines whether the message structure exists in the global session table, and if it does not exist in the global session table, it adds the message structure to the local session table and updates the global session table. If it exists in the global session table but not in the local session table, it adds the message to the handoff buffer queue of the CPU core corresponding to the message in the global session table. If it exists in the local session table, the message is processed.

[0052] The message evidence caching node is used to allocate message structure space, allocate CPU core messages to message structure space, and add messages within a preset time period to the evidence message buffer list.

[0053] The TCP reassembly node is used to maintain a cache list of packets that have failed to be reassembled for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments.

[0054] The threat detection and alarm node is used by the built-in threat signature knowledge base and threat matching engine to determine whether a preset threat exists in the data segment. When a preset threat exists, the evidence collection message buffer list is retrieved and stored as a PCAP file. When no threat is detected after a preset time period, the messages in the evidence collection message buffer list are released, and the messages in the latest preset time period in the message structure space are copied to the evidence collection message buffer list.

[0055] Thirdly, this application provides a high-performance traffic reconstruction detection and forensics device based on VPP, the device comprising:

[0056] processor;

[0057] And a memory containing executable code, which, when executed, causes the processor to execute a VPP-based high-performance traffic reassembly detection and forensics method, as described above.

[0058] Fourthly, this application provides a non-volatile computer storage medium storing computer instructions, which, when executed, implement a high-performance traffic reassembly detection and forensics method based on VPP as described above.

[0059] As can be seen from the above technical solutions, this application has the following advantages:

[0060] Multi-level session table architecture enhances parallel processing capabilities:

[0061] By creating a local session table and a global session table for each CPU core, and combining this with a global hash table for handoff, lock-free transfer of session states is achieved, effectively solving the performance bottleneck caused by global hash table lock contention in traditional solutions. When a packet arrives, the CPU core can independently determine its session state: if it is a local session, it is processed directly; if cross-core collaboration is required, efficient forwarding is achieved through a handoff buffer queue. This design enables linear scaling of session management in multi-core scenarios, significantly improving session matching efficiency in high-speed network environments and avoiding packet loss or latency issues caused by lock contention.

[0062] Memory pre-allocation mechanism reduces fragmentation overhead:

[0063] The message forensics cache node adopts a pre-allocated message structure space approach, replacing the traditional dynamic memory allocation mode. By pre-dividing a fixed-size memory pool to store messages and copying messages within a preset time period to the forensics cache linked list as needed, it satisfies the continuity requirements of forensic data while avoiding fragmentation issues caused by frequent memory operations. This static memory management strategy is particularly suitable for high-frequency message processing scenarios, reducing memory allocation latency and improving cache hit rate, while providing a stable data source for subsequent PCAP file generation.

[0064] Deep integration of flow reconfiguration and threat detection:

[0065] The TCP reassembly node achieves accurate reassembly and merging of out-of-order packets by maintaining an independent packet cache list for each session. When a threat is detected, the complete session data stored in the forensic cache list can be directly used to generate a PCAP file, eliminating the overhead of calling a separate forensic component in traditional solutions. This design, which embeds reassembly, detection, and forensic functions into the VPP data plane, not only reduces the performance loss caused by cross-module communication but also ensures a strict temporal correlation between threat alerts and original packets, improving the accuracy of source tracing analysis. Attached Figure Description

[0066] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart of a high-performance traffic reassembly detection and forensics method based on VPP provided in an embodiment of this application.

[0068] Figure 2 This is a schematic diagram of the internal structure of a high-performance traffic reconstruction detection and forensics system based on VPP provided in an embodiment of this application.

[0069] Figure 3 This is a schematic diagram of the internal structure of a high-performance traffic reconstruction detection and forensics device based on VPP, provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0072] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0073] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0074] The embodiment provides a high-performance traffic reassembly detection and forensics method based on VPP, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps:

[0075] Step 110: Add a session management handoff node, a packet forensics cache node, a TCP reassembly node, and a threat detection and alarm node under the ip4-sv-reassembly-feature node in the VPP framework.

[0076] Step 120: The session management handoff node creates a local session table for each CPU core, a global session table for all CPU cores, and a global handoff hash table.

[0077] It should be noted that the local session table is used to store local sessions of CPU cores, while the global session table is used to store sessions of all CPU cores; the handoff global hash table stores the hash values ​​involved in the message information.

[0078] Step 130: The CPU core determines whether a message exists in the local handoff buffer queue. If a message exists, it determines whether it exists in the global handoff hash table using the hash value corresponding to the message. If it does not exist in the global handoff hash table, it adds the hash value of the message to the global handoff hash table, inserts the message structure into the local session table, and updates the global session table. If it exists in the global handoff hash table, it processes the message.

[0079] Specifically, determining whether a message exists in the global hash table of handoff involves using its corresponding hash value.

[0080] Obtain message information; construct clib_bihash_40_8 matching information based on message information;

[0081] Check if the clib_bihash_40_8 matching information exists in the handoff global hash table.

[0082] Understandably, the hash value matching mechanism achieves precise routing and efficient management of message processing. Specifically, when the CPU core detects a message in its local handoff buffer queue, it quickly determines the message's ownership by comparing the message's hash value with the global hash table. If the hash value does not match the global table, it automatically completes hash value registration, local session table insertion, and global table update operations, ensuring a complete record of the new message session. If the hash value matches, it directly triggers subsequent processing. This design avoids redundant calculations and storage, utilizing the clib_bihash_40_8 matching structure to achieve low-overhead hash lookups. While ensuring session state consistency, it improves message processing efficiency, making it particularly suitable for rapid session allocation and tracking in high-concurrency scenarios.

[0083] Step 140: The CPU core retrieves the packet from the dpdk-input node and determines whether the packet structure exists in the global session table. If it does not exist in the global session table, the packet structure is added to the local session table and updated in the global session table. If it exists in the global session table but not in the local session table, the packet is added to the handoff buffer queue of the CPU core corresponding to the packet in the global session table. If it exists in the local session table, the packet is processed.

[0084] In some embodiments, the CPU core retrieves packets from the dpdk-input node, specifically including:

[0085] Add an intelligent scheduling module to the existing dpdk-input node;

[0086] By adding an intelligent scheduling module, one CPU core is selected from all CPU cores as the distribution core, and the other CPU cores are the working cores; the number of packets received by the working core each time is configured.

[0087] The distribution core retrieves all packets from the network interface hardware queue of the dpdk-input node;

[0088] The intelligent scheduling module creates a software queue for each working core and configures the queue length and queue scheduling threshold for each software queue.

[0089] The distribution core distributes all packets evenly to each software queue;

[0090] During the distribution process, when the queue utilization rate of the software queue exceeds the queue scheduling threshold, the message that should have been distributed to the software queue exceeding the queue scheduling threshold is inserted into the preset shared queue.

[0091] When insertion into the pre-defined shared queue fails, the message is discarded.

[0092] The working core retrieves packets from the software queue corresponding to the dpdk-input node. When the actual number of packets received each time for a preset number of consecutive times is less than the configured number of packets received each time, the control working core retrieves packets from the preset shared queue.

[0093] Those skilled in the art will understand that the intelligent scheduling module works as follows: One dedicated CPU core is allocated as the dispatch core, and several worker cores are allocated. The dispatch core is responsible for receiving packets from the network interface card (NIC) hardware queue, while the worker cores are responsible for processing packets and sending packets from the NIC. A software queue is created for each worker core, specifying the queue length and a warning value, such as a queue length of 2048 and a scheduling threshold of 80%. When the queue utilization rate exceeds 80%, packet scheduling is triggered. At this time, the dispatch core inserts the packets that should have been dispatched to that core into the shared queue. If insertion into the shared queue fails, the packets are discarded. Other worker cores... The kernel core receives packets from its corresponding queue, 32 packets at a time. If it receives fewer than 32 packets four times in a row, it means that the current kernel's queue is not backed up and the load is low. Then, it takes 32 packets from the common queue to start processing. At this point, we temporarily call this kernel the assistant kernel. It searches for a session. When a session is found and the processing kernel is the original processing kernel, the packet is inserted into the queue of the original processing kernel and continues to be processed by the original processing kernel. When no session is found, it means that this packet is the first packet of the session. The kernel then creates the session and designates the kernel that created the session as this kernel. At this point, the session is successfully scheduled from the original kernel to the assistant kernel, and all subsequent packets of this session are processed by the assistant kernel.

[0094] Specifically, this involves determining whether the message structure exists in the global session table. If it does not exist, the message structure is added to the local session table, and the global session table is updated. If it exists in the global session table but not in the local session table, the message is added to the handoff buffer queue of the CPU core corresponding to the message in the global session table. This includes:

[0095] Determine if the message structure exists in the global session table. If it does not exist in the global session table, mark the message as a forward message, add the message structure to the local session table, and update the global session table.

[0096] When the packet exists in the global session table but not in the local session table, obtain the packet direction; when the packet direction is a reverse packet, obtain the CPU core corresponding to the reverse packet from the global session table; when the corresponding CPU core exists, add the packet to the handoff buffer queue of the corresponding CPU core.

[0097] When a reverse message is first obtained but no corresponding CPU core exists, the CPU core of the forward message corresponding to the reverse message is obtained; the message is added to the handoff buffer queue of the CPU core of the forward message; when the message direction is a forward message, the CPU core of the forward message is obtained from the global session table; the message is added to the handoff buffer queue of the CPU core of the forward message.

[0098] Those skilled in the art will understand that, when the network card does not support RSS or a NAT policy exists, forward and reverse packets may be processed by different CPU cores. For example, in an HTTPS session from 192.168.0.2:59356 to 153.3.238.28:443, the forward packet (192.168.0.2:59356->153.3.238.28:443) may be received on CPU0 core, while the reverse packet (153.3.238.28:443->192.168.0.2:59356) may be received on CPU1 core. Since packets of the same session are processed on different cores, locking protection is required, which has the disadvantages of poor performance and complex logic. Therefore, in the VPP framework, the handoff mechanism is used to schedule the reverse packet from CPU1 core to CPU0 core for unified processing by CPU0 core.

[0099] This mechanism has the following advantages: Improved session consistency: All session data (forward and reverse) is processed by the same CPU core, simplifying state management; Avoidance of lock contention: The use of a lock-free buffer queue reduces the risk of cross-core resource contention, and services based on this session do not require locking; Mitigation of out-of-order processing risk: Reverse packets are promptly processed by the original core together with forward packets after handoff scheduling.

[0100] Step 150: Allocate message structure space through the message evidence caching node, allocate the CPU core messages to the message structure space, and add the messages within the preset time period to the evidence caching message buffer list.

[0101] Specifically, the process involves allocating message structure space through message evidence caching nodes, and allocating messages from the CPU core to the message structure space.

[0102] By using the message forensics cache node, initialize the local memory buffer pool of the CPU core and create a global message buffer pool;

[0103] Allocate message structure space in the local memory buffer pool. If allocation in the local memory buffer pool fails, allocate message structure space corresponding to the current CPU core from the global message buffer pool.

[0104] Allocate the message corresponding to the current CPU core to the message structure space corresponding to the current CPU core.

[0105] Step 160: Through the TCP reassembly node, maintain a cache list of packets that have failed to be reassembled for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, merge the reassembled packets into data segments.

[0106] Specifically, through the TCP reassembly node, a cache list of unreassembled packets is maintained for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments, specifically including:

[0107] The TCP reassembly node maintains a message buffer linked list for each direction of each TCP session involved in the CPU core;

[0108] Based on the seq and ack fields in the TCP stream header, determine the corresponding packet buffer list;

[0109] When TCP stream data overlaps, the latest TCP stream data is used to overwrite the old TCP stream data;

[0110] As TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments according to a preset data segment size.

[0111] Step 170: Determine whether a preset threat exists in the data segment using the threat feature knowledge base and threat matching engine built into the threat detection and alarm node; when a preset threat exists, retrieve the evidence collection message buffer list and store it as a PCAP file; when no threat is detected after a preset time period, release the messages in the evidence collection message buffer list, and copy the latest message in the preset time period from the message structure space to the evidence collection message buffer list.

[0112] Specifically, the threat detection and alerting nodes utilize a built-in threat signature knowledge base and threat matching engine to determine whether a data segment contains a preset threat. This includes:

[0113] The threat matching engine performs multi-modal string matching between the data segment and the data in the threat feature knowledge base.

[0114] When a match is successful, the threat corresponding to the matching data in the threat feature knowledge base is determined to be the preset threat;

[0115] If a match fails, it is determined that there is no pre-defined threat.

[0116] Those skilled in the art will understand that successfully reassembled traffic data is used as the overall input to the threat detection and alarm node. For example, a 14,600-byte TCP data block, after being segmented and transmitted through the protocol stack, will be divided into 10 packets for network transmission, each packet having a TCP payload length of 1,460 bytes. After successful reassembly, the data block is reassembled back to a 14,600-byte long database, requiring only one call to the threat detection and alarm node for matching. If reassembly fails, each cached packet needs to be sequentially entered into the threat detection module. In this case of reassembly failure, multiple checks are required, resulting in lower detection performance.

[0117] To further clarify, the threat detection and alerting node employs a modular feature matching engine. Feature matching is based on Snort format rules, supporting regular expressions, strings, and byte sequences. Its feature library is categorized as follows:

[0118] Characteristics of network worms and scanners;

[0119] Trojan horse and spyware protocol identifiers;

[0120] Buffer overflow shellcode detection;

[0121] SQL injection / XSS attack payload pattern.

[0122] The threat detection and alerting node performs signature matching based on the load after TCP stream reassembly, uses the hyperscan engine for efficient multi-pattern string matching, and outputs alert severity levels based on feature ID.

[0123] Additionally, upon detecting a threat, the VPP worker thread encapsulates the alert information and message storage task and inserts it into the lock-free circular queue `evidence_pcap_ring`. This queue is created using the DPDK's `rte_ring_create` interface and is allocated based on a large-page memory heap, resulting in high performance under high concurrency. The alert information includes session information, signature ID, detection time, attack type, and path to forensic files. These forensic files can be downloaded by network security administrators for further threat analysis. The VPP main thread periodically checks the `evidence_pcap_ring` queue through the `PRE_INPUT` node and inserts the alert information into the database upon detection.

[0124] For the evidence collection files, the process is divided into two stages. First, the entire evidence collection message is written to the tmpfs memory file system in the main thread. This file system is built based on memory rather than disk, which can speed up the writing in this stage. In the second stage, the entire evidence collection file is moved to the target path on the disk by an asynchronous thread. This completes the task of storing alarm information and evidence collection files.

[0125] Based on the above description, this application constructs a traffic scheduling, reassembly, detection, and forensics device within the VPP framework to handle high-volume, high-concurrency scenarios. It fully utilizes VPP's vector processing mechanism and efficiently leverages the CPU's i-cache and d-cache, improving system processing performance. Data structures such as session structures, packet structures, and forensic packet structures employ a memory pre-allocation scheme, eliminating the need for dynamic memory allocation and lock-free operation during traffic processing, thus improving system performance. Reverse packet cross-core scheduling ensures that packets from the same session are processed by the same CPU core. In high-concurrency (above 1 million) traffic scenarios, CPU cores process in parallel without locks, improving system performance. TCP stream reassembly combines TCP segments into a large data segment before threat detection, reducing the number of threat detection interface calls and improving threat detection performance. The pre-stored packet mechanism ensures that after a threat is detected, the entire session's packets can be obtained completely without requiring a separate source storage component, resulting in excellent real-time performance and cost-effectiveness.

[0126] In addition, this application Figure 2 This application provides a high-performance traffic reconstruction detection and forensics system based on VPP. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes:

[0127] VPP Framework 200 adds a session management handoff node 210, a packet evidence caching node 220, a TCP reassembly node 230, and a threat detection and alarm node 240 under the ip4-sv-reassembly-feature node of VPP Framework 200.

[0128] The session management handoff node 210 is used to create a local session table for each CPU core 300, a global session table for all CPU cores 300, and a global handoff hash table.

[0129] CPU core 300 is used to determine whether a message exists in the local handoff buffer queue. If the message exists, it uses the hash value corresponding to the message to determine whether it exists in the global handoff hash table. If it does not exist in the global handoff hash table, it adds the hash value of the message to the global handoff hash table, inserts the message structure into the local session table, and updates the global session table. If it exists in the global handoff hash table, the message is processed. CPU core 300 retrieves the message from the dpdk-input node, determines whether the message structure exists in the global session table, and if it does not exist in the global session table, it adds the message structure to the local session table and updates the global session table. If it exists in the global session table but not in the local session table, it adds the message to the handoff buffer queue of the CPU core 300 corresponding to the message in the global session table. If it exists in the local session table, the message is processed.

[0130] The message evidence caching node 220 is used to allocate message structure space, allocate the messages of CPU core 300 to the message structure space, and add the messages within a preset time period to the evidence message buffer list.

[0131] TCP reassembly node 230 is used to maintain a cache list of packets that have failed to be reassembled for each TCP session involved in CPU core 300; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments.

[0132] Threat detection and alarm node 240 is used by the built-in threat feature knowledge base and threat matching engine to determine whether a preset threat exists in the data segment. When a preset threat exists, the evidence collection message buffer list is retrieved and stored as a PCAP file. When no threat is detected after a preset time period, the messages in the evidence collection message buffer list are released, and the messages in the latest preset time period in the message structure space are copied to the evidence collection message buffer list.

[0133] The above are method embodiments of this application. Based on the same inventive concept, this application also provides a high-performance traffic reconstruction detection and forensics device based on VPP. Figure 3 As shown, the device includes: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform a high-performance traffic reassembly detection and forensics method based on VPP as described in the above embodiment.

[0134] Specifically, the server adds a session management handoff node, a packet forensics cache node, a TCP reassembly node, and a threat detection and alarm node under the ip4-sv-reassembly-feature node in the VPP framework;

[0135] The session management handoff node creates a local session table for each CPU core, a global session table for all CPU cores, and a global handoff hash table.

[0136] The CPU core checks if a message exists in the local handoff buffer queue. If the message exists, it uses the hash value corresponding to the message to determine if it exists in the global handoff hash table. If it does not exist in the global handoff hash table, it adds the hash value of the message to the global handoff hash table, inserts the message structure into the local session table, and updates the global session table. If the message exists in the global handoff hash table, it processes the message.

[0137] The CPU core retrieves the packet from the dpdk-input node, determines whether the packet's structure exists in the global session table, and adds the packet's structure to the local session table and updates the global session table if it does not exist in the global session table. If it exists in the global session table but not in the local session table, the packet is added to the handoff buffer queue of the CPU core corresponding to the packet in the global session table. If it exists in the local session table, the packet is processed.

[0138] By using the message evidence caching node, message structure space is allocated, messages from the CPU core are allocated to the message structure space, and messages within a preset time period are added to the evidence message buffer list.

[0139] Through the TCP reassembly node, a cache list of packets that have failed to be reassembled is maintained for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments.

[0140] The threat detection and alerting nodes use a built-in threat signature knowledge base and threat matching engine to determine whether a data segment contains a preset threat.

[0141] When a pre-defined threat is present, the evidence collection message buffer list is retrieved and stored as a PCAP file;

[0142] If no threat is detected after a preset time period, the messages in the forensic message buffer list are released, and the latest message in the preset time period in the message structure space is copied to the forensic message buffer list.

[0143] In addition, this application embodiment also provides a non-volatile computer storage medium storing executable instructions, which, when executed, implement the above-described high-performance traffic reassembly detection and forensics method based on VPP.

[0144] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high-performance flow reorganization detection forensics method based on VPP, characterized in that, The method comprises: Adding a session management handoff node, a message evidence cache node, a TCP reassembly node, and a threat detection and alarm node under an ip4-sv-reassembly-feature node of a VPP framework; The session management handoff node creates a local session table for each CPU core, a global session table for all CPU cores, and a handoff global hash table; The CPU core determines whether the local handoff buffer queue has a message; when the local handoff buffer queue has the message, it is determined whether the message exists in the handoff global hash table through a hash value corresponding to the message; when the message does not exist in the handoff global hash table, the hash value of the message is added to the handoff global hash table, and a structure of the message is inserted into the local session table and updated to the global session table; when the message exists in the handoff global hash table, the message is processed; The CPU core takes out the message from a dpdk-input node, and determines whether a structure of the message exists in the global session table; when the structure of the message does not exist in the global session table, the structure of the message is added to the local session table and updated to the global session table; when the structure of the message exists in the global session table but does not exist in the local session table, the message is added to a handoff buffer queue of a CPU core corresponding to the message in the global session table; when the structure of the message exists in the local session table, the message is processed; Through the message evidence cache node, a message structure space is allocated, the messages of the CPU core are allocated to the message structure space, and messages in a preset time period are added to an evidence message buffer linked list; Through the TCP reassembly node, a message buffer linked list for temporarily storing messages that fail to be reassembled is maintained for each TCP session involved in the CPU core; after the TCP flow data continuously reassembles the messages, the reassembled messages are combined into a data segment; Through a threat feature knowledge base and a threat matching engine provided in the threat detection and alarm node, it is determined whether the data segment has a preset threat; When the preset threat exists, the evidence message buffer linked list is taken out and stored into a PCAP file; When no threat is detected after the preset time period, the messages in the evidence message buffer linked list are released, and the latest messages in the preset time period in the message structure space are copied to the evidence message buffer linked list.

2. The VPP-based high-performance flow reassembly detection forensics method of claim 1, wherein, The CPU core takes out the message from the dpdk-input node, and specifically comprises: An intelligent scheduling module is added in the original dpdk-input node; Through the intelligent scheduling module, one CPU core is determined as a distribution core from all CPU cores, and the other CPU cores are determined as working cores; the number of messages received by each working core is configured; The distribution core takes out all messages from a network card hardware queue of the dpdk-input node; The intelligent scheduling module creates a software queue for each working core, and configures the queue length and the queue scheduling threshold of each software queue; The distribution core evenly distributes all messages to each software queue; In the distribution process, when the queue usage rate of the software queue is greater than the queue scheduling threshold, the messages that should be distributed to the software queue greater than the queue scheduling threshold are inserted into a preset shared queue. Discard the packet when preset shared queue insertion fails; The working core takes out the packet from the software queue corresponding to the dpdk-input node, and when the actual number of received packets each time is less than the configured number of received packets each time for a preset number of times in succession, the working core takes out the packet from the preset shared queue.

3. The VPP-based high-performance flow reassembly detection forensics method of claim 1, wherein, Determine whether the packet exists in the handoff global hash table through the hash value corresponding to the packet, specifically including: Obtain the packet information of the packet; construct clib_bihash_40_8 matching information through the packet information; Query whether the clib_bihash_40_8 matching information exists in the handoff global hash table.

4. The VPP-based high-performance traffic reassembly detection forensics method of claim 1, wherein, Determine whether the packet structure exists in the global session table, when it does not exist in the global session table, add the packet structure to the local session table and update to the global session table; when it exists in the global session table but does not exist in the local session table, add the packet to the handoff buffer queue of the CPU core corresponding to the packet in the global session table, specifically including: Determine whether the packet structure exists in the global session table, when it does not exist in the global session table, mark the packet as a forward packet, add the packet structure to the local session table and update to the global session table; When it exists in the global session table but does not exist in the local session table, obtain the packet direction; when the packet direction is a reverse packet, obtain the CPU core corresponding to the reverse packet from the global session table; when the corresponding CPU core exists, add the packet to the handoff buffer queue of the corresponding CPU core; When the reverse packet does not have a corresponding CPU core for the first time, obtain the CPU core of the forward packet corresponding to the reverse packet; add the packet to the handoff buffer queue of the CPU core of the forward packet; when the packet direction is a forward packet, obtain the CPU core of the forward packet from the global session table; add the packet to the handoff buffer queue of the CPU core of the forward packet.

5. The VPP-based high-performance flow reassembly detection forensics method of claim 1, wherein, Allocate packet structure space through the packet evidence cache node, and allocate the packet of the CPU core to the packet structure space, specifically including: Initialize the local memory buffer pool of the CPU core through the packet evidence cache node, and create a global packet buffer pool; Allocate packet structure space in the local memory buffer pool, and when the local memory buffer pool allocation fails, allocate the packet structure space corresponding to the current CPU core from the global packet buffer pool; Allocate the packet corresponding to the current CPU core to the packet structure space corresponding to the current CPU core.

6. The VPP-based high-performance traffic reassembly detection forensics method of claim 1, wherein, Maintain a packet cache linked list for temporarily storing packets that fail to be recombined for each TCP session involved in the CPU core through the TCP recombination node; After the TCP flow data continuously recombines the packets, combine the recombined packets into data segments, specifically including: The TCP recombination node maintains a packet cache linked list for each TCP session involved in the CPU core in two directions; Determine the corresponding packet cache linked list according to the seq and ack fields of the TCP flow data header; When there is TCP flow data overlap, use the latest TCP flow data to cover the old TCP flow data; When TCP stream data continuously reassembles packets, the reassembled packets are merged into data segments according to a preset data segment size.

7. The VPP-based high-performance flow reassembly detection forensics method of claim 1, wherein, The threat detection and alarm node is internally provided with a threat feature knowledge base and a threat matching engine, and whether the data segment has a preset threat is determined. The threat matching engine is used for multi-mode string matching of the data segment and data in the threat feature knowledge base. When the matching is successful, it is determined that a threat corresponding to the matching data in the threat feature knowledge base is the preset threat. When the matching is unsuccessful, it is determined that there is no preset threat.

8. A high performance flow reassembly detection forensics system based on VPP, characterized in that, The system comprises: A VPP framework, wherein a session management handoff node, a packet evidence cache node, a TCP reassembly node, and a threat detection and alarm node are added under an ip4-sv-reassembly-feature node of the VPP framework; The session management handoff node is used for creating a local session table for each CPU core, creating a global session table for all CPU cores, and creating a handoff global hash table. The CPU core is used for judging whether the local handoff buffer queue has a packet, and when the packet exists, determining whether the packet exists in the handoff global hash table through a hash value corresponding to the packet. When the packet does not exist in the handoff global hash table, the hash value of the packet is added to the handoff global hash table, and a structure of the packet is inserted into the local session table and updated to the global session table. When the packet exists in the handoff global hash table, the packet is processed. The CPU core takes out the packet from a dpdk-input node, and determines whether a structure of the packet exists in the global session table.

9. A high performance flow reassembly forensic device based on VPP, characterized in that, When the structure of the packet does not exist in the global session table, the structure of the packet is added to the local session table and updated to the global session table. When the structure of the packet exists in the global session table but does not exist in the local session table, the packet is added to a handoff buffer queue of the CPU core corresponding to the packet in the global session table. When the structure of the packet exists in the local session table, the packet is processed. The packet evidence cache node is used for allocating a packet structure space, allocating the packet of the CPU core to the packet structure space, and adding packets in a preset time period to an evidence packet buffer linked list. The TCP reassembly node is used for maintaining a packet buffer linked list for temporarily storing packets that fail to be reassembled for each TCP session involved in the CPU core. When TCP stream data continuously reassembles packets, the reassembled packets are merged into data segments. The threat detection and alarm node is internally provided with a threat feature knowledge base and a threat matching engine, and whether the data segment has a preset threat is determined. When the preset threat exists, the evidence packet buffer linked list is taken out and stored into a PCAP file. When no threat is detected after the preset time period, the packets in the evidence packet buffer linked list are released, and the latest packets in the preset time period in the packet structure space are copied to the evidence packet buffer linked list. The device comprises: A processor; and a memory having executable code stored thereon, which, when executed, causes the processor to perform a high-performance flow reassembly detection evidence collection method based on a VPP as claimed in any one of claims 1-7.

10. A non-transitory computer storage medium, comprising, A computer program product comprising a computer readable medium having stored thereon computer instructions that, when executed, implement a VPP-based high-performance flow reassembly detection forensics method as claimed in any of claims 1-7.

Images