High-performance flow recombination detection evidence obtaining method, system and device based on VPP and medium

Abstract

The invention discloses a VPP-based high-performance flow recombination detection evidence obtaining method, system and device and a medium, and mainly relates to the technical field of flow recombination detection evidence obtaining. The method and the device are used for solving the problems of poor multi-core expansibility caused by global hash table lock competition, memory fragments caused by frequent dynamic memory allocation and high performance overhead of a flow evidence obtaining function in an existing scheme. Comprising the following steps: maintaining a message cache linked list of messages which are not temporarily stored and recombined for each TCP session involved in a CPU core; after the TCP stream data continuously recombines the messages, combining the recombined messages into data segments; determining whether the data segment has a preset threat through a threat feature knowledge base and a threat matching engine; when the preset threat exists, taking out the evidence obtaining message buffer chain table, and storing the evidence obtaining message buffer chain table as a PCAP file; and when the threat is not detected after the preset time period, releasing the message in the evidence obtaining message buffer chain table, and copying the message in the latest preset time period in the message structure body space to the evidence obtaining message buffer chain table.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of flow reorganization detection, and in particular to a high-performance flow reorganization detection forensics method, system, device and medium based on VPP. BACKGROUND

[0002] With the increasing complexity of network attack means, traditional threat detection devices often face the bottlenecks of insufficient detection performance, delayed forensics process, and low session management efficiency, and are difficult to meet the demand for real-time processing of massive flows and fine-grained threat analysis.

[0003] Existing solutions mostly use the kernel-based netfilter framework or the dpdk user state collection + snort architecture, which has performance bottlenecks. Especially in a high-speed network environment, when facing complex session matching, TCP reorganization and threat alarm file generation, etc., it is easy to cause packet loss, misjudgment or inability to trace, which seriously affects the timeliness and integrity of network security event response.

[0004] The traditional TCP reorganization scheme (such as nf_conntrack of Linux kernel) has two major bottlenecks: (1) Global hash table lock competition leads to poor multi-core scalability; (2) Frequent dynamic memory allocation causes memory fragmentation; In addition, for threat flow forensics function, the existing solutions in the industry generally provide packet tracing function through an independent tracing component, which has large performance overhead. SUMMARY

[0005] The present application provides a high-performance flow reorganization detection forensics method, system, device and medium based on VPP to solve the problems of the existing solutions, such as global hash table lock competition leading to poor multi-core scalability, frequent dynamic memory allocation causing memory fragmentation, and large performance overhead of threat flow forensics function.

[0006] In a first aspect, the present application provides a high-performance flow reorganization detection forensics method based on VPP, which comprises: adding a session management handoff node, a packet forensics cache node, a TCP reorganization node, and a threat detection and alarm node under the ip4-sv-reassembly-feature node of the VPP framework; The session management handoff node creates a local session table for each CPU core, creates a global session table for all CPU cores, and creates a handoff global hash table; The CPU core judges whether the local handoff buffer queue has a message; when the message exists, the hash value corresponding to the message is determined to determine whether it exists in the handoff global hash table; when it does not exist in the handoff global hash table, the hash value of the message is added to the handoff global hash table, and the structure of the message is inserted into the local session table and updated to the global session table; when it exists in the handoff global hash table, the message processing is performed; The CPU core takes out the message from the dpdk-input node, determines whether the structure of the message exists in the global session table, when it does not exist in the global session table, adds the structure of the message to the local session table and updates to the global session table; when it exists in the global session table and does not exist in the local session table, adds the message to the handoff buffer queue of the CPU core corresponding to the message in the global session table; when it exists in the local session table, the message processing is performed; Through the message evidence cache node, the message structure space is allocated, the message of the CPU core is allocated to the message structure space, and the messages in the preset time period are added to the evidence message buffer linked list; Through the TCP recombination node, each TCP session involved in the CPU core maintains a message cache linked list for temporarily storing messages that fail to be recombined; when the TCP flow data continuously recombines the messages, the recombined messages are combined into data segments; Through the threat detection and alarm node, the threat feature knowledge base and the threat matching engine are determined to determine whether the data segment has a preset threat; When the preset threat exists, the evidence message buffer linked list is taken out and stored into a PCAP file; When no threat is detected after the preset time period, the messages in the evidence message buffer linked list are released, and the latest preset time period messages in the message structure space are copied to the evidence message buffer linked list.

[0007] In an implementation manner of the present application, the CPU core takes out the message from the dpdk-input node, specifically including: An intelligent scheduling module is added in the original dpdk-input node; Through the intelligent scheduling module, one CPU core is determined as a distribution core from all CPU cores, and the other CPU cores are working cores; the number of messages received by each working core is configured; The distribution core takes out all messages from the network card hardware queue of the dpdk-input node; The intelligent scheduling module creates a software queue for each working core, and configures the queue length and queue scheduling threshold of each software queue; The distribution core distributes all messages to each software queue averagely; In the distribution process, when the queue usage of the software queue is greater than the queue scheduling threshold, the packet that should be distributed to the software queue greater than the queue scheduling threshold is inserted into the preset shared queue; When the preset shared queue insertion fails, the packet is discarded; The working core takes out the packet from the software queue corresponding to the dpdk-input node, and when the actual number of received packets each time for a continuous preset number of times is less than the configured number of received packets each time, the working core takes out the packet from the preset shared queue.

[0008] In an implementation manner of the present application, whether the packet exists in the handoff global hash table is determined through the hash value corresponding to the packet, and specifically includes: Packet information of the packet is obtained, and clib_bihash_40_8 matching information is constructed through the packet information; Whether the clib_bihash_40_8 matching information exists in the handoff global hash table is queried.

[0009] In an implementation manner of the present application, whether the packet structure exists in the global session table is determined, when the packet structure does not exist in the global session table, the packet structure is added to the local session table and updated to the global session table, and when the packet structure exists in the global session table and does not exist in the local session table, the packet is added to the handoff buffer queue of the CPU core corresponding to the packet in the global session table, and specifically includes: Whether the packet structure exists in the global session table is determined, when the packet structure does not exist in the global session table, the packet is marked as a forward packet, and the packet structure is added to the local session table and updated to the global session table; When the packet structure exists in the global session table and does not exist in the local session table, the packet direction is obtained, when the packet direction is a reverse packet, the CPU core corresponding to the reverse packet is obtained from the global session table, and when the corresponding CPU core exists, the packet is added to the handoff buffer queue of the corresponding CPU core; When the reverse packet does not have the corresponding CPU core for the first time, the CPU core of the forward packet corresponding to the reverse packet is obtained, the packet is added to the handoff buffer queue of the CPU core of the forward packet, when the packet direction is a forward packet, the CPU core of the forward packet is obtained from the global session table, and the packet is added to the handoff buffer queue of the CPU core of the forward packet.

[0010] In an implementation manner of the present application, the packet structure space is allocated through the packet evidence cache node, and the packet of the CPU core is allocated to the packet structure space, and specifically includes: The local memory buffer pool of the CPU core is initialized through the packet evidence cache node, and a global packet buffer pool is created; Allocating the message structure space in the local memory buffer pool, when the local memory buffer pool allocation fails, allocating the message structure space corresponding to the current CPU core from the global message buffer pool; Allocating the message corresponding to the current CPU core to the message structure space corresponding to the current CPU core.

[0011] In an implementation manner of the application, each TCP session involved in the CPU core is maintained by the TCP reassembly node to temporarily store the message cache linked list which fails to reassemble the message; when the TCP flow data continuously reassembles the message, the reassembled message is merged into a data segment, specifically including: The TCP reassembly node maintains a message cache linked list for each TCP session involved in the CPU core in two directions; According to the TCP flow data header seq and ack fields, the corresponding message cache linked list is determined; When there is TCP flow data overlap, the latest TCP flow data is used to cover the old TCP flow data; When the TCP flow data continuously reassembles the message, the reassembled message is merged into a data segment according to the preset data segment size.

[0012] In an implementation manner of the application, the threat feature knowledge base and the threat matching engine in the threat detection and alarm node are used to determine whether the data segment has a preset threat, specifically including: The threat matching engine is used to perform multi-mode string matching on the data segment and the data in the threat feature knowledge base; When the matching is successful, the threat corresponding to the matching data in the threat feature knowledge base is determined as the preset threat; When the matching is unsuccessful, it is determined that there is no preset threat.

[0013] In a second aspect, the application provides a high-performance flow reassembly detection forensics system based on VPP, which includes: The VPP framework, the session management handoff node, the message forensics cache node, the TCP reassembly node, and the threat detection and alarm node are added under the ip4-sv-reassembly-feature node of the VPP framework; The session management handoff node is used to create a local session table for each CPU core, create a global session table for all CPU cores, and create a handoff global hash table; The CPU core is configured to determine whether the local handoff buffer queue has a message, and when the local handoff buffer queue has the message, determine whether the message exists in the handoff global hash table through a hash value corresponding to the message, and when the message does not exist in the handoff global hash table, add the hash value of the message to the handoff global hash table, insert a structure body of the message into the local session table, and update the global session table, and when the message exists in the handoff global hash table, process the message; the CPU core takes out the message from the dpdk-input node, determines whether the structure body of the message exists in the global session table, when the structure body of the message does not exist in the global session table, adds the structure body of the message to the local session table and updates the global session table, when the structure body of the message exists in the global session table and does not exist in the local session table, adds the message to the handoff buffer queue of the CPU core corresponding to the message in the global session table, and when the structure body of the message exists in the local session table, processes the message. The message evidence cache node is configured to allocate a message structure body space, allocate the message of the CPU core to the message structure body space, and add messages in a preset time period to a cache message buffer linked list. The TCP recombination node is configured to maintain a message cache linked list for temporarily storing messages that fail to be recombined for each TCP session involved in the CPU core, and combine the recombined messages into a data segment when the TCP flow data continuously recombines the messages. The threat detection and alarm node is configured to determine whether the data segment has a preset threat through an internally set threat feature knowledge base and a threat matching engine, take out the cache message buffer linked list and store the cache message buffer linked list into a PCAP file when the preset threat exists, and release the messages in the cache message buffer linked list and copy the latest messages in the message structure body space in a preset time period to the cache message buffer linked list when no threat is detected in the preset time period.

[0014] In a third aspect, the present application provides a high-performance flow recombination detection evidence collection device based on VPP, which comprises: a processor; a memory having executable code stored thereon, wherein the executable code, when executed, causes the processor to perform any one of the above high-performance flow recombination detection evidence collection methods based on VPP.

[0015] In a fourth aspect, the present application provides a non-volatile computer storage medium having computer instructions stored thereon, wherein the computer instructions, when executed, implement any one of the above high-performance flow recombination detection evidence collection methods based on VPP.

[0016] From the above technical solutions, the present application has the following advantages: The multi-level session table architecture improves parallel processing capability: By creating a local session table for each CPU core in combination with a handoff global hash table, the lock-free transfer of session state is achieved, effectively solving the performance bottleneck caused by lock contention in the global hash table in traditional solutions. When a packet arrives, the CPU core can independently determine its session state: if it is a local session, it is directly processed, and if it needs to be cross-core collaboration, it is implemented through the handoff buffer queue to achieve efficient forwarding. This design enables linear expansion of session management in a multi-core scenario, significantly improves the session matching efficiency in a high-speed network environment, and avoids packet loss or delay problems caused by lock contention.

[0017] ‌Memory pre-allocation mechanism reduces fragmentation overhead:‌ The packet evidence cache node adopts a pre-allocated packet structure space, replacing the traditional dynamic memory allocation mode. By pre-dividing a fixed-size memory pool to store packets, and copying the packets of a preset time period to the evidence cache linked list as needed, the continuity requirement of evidence data is met, and the fragmentation problem caused by frequent memory operations is avoided. This static memory management strategy is particularly suitable for high-frequency packet processing scenarios, can reduce memory allocation delay and improve cache hit rate, and provides a stable data source for subsequent PCAP file generation.

[0018] ‌Deep integration of flow reassembly and threat detection:‌ The TCP reassembly node maintains an independent packet cache linked list for each session, enabling accurate reassembly and merging of out-of-order packets. When a threat is detected, the complete session data stored in the evidence cache linked list can be directly used to generate a PCAP file, eliminating the call overhead of the independent evidence component in traditional solutions. This design of embedding reassembly, detection, and evidence functions into the VPP data plane not only reduces the performance loss caused by cross-module communication, but also ensures the strict time sequence association of threat alerts and original packets, improving the accuracy of traceability analysis. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a flowchart of a high-performance traffic reassembly, detection, and evidence method based on VPP provided by an embodiment of the present application.

[0021] Figure 2 is a schematic diagram of the internal structure of a high-performance traffic reassembly, detection, and evidence system based on VPP provided by an embodiment of the present application.

[0022] Figure 3 Fig. 1 is a schematic diagram of an internal structure of a high-performance flow reassembly detection forensics device based on VPP provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present application.

[0024] Those skilled in the art should understand that the embodiments described below are only preferred embodiments of the present disclosure, and do not represent the only way to implement the present disclosure. The preferred embodiments are only used to explain the technical principles of the present disclosure, and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts still fall within the protection scope of the present disclosure.

[0025] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent in such processes, methods, articles, or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article, or device including the element.

[0026] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0027] The embodiments provide a high-performance flow reassembly detection forensics method based on VPP, as shown in Fig. 1, the method provided by the embodiments of the present application mainly includes the following steps. Figure 1 Step 110, adding a session management handoff node, a message forensics cache node, a TCP reassembly node, and a threat detection and alarm node under an ip4-sv-reassembly-feature node of a VPP framework.

[0028] Step 120, the session management handoff node creates a local session table for each CPU core, creates a global session table for all CPU cores, and creates a handoff global hash table.

[0029] ​It should be noted that the local session table is used to store the local session of the CPU core, the global session table is used to store the session of all CPU cores, and the handoff global hash table stores the hash value related to the message information.

[0030] In step 130, the CPU core determines whether the local handoff buffer queue exists a message; when the message exists, whether the message exists in the handoff global hash table is determined through the hash value corresponding to the message; when the message does not exist in the handoff global hash table, the hash value of the message is added to the handoff global hash table, and the structure of the message is inserted into the local session table and updated to the global session table; when the message exists in the handoff global hash table, the message processing is performed.

[0031] Specifically, whether the message exists in the handoff global hash table is determined through the hash value corresponding to the message, and the determination specifically includes: Message information of the message is obtained; and clib_bihash_40_8 matching information is constructed through the message information. Whether the clib_bihash_40_8 matching information exists in the handoff global hash table is queried.

[0032] It can be understood that the hash value matching mechanism is used to realize accurate shunting and efficient management of message processing. The specific effects are as follows: when the CPU core detects that the local handoff buffer queue exists a message, the message attribution can be quickly determined through comparison of the message hash value and the global hash table. If the hash value does not hit the global table, the hash value registration, local session table insertion and global table updating operations are automatically completed, so that the complete record of the new message session is ensured; if the hash value has hit, the subsequent processing flow is directly triggered. This design avoids repeated calculation and redundant storage, realizes low-overhead hash query by using the clib_bihash_40_8 matching structure, ensures the consistency of the session state, improves the message processing efficiency, and is especially suitable for fast session allocation and tracking in a high-concurrency scenario.

[0033] In step 140, the CPU core takes out the message from the dpdk-input node, determines whether the structure of the message exists in the global session table, adds the structure of the message to the local session table and updates to the global session table when the structure of the message does not exist in the global session table, adds the message to the handoff buffer queue of the CPU core corresponding to the message in the global session table when the structure of the message exists in the global session table but does not exist in the local session table, and performs message processing when the structure of the message exists in the local session table.

[0034] In some embodiments, the CPU core takes out the message from the dpdk-input node, and the taking out specifically includes: Intelligent scheduling module is added in the original dpdk-input node; By adding the intelligent scheduling module, one CPU core is determined as a dispatch core and other CPU cores are working cores from all CPU cores; the number of received packets per time of each working core is configured; All packets are taken out from the network card hardware queue of the dpdk-input node by the dispatch core; The intelligent scheduling module creates a software queue for each working core and configures the queue length and queue scheduling threshold of each software queue; The dispatch core distributes all packets to each software queue averagely; During the distribution process, when the queue usage rate of the software queue is greater than the queue scheduling threshold, the packets that should be distributed to the software queue with a queue usage rate greater than the queue scheduling threshold are inserted into a preset shared queue; When the preset shared queue insertion fails, the packet is discarded; The working core takes out the packet from the software queue corresponding to the dpdk-input node, and when the actual number of received packets per time of a continuous preset number of times is less than the configured number of received packets per time, the working core takes out the packet from the preset shared queue.

[0035] As understood by those skilled in the art, the work of the intelligent scheduling module can be specifically: allocating one special CPU core as a dispatch core and allocating a plurality of working cores, the dispatch core is responsible for packet collection from the network card hardware queue, and the working core is responsible for processing the packet and sending the packet from the network card; a software queue is created for each working core, and the queue length and warning value are specified, for example, the queue length is 2048 and the scheduling threshold is 80%, when the queue usage rate is higher than 80%, the packet scheduling is triggered, at this time, the dispatch core inserts the packet that should be distributed to the shared queue, if the shared queue insertion fails, the packet is discarded; other worker cores receive the packet from the queue corresponding to the core, receive 32 packets per time, and continuously receive less than 32 packets for 4 times, indicating that the queue of the current core has no backlog and the load is low, then 32 packets are taken from the public queue to start processing, at this time, we temporarily call the core as an assisting core; find the session, when the session is found and the processing core is the original processing core, the packet is inserted into the queue of the original processing core and continues to be processed by the original processing core, when the session is not found, it indicates that the packet is the first packet of the session, a session is created and the creation core of the session is specified as the current core, at this time, the session is successfully scheduled from the original core to the assisting core, and all subsequent packets of the session are processed by the assisting core.

[0036] Wherein, it is determined whether the structure of the packet exists in the global session table, when not existing in the global session table, the structure of the packet is added to the local session table and updated to the global session table; when existing in the global session table and not existing in the local session table, the packet is added to the handoff buffer queue of the CPU core corresponding to the packet in the global session table, specifically comprising: It is determined whether the structure of the packet exists in the global session table, when not existing in the global session table, the packet is marked as a forward packet, the structure of the packet is added to the local session table and updated to the global session table; When existing in the global session table and not existing in the local session table, the packet direction is obtained; when the packet direction is a reverse packet, the CPU core corresponding to the reverse packet in the global session table is obtained; when the corresponding CPU core exists, the packet is added to the handoff buffer queue of the corresponding CPU core; When the corresponding CPU core does not exist for the first time, the CPU core of the forward packet corresponding to the reverse packet is obtained; the packet is added to the handoff buffer queue of the CPU core of the forward packet; when the packet direction is a forward packet, the CPU core of the forward packet in the global session table is obtained; the packet is added to the handoff buffer queue of the CPU core of the forward packet.

[0037] Those skilled in the art can understand that under the condition that the network card does not support RSS or there is a NAT policy, the forward packet and the reverse packet of the packet may be processed by different CPU cores, such as a https session of 192.168.0.2:59356 to 153.3.238.28:443, the forward packet (192.168.0.2:59356->153.3.238.28:443) may be received by cpu0 core, and the reverse packet (153.3.238.28:443->192.168.0.2:59356) may be received by cpu1 core. Since the same session packet is processed in different cores, locking protection is required, which has the disadvantages of performance difference and logical complexity, therefore, under the vpp framework, the handoff mechanism is used to schedule the reverse packet from the cpu1 core to the cpu0 core for unified processing by the cpu0 core.

[0038] This mechanism has the following advantages: improving session consistency: all session data (forward and reverse) are processed by the same CPU core, simplifying state management; avoiding lock conflict: adopting a lock-free buffer queue, reducing the risk of cross-core resource competition, and the service based on the session does not need to be locked; relieving the risk of out-of-order processing: the reverse direction packet is timely scheduled by the handoff and processed together with the forward packet by the original core.

[0039] Step 150, allocate the message structure body space through the message forensics cache node, allocate the message of the CPU core to the message structure body space, and add the messages of the preset time period to the forensics message buffer linked list.

[0040] The message structure body space is allocated through the message forensics cache node, and the message of the CPU core is allocated to the message structure body space, specifically comprising: The local memory buffer pool of the CPU core is initialized through the message forensics cache node, and a global message buffer pool is created; The message structure body space is allocated in the local memory buffer pool, and when the local memory buffer pool allocation fails, the message structure body space corresponding to the current CPU core is allocated from the global message buffer pool; The message corresponding to the current CPU core is allocated to the message structure body space corresponding to the current CPU core.

[0041] Step 160, maintain a message cache linked list for temporarily storing messages that fail to be recombined through the TCP recombination node for each TCP session involved in the CPU core; and when the TCP stream data continuously recombines messages, the recombined messages are merged into data segments.

[0042] The message cache linked list for temporarily storing messages that fail to be recombined is maintained through the TCP recombination node for each TCP session involved in the CPU core; and when the TCP stream data continuously recombines messages, the recombined messages are merged into data segments, specifically comprising: The TCP recombination node maintains a message cache linked list for each direction of each TCP session involved in the CPU core; According to the TCP stream data header seq and ack fields, the corresponding message cache linked list is determined; When there is TCP stream data overlap, the latest TCP stream data is used to cover the old TCP stream data; When the TCP stream data continuously recombines messages, the recombined messages are merged into data segments according to the preset data segment size.

[0043] Step 170, determine whether the data segment has a preset threat through the threat feature knowledge base and threat matching engine built in the threat detection and alarm node; when there is a preset threat, the forensics message buffer linked list is taken out and stored as a PCAP file; when no threat is detected after a preset time period, the messages in the forensics message buffer linked list are released, and the latest messages of the preset time period in the message structure body space are copied to the forensics message buffer linked list.

[0044] The threat feature knowledge base and threat matching engine built in the threat detection and alarm node are used to determine whether the data segment has a preset threat, specifically comprising: The data segment is subjected to multi-mode string matching with data in the threat feature knowledge base through a threat matching engine. When the matching is successful, it is determined that a threat corresponding to the matching data in the threat feature knowledge base is a preset threat. When the matching is unsuccessful, it is determined that there is no preset threat.

[0045] As can be understood by those skilled in the art, the recombined traffic data is input as a whole into a threat detection and alarm node, such as a TCP data block of 14600 bytes, which is divided into 10 packets after being subjected to protocol stack block transmission, and each packet has a TCP load length of 1460 bytes. After recombination, the data block is recovered to a database with a length of 14600 bytes, and only one call to the threat detection and alarm node is required to complete the matching. If recombination fails, each buffered packet needs to be sequentially input into the threat detection module. In this case of recombination failure, multiple detections are required, and the detection performance is low.

[0046] Further, the threat detection and alarm node adopts a modular feature matching engine, supports matching based on snort format rules, supports regular expression, string, and byte sequence detection, and the feature library classification includes: Network worm and scanner features; Trojan and spyware protocol identification; Buffer overflow shellcode detection; SQL injection / XSS attack payload mode.

[0047] The threat detection and alarm node matches the feature library based on the load after TCP flow recombination, uses the hyperscan engine for efficient multi-mode string matching, and outputs the alarm severity based on the feature ID classification.

[0048] In addition, after detecting a threat, the vpp worker thread encapsulates and inserts alarm information and packet storage tasks into a lock-free ring queue evidence_pcap_ring. The queue is created using the dpdk rte_ring_create interface, is allocated based on a large page memory heap, and has high performance in a high-concurrency situation. The alarm information includes session information, feature ID, detection time, attack type, and evidence file path. The evidence file can be downloaded by a network security administrator for further threat analysis. The vpp main thread regularly checks the evidence_pcap_ring queue through a PRE_INPUT node, inserts alarm information into a database after detecting an alarm, and outputs the alarm severity based on the feature ID classification.

[0049] For the forensic file, the process is divided into two stages, first, the whole forensic packet is written into the tmpfs memory file system in the main thread, the file system is built based on memory rather than disk, which can accelerate the writing in this stage, the second stage, the asynchronous thread is used to move the whole forensic file to the target path on the disk, thus completing the whole alarm information and forensic file storage task.

[0050] Based on the above description, the application constructs a set of traffic scheduling and reorganization detection and evidence collection device in the vpp framework to deal with large flow high concurrency scene, fully utilizes the mechanism of vector processing of vpp, efficiently utilizes the i-cache and d-cache cache of cpu, improves the system processing performance; The data structure such as session structure, packet structure and forensic packet structure adopts the memory pre-allocation scheme, and when processing traffic, it is not necessary to allocate memory dynamically, so as to avoid locking and improve the system processing performance; Reverse packet cross-core scheduling makes the packets of the same session processed by the same cpu core, in the large concurrency (more than 100w) flow scene, the cpu cores are processed in parallel and are free of locking, which improves the system performance; TCP flow reorganization combines TCP segments into a large data segment before threat detection, reduces the number of threat detection interface calls, and improves the threat detection performance; The pre-stored packet mechanism ensures that the whole session packet can be obtained after detecting the threat, and no separate additional trace storage component is needed, which has good real-time effect and good economy.

[0051] In addition, the application Figure 2 A high-performance flow reorganization detection and evidence collection system based on VPP is provided for the embodiment of the application. As shown in Figure 2 The system provided by the embodiment of the application mainly comprises: The VPP framework 200, the session management handoff node 210, the packet forensic cache node 220, the TCP reorganization node 230 and the threat detection and alarm node 240 are added under the ip4-sv-reassembly-feature node of the VPP framework 200; The session management handoff node 210 is used to create a local session table for each CPU core 300, create a global session table for all CPU cores 300, and create a handoff global hash table; The CPU core 300 is configured to determine whether the local handoff buffer queue has a packet, and when the packet exists, determine whether the packet exists in the handoff global hash table through a hash value corresponding to the packet. When the packet does not exist in the handoff global hash table, the hash value of the packet is added to the handoff global hash table, and a structure of the packet is inserted into the local session table and updated to the global session table. When the packet exists in the handoff global hash table, the packet is processed. The CPU core 300 takes out the packet from the dpdk-input node, determines whether the structure of the packet exists in the global session table, and when the structure of the packet does not exist in the global session table, the structure of the packet is added to the local session table and updated to the global session table. When the structure of the packet exists in the global session table and does not exist in the local session table, the packet is added to the handoff buffer queue of the CPU core 300 corresponding to the packet in the global session table. When the structure of the packet exists in the local session table, the packet is processed. The packet evidence cache node 220 is configured to allocate a packet structure space, allocate the packet of the CPU core 300 to the packet structure space, and add the packet in a preset time period to the evidence packet buffer linked list. The TCP reassembly node 230 is configured to maintain a packet buffer linked list for temporarily storing a packet that fails to be reassembled for each TCP session involved in the CPU core 300. When the TCP flow data continuously reassembles the packet, the reassembled packet is combined into a data segment. The threat detection and alarm node 240 is configured to determine whether the data segment has a preset threat through an internally set threat feature knowledge base and a threat matching engine. When the preset threat exists, the evidence packet buffer linked list is taken out and stored into a PCAP file. When no threat is detected after a preset time period, the packet in the evidence packet buffer linked list is released, and the latest packet in the preset time period in the packet structure space is copied to the evidence packet buffer linked list.

[0052] The above is the method embodiment in the application. Based on the same inventive concept, the application embodiment also provides a high-performance flow reassembly detection evidence collection device based on VPP. As shown in the figure, the device comprises a processor and a memory having executable code stored thereon, which, when executed, causes the processor to perform a high-performance flow reassembly detection evidence collection method based on VPP as described in the above embodiment. Figure 3

[0053] Specifically, the server side adds a session management handoff node, a packet evidence cache node, a TCP reassembly node, and a threat detection and alarm node under an ip4-sv-reassembly-feature node of the VPP framework. ​The session management handoff node creates a local session table for each CPU core, a global session table for all CPU cores, and a handoff global hash table; The CPU core determines whether the local handoff buffer queue has a message, and when the message exists, determines whether the message exists in the handoff global hash table through a hash value corresponding to the message; when the message does not exist in the handoff global hash table, adds the hash value of the message to the handoff global hash table, inserts a structure body of the message into the local session table, and updates the global session table; when the message exists in the handoff global hash table, processes the message; The CPU core takes out the message from the dpdk-input node, determines whether the structure body of the message exists in the global session table, when the structure body does not exist in the global session table, adds the structure body of the message to the local session table and updates the global session table; when the structure body exists in the global session table but does not exist in the local session table, adds the message to the handoff buffer queue of the CPU core corresponding to the message in the global session table; when the structure body exists in the local session table, processes the message; Through the message evidence cache node, a message structure body space is allocated, the message of the CPU core is allocated to the message structure body space, and messages in a preset time period are added to an evidence message buffer linked list; Through the TCP recombination node, a message buffer linked list for temporarily storing messages that fail to be recombined is maintained for each TCP session involved in the CPU core; when the TCP flow data continuously recombines the messages, the recombined messages are combined into a data segment; Through a threat feature knowledge base and a threat matching engine arranged in the threat detection and alarm node, it is determined whether the data segment has a preset threat; When the preset threat exists, the evidence message buffer linked list is taken out and stored into a PCAP file; When no threat is detected after a preset time period, the messages in the evidence message buffer linked list are released, and the latest messages in the preset time period in the message structure body space are copied to the evidence message buffer linked list.

[0054] In addition, the embodiment of the application further provides a nonvolatile computer storage medium, which has executable instructions stored thereon, and the executable instructions, when executed, implement the high-performance flow recombination detection evidence method based on VPP.

[0055] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high-performance traffic reassembly detection and forensics method based on VPP, characterized in that, The method includes: Add a session management handoff node, a packet forensics cache node, a TCP reassembly node, and a threat detection and alarm node under the ip4-sv-reassembly-feature node in the VPP framework; The session management handoff node creates a local session table for each CPU core, a global session table for all CPU cores, and a global handoff hash table. The CPU core checks if a message exists in the local handoff buffer queue. If the message exists, it uses the hash value corresponding to the message to determine if it exists in the global handoff hash table. If it does not exist in the global handoff hash table, it adds the hash value of the message to the global handoff hash table, inserts the message structure into the local session table, and updates the global session table. If the message exists in the global handoff hash table, it processes the message. The CPU core retrieves the packet from the dpdk-input node, determines whether the packet's structure exists in the global session table, and adds the packet's structure to the local session table and updates the global session table if it does not exist in the global session table. If it exists in the global session table but not in the local session table, the packet is added to the handoff buffer queue of the CPU core corresponding to the packet in the global session table. If it exists in the local session table, the packet is processed. By using the message evidence caching node, message structure space is allocated, messages from the CPU core are allocated to the message structure space, and messages within a preset time period are added to the evidence message buffer list. Through the TCP reassembly node, a cache list of packets that have failed to be reassembled is maintained for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments. The threat detection and alerting nodes use a built-in threat signature knowledge base and threat matching engine to determine whether a data segment contains a preset threat. When a pre-defined threat is present, the evidence collection message buffer list is retrieved and stored as a PCAP file; If no threat is detected after a preset time period, the messages in the forensic message buffer list are released, and the latest message in the preset time period in the message structure space is copied to the forensic message buffer list.

2. The high-performance traffic reassembly detection and forensics method based on VPP according to claim 1, characterized in that, The CPU core retrieves packets from the dpdk-input node, specifically including: Add an intelligent scheduling module to the existing dpdk-input node; By adding an intelligent scheduling module, one CPU core is selected from all CPU cores as the distribution core, and the other CPU cores are the working cores; the number of packets received by the working core each time is configured. The distribution core retrieves all packets from the network interface hardware queue of the dpdk-input node; The intelligent scheduling module creates a software queue for each working core and configures the queue length and queue scheduling threshold for each software queue. The distribution core distributes all packets evenly to each software queue; During the distribution process, when the queue utilization rate of the software queue exceeds the queue scheduling threshold, the message that should have been distributed to the software queue exceeding the queue scheduling threshold is inserted into the preset shared queue. When insertion into the pre-defined shared queue fails, the message is discarded. The working core retrieves packets from the software queue corresponding to the dpdk-input node. When the actual number of packets received each time for a preset number of consecutive times is less than the configured number of packets received each time, the control working core retrieves packets from the preset shared queue.

3. The high-performance traffic reassembly detection and forensics method based on VPP according to claim 1, characterized in that, The existence of a message in the global hash table is determined by its corresponding hash value. This includes: Obtain message information; construct clib_bihash_40_8 matching information based on message information; Check if the clib_bihash_40_8 matching information exists in the handoff global hash table.

4. The high-performance traffic reassembly detection and forensics method based on VPP according to claim 1, characterized in that, Determine if the message structure exists in the global session table. If it does not exist, add the message structure to the local session table and update the global session table. If it exists in the global session table but not in the local session table, add the message to the handoff buffer queue of the CPU core corresponding to the message in the global session table. Specifically, this includes: Determine if the message structure exists in the global session table. If it does not exist in the global session table, mark the message as a forward message, add the message structure to the local session table, and update the global session table. When the packet exists in the global session table but not in the local session table, obtain the packet direction; when the packet direction is a reverse packet, obtain the CPU core corresponding to the reverse packet from the global session table; when the corresponding CPU core exists, add the packet to the handoff buffer queue of the corresponding CPU core. When a reverse message is first obtained but no corresponding CPU core exists, the CPU core of the forward message corresponding to the reverse message is obtained; the message is added to the handoff buffer queue of the CPU core of the forward message; when the message direction is a forward message, the CPU core of the forward message is obtained from the global session table; the message is added to the handoff buffer queue of the CPU core of the forward message.

5. The high-performance traffic reassembly detection and forensics method based on VPP according to claim 1, characterized in that, Through the message forensics cache node, message structure space is allocated, and messages from the CPU core are allocated to the message structure space, specifically including: By using the message forensics cache node, initialize the local memory buffer pool of the CPU core and create a global message buffer pool; Allocate message structure space in the local memory buffer pool. If allocation in the local memory buffer pool fails, allocate message structure space corresponding to the current CPU core from the global message buffer pool. Allocate the message corresponding to the current CPU core to the message structure space corresponding to the current CPU core.

6. The high-performance traffic reassembly detection and forensics method based on VPP according to claim 1, characterized in that, Through the TCP reassembly node, a cache list of packets that failed to be reassembled is maintained for each TCP session involved in the CPU core. As TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments, specifically including: The TCP reassembly node maintains a message buffer linked list for each direction of each TCP session involved in the CPU core; Based on the seq and ack fields in the TCP stream header, determine the corresponding packet buffer list; When TCP stream data overlaps, the latest TCP stream data is used to overwrite the old TCP stream data; As TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments according to a preset data segment size.

7. The high-performance traffic reassembly detection and forensics method based on VPP according to claim 1, characterized in that, By utilizing the threat signature knowledge base and threat matching engine built into the threat detection and alerting node, it is determined whether a data segment contains a preset threat, specifically including: The threat matching engine performs multi-modal string matching between the data segment and the data in the threat feature knowledge base. When a match is successful, the threat corresponding to the matching data in the threat feature knowledge base is determined to be the preset threat; If a match fails, it is determined that there is no pre-defined threat.

8. A high-performance traffic reassembly detection and forensics system based on VPP, characterized in that, The system includes: The VPP framework adds a session management handoff node, a packet forensics cache node, a TCP reassembly node, and a threat detection and alarm node under the ip4-sv-reassembly-feature node. The session management handoff node is used to create a local session table for each CPU core, a global session table for all CPU cores, and a global handoff hash table. The CPU core is used to determine whether a message exists in the local handoff buffer queue. If the message exists, it uses the hash value corresponding to the message to determine whether it exists in the global handoff hash table. If it does not exist in the global handoff hash table, it adds the hash value of the message to the global handoff hash table, inserts the message structure into the local session table, and updates the global session table. If it exists in the global handoff hash table, the message is processed. The CPU core retrieves the message from the dpdk-input node, determines whether the message structure exists in the global session table, and if it does not exist in the global session table, it adds the message structure to the local session table and updates the global session table. If it exists in the global session table but not in the local session table, it adds the message to the handoff buffer queue of the CPU core corresponding to the message in the global session table. If it exists in the local session table, the message is processed. The message evidence caching node is used to allocate message structure space, allocate CPU core messages to message structure space, and add messages within a preset time period to the evidence message buffer list. The TCP reassembly node is used to maintain a cache list of packets that have failed to be reassembled for each TCP session involved in the CPU core; as TCP stream data continues to reassemble packets, the reassembled packets are merged into data segments. The threat detection and alarm node is used by the built-in threat signature knowledge base and threat matching engine to determine whether a preset threat exists in the data segment. When a preset threat exists, the evidence collection message buffer list is retrieved and stored as a PCAP file. When no threat is detected after a preset time period, the messages in the evidence collection message buffer list are released, and the messages in the latest preset time period in the message structure space are copied to the evidence collection message buffer list.

9. A high-performance flow reconstruction detection and evidence collection device based on VPP, characterized in that, The device includes: processor; And a memory having executable code stored thereon, which, when executed, causes the processor to perform a high-performance traffic reassembly detection and forensics method based on VPP as described in any one of claims 1-7.

10. A non-volatile computer storage medium, characterized in that, It stores computer instructions, which, when executed, implement a high-performance traffic reassembly detection and forensics method based on VPP as described in any one of claims 1-7.

Images