A sketch-guided adaptive network traffic storage and retrieval method and system

Abstract

The application relates to a Sketch-guided adaptive network traffic storage and retrieval method and system, the system comprising a data packet capturing module, a data packet analysis module, a data packet filtering module, a Sketch shunting decision module, a flow table establishment and updating module, a data packet write cache module, a data flow write cache module, an index establishment and updating module and a data packet retrieval module, each module working in a pipeline mode. The application uses an efficient Sketch data structure to distinguish long flows and short flows, the long flows adopt flow-level storage to ensure retrieval efficiency, and the short flows return to packet-level storage to reduce memory pressure, so as to solve the fundamental contradiction that retrieval efficiency and memory consumption cannot be compatible in high-speed network traffic storage.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, and in particular relates to a Sketch-oriented adaptive network traffic storage and retrieval method and system. Background Technology

[0002] Real-time capture and efficient retrieval of massive data packets has become a core challenge in the fields of network forensics and traffic analysis. Existing research can be divided into per-packet storage and retrieval and per-flow storage and retrieval, depending on the granularity of data processing.

[0003] I. Per-packet Storage and Retrieval

[0004] In scenarios requiring complete data recording, data packets are typically persisted to disk strictly in time-ordered fashion. This storage model, which does not alter the arrival order of data packets, is known as a packet-based storage and retrieval model, such as... Figure 1 As shown in (a), upon arrival, data packets are directly stored in a pre-allocated data packet write buffer, and a data packet index is built. Then, the original data packets and the index are permanently stored in a disk file. While this method can accurately recreate network events, it faces the daunting challenge of finding a needle in a haystack. To achieve interactive responses in TB-level or even PB-level data, research has focused primarily on the design of index structures, from general indexes to bitmap indexes, and then to sorted array indexes and inverted indexes optimized for network characteristics.

[0005] The core concept of bitmap indexing is to leverage native bitwise operations to accelerate multi-condition combined queries. pcapIndex was the first to introduce bitmap indexing into this field, achieving sub-second-faster retrieval responses by establishing a mapping between attribute values ​​and bits. However, this method, when used for high cardinality attributes (such as port numbers or IP addresses), leads to index size bloat. Therefore, the academic community has proposed methods to compress bitmap indexes using algorithms such as WAH, PLWAH, and COMPAX. However, these compression algorithms are very CPU-intensive, requiring a balance between index memory usage and CPU utilization. Unlike bitmap indexing's reliance on Boolean operations, some researchers have proposed an indexing path based on sorted attribute arrays (SAA). When processing metadata with fixed patterns, sorted arrays directly maintain multiple sorted field arrays and perform binary search, effectively solving the index size bloat problem. However, as data volume surges, the sorting overhead required to maintain the array's order and the frequency of memory accesses significantly drag down real-time capture performance. Therefore, new research is shifting towards more space-efficient inverted index variants. For example, Compact-Index maps the shortest disjoint prefixes to reduce memory accesses and improve cache hit rate. By dynamically extracting prefixes through a compressed hash trie and combining it with offset list compression technology based on common byte prefixes, disk I / O bottlenecks are greatly alleviated.

[0006] However, storing data packets results in extremely scattered data distribution, and retrieving specific traffic often triggers a large number of random seeks, severely limiting retrieval speed. To address this issue, researchers have leveraged the long-tail effect of network traffic and shifted to stream-level storage.

[0007] II. Per-flow storage and retrieval

[0008] For data stream-level storage and retrieval, the indexing and retrieval methods can reuse those for data packets. The key issue is how to store network traffic information at the stream level. Based on the content being stored, strategies are mainly divided into: truncation strategies, filtering strategies, and full-traffic storage strategies.

[0009] The Time Machine method assumes that subsequent loads in a large flow are often redundant, therefore it employs a truncation restriction strategy for each flow, recording only the first N bytes of each connection. In actual testing, by setting a 20 KB truncation threshold, the system can discard an average of 98% of the total traffic, retaining only the connection start portion most critical for forensics. This strategy significantly extends the retention time of raw data packets on disk; in laboratory deployments, data availability increased from 3 hours to 4 days, a 32-fold improvement. FlowScope employs a filtering design strategy similar to a digital storage oscilloscope. Its core logic is to continuously store captured raw data packets into a circular buffer in memory. Only when a preset trigger event occurs will it jump to a specified historical time point according to a defined dump filter, extract specific flow data that meets the filtering conditions, and write it to persistent storage. Before this, the data exists in memory only as a raw packet sequence. FloSIS, on the other hand, argues that this lossy capture would miss key characteristics of slow attacks or data leaks, therefore advocating for a lossless full-flow capture method based on the data stream. The discontinuous distribution of data packets within the same stream on the disk is considered the root cause of slow retrieval. Therefore, by buffering all data packets belonging to the same stream in memory and storing them contiguously on the physical disk, the goal of reading the entire stream in a single disk seek can be achieved.

[0010] However, with the increase in network traffic, the use of the core mechanism of data stream-level storage and retrieval, the stream buffer, has exposed a serious memory management bottleneck. Figure 1 The area shown in the red dashed box in (b) highlights: (1) The dramatic increase in stream buffers. To achieve full data storage at the stream level, it is necessary to maintain an independent buffer for each active stream in memory, and the buffer is generally maintained by pre-allocation + dynamic allocation. As network traffic increases, the number of concurrent streams grows exponentially, and the memory consumption of the buffer will increase rapidly with the large number of stream nodes. (2) Serious waste of stream buffers. The size of a single stream buffer is generally set to the average length of the stream, such as FloSIS, where the size of a single stream buffer is set to 32KB. However, statistics show that about 60.6% of streams are smaller than 4KB, which are "rat streams". Allocating independent stream buffers for these fleeting small streams not only consumes most of the memory overhead, but also easily causes random memory processing delays and increases the risk of packet loss. Summary of the Invention

[0011] The purpose of this invention is to provide a Sketch-oriented adaptive network traffic storage and retrieval method and system, such as... Figure 1 As shown in (c), sketch distinguishes data streams into two types: packet-based storage and stream-based storage. Only when the number of data packets exceeds the sketch's preset value is a stream table created for stream-based storage. This reduces... Figure 1The memory portion shown in the red box in (b) is used to address the trade-off between retrieval efficiency and memory consumption in existing high-speed network traffic storage.

[0012] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a Sketch-oriented adaptive network traffic storage method, the method comprising:

[0013] Capture network traffic in real time and extract the 5-tuple of the data packets. The 5-tuple includes the source IP address, destination IP address, source port, destination port, and protocol.

[0014] Using the quintuple of data packets as input, the storage decision results are generated based on the probabilistic sketch decision algorithm. The storage decision results include "storage by packet" and "storage by stream".

[0015] In response to storage decisions, network traffic is stored and an index file is created.

[0016] As a further improvement to one embodiment of the present invention, the step of generating and storing decision results based on the probabilistic sketch decision algorithm, using the quintuple of a data packet as input, includes:

[0017] Create a two-dimensional counter array, where each row of the array is associated with an independent hash function;

[0018] In response to the hash function associated with the array, calculate the hash value corresponding to the quintuple of the data packet to update the array;

[0019] If none of the counter values ​​corresponding to the stream have reached the set decision threshold, the stream is determined to be a "short stream candidate" and the storage decision result is "stored by packet"; otherwise, the stream is determined to be a "long stream" and the storage decision result is "stored by stream".

[0020] As a further improvement to one embodiment of the present invention, the network traffic storage in response to a storage decision includes,

[0021] ① The storage decision result is "stream-based storage".

[0022] Create or find a "stream node" structure. Stream nodes are used to continuously cache all subsequent data packets belonging to the stream in memory until the conditions for writing to disk are met.

[0023] Write the consecutive data packets of the stream to a global, shared data stream write buffer, and at the same time generate a stream-level metadata record pointing to the consecutive data packets;

[0024] ② The storage decision result is "store by package".

[0025] Data packets are stored in a global, shared data packet write buffer in the order of arrival, and a discrete packet-level metadata record is generated for each data packet.

[0026] As a further improvement to one embodiment of the present invention, the step of storing network traffic and establishing an index file in response to a storage decision includes,

[0027] In response to storage decisions, network traffic is stored into corresponding storage blocks. Each storage block includes an index file, a Packet-data.pcap file that stores network traffic in the order of packet arrival, and a Flow-data.pcap file that stores network traffic sequentially by flow.

[0028] The index files include source IP index files, destination IP index files, source port index files, and destination port index files.

[0029] As a further improvement to one embodiment of the present invention, the dynamic adjustment step of the decision threshold includes,

[0030] Periodically monitor the utilization rate of the stream storage unit memory pool;

[0031] If the current utilization rate is greater than the preset maximum value, the current decision threshold will be increased by the set step size.

[0032] If the current usage rate is less than the preset minimum, check whether the adjustment direction of the decision threshold in the historical monitoring period is not decreasing. If it is not decreasing, reduce the current decision threshold by the set step size; otherwise, keep the current decision threshold.

[0033] If the current usage rate is within the range of [preset minimum value, preset maximum value], then maintain the current decision threshold.

[0034] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a Sketch-oriented adaptive network traffic storage system, the system comprising:

[0035] The packet capture module is used to capture network traffic in real time.

[0036] The packet parsing and filtering module is used to extract the five-tuple of data packets and filter traffic according to preset rules;

[0037] The Sketch triage decision module is used to generate and store decision results based on the probabilistic Sketch decision algorithm, taking the five-tuple of the data packet as input.

[0038] The caching module is used to store network traffic and build index files in response to storage decisions.

[0039] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a Sketch-oriented adaptive network traffic retrieval method, comprising:

[0040] Receive user search instructions, the search instructions including source IP address, destination IP address, source port address, and destination port address;

[0041] The target network traffic is retrieved and located from the storage results of the storage method described above.

[0042] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a Sketch-oriented adaptive network traffic retrieval system, the system comprising:

[0043] The instruction receiving module is used to receive user search instructions;

[0044] The packet retrieval module is used to respond to retrieval commands, retrieve and locate target network traffic.

[0045] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, the steps in the storage and retrieval method described above are implemented.

[0046] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps in the storage and retrieval method described above.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) This invention achieves hybrid storage through real-time decision-making via Sketch. It uses efficient packet storage for massive short streams to save memory, and continuous stream storage for critical long streams to optimize retrieval. This effectively solves the problems of low performance in retrieving long streams by packet storage (such as n2disk) and linear memory growth with the number of streams by stream storage (such as FloSIS).

[0049] (2) The core decision module of this invention has a simple structure and extremely low overhead, and can be integrated into the parallel pipeline of existing high-performance traffic capture architectures (such as FloSIS) in a minimally invasive manner. It does not change the final storage format and index structure of the underlying data, but mainly affects the temporary storage path of the data in memory, so it is easy to implement, verify and deploy in actual systems.

[0050] (3) This invention not only provides static decision-making, but also integrates dynamic threshold adjustment. The system can automatically adjust the decision threshold according to the utilization rate of streaming storage resources, maintaining the optimal balance between performance and memory when traffic characteristics change. Attached Figure Description

[0051] Figure 1 This is a diagram comparing three types of traffic storage and retrieval, where (a) is packet storage, (b) is stream storage, and (c) is hybrid storage.

[0052] Figure 2 This is a flowchart of the data storage process described in this invention.

[0053] Figure 3 This is a flowchart of the data retrieval process described in this invention. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0055] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0056] In Embodiment 1 of the present invention, the present invention provides a Sketch-oriented adaptive network traffic storage and retrieval method and system. The core is to transform the probabilistic sketch (Sketch) from a traditional background measurement tool into a decision engine that controls the storage path in real time on the data plane, thereby realizing adaptive memory resource allocation based on flow length.

[0057] The system described in this invention includes: a packet capture module, a packet parsing module, a packet filtering module, a Sketch traffic splitting decision module, a flow table creation and update module, a packet write cache module, a data stream write cache module, an index creation and update module, and a packet retrieval module. These modules work collaboratively in a pipeline manner, and the data storage process is as follows: Figure 2 As shown, the data packet retrieval process is as follows: Figure 3 As shown.

[0058] (1) Packet capture module: used to capture network packets at line speed from the high-speed network interface. RSS technology is used to ensure that packets belonging to the same data stream (defined by the 5-tuple) are distributed to the same processing core, laying the foundation for subsequent flow-aware processing.

[0059] (2) Packet parsing and filtering module: The parsing module extracts key metadata from the captured raw packets, including at least the source IP address, destination IP address, source port, destination port, and protocol (i.e., a 5-tuple) to identify unique network flows. The filtering module can perform preliminary filtering of traffic according to preset rules (such as BPF filters), sending only "interest flows" to the subsequent processing flow.

[0060] (3) Sketch Triage Decision Module: This module is key to the adaptive implementation of this scheme. Internally, it maintains a lightweight probabilistic sketch data structure (such as an improved CM Sketch). For each arriving data packet, the following operations are performed:

[0061] a. Query and Statistics: Use the 5-tuple hash value of the data packet as the key to query the corresponding counter in the sketch.

[0062] b. Real-time decision-making: The decision is made based on the current value of the counter and a preset, dynamically adjusted decision threshold (N). If none of the counter values ​​corresponding to the flow have reached the threshold N, the flow is determined to be a "short flow candidate", and the decision result is "stored by packet"; if the counter value reaches or exceeds the threshold N, the flow is determined to be a "long flow", and the decision result is "stored by stream".

[0063] c. Controlled Traffic Distribution: Based on the above binary decision-making results, data packets are directly guided into the corresponding storage paths. This achieves a functional shift from "measurement" to "control," leveraging Sketch's features to complete real-time storage strategy decisions at the data packet level with extremely low additional latency and memory overhead.

[0064] (4) Flow Table Creation and Update Module: Receives the "Store by Stream" instruction from the Sketch flow splitting decision module. For a 5-tuple determined to be a long flow, it creates or finds a "flow node" structure. This flow node is used to continuously cache all subsequent data packets belonging to this flow in memory until the write conditions are met (such as buffer full, flow end, or timeout). This module is also responsible for the lifecycle management of flow nodes, including a reclamation mechanism based on the last active time to release resources occupied by silent flows.

[0065] (5) Packet Write Buffer Module: Receives the "store by packet" instruction from the Sketch traffic splitting decision module. Stores packets in a global, shared packet write buffer in the order of arrival. At the same time, generates a discrete packet-level metadata record for each packet.

[0066] (6) Data Stream Write Buffer Module: Receives complete or partial stream data cached in the stream node from the stream table creation and update module. When the stream node data is ready, it writes the entire stream's continuous data block into a global, shared data stream write buffer. At the same time, it generates a stream-level metadata record pointing to that continuous data block.

[0067] (7) Index Building and Update Module: This module uniformly manages packet-level metadata from the packet write buffer module and stream-level metadata from the data stream write buffer module. It organizes this metadata into efficient index structures, such as bitmaps, sorted arrays, and inverted sorted arrays. The index records the data packets or data streams in the final storage block (…). Figure 2 The location information (data offset and length) within the block. All metadata is uniformly written to index files: source IP index file, destination IP index file, source port index file, and destination port index file, enabling subsequent retrieval processes to directly locate the mixed-store data.

[0068] (8) Packet Retrieval Module: Responsible for responding to user queries and quickly locating and extracting target network traffic from the mixed-storage disk data. Its core is to achieve efficient retrieval based on a three-layer index filtering mechanism using Block. The specific process is detailed in (see...). Figure 3 )as follows:

[0069] a. Search Command Parsing: The system receives the search command input by the user. The command may include filtering conditions such as time range, source / destination IP address, and source / destination port. In this embodiment, the search command is: / retrieve -s 2026-03-01 12:00:00 -e 2026-03-01 13:00:00 -sip 192.168.1.1 / 16 -dip 128.142.33.1 / 24.

[0070] b. First-level retrieval: Time range filtering. Based on the time range in the query command, the system uses the timestamp index of each block maintained in memory to quickly filter out all blocks with overlapping timestamps, significantly narrowing the search scope. In this embodiment, the timestamps are: start time: 2026-03-01 12:00:00, end time: 2026-03-01 13:00:00.

[0071] c. Second-level retrieval: Five-tuple Bloom filter filtering. For the Block set filtered in the previous layer, the system reads the five-tuple Bloom filter maintained in memory for each Block. The Bloom filter is used to quickly determine whether a Block might contain a stream that meets the query conditions. Blocks that cannot contain the target stream are excluded, resulting in a candidate Block set. In this embodiment, the Bloom filter is for source IP: 192.168.1.1 / 16, destination IP: 128.142.33.1 / 24.

[0072] d. Third-level retrieval: Precise metadata location. For a set of candidate blocks, the system reads its unified index file on disk. This index file contains a mixture of packet-level and flow-level metadata based on source IP index, destination IP index, source port index, and destination port index, as well as the offsets and lengths of this metadata within the block's data file. The system performs precise matching on this index, such as bitmap, binary search, and inverted indexing, to ultimately determine the specific location of the target data packet or data stream in the Packet-data.pcap or Flow-data.pcap file. In this embodiment, the index file is: Source IP: 192.168.1.1 / 16, Destination IP: 128.142.33.1 / 24.

[0073] This invention can be integrated as an optimization module into existing high-performance traffic capture frameworks (such as the FloSIS architecture) in a minimally invasive manner. High-performance traffic capture frameworks typically employ a multi-threaded parallel pipeline design, mainly comprising the following threads:

[0074] (1) Engine Thread. Bound to a specific CPU core and network card receive queue, responsible for packet capture, parsing, classification, and Sketch traffic splitting decisions. Each engine thread has a thread-local decision maker instance and flow table cache to avoid cross-core synchronization overhead.

[0075] (2) Write thread. Responsible for writing the data buffers (including packet buffers and stream buffers) filled by the engine thread to disk in batches sequentially.

[0076] (3) Index thread. Responsible for building the final disk index (such as a bitmap, sorted array, or inverted index).

[0077] (4) Control Thread. An independent low-priority thread is added to the high-performance traffic capture framework to run the dynamic threshold adjuster and the lazy recycling daemon thread, so as to realize the adaptive optimization of the system.

[0078] In one specific embodiment of the present invention, the data packet processing flow of the engine thread is implemented as follows:

[0079] Step 1: Thread Initialization and Resource Binding. Upon system startup, each engine thread is allocated an independent thread context, including: a Sketch decision-maker instance, a flow table (hash table), a packet storage write buffer, and a flow node memory pool. The engine thread is bound to a fixed CPU core and exclusively uses a network interface card receive queue (implemented through symmetric RSS technology) to ensure lock-free parallel processing.

[0080] Step 2: Capture data packets in batches from the network interface card (NIC) queue. The engine thread enters the main loop and reads network data packets in batches from the bound NIC receive queue.

[0081] Step 3: Parse the data packets and extract the 5-tuple information. For each captured data packet, parse it and generate a unique flow key based on the extracted source IP address, destination IP address, source port number, destination port number, and protocol type.

[0082] Step 4: Call the Sketch decision-maker for real-time path decision-making. Using the flow quintuple as input, the Sketch decision-making algorithm (corresponding to the core of this invention) is invoked. Internally, this decision-maker queries a very lightweight counting sketch and, based on a real-time comparison of the currently observed packet count of the flow with the system's global dynamic threshold N, generates a binary decision result: packet storage path or flow storage path.

[0083] Step 5: Based on the decision results, execute storage path splitting, importing data packets into different memory buffers. Stream storage path: Query the thread-local stream table using the stream 5-tuple. If a match is found, obtain the corresponding stream node; otherwise, allocate a new stream node from the memory pool and insert it into the stream table. Then, append the content of the current data packet to the dedicated contiguous buffer of that stream node. Packet storage path: Directly append the current data packet to the engine thread's local, globally shared packet storage write buffer.

[0084] Step 6: Stream Metadata Maintenance and Status Update. For data packets in the stream storage path, update the metadata of their corresponding stream nodes, including: packet count, last active timestamp, buffer write position, etc. This timestamp is used for subsequent lazy garbage collection mechanism checks.

[0085] Step 7: Check and trigger the write operation. After processing a batch of data packets, the engine thread checks the status of the buffers it manages. If the packet storage write buffer is full, or the dedicated buffer of a certain stream node is full (or the corresponding stream has ended), the engine thread transfers ownership of the prepared buffer to the dedicated write thread and triggers the write event. The write thread is responsible for writing the data in the buffer to the disk in batches and sequentially, thereby releasing the memory buffer back to the pool for reuse.

[0086] In one specific embodiment of the present invention, the pseudocode integrating the decision logic flow of the present invention is as follows:

[0087] Input: ctx: Thread-local context, which contains resources such as decision makers, flow tables, and buffers.

[0088] Output: The data packet is stored in the corresponding memory buffer and an asynchronous disk write is triggered.

[0089] 1.procedure ENGINE_THREAD_PROCESSING_LOOP(ctx)

[0090] 2. while SYSTEM_RUNNING do

[0091] 3. / / Step 1: Packet Capture and Parsing

[0092] 4. pkt ← CAPTURE_PACKET(ctx.rx_ring)

[0093] 5. key ← EXTRACT_FIVE_TUPLE(pkt)

[0094] 6. / / Step 2: Sketch Real-time Traffic Distribution Decision (Core)

[0095] 7. storage_path ← SKETCH_DECISION(ctx.sketch_decision, key)

[0096] 8. / / Step 3: Divide the traffic according to the decision path

[0097] 9. if storage_path == PATH_FLOW then

[0098] 10. flow_node←GET_OR_CREATE_FLOW_NODE(ctx.flow_table,key)

[0099] 11. APPEND_PACKET_TO_FLOW_NODE(flow_node, pkt)

[0100] 12. else

[0101] 13. APPEND_PACKET_TO_PACKET_BUFFER(ctx.packet_buffer, pkt)

[0102] 14. end if

[0103] 15. / / Step 4: Buffer Management and Asynchronous Disk Flush Trigger

[0104] 16. if BUFFER_NEEDS_FLUSH(ctx.packet_buffer) or

[0105] 17. BUFFER_NEEDS_FLUSH(ctx.flow_buffer_pool) then

[0106] 18. NOTIFY_WRITER_THREAD(ctx)

[0107] 19. end if

[0108] 20. end while

[0109] end procedure

[0110] In one specific embodiment of the present invention, the implementation method of Sketch traffic splitting decision is as follows:

[0111] (1) Data Structure Definition: The decision maker consists of a two-dimensional counter array Count[d][w] with d rows and w columns. Each row i is associated with an independent hash function hash_i. Unlike traditional counting sketches that use 32 or 64-bit counters, the counter in this invention only needs to take the largest integer bit of log2(N) (N is the decision threshold; for example, only 3 bits are needed when N=8), which can save memory and improve cache hit rate. Its data structure can be defined as follows:

[0112] #define SGAS_L 4 / / Number of sketch lines

[0113] #define SGAS_C 1024 / / Number of sketch columns

[0114] #define SGAS_BIT_CNT 3 / / Counter bit width N=7 (2^3-1)

[0115] struct Sketch_Decision {

[0116] uint8_t Count[SGAS_L][SGAS_C];

[0117] uint32_t seed[SGAS_L];

[0118] _Atomic uint8_t N;

[0119] };

[0120] (2) Decision Algorithm: If none of the counter values ​​corresponding to the flow have reached the threshold N, the flow is determined to be a "short flow candidate", and the decision result is "store by packet"; if the counter value reaches or exceeds the threshold N, the flow is determined to be a "long flow", and the decision result is "store by flow". The implementation steps of the decision method are as follows:

[0121] Step 1: Initialize the decision path. When the system captures a network data packet, it initializes the default storage path corresponding to that data packet to the packet storage path. It also sets a flag indicating whether the conditions for switching to the stream storage path are met.

[0122] Step 2: Calculate the hash position of the flow node. Extract the five-tuple information of the flow (source IP address, destination IP address, source port number, destination port number, protocol type) and generate a unique key. Calculate d hash functions to obtain d hash values, and determine the d column positions of the flow hash value in the sketch two-dimensional counter array.

[0123] Step 3: Query and update the sketch counters. Check the counter values ​​of the stream hash value at the d column positions. If the counter value is less than the current system-set decision threshold N, increment the counter value by 1; if the counter value is greater than or equal to the threshold N, keep the counter value unchanged and continue to check the next counter.

[0124] Step 4: Determine the storage path. After checking d counters, if the counter values ​​on all d rows are greater than or equal to the threshold N, the data packet is determined to be a long flow, and step 5 is executed; otherwise, it is determined to be a short flow or a long flow that has not yet reached the threshold, and step 6 is executed.

[0125] Step 5: Bind the stream storage path and store data packets by stream. Create a stream table node and allocate a 32KB buffer for that stream node. Subsequent data packets of this stream will be determined as the stream storage path and saved to the 32KB buffer.

[0126] Step 6: Store data packets according to the packet storage path. Data packets are stored in the packet storage buffer.

[0127] In one specific embodiment of the present invention, the pseudocode for the real-time decision-making process is as follows:

[0128] Input: data packet pkt, its stream key key, current threshold N, sketch C[d][w], hash function family {h_i}

[0129] Output: Storage path decision (packet mode or streaming mode)

[0130] 1.procedure ONLINE_SKETCH_DECISION(pkt, key, N, C, {h_i})

[0131] 2. / / Initialization: Default packet storage path

[0132] 3. decision ← PATH_PACKET

[0133] 4. all_counters_ge_N ← true

[0134] 5. / / Single traversal, combining query and update operations

[0135] 6. for i ← 0 to d-1 do

[0136] 7. pos ← h_i(key) mod w

[0137] 8. if C[i][pos] < N then

[0138] 9. all_counters_ge_N ← false

[0139] 10. C[i][pos] ← C[i][pos] + 1

[0140] 11. end if

[0141] 12. end for

[0142] 13. / / Decision point: Determine whether to trigger stream storage binding.

[0143] 14. if all_counters_ge_N == true then

[0144] 15. decision ← PATH_FLOW

[0145] 16. / / Instant Binding: Enables efficient streaming storage for this stream.

[0146] 17. flow_node ← GET_OR_CREATE_FLOW_NODE(key)

[0147] 18. REDIRECT_SUBSEQUENT_PACKETS(key, flow_node)

[0148] 19. end if

[0149] 20. / / Store the current data packet into the corresponding buffer based on the decision.

[0150] 21. if decision == PATH_FLOW then

[0151] 22. APPEND_PACKET_TO_FLOW_NODE(flow_node, pkt)

[0152] 23. else

[0153] 24.APPEND_PACKET_TO_PACKET_BUFFER(pkt)

[0154] 25. end if

[0155] 26. return decision

[0156] end procedure

[0157] In one specific embodiment of the present invention, to enable the system to adapt to dynamically changing network traffic, the present invention implements dynamic threshold adjustment, which is executed periodically by a control thread. The system monitors the core indicator, streaming storage unit utilization (U), and dynamically adjusts the threshold N to stabilize U within a target range (e.g., [70%, 80% of device content capacity]). N is rapidly increased when memory is scarce and cautiously decreased when memory is plentiful, and strategies are set to prevent frequent oscillations of the threshold N. The specific implementation is as follows:

[0158] Step 1: Periodically monitor resource status. The control thread periodically collects current system resource status metrics, including the utilization rate (U) of the stream storage unit memory pool. U is the ratio of the currently allocated stream nodes to the pre-allocated memory capacity.

[0159] Step 2: Calculate and determine the memory usage range. Obtain the current memory usage U and compare it with the system's preset target memory usage range [U_low, U_high]. If U > U_high, proceed to Step 3 (high memory usage branch); if U < U_low, proceed to Step 4 (low memory usage branch); if U_low ≤ U ≤ U_high, proceed to Step 5 (normal memory usage branch).

[0160] Step 3: Increase the threshold. When high memory usage is detected, the system immediately increases the threshold to quickly alleviate memory pressure. The current threshold N_current is increased by a fixed step (usually 1), resulting in a new threshold N_new. The direction of this adjustment is recorded as an "increasing" trend.

[0161] Step 4: Lower the threshold. When memory usage is determined to be too low, the system does not immediately implement a threshold lowering strategy. Instead, it first confirms the trend to prevent frequent threshold oscillations caused by instantaneous traffic fluctuations. It checks whether the system's adjustment direction has not been "decreasing" over the past k consecutive monitoring periods. If this condition is met, it indicates that the memory slack state has been consistently stable, and a threshold lowering adjustment is performed: N_current is reduced by a fixed step (usually 1) to obtain N_new, and the adjustment direction is recorded as a "decreasing" trend. If the condition is not met, N_current remains unchanged.

[0162] Step 5: Keep the threshold unchanged. When memory usage is determined to be within the ideal range, the system considers the current operating point to be optimal and no threshold adjustment is necessary.

[0163] Step 6: Perform boundary protection. Perform a validity check on the new threshold candidate value N_new obtained from the above steps. Limit it to the system's preset range [N_min, N_max].

[0164] In one specific embodiment of the present invention, the pseudocode for the dynamic threshold adjustment process is as follows:

[0165] Input: U: Current utilization rate of the stream memory pool (0 ≤ U ≤ 1). [U_low, U_high]: Target utilization range, representing the ideal operating point of the system (e.g., [0.7, 0.8]). N_current: Current decision threshold. last_trend: Direction of the last threshold adjustment (INCREASING or DECREASING). k: Number of consecutive periods required for trend confirmation (e.g., k=3). [N_min, N_max]: Valid range of threshold values ​​(e.g., [1, 10]).

[0166] Output: N_new: The adjusted new decision threshold. last_trend: The updated adjustment direction.

[0167] 1.procedure DYNAMIC_THRESHOLD_ADAPTATION(U, U_low, U_high, N_current,last_trend, k)

[0168] 2. N_new ← N_current

[0169] 3. new_trend ← last_trend

[0170] 4. / / Scenario A: Memory usage is too high, a "conservative" strategy is needed.

[0171] 5. If U > U_high then

[0172] 6. N_new ← N_current + 1

[0173] 7. new_trend ← INCREASING

[0174] 8. goto BOUNDARY_CHECK

[0175] 9. / / Scenario B: Memory usage is ample; consider switching to an "aggressive" strategy.

[0176] 10. else if U < U_low then

[0177] 11. if last_trend != DECREASING for k consecutive monitoringintervals then

[0178] 12. N_new ← N_current - 1

[0179] 13. new_trend ← DECREASING

[0180] 14. end if

[0181] 15. goto BOUNDARY_CHECK

[0182] 16. / / Scenario C: Memory usage is within the ideal range and remains stable.

[0183] 17. else

[0184] 18. new_trend ← last_trend

[0185] 19. goto BOUNDARY_CHECK

[0186] 20. end if

[0187] 21. BOUNDARY_CHECK:

[0188] 22. / / Ensure the new threshold is within a reasonable preset range to prevent extreme values.

[0189] 23. N_new ← MAX(N_min, MIN(N_new, N_max))

[0190] 24. return (N_new, new_trend)

[0191] end procedure

[0192] In one specific embodiment of the present invention, a lazy garbage collection mechanism is also implemented. This mechanism is executed periodically by a control thread to actively identify and reclaim memory resources occupied by completed or long-term silent streams, converting their data logic into packet storage format and releasing the buffer. This directly optimizes resource utilization and improves resource pool turnover rate through an adaptive optimization method. Specific implementation steps include:

[0193] Step 1: Periodic triggering of scans. This is executed periodically by the control thread, with a round of recycling scans performed at regular intervals (e.g., every 30 seconds). This low-frequency design ensures that the CPU overhead of this mechanism is extremely low.

[0194] Step 2: Identify and collect reclaimable candidate stream nodes. The control thread accesses a global linked list sorted by the last active timestamp of each stream node, checking sequentially starting from the head of the list (i.e., the least active node). If the difference between the current time and the last active timestamp of the stream node is greater than a quiescent threshold (e.g., 30 seconds), it indicates that the stream has ended or has been idle for a long time, and this stream node is marked as a candidate.

[0195] Step 3: Safely remove candidate nodes from the active flow table. For each identified candidate flow node, use atomic operations or a read-copy-update (RCU) mechanism to remove it from the active hash table accessed by the engine thread or mark it as "deleted".

[0196] Step 4: Perform data logic transformation. The cached, contiguous data segments within the hybrid storage block of the candidate stream node are converted into streaming data records. This process only generates new streaming data and does not move or change the physical location or content of the original data packets on disk, ensuring data integrity.

[0197] Step 5: Merge metadata into the global index. The streaming data records generated in the previous step are merged into the streaming storage index structure, and this data will be retrieved through the streaming index path.

[0198] Step 6: Release resources back to the memory pool. Release the stream node structure and its dedicated contiguous data buffer, returning them to the thread-local memory pool. These resources can be immediately reused by newly arriving streams, thereby improving resource pool turnover and overall system throughput.

[0199] In one specific embodiment of the present invention, the pseudocode for the lazy recycling process is as follows:

[0200] Input: global_lru_list: A globally linked list of FlowNodes sorted by their last active timestamp (t_last). T_silent: The silent timeout threshold (e.g., 30 seconds).

[0201] Output: Successfully recycled stream nodes are released back into the memory pool, and their data is converted into packet metadata and added to the main index.

[0202] 1.procedure LAZY_RECLAMATION_DAEMON_ENHANCED(global_lru_list, T_silent)

[0203] 2. while SYSTEM_RUNNING do

[0204] 3. SLEEP(T_scan_interval)

[0205] 4. / / Identify and collect reclaimable candidate nodes

[0206] 5. recycle_list ← EMPTY_LIST()

[0207] 6. LOCK(global_lru_list.lock)

[0208] 7. for each flow_node in global_lru_list from head (oldest) do

[0209] 8. current_time ← GET_CURRENT_TIME()

[0210] 9. / / Judgment condition: Silent timeout (last active time exceeds the threshold)

[0211] 10. if (current_time - flow_node.last_active_ts) > T_silent then

[0212] 11. / / Safely remove from the activity flow table

[0213] 12. ATOMIC_MARK_AS_DELETED(flow_node)

[0214] 13. / / Remove it from the LRU list and add it to the list of items to be reclaimed.

[0215] 14. LIST_DEL(&flow_node.lru_link)

[0216] 15. LIST_ADD_TAIL(&flow_node.lru_link, &recycle_list)

[0217] 16. end if

[0218] 17. if LENGTH(recycle_list) >= MAX_CANDIDATES_PER_SCANthen

[0219] 18. break

[0220] 19. end if

[0221] 20. End for

[0222] 21. UNLOCK(global_lru_list.lock)

[0223] 22. / / Batch processing of candidate nodes

[0224] 23. for each flow_node in recycle_list do

[0225] 24. / / Generate stream data records

[0226] 25. flow_record ← CONVERT_TO_FLOW_RECORD(flow_node)

[0227] 26. / / Merge stream records into the global stream storage index

[0228] 27. MERGE_INTO_FLOW_INDEX(flow_record)

[0229] 28. / / Release resources back to the memory pool

[0230] 29. RELEASE_FLOW_NODE_RESOURCES(flow_node)

[0231] 30. end for

[0232] 31. end while

[0233] end procedure

[0234] In one specific embodiment of the present invention, regardless of whether the data is stored "by packet" or "by stream," it is ultimately organized into a unified storage block. Figure 2 The Block contains: (1) an index file: including source IP index file, destination IP index file, source port index file, and destination port index file, which respectively record packet-level and flow-level metadata, as well as the offset and length of these metadata in the pcap file within the Block. (2) a Packet-data.pcap file: storing the raw data saved in the order of packet arrival. (3) a Flow-data.pcap file: storing the raw data saved continuously in the flow.

[0235] To ensure optimal storage and retrieval efficiency under varying network sizes and traffic characteristics, this invention implements adaptive configuration for storage blocks. The size, quantity, and storage path organization of each file within a block (four index files, Packet-data.pcap, and Flow-data.pcap) are determined through a combination of pre-configuration and dynamic adjustment based on real-time data capture rate, traffic composition analysis, and underlying disk I / O characteristics. The block configuration implementation steps are as follows:

[0236] Step 1: Determine the storage path organization rules for Blocks. The system assigns a directory path with clear semantics to each generated Block. The naming format of this path is: / year / month / day / hour / minute / second / thread number / block. This time-based multi-level directory structure naturally constitutes the first-level time index, enabling subsequent searches to quickly locate data blocks within a specific time range.

[0237] Step 2: Adaptively determine the path depth based on the data capture rate. The path depth (i.e., the time granularity it includes) is directly related to the system's data capture rate to ensure that the amount of data in each time directory is within a reasonable range. For example, if the packet capture rate is approximately 1 Gbps, about 7.5 GB of data is generated per minute. In this case, the path depth can be set to the minute level ( / year / month / day / hour / minute / ), with each minute directory storing a set of approximately 7.5 GB of data blocks. If the packet capture rate reaches 100 Gbps, about 12.5 GB of data is generated per second. In this case, the path depth needs to be set to the second level ( / year / month / day / hour / minute / second / ) to prevent the data volume in a single directory from being too large and affecting the file system performance.

[0238] Step 3: Configure the thread directories based on the number of packet-receiving threads. To fully utilize the multi-core parallel capture capability, the system employs multi-threaded packet capture. The thread number directories in the path are used to distinguish data captured by different engine threads, achieving physical isolation and avoiding write conflicts. For example, to achieve a 1Gbps capture rate, if two packet-receiving threads are needed, two subdirectories, thread0 and thread1, are created under the corresponding time directory. To achieve a 100Gbps capture rate, if 20 packet-receiving threads are needed, 20 subdirectories, thread0 to thread19, are created under the corresponding time directory.

[0239] Step 4: Based on network traffic analysis, disk I / O performance, and system resource settings, determine the size and number of files within the block. The system analyzes representative traffic samples from the target network environment (network traffic can be sampled periodically) to obtain statistical patterns in traffic composition. Combined with the I / O characteristics of the disk device and system resource settings, it determines the size and proportion of each file within the block. This includes the following:

[0240] 1) Traffic Sample Analysis: Acquire and analyze a fixed-size segment of network traffic data. Analyze key parameters to obtain, including: total number of flows, the proportion of long flows with different threshold values ​​N, the average length of long flows, and the proportion of long flows in the total number of bytes. For example, a 1GB pcap file from a campus network has over 100,000 total flows (five-tuple flows). Table 1 shows the proportion of long flows, the proportion of bytes in each flow, and the average length of each flow with different threshold values ​​N.

[0241] Table 1. Analysis of 1GB Traffic on a Campus Network

[0242]

[0243] 2) Block export strategy and size configuration: Based on the I / O characteristics of disk devices, system resource settings, and traffic sample analysis, set the data export strategy and file size limits in the block.

[0244] To intuitively illustrate the Block export strategy and size configuration method, the following assumptions are made regarding system resources and network traffic sampling, and the Block configuration content is obtained based on these assumptions:

[0245] Assumption 1: When exporting 512MB of cache, the disk device's I / O performance is optimal, so the maximum size of the index file and pcap file in the Block is set to 512MB.

[0246] Assumption 2: The packet capture rate is about 1Gbps, and the system only sets up 1 engine thread. A 1GB block storage area can be set up for this engine thread. Then a block storage area can be generated in about 8 seconds. Therefore, 1GB of network traffic can be analyzed in advance or periodically, and the pcap file and index file export strategy can be set for this 1GB block.

[0247] Assumption 3: The threshold N in the current system is 1.

[0248] Based on the above assumptions, the export strategy for the Block storage area is set to export all data in the Block (including the four index files, Flow-data.pcap, and Packet-data.pcap) if one of the following conditions is met:

[0249] 1. Once the number of flow nodes maintained in the flow table reaches 100,000 * 47.68% of the number of long flows, export all data from the Block. Here, 100,000 is the total number of flows obtained from the pre-analysis, and 47.68% is the percentage of long flows when N=1 in Table 1.

[0250] 2. After the system has exported a total of 1GB of cached data, it generates two combined files of 512MB each: Flow-data.pcap or Packet-data.pcap. Then, it exports all the data in the Block.

[0251] It should be noted that condition "1" is set because flow tables are generally preset to a fixed size, and condition "1" is set to prevent flow tables from exceeding the preset size; condition "2" is set to keep the size of the Block storage area stable at around 1GB.

[0252] In summary, to address the contradiction between high-performance retrieval and high memory consumption in streaming storage, this invention uses an efficient Sketch data structure to distinguish between long and short streams. Long streams are stored at the stream level to ensure retrieval efficiency, while short streams revert to packet-level storage to reduce memory pressure, thus achieving hybrid storage. Furthermore, it integrates dynamic threshold adjustment and lazy reclamation mechanisms, enabling the system to automatically adjust the decision threshold (N) based on the utilization rate of streaming storage resources. This maintains an optimal balance between performance and memory usage when traffic characteristics change. Simultaneously, by reclaiming silent streaming resources, resource waste is effectively reduced. This allows the invention to maintain near-optimal performance in long-term, non-stationary real-world network environments.

[0253] In one specific embodiment of the present invention, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the execution of the program on the processor implements the steps in the Sketch-oriented adaptive network traffic storage and retrieval method.

[0254] In one specific embodiment of the present invention, the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps in the Sketch-oriented adaptive network traffic storage and retrieval method.

[0255] Those skilled in the art will clearly understand that the modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0256] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0257] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0258] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A Sketch-oriented adaptive network traffic storage method, characterized in that: include, Capture network traffic in real time and extract the 5-tuple of the data packets. The 5-tuple includes the source IP address, destination IP address, source port, destination port, and protocol. Using the quintuple of data packets as input, the storage decision results are generated based on the probabilistic sketch decision algorithm. The storage decision results include "storage by packet" and "storage by stream". In response to storage decisions, network traffic is stored and an index file is created.

2. The storage method according to claim 1, characterized in that: The process of taking the quintuple of data packets as input and generating and storing decision results based on the probabilistic sketch decision algorithm includes: Create a two-dimensional counter array, where each row of the array is associated with an independent hash function; In response to the hash function associated with the array, calculate the hash value corresponding to the quintuple of the data packet to update the array; If none of the counter values ​​corresponding to the stream have reached the set decision threshold, the stream is determined to be a "short stream candidate" and the storage decision result is "store by packet"; otherwise, the stream is determined to be a "long stream" and the storage decision result is "store by stream".

3. The storage method according to claim 2, characterized in that: The process of storing network traffic in response to a storage decision includes, ① The storage decision result is "stream-based storage". Create or find a "stream node" structure. Stream nodes are used to continuously cache all subsequent data packets belonging to the stream in memory until the conditions for writing to disk are met. Write the consecutive data packets of the stream to a global, shared data stream write buffer, and at the same time generate a stream-level metadata record pointing to the consecutive data packets; ② The storage decision result is "store by package". Data packets are stored in a global, shared data packet write buffer in the order of arrival, and a discrete packet-level metadata record is generated for each data packet.

4. The storage method according to claim 3, characterized in that: In response to a storage decision, network traffic is stored and an index file is created, including: In response to storage decisions, network traffic is stored in corresponding storage blocks. Each storage block includes an index file, a Packet-data.pcap file that stores network traffic in the order of packet arrival, and a Flow-data.pcap file that stores network traffic sequentially by flow. The index files include source IP index files, destination IP index files, source port index files, and destination port index files.

5. The storage method according to claim 2, characterized in that: The dynamic adjustment steps for the decision threshold include, Periodically monitor the utilization rate of the stream storage unit memory pool; If the current utilization rate is greater than the preset maximum value, the current decision threshold will be increased by the set step size. If the current usage rate is less than the preset minimum, check whether the adjustment direction of the decision threshold in the historical monitoring period is not decreasing. If it is not decreasing, reduce the current decision threshold by the set step size; otherwise, keep the current decision threshold. If the current usage rate is within the range of [preset minimum value, preset maximum value], then maintain the current decision threshold.

6. A Sketch-oriented adaptive network traffic storage system as described in any one of claims 1 to 5, characterized in that: include The packet capture module is used to capture network traffic in real time. The packet parsing and filtering module is used to extract the five-tuple of data packets and filter traffic according to preset rules; The Sketch triage decision module is used to generate and store decision results based on the probabilistic Sketch decision algorithm, taking the five-tuple of the data packet as input. The caching module is used to store network traffic and build index files in response to storage decisions.

7. A Sketch-oriented adaptive network traffic retrieval method, characterized in that: include, Receive user search instructions, the search instructions including source IP address, destination IP address, source port address, and destination port address; In the storage result of the storage method as described in any one of claims 1 to 5, the target network traffic is retrieved and located.

8. A Sketch-oriented adaptive web traffic retrieval system applying the retrieval method as described in claim 7, characterized in that: include, The instruction receiving module is used to receive user search instructions; The packet retrieval module is used to respond to retrieval commands, retrieve and locate target network traffic.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps of the storage method as described in any one of claims 1 to 5 or the retrieval method as described in claim 7.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the storage method as described in any one of claims 1 to 5 or the retrieval method as described in claim 7.

Images