Packet flow mixed storage method and device based on first packet identification

Abstract

The invention relates to a packet flow hybrid storage method and device based on first packet identification, and the method comprises the steps: receiving a network data packet from a network interface in real time, analyzing quintuple information, searching a flow table, and judging whether the data packet belongs to an existing flow first packet or a new flow first packet; extracting characteristics of a quintuple, a transmission control protocol flag bit, a message length and an internetwork control message protocol type from the first packet of the new stream, matching through a preloaded configuration file rule, and judging whether the new stream is a single-packet stream or a multi-packet stream; and respectively storing the single packet stream or the multi-packet stream in a unified buffer area or a stream buffer area according to a judgment result, generating a corresponding packet level or stream level index, and moving data in the stream buffer area to the unified buffer area after the stream ends or is overtime. According to the method, memory occupation is remarkably reduced, invalid flow table nodes are reduced, and the memory overflow risk is relieved; unifying buffer area batch sequence disk writing to optimize disk throughput; the dynamic rule is matched with various network environments; a fault-tolerant mechanism corrects misjudgment through subsequent packet detection, and data integrity is ensured.

Description

Technical Field

[0001] This invention belongs to the field of network traffic acquisition technology, and in particular relates to a method and apparatus for packet flow hybrid storage based on first packet identification. Background Technology

[0002] Current network traffic acquisition and backtracking systems generally adopt a stream-based storage architecture. Upon receiving the first packet of a data stream, a flow table node is created, a flow buffer is pre-allocated, and data packets are continuously written to this buffer. After the flow ends or times out, the data is moved to a larger buffer, written to disk in batches, and an index structure based on five-tuples is generated. However, in high-concurrency scenarios such as backbone network egress or network security monitoring, the proportion of single-packet flows (such as TCP handshake failures, UDP probes, and ICMP probes) increases significantly, leading to several drawbacks: First, pre-allocating buffers for each single-packet flow results in low memory utilization and significant waste; second, the surge in a large number of flow table nodes increases memory usage, easily triggering memory overflow or packet loss limiting mechanisms, affecting system stability; furthermore, maintaining a large number of invalid flow nodes that only carry single data packets significantly increases the burden of memory management and reclamation, reducing overall efficiency.

[0003] Therefore, there is an urgent need for a hybrid storage solution that combines stream-based and packet-based storage methods to optimize memory usage, improve system stability, and adapt to the needs of high-speed network environments. Summary of the Invention

[0004] The purpose of this invention is to provide a packet-stream hybrid storage method and apparatus based on first packet identification, so as to solve the problems of memory waste, memory overflow and decreased system stability caused by traditional stream-based storage in high-proportion single-packet-stream scenarios.

[0005] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a packet stream hybrid storage method based on first packet identification, the method comprising:

[0006] The system receives network data packets in real time from the network interface, parses the 5-tuple information, performs flow table lookups, and determines whether the data packet belongs to an existing flow or is the first packet of a new flow.

[0007] The features of the first packet of a new flow are extracted, including the five-tuple, transmission control protocol flag, message length, and Internet control message protocol type. These features are then matched using pre-loaded configuration file rules to determine whether the flow is a single-packet flow or a multi-packet flow.

[0008] Based on the judgment result, single-packet streams or multi-packet streams are stored in a unified buffer or a stream buffer respectively, and corresponding packet-level or stream-level indexes are generated. Stream buffer data is moved to the unified buffer after the stream ends or times out.

[0009] As a further improvement to one embodiment of the present invention, the method further includes, in which the determination of whether the data packet belongs to an existing flow or the first packet of a new flow includes,

[0010] After receiving network data packets in real time from the network interface, zero-copy parsing technology is used to extract the quintuple information, and a hash table is used to store the flow table. The quintuple is used to determine whether the data packet hits an existing flow table node.

[0011] The 5-tuple information includes the source IP address, destination IP address, source port, destination port, and protocol type.

[0012] As a further improvement to one embodiment of the present invention, the method further includes the following features for extracting the five-tuple, Transmission Control Protocol (TCP) flags, message length, and Internet Control Message Protocol (ICP) type from the first packet of the new flow:

[0013] Parse the Ethernet frame header and Internet Protocol header to obtain the 5-tuple;

[0014] Parse the Transmission Control Protocol header to obtain the flag bits;

[0015] Parse the Internet Control Message Protocol header to obtain the type and code;

[0016] Extract the first few bytes of the Transmission Control Protocol or User Datagram Protocol payload for deep protocol identification, and perform rule matching to determine whether the data packet is a single-packet stream or a multi-packet stream.

[0017] As a further improvement to one embodiment of the present invention, the method further includes, in which the matching based on preloaded configuration file rules to determine whether a stream is a single-packet flow or a multi-packet flow includes,

[0018] The preloaded configuration file rules include single or combined rules for protocol type, destination port range, and transmission control protocol flags.

[0019] By comparing the first packet's features with the rules one by one according to the priority of the rules, if any rule is met, it is determined to be a single packet flow; otherwise, it is determined to be a multi-packet flow.

[0020] The system dynamically loads and hot-updates rules to adapt to the single-packet and multi-packet flow requirements in different network environments.

[0021] As a further improvement to one embodiment of the present invention, the method further includes, in which the step of storing single-packet streams or multi-packet streams into a unified buffer or a stream buffer and generating corresponding packet-level or stream-level indexes includes,

[0022] Store single packet streams into a unified buffer and generate packet-level indexes containing 5-tuples, timestamps, and data locations;

[0023] Multi-packet streams are stored in an allocated stream buffer and a stream-level index containing a 5-tuple and a time range is generated; multi-packet stream data is transferred to a unified buffer after the stream terminates or times out.

[0024] As a further improvement to one embodiment of the present invention, the method further includes writing data from the unified buffer sequentially in batches to the storage medium, and when the first packet is mistakenly identified as a single packet flow, triggering flow table reconstruction and index merging by detecting subsequent packets to ensure data integrity and system stability. Specifically, this includes...

[0025] If the first packet is misidentified as a single-packet flow, subsequent packets trigger the creation of new flow table nodes due to flow table mismatch. The first packet is retained in the unified buffer and maintains the packet-level index. Subsequent packets are written to the newly allocated flow buffer and a flow-level index is generated. The packet-level and flow-level indexes are merged through the unified index interface to ensure data integrity.

[0026] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a packet flow hybrid storage device based on first packet identification, the device comprising a flow table lookup module, a feature matching module, and a storage module;

[0027] The flow table lookup module is used to receive network data packets from the network interface in real time, parse the five-tuple information and perform flow table lookup to determine whether the data packet belongs to an existing flow or the first packet of a new flow.

[0028] The feature matching module is used to extract features such as the five-tuple, transmission control protocol flag, message length, and Internet Control Message Protocol type from the first packet of a new flow, and to match them according to the pre-loaded configuration file rules to determine whether it is a single-packet flow or a multi-packet flow.

[0029] The storage module is used to store single-packet streams or multi-packet streams into a unified buffer or a stream buffer respectively according to the determination result and generate corresponding packet-level or stream-level indexes. The stream buffer data is moved to the unified buffer after the stream ends or times out.

[0030] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the packet stream hybrid storage method based on first packet identification as described above.

[0031] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps in the packet-stream hybrid storage method based on first packet identification as described above.

[0032] Compared with existing technologies, this invention provides a packet flow hybrid storage method and apparatus based on first packet identification. It distinguishes between single-packet and multi-packet flows by identifying the first packet. Single-packet flows are directly stored in a unified buffer, while multi-packet flows are allocated to different buffers and then transferred. This significantly reduces memory usage and invalid flow table nodes, mitigating the risk of memory overflow. Batch sequential writing to disk through the unified buffer optimizes disk throughput. Dynamic rule matching adapts to various network environments. A fault-tolerance mechanism corrects misjudgments through subsequent packet detection, ensuring data integrity. A unified index interface supports efficient retrieval, meeting the demands of high-speed networks up to 100Gbps. Attached Figure Description

[0033] Figure 1 This is an overall flowchart of the packet stream hybrid storage method based on first packet identification described in this invention.

[0034] Figure 2 This is a flowchart illustrating the packet-stream hybrid storage method based on first packet identification as described in this invention.

[0035] Figure 3 This is a schematic diagram of the architecture of the packet stream hybrid storage device based on first packet identification as described in this invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0037] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0038] In Embodiment 1 of the present invention, the present invention provides a packet stream hybrid storage method based on first packet identification, such as... Figure 1 As shown, the method includes,

[0039] S1: Receive network data packets from the network interface in real time, parse the 5-tuple information and perform flow table lookup to determine whether the data packet belongs to an existing flow or the first packet of a new flow;

[0040] S2: Extract the five-tuple, Transmission Control Protocol (TCP) flag, message length, and Internet Control Message Protocol (ICP) type features from the first packet of a new flow, and match them using pre-loaded configuration file rules to determine whether it is a single-packet flow or a multi-packet flow;

[0041] S3: Based on the judgment result, store the single-packet stream or multi-packet stream into the unified buffer or stream buffer respectively and generate the corresponding packet-level or stream-level index. The stream buffer data is moved to the unified buffer after the stream ends or times out.

[0042] In one specific embodiment of the present invention, determining whether a data packet belongs to an existing flow or the first packet of a new flow specifically involves:

[0043] After receiving network data packets in real time from the network interface, zero-copy parsing technology is used to extract the quintuple information, and a hash table is used to store the flow table. The quintuple is used to determine whether the data packet hits an existing flow table node.

[0044] The 5-tuple information includes the source IP address, destination IP address, source port, destination port, and protocol type.

[0045] It should be noted that in the step of determining whether a data packet belongs to an existing flow or the first packet of a new flow, the following specific process is used to achieve efficient data acquisition and flow table lookup, so as to ensure that the processing path of the data packet can be quickly determined in a 100Gbps high-speed network environment.

[0046] Furthermore, the network interface used in this invention is typically a network interface card (NIC) that supports high-performance data processing, such as a NIC using the Data Plane Development Kit (DPDK) technology, to meet the throughput requirements of high-speed networks. Received data packets first enter a temporary buffer, awaiting subsequent parsing and processing. To improve data processing efficiency, this invention employs zero-copy parsing technology, avoiding the multiple memory copying operations involved in traditional packet parsing. Specifically, zero-copy parsing quickly extracts the five-tuple information of the data packet, including the source IP address, destination IP address, source port, destination port, and protocol type, by directly accessing the data in the NIC's receive buffer and combining it with a predefined protocol header offset. This information is obtained by parsing the Ethernet frame header and the Internet Protocol (IP) header, ensuring that the parsing process is efficient and accurate.

[0047] Furthermore, based on the extracted quintuple information, the system performs a flow table lookup to determine whether the data packet belongs to an existing flow. The flow table is stored using a hash table structure, where the key is the quintuple and the value is the corresponding flow table node. Each node records the flow's metadata, such as the flow buffer address and status information. The flow table lookup process quickly determines whether an existing flow table node is matched by matching the data packet's quintuple with the key in the hash table. If a match is found, it indicates that the data packet belongs to an existing flow (multi-packet flow), and the system directly writes it to the corresponding flow buffer to maintain data continuity. If a match is not found, it indicates that the data packet is the first packet of a new flow, and further processing of first packet feature extraction and rule matching is required.

[0048] Preferably, to ensure efficient flow table lookup, this invention optimizes the hash table design by selecting efficient hash functions and collision resolution mechanisms (such as linked lists or open addressing) to reduce lookup latency. Furthermore, flow table management supports dynamic expansion and node reclamation to cope with the rapid increase in the number of flows under high concurrency scenarios, reducing memory fragmentation and improving system stability.

[0049] In one specific embodiment of the present invention, the features of extracting the five-tuple, Transmission Control Protocol (TCP) flags, message length, and Internet Control Message Protocol (ICP) type from the first packet of a new flow are specifically as follows:

[0050] Parse the Ethernet frame header and Internet Protocol header to obtain the 5-tuple;

[0051] Parse the Transmission Control Protocol header to obtain the flag bits;

[0052] Parse the Internet Control Message Protocol header to obtain the type and code;

[0053] Extract the first few bytes of the Transmission Control Protocol or User Datagram Protocol payload for deep protocol identification, and perform rule matching to determine whether the data packet is a single-packet stream or a multi-packet stream.

[0054] It should be noted that in the step of extracting features from the first packet of a new flow, features such as the five-tuple, Transmission Control Protocol (TCP) flags, message length, and Internet Control Message Protocol (ICMP) type are obtained through deep parsing of the packet's protocol header. Optional features of the Transmission Control Protocol or User Datagram Protocol (UDP) payload can also be extracted for deep protocol identification, thus providing an accurate basis for rule matching to determine whether the packet is a single-packet or multi-packet flow. This process achieves efficient feature extraction in a 100Gbps high-speed network environment, ensuring that the system can quickly distinguish between different types of traffic.

[0055] Furthermore, the feature extraction process first obtains the five-tuple information, including the source IP address, destination IP address, source port, destination port, and protocol type, by parsing the Ethernet frame header and Internet Protocol (IP) header. The parsing process employs zero-copy parsing technology, directly accessing the data in the network interface card's receive buffer and quickly locating the five-tuple fields using predefined protocol header offsets. This avoids the memory copying overhead of traditional parsing, significantly improving parsing efficiency. When parsing the Ethernet frame header, the system extracts the IP protocol field to determine the subsequent protocol type; when parsing the IP header, it extracts the source IP, destination IP, source port, destination port, and protocol type fields to form a five-tuple, which serves as the basic identifier for flow table lookup and rule matching.

[0056] Furthermore, for TCP packets, the system further parses the Transmission Control Protocol header to extract flag information, including SYN, ACK, FIN, RST, etc., to determine the connection status of the packets. For example, the first packet with the SYN flag set to 1 and the ACK flag set to 0 usually indicates the initiation of a TCP connection, which may be a single-packet stream (such as in a handshake failure scenario). The parsing process quickly obtains relevant information by locating the flag fields in the TCP header, ensuring high efficiency.

[0057] Furthermore, for ICMP packets, the system parses the Internet Control Message Protocol header to extract the type and code fields to identify common single-packet flow behaviors (such as network probing). ICMP header parsing is also based on offset positioning, directly reading the type and code fields, and combining them with 5-tuple information to support subsequent rule matching.

[0058] Preferably, to enhance the accuracy of protocol identification, the system can optionally extract the first few bytes of the TCP or UDP payload for deep protocol identification. For example, for UDP packets, the first few bytes of the payload can be extracted to identify specific protocol features; for TCP packets, the payload can be analyzed to identify application layer protocol features. These payload features are compared with predefined protocol templates to further confirm whether the packet belongs to a single-packet flow scenario (such as short DNS packets). Deep protocol identification employs an efficient byte matching algorithm to ensure rapid feature extraction in high-throughput environments.

[0059] In one specific implementation scenario of the present invention, a set of predefined rules is loaded from an external YAML configuration file when the system starts up. An example of the rules is as follows:

[0060] rules:

[0061] - name: "TCP SYN without ACK"

[0062] protocol: "TCP"

[0063] conditions:

[0064] tcp_flags:

[0065] {

[0066] syn: 1,

[0067] ack: 0,

[0068] fin: 0,

[0069] rst: 0

[0070] }

[0071] length_lt: 0

[0072] action: 1

[0073] - name: "UDP short packet common ports"

[0074] protocol: "UDP"

[0075] conditions:

[0076] dst_ports: [53, 123, 161]

[0077] length_lt: 80

[0078] action: 1

[0079] - name: "ICMP echo or unreachable"

[0080] protocol: "ICMP"

[0081] conditions:

[0082] icmp_types: [8, 3]

[0083] length_lt: 0

[0084] action: 1

[0085] Where length_lt: 0 represents a mismatch length, action: 1 represents a single packet stream, and 0 represents a multi-packet stream.

[0086] In one specific embodiment of the present invention, matching is performed using preloaded configuration file rules to determine whether it is a single-packet flow or a multi-packet flow. Specifically,

[0087] The preloaded configuration file rules include single or combined rules for protocol type, destination port range, and transmission control protocol flags.

[0088] By comparing the first packet's features with the rules one by one according to the priority of the rules, if any rule is met, it is determined to be a single packet flow; otherwise, it is determined to be a multi-packet flow.

[0089] The system dynamically loads and hot-updates rules to adapt to the single-packet and multi-packet flow requirements in different network environments.

[0090] It should be noted that in the step of matching data packets to determine whether they are single-packet or multi-packet flows using pre-loaded configuration file rules, efficient and accurate traffic classification is achieved through processes such as rule loading, feature comparison, priority matching, and dynamic updates. This supports the differentiated storage requirements of single-packet and multi-packet flows in high-speed network environments of hundreds of Gbps. This step, based on the results of first packet feature extraction, utilizes the flexible design and dynamic adjustment mechanism of configuration file rules to ensure that the system can adapt to diverse network environments and protocol types.

[0091] Furthermore, upon startup, the system loads a set of predefined rules from an external configuration file, preferably stored in YAML format to ensure the rules' structure, readability, and extensibility. The configuration file rules include single or combined conditions such as protocol type (e.g., TCP, UDP, ICMP), destination port range, and TCP flag combinations. A single condition refers to a constraint on a single feature, such as the protocol type being ICMP; a combined condition refers to a joint constraint on multiple features, such as the TCP flag SYN=1 and the packet length being below a certain threshold. These rules are designed based on common single-packet flow behaviors, such as TCP handshake failures, DNS queries, and ICMP probes, effectively distinguishing between single-packet flows (e.g., a single request packet) and multi-packet flows (e.g., a persistent session flow). The diversity and flexibility of the rules ensure that the system can cover various network traffic scenarios.

[0092] Furthermore, during the rule matching process, the system compares the features extracted from the first packet (such as the 5-tuple, TCP flags, packet length, ICMP type, etc.) with the conditions of each rule one by one, according to the priority order of the rules in the configuration file. The comparison process employs efficient matching algorithms, such as fast lookups based on hash or tree structures, to reduce computational overhead in high-throughput environments. If the features of the first packet satisfy all conditions of any rule (such as matching TCP SYN without ACK or UDP short packets), the packet is determined to be a single-packet flow; if none of the rules are satisfied, it is determined to be a multi-packet flow. The matching result directly determines the subsequent storage path: single-packet flows are stored in a unified buffer, and multi-packet flows are stored in a stream buffer. This priority matching mechanism ensures that high-priority rules (such as short packet rules for specific protocols) take precedence over general rules, improving the accuracy of the determination.

[0093] Preferably, to adapt to the single-packet and multi-packet flow judgment requirements of different network environments, the system supports dynamic loading and hot updating of rules. Dynamic loading allows rules to be reloaded from external configuration files while the system is running, without interrupting service; hot updating supports real-time modification of rule content without restarting the system, such as adding feature conditions for emerging protocols (e.g., QUIC, HTTP / 3) or adjusting port ranges to adapt to specific network scenarios (e.g., 5G core network or data center). Rule updates are implemented through efficient memory management and synchronization mechanisms, ensuring that the impact of the update process on system performance is minimized. The dynamic loading and hot updating mechanisms enhance the system's adaptability to new protocols, abnormal traffic, or changes in the network environment by periodically or triggering updates to the rule base, thereby improving the robustness and universality of judgments.

[0094] In one specific embodiment of the present invention, single-packet streams or multi-packet streams are stored in a unified buffer or a stream buffer, and corresponding packet-level or stream-level indexes are generated. Specifically,

[0095] Store single packet streams into a unified buffer and generate packet-level indexes containing 5-tuples, timestamps, and data locations;

[0096] Multi-packet streams are stored in an allocated stream buffer and a stream-level index containing a 5-tuple and a time range is generated; multi-packet stream data is transferred to a unified buffer after the stream terminates or times out.

[0097] It should be noted that in the steps of storing single-packet or multi-packet streams to a unified buffer or stream buffer and generating corresponding packet-level or stream-level indexes, optimized storage of single-packet and multi-packet streams is achieved through differentiated storage path selection and an efficient index generation mechanism, minimizing memory consumption and ensuring efficient data retrieval. This step, based on the first packet identification result, designs different storage strategies for single-packet and multi-packet streams while generating a unified index structure to support traffic storage and backtracking requirements in a 100Gbps high-speed network environment.

[0098] Furthermore, for packets identified as single-packet flows, the system does not create flow table nodes but directly stores them in a unified buffer. This unified buffer, a large-capacity memory area, aggregates all single-packet flows and migrated multi-packet flow data to achieve efficient batch writes to disk. During storage, the system generates a packet-level index for each single-packet flow packet. The index includes a 5-tuple (source IP address, destination IP address, source port, destination port, protocol type), a timestamp, and the data's storage location in the unified buffer (such as offset and length). The packet-level index uses a hash table combined with a linked list structure, using the 5-tuple as a key to support fast insertion and query operations, ensuring efficient recording of single-packet flow location information in high-concurrency scenarios, facilitating subsequent retrieval and backtracking. The design of directly storing single-packet flows in a unified buffer avoids allocating independent buffers for individual packets, significantly reducing memory waste, and is particularly suitable for network environments with a high proportion of single-packet flows.

[0099] Furthermore, for packets identified as multi-packet flows, the system first creates a flow table node and allocates a fixed-size flow buffer to temporarily store all subsequent packets of that flow. The flow buffer is managed using a memory pool technique, which avoids frequent memory allocation and deallocation operations by pre-allocating fixed-size memory blocks, thereby reducing memory fragmentation and improving allocation efficiency. Subsequent packets are written to the flow buffer in sequence to maintain flow continuity and support flow-level data analysis and backtracking requirements. When a flow terminates (e.g., upon receiving a TCP FIN or RST flag), times out, or the flow buffer is full, the system transfers all data in the flow buffer to a unified buffer and generates a flow-level index for that flow. The flow-level index records the 5-tuple, the start and end time range of the flow, and the data's storage location in the unified buffer (e.g., the offset of the data block). The flow-level index also uses a hash table structure to ensure fast lookups and is compatible with the packet-level index, supporting calls to the unified retrieval interface. The transfer process from the flow buffer to the unified buffer is implemented through efficient memory copying or pointer operations, ensuring data continuity and reducing processing overhead.

[0100] Preferably, to further optimize storage efficiency, the unified buffer adopts a ring buffer structure design, supporting efficient sequential write operations. When the amount of data in the unified buffer reaches a preset threshold or a disk write cycle is triggered, the system writes the data to the disk in batches, reducing random input / output operations and improving disk throughput. The design of the storage path selection and index generation modules fully considers the differentiated needs of single-packet streams and multi-packet streams. Ineffective memory allocation is avoided through direct storage of single-packet streams, and data continuity is maintained through stream buffer management of multi-packet streams. Combined with the batch disk write mechanism of the unified buffer, the utilization efficiency of memory and disk resources is significantly optimized.

[0101] In one specific embodiment of the present invention, data in a unified buffer is sequentially written to a storage medium in batches. When the first packet is mistakenly identified as a single-packet flow, subsequent packets trigger flow table reconstruction and index merging to ensure data integrity and system stability. Specifically,

[0102] If the first packet is misidentified as a single-packet flow, subsequent packets trigger the creation of new flow table nodes due to flow table mismatch. The first packet is retained in the unified buffer and maintains the packet-level index. Subsequent packets are written to the newly allocated flow buffer and a flow-level index is generated. The packet-level and flow-level indexes are merged through the unified index interface to ensure data integrity.

[0103] It should be noted that in the step of sequentially writing data from the unified buffer to the storage medium in batches and handling the scenario of initial packet misjudgment, a highly efficient batch disk writing mechanism and a robust fault-tolerant processing flow ensure high data storage performance and system stability. This step integrates data storage for single-packet and multi-packet streams, optimizes disk input / output efficiency, and ensures data integrity and retrieval consistency in the case of initial packet misjudgment through a misjudgment detection and correction mechanism. It is particularly suitable for handling high-concurrency and complex traffic processing needs in 100Gbps high-speed network environments.

[0104] Furthermore, the unified buffer, serving as the final convergence point for single-packet and multi-packet stream data, employs a ring buffer structure design to support efficient sequential write operations. All single-packet stream data is directly stored in the unified buffer, while multi-packet stream data is transferred from the stream buffer to the unified buffer after stream termination or timeout. The system periodically checks the fill status of the unified buffer. When the data volume reaches a preset threshold or a predefined disk write cycle is triggered, the data in the unified buffer is written to the storage medium (such as a hard drive or solid-state drive) in batches via sequential write operations. Compared to random writes, batch sequential writes significantly reduce the overhead of disk I / O operations, improve throughput, and reduce disk access latency. In addition, the ring buffer design reduces memory fragmentation by recycling memory space, ensuring efficient memory management during long-term operation.

[0105] Furthermore, regarding fault tolerance, this invention specifically designs a detection and correction mechanism for initial packet misjudgment to address situations where a single-packet flow may be misjudged as a multi-packet flow or vice versa during rule matching. Particularly in the case where the initial packet is misjudged as a single-packet flow, the system ensures data integrity and system stability through the following process: If the initial packet is determined to be a single-packet flow and stored in a unified buffer, generating a packet-level index containing a 5-tuple, timestamp, and storage location, but subsequent arriving data packets indicate they belong to a multi-packet flow due to flow table mismatch, the system will trigger a flow table reconstruction mechanism. Specifically, the system creates a new flow table node for the 5-tuple and allocates a new flow buffer to store subsequent data packets. The initial packet data remains in the unified buffer, and its packet-level index remains valid to ensure that the initial packet data can be retrieved. Subsequent data packets are written sequentially to the newly allocated flow buffer and transferred to the unified buffer after flow termination or timeout, while simultaneously generating a flow-level index containing a 5-tuple and time range.

[0106] Furthermore, to ensure the integrity of data retrieval, the system designed a unified index interface that supports seamless access to both packet-level and stream-level indexes. In cases of misjudgment, the unified index interface uses a 5-tuple to associate the packet-level index of the first packet with the stream-level index of subsequent streams, merging the query results to reconstruct the complete data stream content. The index merging process is based on an efficient hash table query mechanism, quickly locating packet-level and stream-level index entries through the 5-tuple, ensuring that retrieval efficiency is not affected by misjudgments. Experiments show that the data redundancy rate in misjudgment scenarios is extremely low, and the impact on storage efficiency and retrieval performance is negligible, thus ensuring the system's robustness in high-concurrency environments.

[0107] In a second embodiment of the present invention, the present invention provides a packet-flow hybrid storage device based on first packet identification, the device comprising a flow table lookup module 1, a feature matching module 2, and a storage module 3;

[0108] The flow table lookup module 1 is used to receive network data packets from the network interface in real time, parse the five-tuple information and perform flow table lookup to determine whether the data packet belongs to an existing flow or the first packet of a new flow.

[0109] The feature matching module 2 is used to extract the features of the first packet of a new flow, including the five-tuple, transmission control protocol flag, message length, and Internet control message protocol type, and to match them according to the pre-loaded configuration file rules to determine whether it is a single-packet flow or a multi-packet flow.

[0110] The storage module 3 is used to store single-packet streams or multi-packet streams into a unified buffer or a stream buffer according to the determination result and generate corresponding packet-level or stream-level indexes. The stream buffer data is moved to the unified buffer after the stream ends or times out.

[0111] In a third embodiment of the present invention, the present invention provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the packet stream hybrid storage method based on first packet identification as described above.

[0112] In Embodiment 4 of the present invention, the present invention provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps in the packet stream hybrid storage method based on first packet identification as described above.

[0113] In summary, the present invention provides a packet flow hybrid storage method and apparatus based on first packet identification. By identifying the first packet, it distinguishes between single-packet flows and multi-packet flows. Single-packet flows are directly stored in a unified buffer, while multi-packet flows are allocated to a separate flow buffer and then transferred. This significantly reduces memory usage and invalid flow table nodes, mitigating the risk of memory overflow. Batch sequential writing to disk through the unified buffer optimizes disk throughput. Dynamic rule matching adapts to various network environments. A fault-tolerance mechanism corrects misjudgments through subsequent packet detection, ensuring data integrity. A unified index interface supports efficient retrieval, meeting the demands of high-speed networks up to 100 Gbps.

[0114] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0115] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0116] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0117] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A packet stream hybrid storage method based on first packet identification, characterized in that: The application relates to a network packet processing method and device. Real-time network packet receiving, quintuple information parsing and flow table lookup are carried out on a network interface, and it is judged whether the packet belongs to an existing flow or a new flow first packet; Five-tuple, transmission control protocol flag, message length and internet control message protocol type features of the new flow first packet are extracted, and preloaded configuration file rules are matched to determine whether the packet is a single packet flow or a multi-packet flow; According to the determination result, the single packet flow or the multi-packet flow is respectively stored in a unified buffer or a flow buffer, and corresponding packet-level or flow-level indexes are generated, and the flow buffer data is moved to the unified buffer after the flow ends or times out.

2. The packet stream hybrid storage method based on first packet identification according to claim 1, characterized in that: The judgment whether the packet belongs to an existing flow or a new flow first packet comprises: After real-time network packet receiving on the network interface, quintuple information is extracted by using zero-copy parsing technology, and a hash table is used to store a flow table, and it is judged whether the packet hits an existing flow table node by quintuple lookup; The quintuple information comprises a source IP address, a destination IP address, a source port, a destination port and a protocol type.

3. The packet stream hybrid storage method based on first packet identification according to claim 2, characterized in that: The extraction of the five-tuple, the transmission control protocol flag, the message length and the internet control message protocol type features of the new flow first packet comprises: Ethernet frame header and internet protocol header are parsed to obtain the five-tuple; Transmission control protocol header is parsed to obtain the flag; Internet control message protocol header is parsed to obtain the type and code; A plurality of bytes before a transmission control protocol or a user datagram protocol load are extracted for deep protocol identification, and the packet is determined to be a single packet flow or a multi-packet flow by rule matching.

4. The packet stream hybrid storage method based on first packet identification according to claim 3, characterized in that: The matching by the preloaded configuration file rules to determine whether the packet is a single packet flow or a multi-packet flow comprises: The preloaded configuration file rules comprise single or combined rules of protocol type, destination port range and transmission control protocol flag; The rules are compared with the first packet features in the order of rule priority, and if any rule is met, the packet is determined to be a single packet flow, otherwise, the packet is determined to be a multi-packet flow; Dynamic loading and hot updating of rules are adopted to adapt to the single packet flow and multi-packet flow determination requirements of different network environments.

5. The packet stream hybrid storage method based on first packet identification according to claim 3, characterized in that: The single packet flow or the multi-packet flow is respectively stored in the unified buffer or the flow buffer, and corresponding packet-level or flow-level indexes are generated, and the flow buffer data is moved to the unified buffer after the flow ends or times out. The single packet flow is stored in the unified buffer and packet-level indexes containing the five-tuple, a timestamp and a data position are generated; The multi-packet flow is stored in the allocated flow buffer and flow-level indexes containing the five-tuple and a time range are generated; wherein the multi-packet flow data is transferred to the unified buffer after the flow terminates or times out.

6. The packet stream hybrid storage method based on first packet identification according to claim 1, characterized in that: Further comprising: Batch sequential writing of data in the unified buffer to a storage medium is carried out, and when the first packet is misjudged as a single packet flow, flow table reconstruction is triggered by detecting subsequent packets and indexes are combined to ensure data integrity and system stability, and the method specifically comprises: If the first packet is misjudged as a single packet flow, subsequent packets trigger new flow table nodes because the flow table is not hit, the first packet is kept in the unified buffer and packet-level indexes are maintained, subsequent packets are written into newly allocated flow buffers and flow-level indexes are generated, and packet-level and flow-level indexes are combined through a unified index interface to ensure data integrity.

7. A packet stream hybrid storage apparatus based on first packet recognition, characterized by: The device comprises a flow table lookup module, a feature matching module and a storage module; The flow table lookup module is used for real-time network packet receiving on a network interface, quintuple information parsing and flow table lookup, and judgment whether the packet belongs to an existing flow or a new flow first packet; The feature matching module is used for extracting the features of the new flow first packet, including quintuple, transmission control protocol flag bit, message length and internet control message protocol type, and matching through the preloaded configuration file rule to determine as single packet flow or multi-packet flow. The storage module is used for storing the single packet flow or multi-packet flow to the unified buffer or flow buffer according to the determination result and generating the corresponding packet level or flow level index, and the flow buffer data is relocated to the unified buffer after the flow ends or times out.

8. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program which can run on the processor, and the processor executes the program to implement the steps in the packet flow mixed storage method based on first packet identification according to any one of claims 1-6.

9. A storage medium, storing a computer program, characterized in that: The computer program is executed by the processor to implement the steps in the packet flow mixed storage method based on first packet identification according to any one of claims 1-6.

Images