VPP-based full-amount network message storage and retrieval method

Abstract

The invention relates to a full-quantity network message storage and retrieval method based on a VPP (Virtual Private Protocol), which is characterized in that message storage and preprocessing are completed by means of a high-performance message capturing and processing framework of the VPP, and message index information obtained after preprocessing is stored through a column database; and performing write completion message storage on the plurality of disks through the optimized write method. According to the message retrieval, index information of a target message is quickly retrieved by means of retrieval efficiency of a column database and complete information of index data, the target message is quickly positioned according to the index information, and a final target PCAP file is extracted and produced.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, and in particular relates to a method for storing and retrieving full network packets based on VPP. Background Technology

[0002] VPP (Vector Packet Processing) is an open-source version of Cisco's vector packet processing technology. It provides a high-performance packet processing stack that can run on commercial CPUs and deliver production-quality switch / router functionality. Due to VPP's rich features, production-grade quality, and excellent framework design, an increasing number of network products based on VPP are being used in live network environments (physical and virtual networks).

[0003] Full network packet storage and retrieval is a technology that preserves all raw traffic data passing through a network environment completely and without loss, and enables rapid retrieval of massive amounts of stored network packet data based on different conditions (IP, port, protocol, etc.).

[0004] A full network packet storage and retrieval system mainly consists of two parts: storage and retrieval. The storage module is responsible for capturing passing network packets and preprocessing them before storing the network packets and their metadata. Storing the metadata improves retrieval efficiency when searching for network packets. The retrieval module can directly retrieve the metadata. The retrieval module retrieves network packets that meet certain criteria by directly searching the metadata (or the stored raw network packets) and then assembles these matching packets into a new file (e.g., PCAP format) and returns it to the user. The performance of the packet storage module is primarily affected by disk write performance and packet preprocessing. Optimizing disk write performance can often be achieved by replacing hard disk drives (HDDs) with solid-state drives (SSDs). However, SSDs are expensive, and continuous writing of large volumes of data requires significant storage space. Furthermore, continuous writing of large volumes of data also affects the lifespan of SSDs. Additionally, data cannot be recovered if an SSD fails, while some (or all) of the data can be recovered if an HDD fails. Some implementations use SSDs as a front-end cache while still using HDDs for back-end storage, but this only solves temporary bursts of high traffic and cannot handle sustained high traffic. The optimization of preprocessing is achieved by simplifying (or omitting) preprocessing, and the quality of preprocessing directly impacts the packet retrieval performance of the retrieval module. The performance of the packet retrieval module is primarily affected by the structure and volume of the retrieved data. For example, some implementations directly retrieve the stored raw packets, resulting in very poor retrieval performance under high traffic conditions. Even with partial implementations that query packet metadata, the amount of data retrieved under continuous full-traffic storage is extremely large, causing retrieval performance to significantly decrease as the data volume increases. For these reasons, some products offer filtering functions to filter network traffic before packet retrieval, storing only network packets that meet the criteria, reducing the amount of data stored and retrieved. However, this does not allow for continuous, long-term full-traffic storage under high traffic (10G levels). Summary of the Invention

[0005] The purpose of this invention is to provide a method for storing and retrieving full network packets based on VPP, so as to meet the needs of continuous storage and retrieval of large volumes of full network packets.

[0006] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a method for storing all network packets based on VPP, the method comprising: Step 1: Capture network packets from the network interface; Step 2: Parse the network packets captured in Step 1; Step 3: Retrieve the packet cache from the packet cache pool, copy the network packets captured in Step 1 into the packet cache, and obtain the storage location and file name of the captured network packets in the file; Step 4: Determine if the message buffer is full or has timed out. If so, proceed to Step 5; otherwise, proceed to Step 7. Step 5: After aligning the message buffer, send it to the disk write thread corresponding to the message buffer through a lock-free queue; Step 6: Messages are sequentially written to the pre-allocated file; Step 7: Preprocess the parsing results from Step 2 and the storage location and file name of the captured network packets obtained from Step 3 in the file; Step 8: Temporarily store the preprocessing results from Step 7 in the message index cache; Step 9: Determine if the message index cache is full or has timed out. If so, proceed to step 10; otherwise, the process ends. Step 10: Persist the message index cache to the column database. The process ends.

[0007] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a method for full network packet retrieval based on VPP, including: Step S1: Receive the database of conditional search columns for the target message input by the user; Step S2: Determine if there is index information in the column database that meets the conditions. If it exists, proceed to step S3; otherwise, the process ends. Step S3: Obtain the storage location, file name, and message length of the message from the index information that meets the conditions, and read the data from the file to obtain the target message; Step S4: Sort the acquired target messages by time, encapsulate the PCAP header information, and generate the target PCAP file; Step S5: Return the generated PCAP file to the user.

[0008] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the storage method or retrieval method described above.

[0009] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps in the storage method or retrieval method described above.

[0010] Compared with the prior art, the present invention has the following beneficial effects: 1. Supports continuous full storage of large traffic (10G level) and can be horizontally scaled to support higher traffic based on the user's traffic volume. Through VPP's high-performance packet capture technology and parallel writing of multiple disks, it ensures continuous and stable storage of packets under high traffic conditions.

[0011] 2. The overall cost is low. Multiple mechanical hard drives are used to horizontally expand the solid-state drive, ensuring that more packets are stored and larger volumes of continuous full storage are supported at a lower cost while meeting performance requirements. Block compression is used to reduce disk usage while ensuring that compression has little impact on packet retrieval (no need to compress the entire file).

[0012] 3. It has high data security. RAID ensures that even if some disks fail, the stored network packets can still be retrieved and used.

[0013] 4. Supports retrieval of massive network packets. Through VPP's high-performance packet processing framework, network packets are fully preprocessed and stored in a column database, which greatly improves the retrieval efficiency of massive packets. In addition, the information in the index, such as the file name, offset, and length of the packet, allows for quick location and extraction of the target packet. Attached Figure Description

[0014] Figure 1 This is the initialization flowchart of the present invention.

[0015] Figure 2 This is a flowchart of the message storage process of the present invention.

[0016] Figure 3 This is a flowchart of the message retrieval process of this invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] A full network packet storage and retrieval system mainly consists of two parts: storage and retrieval. The performance of the packet storage module is primarily affected by disk write performance and packet preprocessing. Optimizing disk write performance can often be achieved by replacing hard disk drives (HDDs) with solid-state drives (SSDs). However, SSDs are expensive, and continuous high-volume data writing requires significant storage space and also impacts the lifespan of SSDs. Furthermore, data loss due to SSD failure is irreversible, while partial (or complete) data recovery is possible from HDD failure. Some implementations use SSDs as front-end caches while retaining HDDs for back-end storage, but this only addresses temporary bursts of high traffic, not sustained high traffic. Optimizing preprocessing is achieved by simplifying (or omitting) preprocessing, and the quality of preprocessing directly affects the packet retrieval performance of the retrieval module. The performance of the packet retrieval module is mainly affected by the structure and volume of the retrieved data. For example, some implementations directly retrieve the original stored packets, resulting in very poor retrieval performance under high traffic conditions. Even if querying message metadata is partially implemented, the amount of data retrieved will be very large under continuous full traffic storage, so the retrieval performance will decrease significantly as the amount of data continues to increase.

[0020] For the reasons mentioned above, some products provide filtering functions to filter network traffic before packet acquisition and store only network packets that meet the conditions, thereby reducing the amount of data to be stored and retrieved. Therefore, they cannot achieve continuous long-term full storage of large traffic (10G level). Therefore, this invention proposes a full network packet storage and retrieval method based on VPP.

[0021] In Embodiment 1 of this invention, the invention mainly consists of two parts: message storage and message retrieval. The message storage process primarily completes message capture, preprocessing, index information generation and storage, and message storage. The message storage module is developed based on VPP, utilizing VPP's high-performance message capture and processing framework to complete message storage and preprocessing. The preprocessed message index information is stored in a column database, and message storage is completed by writing to multiple disks using an optimized write method. The message retrieval module primarily completes message retrieval and generates a target PCAP file composed of messages that meet the criteria based on the retrieval results. Leveraging the retrieval efficiency of the column database and the complete information of the index data, the message retrieval module quickly retrieves the target message index information, quickly locates the target message based on the index information, extracts it, and generates the final target PCAP file.

[0022] It's important to note that PCAP files are a standard format for network packet capture, used to store raw data packet information from network communications. This file format is widely used in network analysis, security monitoring, and troubleshooting.

[0023] The implementation method of this solution will be introduced in three parts: initialization, message storage process, and message retrieval process.

[0024] The initialization section is mainly used to initialize the resources required for message storage and retrieval, as well as the message storage module and message retrieval module. This includes pre-allocating message storage files, initializing the message capture network interface, initializing the corresponding number of disk write threads based on the number of disks, and initializing the column database.

[0025] as follows Figure 1 As shown, the initialization process includes: 1. Check the configured disk. If the disk has pre-allocated files and the pre-allocated files are physically contiguous on the disk, proceed to step 3; otherwise, proceed to step 2. 2. Delete files on the disk and reallocate files to each disk; 3. Initialize physical network interface cards, memory, and cache resources; 4. Start packet capture and preprocessing threads. Multiple threads can be started according to the traffic volume, and multiple disk write threads can be started according to the number of physical hard drives to ensure that only one thread writes to each physical hard drive. 5. Initialize the column database and start the message retrieval thread.

[0026] II. Message storage primarily handles message capture, preprocessing, index generation, and disk persistence of messages and index information. Leveraging the high-performance message capture and processing framework provided by VPP, along with an optimized disk write process, it can meet the continuous storage requirements for large volumes of full messages. The message storage module consists of disk write threads and message processing threads, and there is no one-to-one correspondence between them. The number of disk write threads can be configured based on the number of disks, and the number of message processing threads can be configured based on the actual traffic volume.

[0027] like Figure 2 As shown, the message storage processing flow includes: 1. Capture network packets from the network interface using the PMD driver provided by DPDK, and then proceed to step 2; 2. Parse the captured network packets to obtain the metadata information of the network packets for subsequent index information generation. The metadata of the packets includes: MAC, VLAN ID, protocol, IP, port, packet length, etc., and proceed to step 3. 3. Retrieve packet cache from the packet cache pool and copy captured network packets into the packet cache. The packet cache is bound to the disk write thread, which in turn is bound to the file. Therefore, once a packet is copied into the packet cache, its storage location and filename within the file are obtained. If compression is configured, the packet cache is compressed in blocks. The storage location information at this point consists of two parts: the location of the compressed block within the file and the location of the packet within the decompressed data block. Proceed to step 4; 4. Determine if the message buffer is full or has timed out. If so, proceed to step 5; otherwise, proceed to step 7. Message buffering is used to consolidate disk I / O, fully utilizing hard drive characteristics to improve disk write performance. 5. In order to improve disk write performance by utilizing the characteristics of hard disks, after the packet buffer is aligned, it is sent to the disk write thread corresponding to the packet buffer through a lock-free queue, and then step 6 is executed; 6. Message buffer is sequentially written to the pre-allocated file. To ensure the physical contiguousness of the disk sectors corresponding to the files and to reduce disk fragmentation, the message buffer is written to the pre-allocated file. By establishing a one-to-one correspondence between message threads and disks, it is ensured that data is written to the disk sequentially, and only one thread operates on a single disk, thus guaranteeing disk write performance. Proceed to step 7. 7. Preprocess the parsed message information and the filename and position information of the message obtained in step 3 to merge multiple messages from the same stream into one. Since the same stream differs only in message length, timestamp, and position in the file, redundant information can be compressed. Furthermore, differential compression can be used to further compress message length, timestamp, and position in the file. Proceed to step 8. It should be noted that preprocessing includes merging and compression; merging means combining multiple messages from the same stream into one.

[0028] 8. Temporarily store the information obtained in step 7 into the message index cache, and then proceed to step 9; 9. Determine if the index cache is full or has timed out. If so, proceed to step 10; otherwise, end the process. 10. Persist the index cache to the column database. To prevent the index cache persistence from affecting message storage, a separate hard disk is needed to persist the index cache.

[0029] It's important to note that the Data Plane Development Kit (DPDK) consists of a set of user-space function libraries and drivers that provide high-performance data transmission, reception, and processing for the data plane. IPSEC (Internet Protocol Security) is a protocol suite (a collection of interconnected protocols) that protects network transport protocols using the IP protocol by encrypting and authenticating IP protocol packets.

[0030] III. Message Retrieval: This function primarily handles message retrieval, extraction, and PCAP file generation. It retrieves column data based on input criteria, extracts messages from specified files using the retrieved index information, and ultimately generates the target PCAP file. The index is stored in the column data to improve the efficiency of message index information retrieval. Each index entry contains multiple messages from a single flow and includes information such as the message's location in the file, filename, and message length, enabling rapid extraction of the original message.

[0031] like Figure 3 As shown, the message retrieval process is as follows: 1. Retrieve the column database based on the conditions entered by the user. The conditions entered by the user include: time period, MAC, IP, protocol, port, stream ID, etc., and then proceed to step 2; 2. Determine if the column database contains index information that meets the conditions. If it exists, proceed to step 3; otherwise, end the process. 3. Based on the obtained target message index information, determine whether the message is compressed. If the message is uncompressed, obtain the message filename, message position in the file, and message length from the index information. Then, directly read the specified length of data from the specified position in the file to obtain the original message. If the message is compressed, obtain the message filename, the position of the compressed block in the file, the position of the decompressed data block, and message length from the index information. Then, first extract the compressed block from the specified position in the specified file, then decompress the compressed block to obtain the decompressed data block, and finally read the specified length from the specified position in the decompressed data block to obtain the original message. Proceed to step 4. 4. Sort the acquired target messages by time, encapsulate the PCAP header information, generate the target PCAP file, and proceed to step 5; 5. The generated PCAP file is returned to the user, who can use it for data auditing, security analysis, message playback, and backtracking analysis.

[0032] It should be noted here that, in order to bring disk write performance close to the physical limits of the disk, the following methods can be used to further improve disk write performance: a. Use user-space caching to merge disk write I / O; b. By using the write pre-allocated file method, we can ensure that the file data is physically contiguous on the disk and that long-term message storage will not generate disk fragmentation. c. The number of disk write threads is one-to-one with the number of disks, and disk writes are performed sequentially; d. Message data and index data are stored on different disks.

[0033] In a second embodiment of the present invention, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the storage method or retrieval method described above.

[0034] In a third embodiment of the present invention, the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the steps in the storage method or retrieval method described above.

[0035] In summary, after comprehensively considering factors such as implementation complexity, usage and maintenance costs, performance of continuous storage and retrieval of large volumes of full network packets, and data security, this invention proposes a VPP-based method for storing and retrieving full volumes of network packets, which can meet the industry's needs for continuous storage and retrieval of large volumes of full network packets.

[0036] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0037] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0038] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0039] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for storing all network packets based on VPP, characterized in that: include, Step 1: Capture network packets from the network interface; Step 2: Parse the network packets captured in Step 1; Step 3: Retrieve the packet cache from the packet cache pool, copy the network packets captured in Step 1 into the packet cache, and obtain the storage location and file name of the captured network packets in the file; Step 4: Determine if the message buffer is full or has timed out. If so, proceed to Step 5; otherwise, proceed to Step 7. Step 5: After aligning the message buffer, send it to the disk write thread corresponding to the message buffer through a lock-free queue; Step 6: Messages are sequentially written to the pre-allocated file; Step 7: Use the parsing results from Step 2 and the storage location and file name of the captured network packets obtained from Step 3 as packet index information, and perform preprocessing. Step 8: Temporarily store the preprocessing results from Step 7 in the message index cache; Step 9: Determine if the message index cache is full or has timed out. If so, proceed to step 10; otherwise, the process ends. Step 10: Persist the message index cache to the column database. The process ends.

2. The storage method according to claim 1, characterized in that: In step 1, network packets are captured from the network interface using the PMD driver provided by DPDK.

3. The storage method according to claim 1, characterized in that: In step 3, the message cache is bound to the disk write thread.

4. The storage method according to claim 1, characterized in that: In step 2, the network packets captured in step 1 are parsed to obtain the metadata of the network packets, including: MAC, VLAN ID, protocol, IP, port, and packet length.

5. The storage method according to claim 1, characterized in that: Step 3 also includes, Compress the message buffer, and simultaneously obtain the location of the compressed block in the file and the location of the network packet in the decompressed data block; Step 7 includes, The parsing results from step 2, the storage location and file name of the captured network packets obtained from step 3, the location of the compressed blocks in the file, and the location of the network packets in the decompressed data blocks are used as packet index information and preprocessed.

6. The storage method according to claim 1 or 5, characterized in that: The preprocessing includes merging and compression, wherein merging is: merging the message index information of the same stream into one.

7. The storage method according to claim 1, characterized in that: It also includes initialization, Step (1): Check the configured disk. If there are pre-allocated files on the disk and the pre-allocated files are physically contiguous on the disk, then proceed to step (3); otherwise, proceed to step (2). Step (2): Delete the files on the disk and reassign files to the disk; Step (3): Initialize the physical network card, memory, and cache; Step (4): Start the message capture and preprocessing thread; Step (5): Initialize the column database and start the message retrieval thread.

8. A method for full network packet retrieval based on VPP, characterized in that: The retrieval method is applied to messages stored by any of the storage methods described in claims 1 to 6, including: Step S1: Receive the database of conditional search columns for the target message input by the user; Step S2: Determine if there is index information in the column database that meets the conditions. If it exists, proceed to step S3; otherwise, the process ends. Step S3: Obtain the storage location, file name, and message length of the message from the index information that meets the conditions, and read the data from the file to obtain the target message; Step S4: Sort the acquired target messages by time, encapsulate the PCAP header information, and generate the target PCAP file; Step S5: Return the generated PCAP file to the user.

9. The retrieval method according to claim 8, characterized in that: The target message condition search column data input by the user in step S1 includes: time period, MAC, IP, protocol, port, and flow ID.

10. The retrieval method according to claim 8, characterized in that: Step 3 also includes, If the index information meets the conditions, determine whether the message is compressed: If it is not compressed, the storage location, file name and message length of the message are obtained from the index information that meets the conditions, and the data is read from the file, which is the target message; If compression is used, the filename of the message, the location of the compressed block of the message in the file, the location of the data block of the message after decompression, and the message length information are obtained from the index information that meets the conditions. Then, the compressed block is extracted from the file, the compressed block is decompressed to obtain the decompressed data block, and finally the data is read from the decompressed data block, which is the target message.

Images