A method for quickly segmenting a Pcap file
By combining dynamic segmentation and adaptive sliding verification with thread pool optimization, the problems of slow parsing speed and insufficient memory for giant Pcap files are solved, achieving fast and accurate file segmentation and improving processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINESE PEOPLES LIBERATION ARMY UNIT 32076
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies are slow to parse large Pcap files and are prone to crashing due to insufficient memory, failing to strike a balance between efficiency, accuracy, and resource consumption.
A dynamic segmentation method combined with adaptive sliding verification is adopted. By obtaining the pcap file header information and constructing a global byte index, the initial segmentation point is calculated and corrected. The correctness of the segmentation point is verified by the sliding verification method, and the segmentation process is optimized by combining thread pool and heterogeneous read and write.
It enables rapid splitting of TB-level pcap files, with splitting speed approaching the upper limit of disk read/write speed, improving the loading, searching, and parsing speed of analysis tools, and avoiding memory shortage problems.
Smart Images
Figure CN122173024A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of network data processing technology, specifically relating to a method for fast segmentation of Pcap files. Background Technology
[0002] Pcap (Packet Capture) files are a common storage format for network packet captures, defined by the libpcap / winpcap library and widely used in network analysis tools such as Wireshark and tcpdump. Traditional pcap file processing methods mainly fall into two categories: one is complete parsing based on the libpcap / winpcap library, which is feature-rich but dependent on specific libraries; the other is direct binary reading, which is highly flexible but complex to implement. When processing large pcap files, these methods slow down the loading, searching, and parsing speed of analysis tools, and may even cause crashes due to insufficient memory. Using pcap file splitting methods, large files are divided into N smaller files, and these smaller files are distributed to different tools or threads for parallel analysis, thereby improving parsing and processing efficiency.
[0003] Currently, publicly available pcap segmentation techniques mainly fall into several categories: libpcap-based segmentation, time-window-based segmentation, equal-part segmentation, and machine learning-assisted segmentation. Libpcap-based segmentation requires complete parsing of the pcap file before segmentation, resulting in the longest processing time but the highest accuracy. Time-window-based segmentation similarly requires parsing all timestamps, with similar advantages and disadvantages. Equal-part segmentation divides the pcap file into equal-length sub-files using a uniform partitioning method, offering speed but lower accuracy due to a lack of validation. Machine learning-assisted segmentation utilizes an LSTM model to identify packet boundary features, combined with a sliding window and dynamically adjusted segmentation strategy, achieving high accuracy but also high hardware requirements and time consumption. Therefore, a method is needed that strikes a balance between efficiency, accuracy, and resource consumption to achieve fast and accurate pcap file segmentation. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies, this invention provides a fast pcap file segmentation method. First, in the preprocessing stage, the pcap file header information, text byte count, and time range are obtained, and a global byte index is constructed. Then, a dynamic segmentation method is used to find the theoretical segmentation point, and an adaptive sliding verification method is used to verify the correctness of the theoretical segmentation point. Finally, the segmented pcap file is saved. This invention can quickly segment pcap format network data files, with a segmentation speed approaching the upper limit of disk read / write speed.
[0005] The technical solution adopted by this invention to solve its technical problem is as follows: Step 1: Scan the first 24 bytes of the pcap file to obtain the pcap file. Information and maximum storage length ; The information is used to identify the file format and determine the byte order; Step 2: Read the pcap file to build a global byte index and obtain the number of bytes in the pcap file. and time range ; Step 3: Calculate the initial split point tuple; Based on the input target segmentation number And obtaining the file byte count in step 2 Calculate the initial split point tuple ; Step 4: Calculate the theoretical split point tuple; The initial segmentation point is corrected until a valid packet header is found; the calculation method is as follows: read the last 16 bytes of the initial segmentation point and combine them with the data obtained in step 1. Information calculation second-level timestamp Millisecond-level timestamps Capture length and actual length The correction function is defined as follows:
[0006] During the correction process, the device slides unidirectionally in 1-byte increments until a valid packet header is found.
[0007] Step 5: Adaptive sliding verification; The theoretical split point is verified using a sliding verification method, taking the data from the Packet header of the current theoretical split point as an example. Given the sliding length, find the next Packet header and verify whether it meets the correction function, and so on, until the verification ends; An adaptive sliding verification count is achieved by combining file size, number of splits, and theoretical split point error rate. The verification count calculation function is as follows:
[0008] in Represents the final number of verifications. It is the baseline number of verifications. This is the maximum file size. It is the index of the current split point in the split point tuple. It is the average number of partitions. It is the error rate of the verification segmentation point. All are adjustment coefficients; As file size As the value increases, the number of verifications increases; with the current segmentation point... near The number of verifications decreases; with the number of segmentations... As the error rate increases, the number of verifications decreases; The increase in the number of verifications leads to an increase in the error rate. The calculation function is as follows:
[0009] in This indicates whether the current split point encountered an error during the first validation; 1 represents an error, and 0 represents a correct result. The error rate of the theoretical split point changes as the theoretical split point is verified, enabling real-time monitoring of the theoretical split point error rate; Step 6: Combine and save the theoretical split points and pcap header information as... Two different pcap files; Step 7: Improve processing speed from three perspectives: initializing the thread pool, task allocation, and heterogeneous read / write; The initial process pool is set to the same number of CPU cores, and then adjusted based on performance monitoring results during actual operation. Task allocation adopts a producer + consumer model. The producer thread is responsible for obtaining the global byte index and initial split point in steps 1-3 and putting them into the queue. Multiple consumer threads obtain tasks from the queue and process them in steps 4-6. Heterogeneous read and write achieves non-blocking file writing.
[0010] An electronic device includes: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to enable the electronic device to perform the above-described Pcap file fast segmentation method.
[0011] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for fast segmentation of Pcap files.
[0012] A chip includes a processor for retrieving and running a computer program from memory, causing a device equipped with the chip to perform the aforementioned fast Pcap file segmentation method.
[0013] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described fast Pcap file segmentation method.
[0014] The beneficial effects of this invention are as follows: 1. This invention enables rapid segmentation of TB-level pcap files. Compared to traditional methods, the theoretical segmentation speed improvement is proportional to the number of consumer threads, and in practical use, it approaches the upper limit of disk read / write speed. As a front-end tool, this invention can accelerate various back-end pcap-based analysis tools, such as offline playback and protocol parsing.
[0015] 2. This invention can quickly segment pcap format network data files, with a segmentation speed approaching the upper limit of disk read / write speed. Attached Figure Description
[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 These are schematic diagrams comparing the time taken for pcap segmentation and the time taken for pcap segmentation and saving, respectively, according to the present invention. Detailed Implementation The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0017] This invention provides a fast pcap file segmentation method. To solve the problem of fast pcap file segmentation, this invention implements a fast segmentation method that combines dynamic segmentation based on file bytes, adaptive sliding verification, and available resource scheduling. A segmentation method combining dynamic segmentation and adaptive sliding verification is designed for segmentation point calculation, and fully utilizes hardware and software resources to achieve a balance between efficiency, accuracy, and resource consumption, thus achieving the goal of fast pcap file segmentation.
[0018] Its implementation steps include: Steps 1 and 2 are the preprocessing stage, which mainly involves obtaining basic information about the pcap file.
[0019] Step 1: Quickly scan the first 24 bytes of the pcap file to obtain the pcap file. Information and maximum storage length , The information is primarily used to identify file formats and determine byte order. It is mainly used as one of the verification conditions for the theoretical split point in the subsequent step 4.
[0020] Step 2: Read the pcap file to build a global byte index and obtain the number of bytes in the pcap file. and time range .
[0021] Steps 3-4: Calculate the theoretical segmentation points using the dynamic segmentation method.
[0022] Step 3: Calculate the initial split point tuple, based on the input target number of splits. And obtaining the file byte count in step 2 To calculate the initial split point tuple .
[0023] Step 4: Calculate the theoretical split point tuple. The initial split point may be located in the middle of a single packet. Directly splitting the packet will result in incorrect packets that cannot be parsed and processed by other tools. Therefore, the initial split point needs to be corrected until a valid packet header is found. The calculation method is as follows: Read the last 16 bytes of the initial split point and combine them with the data obtained in Step 1. Calculate second-level timestamps in byte order Millisecond-level timestamps Capture length and actual length The correction function is defined as follows:
[0024] During the correction process, the device slides unidirectionally in 1-byte increments until a valid packet header is found.
[0025] Step 5: Adaptive sliding verification method. This method verifies the theoretical split point using a sliding verification approach, taking the data from the Packet header of the current theoretical split point as an example. The sliding length is used to find the next packet header and verify if it meets the correction function, and so on, until the verification ends. A fixed number of sliding verifications may not be effective for verifying the theoretical split point, so an adaptive sliding verification count is achieved by combining file size, number of splits, and theoretical split point error rate. The verification count calculation function is as follows:
[0026] in Represents the final number of verifications. It is the baseline number of verifications. This is the maximum file size. It is the index of the current split point in the split point tuple. It is the average number of partitions. It is the error rate of verifying the split points. All are adjustment factors (positive values). These vary with file size. As the value increases, the number of verifications increases; with the current segmentation point... near The number of verifications decreases; with the number of segmentations... As the error rate increases, the number of verifications should be appropriately reduced, and the error rate should be adjusted accordingly. The increase in [something] leads to an increase in the number of verifications. Error rate. The calculation function is as follows:
[0027] in This indicates whether the current split point encountered an error during the first validation; 1 represents an error, and 0 represents a correct result. The error rate of the theoretical split point changes as it is verified, enabling real-time monitoring of the theoretical split point error rate.
[0028] Step 6: Combine and save the theoretical split points and pcap header information as... Two different pcap files.
[0029] Step 7: When splitting large pcap files, to improve the splitting speed within limited hardware resources, a method combining CPU core count, memory space, and I / O speed is used to implement batch execution via a thread pool. The size of the thread pool is mainly determined by the number of CPU cores, but memory space and I / O speed still need to be considered. If too many threads are set up, and memory space and I / O speed cannot keep up, it will lead to a bottleneck and reduce efficiency.
[0030] The main improvements to processing speed come from three aspects: thread pool initialization, task allocation, and heterogeneous read / write. The initial process pool is set to the same number of CPU cores, and adjustments are made based on performance monitoring results during actual operation. Task allocation adopts a producer + consumer model. The producer thread is responsible for obtaining the global index and initial split point in steps 1-3 and putting them into the queue. Multiple consumer threads obtain tasks from the queue and process them in steps 4-6. Heterogeneous read and write achieves non-blocking file writing.
[0031] Example: Figure 1 The processing flow of this invention is presented, including header information preprocessing, dynamic segmentation, sliding verification, and output. First, in the preprocessing stage, header information of the pcap file, such as Magic, Snplen, byte count, and time range, is obtained, and a global byte index is constructed. An initial segmentation point tuple is generated using a producer thread. Then, a consumer thread pool and a dynamic segmentation method are used to find the theoretical segmentation point, and an adaptive sliding verification method is used to calculate the number of verifications to verify the correctness of the theoretical segmentation point. If correct, it is output to the final saving stage; if incorrect, a new theoretical segmentation point is recalculated and verified again until the correct packet header is found. Finally, the segmented pcap file is saved.
[0032] The method of this invention was implemented in C++ on a Windows system to split a 100GB pcap file. The table below compares the performance of this invention with that of common pcap file splitting tools tshark and splitcap. The values in the table are the average after 10 runs. As can be seen from the figure, when using the same hardware resources for splitting and saving, the method of this invention quickly locates the split point of the pcap file, and the write performance when storing based on this point is approximately 540MB / s, approaching the upper limit of disk write performance.
Claims
1. A method for fast segmentation of Pcap files, characterized in that, Includes the following steps: Step 1: Scan the first 24 bytes of the pcap file to obtain the pcap file. Information and maximum storage length ; The information is used to identify the file format and determine the byte order; Step 2: Read the pcap file to build a global byte index and obtain the number of bytes in the pcap file. and time range ; Step 3: Calculate the initial split point tuple; Based on the input target segmentation number And obtaining the file byte count in step 2 Calculate the initial split point tuple ; Step 4: Calculate the theoretical split point tuple; The initial segmentation point is corrected until a valid packet header is found; the calculation method is as follows: read the last 16 bytes of the initial segmentation point and combine them with the data obtained in step 1. Information calculation second-level timestamp Millisecond-level timestamps Capture length and actual length The correction function is defined as follows: During the correction process, the device slides unidirectionally in 1-byte increments until a valid packet header is found. Step 5: Adaptive sliding verification; The theoretical split point is verified using a sliding verification method, taking the data from the Packet header of the current theoretical split point as an example. Given the sliding length, find the next Packet header and verify whether it meets the correction function, and so on, until the verification ends; An adaptive sliding verification count is achieved by combining file size, number of splits, and theoretical split point error rate. The verification count calculation function is as follows: in Represents the final number of verifications. It is the baseline number of verifications. This is the maximum file size. It is the index of the current split point in the split point tuple. It is the average number of partitions. It is the error rate of the verification segmentation point. All are adjustment coefficients; As file size As the value increases, the number of verifications increases; with the current segmentation point... near The number of verifications decreases; with the number of segmentations... As the error rate increases, the number of verifications decreases; The increase in the number of verifications leads to an increase in the error rate. The calculation function is as follows: in This indicates whether the current split point encountered an error during the first validation; 1 represents an error, and 0 represents a correct result. The error rate of the theoretical split point changes as the theoretical split point is verified, enabling real-time monitoring of the theoretical split point error rate; Step 6: Combine and save the theoretical split points and pcap header information as... Two different pcap files; Step 7: Improve processing speed from three perspectives: initializing the thread pool, task allocation, and heterogeneous read / write; The initial process pool is set to the same number of CPU cores, and then adjusted based on performance monitoring results during actual operation. Task allocation adopts a producer + consumer model. The producer thread is responsible for obtaining the global byte index and initial split point in steps 1-3 and putting them into the queue. Multiple consumer threads obtain tasks from the queue and process them in steps 4-6. Heterogeneous read and write achieves non-blocking file writing.
2. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in claim 1.
3. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in claim 1.
4. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in claim 1.
5. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in claim 1.