Space-time multiple encoding joint compression storage method

Abstract

The application provides a kind of spatiotemporal multiple encoding joint compression storage method, belongs to computer data processing technical field, to adopt dynamic prediction error correction combined with the mechanism of hierarchical index, can realize the efficient compression of massive network traffic, real-time write and fast random access.The application applies multiple encoding of time dimension and space dimension to the field of network traffic storage, converts absolute timestamp and absolute offset into small differential value, and cooperates with variable-length integer encoding, completely eliminates the massive redundancy caused by traditional PCAP fixed-length storage.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a spatiotemporal multi-encoding joint compression storage method. Background Technology

[0002] Network traffic storage and analysis is a key foundational technology in fields such as network security, performance monitoring, and troubleshooting. Its core lies in the efficient and reliable persistent storage of every data packet captured on a high-speed network link, along with its metadata, such as precise timestamps and storage offsets within files, for subsequent querying, backtracking, and in-depth analysis. With the rapid growth of network bandwidth, the amount of traffic data requiring storage has reached terabytes (TB) or even petabytes (PB) per day. How to achieve long-term preservation of massive amounts of traffic with limited storage resources while ensuring timely data retrieval has become a core challenge in this field.

[0003] Currently, there are two main technical solutions in the industry for storing massive amounts of network traffic. The first is based on general compression algorithms, such as using tools like Gzip or Snappy to compress captured traffic files as a whole, reducing storage size. The second is based on databases, storing key metadata such as packet timestamps and offsets in dedicated time-series or relational databases, and using the database's indexing mechanism for data management and retrieval. Both solutions alleviate storage pressure or provide retrieval capabilities to some extent.

[0004] Existing technical solutions all have obvious drawbacks. While general compression-based solutions can compress data, the entire compressed file needs to be decompressed during querying, making it impossible to achieve random access within a time range, resulting in extremely low query efficiency and making them unsuitable for real-time analysis scenarios. Database-based solutions, although supporting indexed queries, introduce significant database management overhead and high storage costs. When the data volume reaches massive levels, the maintenance cost of database indexes and query performance will drop sharply, indicating room for improvement. Summary of the Invention

[0005] This invention provides a spatiotemporal multi-coding joint compression and storage method, which uses a dynamic prediction error correction combined with a hierarchical index mechanism to achieve efficient compression, real-time writing and fast random access of massive network traffic.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A spatiotemporal multi-coding joint compression and storage method, applied to electronic devices, the method comprising: Obtain the network data packets to be stored, and extract the absolute timestamp of the network data packets and their absolute byte offset in the storage file; Perform time dimension encoding, divide the consecutive absolute timestamps into corresponding time windows, record the base time of the time window, and calculate the time difference between the absolute timestamp of the data packet in the time window and the base time or the timestamp of the previous data packet to obtain the time dimension encoded value. Perform spatial dimension encoding. If it is the first inserted data packet, record its absolute offset as the base offset. Otherwise, calculate the offset difference between the absolute offset of the current data packet and the absolute offset of the previous data packet to obtain the spatial dimension encoded value. Variable-length integer encoding compression is performed on the time dimension encoding value and the spatial dimension encoding value respectively to obtain compressed differential data; Execute storage block management, sequentially write the generated compressed differential data into pre-allocated memory blocks, and allocate new memory blocks when the memory blocks are full; Batch disk write is performed. When the memory usage reaches a set threshold, compressed differential data from multiple memory blocks is written to the disk in batches to form a segmented data file.

[0007] Furthermore, the data generated by performing time-dimension encoding and spatial-dimension encoding is managed using a data structure of baseline values ​​and difference values; The first inserted data packet records the reference time and the reference offset, and sets a first packet flag; The subsequently inserted data packet records the time difference and the offset difference between the previous data packet and the previous data packet.

[0008] Furthermore, the step of writing data from multiple memory blocks to disk in batches to form a data file includes: The data file adopts a segmented organizational structure, including a file header, data segments, and block storage areas; The file header records the global identifier and start time; the data segment records the segment size, number of packets, reference time, and reference offset. The block storage area is used to store the time dimension encoded value and the spatial dimension encoded value after variable-length integer encoding.

[0009] Furthermore, when dividing consecutive absolute timestamps into corresponding time windows, an adaptive window mechanism is adopted, specifically including: The moving average interval data is obtained by calculating the time interval between adjacent data packets. A dynamic threshold is generated based on the moving average interval data and a preset multiple. When the time interval between the current data packet and the previous data packet exceeds the dynamic threshold, or when the number of data packets in the current time window reaches the packet number threshold, the current time window is closed and a new time window is opened to adaptively adjust the window size.

[0010] Furthermore, when calculating the difference between the time difference and the offset, optimization is further achieved by incorporating a dynamic prediction and error correction mechanism, specifically including: The average bandwidth is calculated based on the historical data packet size and absolute timestamp within the time window and used as a parameter for the prediction model. Based on the prediction model parameters and the size of the previous data packet, the time interval and offset increment of the current data packet are jointly predicted to obtain the prediction increment data. The difference between the actual time interval of the current data packet and the actual offset increment and the predicted increment data is calculated to obtain joint error data, which is then used as the input object for the final variable-length integer encoding.

[0011] Furthermore, the calculation of the actual time interval of the current data packet and the difference between the actual offset increment and the predicted increment data yields joint error data, including: The time error component is obtained by calculating the difference between the actual time interval and the predicted time interval, and the offset error component is obtained by calculating the difference between the actual offset increment and the predicted offset increment. After scaling the time error component using a preset scaling factor, it is added to the offset error component to obtain the single-dimensional joint error data.

[0012] Furthermore, the formula for converting the absolute timestamp into a microsecond value is as follows: in and These are the second and microsecond portions of the timestamp for the i-th data packet, respectively.

[0013] Secondly, a spatiotemporal multi-coding joint compression storage device is provided, the device comprising: The data acquisition module is used to acquire network data packets to be stored, and extract the absolute timestamp of the network data packets and their absolute byte offset in the storage file; The time dimension encoding module is used to perform time dimension encoding, divide the consecutive absolute timestamps into corresponding time windows, record the reference time of the time window, and calculate the time difference between the absolute timestamp of the data packet in the time window and the reference time or the timestamp of the previous data packet to obtain the time dimension encoded value. The spatial dimension encoding module is used to perform spatial dimension encoding. If it is the first inserted data packet, its absolute offset is recorded as the base offset. Otherwise, the offset difference between the absolute offset of the current data packet and the absolute offset of the previous data packet is calculated to obtain the spatial dimension encoded value. The variable-length integer encoding compression module is used to perform variable-length integer encoding compression, and to perform variable-length integer encoding on the time dimension encoding value and the spatial dimension encoding value respectively to obtain compressed differential data; The storage block management module is used to perform storage block management, sequentially write the generated compressed differential data into pre-allocated memory blocks, and allocate new memory blocks when the memory blocks are full; The batch disk write module is used to perform batch disk writes. When the memory occupied reaches a set threshold, it writes the compressed differential data in multiple memory blocks to the disk in batches to form a segmented data file.

[0014] Thirdly, an electronic device is provided, comprising: a processor and a memory; the memory is used to store a computer program, which, when executed by the processor, causes the electronic device to perform the spatiotemporal multi-coding joint compression storage method described in the first aspect.

[0015] In one possible design, the electronic device described in the third aspect may further include a transceiver. This transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the electronic device described in the third aspect and other electronic devices.

[0016] In the embodiments of the present invention, the electronic device described in the third aspect may be a terminal, or a chip (system) or other component or assembly disposed in the terminal, or a system containing the terminal.

[0017] Fourthly, a computer-readable storage medium is provided, comprising: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer causes the computer to perform the spatiotemporal multi-coding joint compression storage method described in the first aspect.

[0018] In summary, the above methods and systems have the following technical effects: 1. This invention applies multiple encodings of time and space dimensions to the field of network traffic storage. By converting absolute timestamps and absolute offsets into extremely small differential values ​​and combining them with variable-length integer encoding, it completely eliminates the massive redundancy caused by traditional PCAP fixed-length storage.

[0019] 2. This invention adopts a block-based pre-allocated memory management and batch disk writing strategy. During data processing, compressed differential data is directly written to the pre-allocated memory blocks, avoiding the frequent memory allocation overhead of the underlying system. Batch disk writing is only triggered when memory reaches a threshold, reducing the number of disk I / O operations and meeting the real-time storage needs of massive traffic under high-speed network links.

[0020] 3. This invention designs a segmented data file organization structure based on benchmark values ​​and differential values. While maintaining the lossless and complete restoration of data, the benchmark parameters recorded in the header of the data segment enable the system to quickly locate the target data block without decompressing the entire file when performing data retrieval, thus achieving near real-time random access to the compressed data stream. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the spatiotemporal multi-coding joint compression and storage method provided in an embodiment of the present invention. Figure 2 This is a system block diagram of a spatiotemporal multi-coding joint compression storage device provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the spatiotemporal multi-coding joint compression and storage method provided in this embodiment of the invention; Figure 4 A time-dimensional encoding logic block diagram of the spatiotemporal multi-encoding joint compression and storage method provided in this embodiment of the invention; Figure 5 The process of converting the offset from the original value to the differential value in the spatiotemporal multi-coding joint compression and storage method provided in the embodiments of the present invention; Figure 6 A logical block diagram of the storage data structure of the spatiotemporal multi-coding joint compression storage device provided in an embodiment of the present invention; Figure 7 A storage process logic block diagram of the spatiotemporal multi-coding joint compression storage method provided in the embodiments of the present invention; Figure 8 This is a data file organization diagram for the spatiotemporal multi-coding joint compression and storage method provided in an embodiment of the present invention. Detailed Implementation

[0022] The following will be combined with the appendix Figures 1-8 The technical solutions in this invention will be described below.

[0023] Furthermore, the specific indication method can also be any existing indication method, such as, but not limited to, the above-mentioned indication methods and their various combinations. Specific details of various indication methods can be found in existing technologies, and will not be elaborated upon here. As described above, for example, when multiple pieces of information of the same type need to be indicated, the indication methods for different pieces of information may differ. In specific implementation, the required indication method can be selected according to specific needs. This embodiment of the invention does not limit the selected indication method; therefore, the indication methods involved in this embodiment of the invention should be understood to cover various methods that enable the party to be indicated to obtain the information to be indicated.

[0024] It should be understood that the information to be indicated can be sent as a whole or divided into multiple sub-information messages sent separately, and the sending period and / or timing of these sub-information messages can be the same or different. The specific sending method is not limited in this embodiment of the invention. The sending period and / or timing of these sub-information messages can be predefined, for example, according to a protocol, or configured by the sending device by sending configuration information to the receiving device.

[0025] "Predefined" or "pre-configured" can be achieved by pre-saving corresponding codes, tables, or other means that can be used to indicate relevant information in the device. This embodiment of the invention does not limit the specific implementation method. "Saving" can refer to saving in one or more memories. These memories can be separate installations or integrated into the encoder, decoder, processor, or electronic device. Alternatively, some memories can be separately installed, while others are integrated into the decoder, processor, or electronic device. The type of memory can be any form of storage medium, and this embodiment of the invention does not limit this.

[0026] In the embodiments of this invention, "protocol" may refer to a protocol family in the field of communication, a standard protocol with a similar protocol family frame structure, or a related protocol applied to a future spatiotemporal multi-coding joint compression and storage method system. The embodiments of this invention do not specifically limit this.

[0027] In this embodiment of the invention, descriptions such as "when," "under the circumstances," "if," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not limited to a specific time. They do not require the device to make a judgment action during implementation, nor do they imply any other limitations.

[0028] In the description of the embodiments of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in the embodiments of the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of the embodiments of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or order of execution, and that "first," "second," etc., are not necessarily different. Furthermore, in the embodiments of this invention, words such as "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0029] The network architecture and business scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0030] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention. This spatiotemporal multi-coding joint compression and storage method, applied to electronic devices, includes: Obtain the network data packets to be stored, and extract the absolute timestamp of the network data packets and their absolute byte offset in the storage file; Perform time dimension encoding, divide the consecutive absolute timestamps into corresponding time windows, record the base time of the time window, and calculate the time difference between the absolute timestamp of the data packet in the time window and the base time or the timestamp of the previous data packet to obtain the time dimension encoded value. Perform spatial dimension encoding. If it is the first inserted data packet, record its absolute offset as the base offset. Otherwise, calculate the offset difference between the absolute offset of the current data packet and the absolute offset of the previous data packet to obtain the spatial dimension encoded value. Variable-length integer encoding compression is performed on the time dimension encoding value and the spatial dimension encoding value respectively to obtain compressed differential data; Execute storage block management, sequentially write the generated compressed differential data into pre-allocated memory blocks, and allocate new memory blocks when the memory blocks are full; Batch disk write is performed. When the memory usage reaches a set threshold, compressed differential data from multiple memory blocks is written to the disk in batches to form a segmented data file.

[0031] Furthermore, the data generated by performing time-dimension encoding and spatial-dimension encoding is managed using a data structure of baseline values ​​and difference values; The first inserted data packet records the reference time and the reference offset, and sets a first packet flag; The subsequently inserted data packet records the time difference and the offset difference between the previous data packet and the previous data packet.

[0032] Furthermore, the step of writing data from multiple memory blocks to disk in batches to form a data file includes: The data file adopts a segmented organizational structure, including a file header, data segments, and block storage areas; The file header records the global identifier and start time; the data segment records the segment size, number of packets, reference time, and reference offset. The block storage area is used to store the time dimension encoded value and the spatial dimension encoded value after variable-length integer encoding.

[0033] Furthermore, after obtaining the compressed differential data after variable-length integer encoding, the present invention implements storage block management and batch disk writing strategies to meet the real-time storage needs of massive traffic under high-speed network links.

[0034] Specifically, when performing storage block management, the system adopts a block-based pre-allocated memory management strategy. Before data processing begins, the system pre-allocates multiple fixed-size (e.g., 4MB or 8MB) contiguous memory blocks in memory. Due to the extremely high data throughput of network traffic, triggering dynamic memory allocation for the compressed differential data of each data packet would incur significant system kernel overhead. This invention sequentially appends the generated variable-length integer encoded data stream to the currently active memory block using pure memory operations. When the system detects that the remaining space in the currently active memory block is insufficient or full, it quickly retrieves a new free memory block from the pre-allocated memory pool as the subsequent active block, thereby completely avoiding frequent memory allocation and deallocation during runtime and ensuring high throughput performance for real-time encoding and writing.

[0035] During batch disk writes, the system constructs a buffer queue in memory for the multiple filled memory blocks. To avoid a sharp drop in disk I / O performance caused by frequent small data writes to disk, the system monitors the total memory occupied by this buffer queue in real time. When the memory usage reaches a set threshold (e.g., 60% of the system's pre-allocated memory pool) or a set maximum latency limit, the system merges the compressed differential data in the multiple memory blocks through an asynchronous I / O thread, performing a one-time batch sequential disk write operation to form a segmented data file. This mechanism combines hundreds of thousands of fragmented I / O operations in the traditional model into a single, continuous large-block I / O, greatly improving the throughput efficiency of the underlying storage medium.

[0036] Furthermore, when dividing consecutive absolute timestamps into corresponding time windows, an adaptive window mechanism is adopted, specifically including: The moving average interval data is obtained by calculating the time interval between adjacent data packets. A dynamic threshold is generated based on the moving average interval data and a preset multiple. When the time interval between the current data packet and the previous data packet exceeds the dynamic threshold, or when the number of data packets in the current time window reaches the packet number threshold, the current time window is closed and a new time window is opened to adaptively adjust the window size.

[0037] Detailed implementation: After acquiring the raw time data, to quantify the instantaneous characteristics of network traffic and smooth noise, the system extracts the absolute timestamps of adjacent data packets and calculates the instantaneous time interval. A moving average is then calculated by maintaining a time interval queue of fixed length N, yielding the moving average interval data A. The calculation formula is as follows: Where A is the moving average interval data; N is the fixed length of the time interval queue (preferably ranging from 64 to 256). is the instantaneous time interval between the j-th data packet and its predecessor; i is the sequence number of the current data packet. This dynamic feature ensures that the boundaries of the time window can be smoothly scaled according to the actual traffic.

[0038] Furthermore, a dynamic threshold is generated based on the moving average interval data and the multiple used to limit window jitter, and adaptive window parameters are obtained by grouping the original time data according to the packet number threshold used to limit the window capacity, including: Determine whether the time interval between the current data packet and the previous data packet exceeds the dynamic threshold obtained by multiplying the moving average interval data by a multiple, or determine whether the number of data packets processed in the current window has reached the packet count threshold. If any judgment is true, the current window is closed and a new window is opened, generating adaptive window parameters containing the start and end information of each window to segment the window when traffic changes suddenly and to prevent the window from becoming too large when traffic is stable.

[0039] In order to achieve true adaptation to changes in network traffic, the "multiple" used to limit window jitter and the "packet number threshold" used to limit window capacity are not static configurations that remain unchanged. Instead, they are variables that are dynamically fine-tuned in a closed loop by the system status monitoring module based on the current system throughput and memory pressure.

[0040] Regarding the adaptive adjustment of the "multiplier," the system periodically samples the current overall traffic rate, such as the number of packets processed per second. When the system detects that the traffic rate is continuously rising and tending to stabilize and smooth out, the system will automatically reduce the multiplier, for example, from the initial 3x to 2x, to improve the system's sensitivity to minor network fluctuations and thus more accurately segment traffic characteristics. Conversely, if the system detects that the traffic itself is in an extremely unstable pulse jitter state, the system will dynamically increase the multiplier, such as to 4x or 5x, to prevent the adaptive window from being overly fragmented due to frequent reasonable fluctuations and to avoid introducing excessive window metadata overhead. The adaptive adjustment of the "packet quantity threshold" is controlled by the system's remaining available memory capacity and the backlog length of the disk write queue. When the system has sufficient memory resources and the disk I / O channels are idle, the threshold is automatically maintained at a high level (such as a set upper limit of 4096 packets) to maximize the local historical data sample and improve the accuracy of the prediction model. Conversely, when the system encounters a sudden surge in traffic, detects a significant backlog in the write queue, or the memory usage rate approaches the warning line, the adaptive logic will quickly lower the threshold (for example, by stepping down to 1024 or even 512). This operation aims to force all currently active windows to accelerate to meet the closure conditions and trigger the batch write operation of subsequent storage blocks to disk more frequently. By temporarily sacrificing a very small percentage of the compression rate, the system quickly releases resident memory, ensuring the stable operation and high availability of the core system under extreme pressure.

[0041] Furthermore, when calculating the difference between the time difference and the offset, optimization is further achieved by incorporating a dynamic prediction and error correction mechanism, specifically including: The average bandwidth is calculated based on the historical data packet size and absolute timestamp within the time window and used as a parameter for the prediction model. Based on the prediction model parameters and the size of the previous data packet, the time interval and offset increment of the current data packet are jointly predicted to obtain the prediction increment data. The difference between the actual time interval of the current data packet and the actual offset increment and the predicted increment data is calculated to obtain joint error data, which is then used as the input object for the final variable-length integer encoding.

[0042] Specifically: Within the adaptive window, to establish a predictive model reflecting local traffic characteristics, the system dynamically calculates average bandwidth data as parameters for the predictive model. For the i-th data packet within the window, the system extracts data from the preceding packets. The average bandwidth B is calculated by taking the size and absolute timestamp of each historical data packet, using the following formula: Where B is the average bandwidth, i.e., the prediction model parameter; Lk is the size of the k-th historical data packet. .

[0043] Furthermore, the calculation of the actual time interval of the current data packet and the difference between the actual offset increment and the predicted increment data yields joint error data, including: The time error component is obtained by calculating the difference between the actual time interval and the predicted time interval, and the offset error component is obtained by calculating the difference between the actual offset increment and the predicted offset increment. After scaling the time error component using a preset scaling factor, it is added to the offset error component to obtain the single-dimensional joint error data.

[0044] Detailed implementation: Based on the size of the previous data packet and the prediction model parameters, the system makes a forward-looking estimation of the spatiotemporal position of the current data packet. Since the offset increment of the subsequent data packet in a continuous data stream is theoretically equal to the total length of the previous data packet, the system generates a first prediction component time interval prediction value C and a second prediction component offset increment prediction value D. The calculation formula is as follows: Where Lp is the predicted packet size of the current data packet, which is equal to the size of the previous data packet; where B is the prediction model parameter calculated in the above steps, and C and D are combined into the prediction increment data.

[0045] Furthermore, the actual time interval of the current data packet and the difference between the actual offset increment and the predicted increment data are calculated to obtain joint error data, including: Separate the first prediction component and the second prediction component from the predicted incremental data; The time error component is obtained by calculating the difference between the actual time interval and the first predicted component, and the offset error component is obtained by calculating the difference between the actual offset increment and the second predicted component. The time error component is scaled using a scaling factor to avoid numerical conflicts, and the scaled result is added to the offset error component to obtain the joint error data.

[0046] Specifically: After obtaining the predicted increment data, the system calculates the difference between the actual time interval At and the first predicted component C (time error component E), and the difference between the actual offset increment Ao and the second predicted component D (offset error component F). To facilitate subsequent variable-length encoding, the two components are merged into one-dimensional joint error data G using a scaling factor K. The calculation formula is as follows: Wherein, K is a preset scaling factor, and its value should be greater than the upper limit of the offset error component F (preferably 4096 or 8192) to prevent carry interference from the merging operation and ensure the reversibility of subsequent Varint encoding and decoding.

[0047] When performing joint error merging, the preset scaling factor K plays a crucial role in isolating the two error components and avoiding numerical carry conflicts. Under normal network conditions, the value of the offset error component F is extremely small, so the empirical value of K (e.g., 8192) is usually much larger than the upper limit of F. However, in cases of extreme network congestion, severe packet loss and retransmission, or the presence of massive out-of-order frame writes, the actual observed offset increment may deviate significantly from the predicted value, causing the actual F value to exceed the set upper limit, i.e., error occurs. In this situation, without intervention, once F≥K, the formula: This overflow can cause the carry-over of the time error component E to overlap with the higher bits of F, making it impossible to uniquely and accurately separate the true E and F from the joint error data G during subsequent decoding, resulting in irreversible data corruption. To prevent this overflow risk, this system introduces an overflow degradation mechanism in the encoding module: before each calculation and merging, the system strictly checks whether the absolute value of the actual F is less than K. If |F| < K, then the conventional merging formula is executed to generate the one-dimensional joint error data G. The system then triggers overflow protection interception, abandons the merging operation, and first inserts a pre-agreed escape identifier into the variable-length integer encoded data stream. Then, it treats E and F as two independent integers, encodes them separately using Varint, and writes them sequentially.

[0048] This protection mechanism ensures that while pursuing the ultimate compression rate, the system can achieve 100% lossless data restoration even under any extreme network conditions.

[0049] Furthermore, during the process of writing data to the storage block, the original time data and original offset data of the corresponding data packet are extracted as snapshot data according to the interval value used to determine the snapshot density, including: Maintain a packet write counter within the storage block; When the value of the data packet write counter reaches an integer multiple of the interval value, a snapshot generation operation is triggered. The uncompressed original time data and original offset data of the current data packet are used as snapshot data so that the snapshot data can work together with the compressed differential data as an absolute coordinate reference.

[0050] In this invention, absolute offset refers to the absolute byte offset of a network data packet within a continuously captured file (e.g., a standard PCAP file format). In the linear storage structure of the file, the offset of the first byte at the beginning of the file is denoted as 0. When the system extracts the i-th data packet, its absolute offset is the linear physical addressing coordinate of the first byte of the packet header within that file. This parameter objectively reflects the physical continuity of the data packet's distribution in the underlying storage medium. Since data packets are stored sequentially in the PCAP file, the offset difference between adjacent data packets is theoretically equal to the physical size of the previous data packet, forming the physical coordinate basis for spatial dimension differential encoding and the construction of a fast random access benchmark in this invention.

[0051] The above combination Figure 1 The method provided by the embodiments of the present invention is described in detail below. Figure 2 A spatiotemporal multi-coding joint compression storage device is described in detail for performing the method provided in the embodiments of the present invention, applied to an electronic device, the device comprising: The data acquisition module is used to acquire network data packets to be stored, and extract the absolute timestamp of the network data packets and their absolute byte offset in the storage file; The time dimension encoding module is used to perform time dimension encoding, divide the consecutive absolute timestamps into corresponding time windows, record the reference time of the time window, and calculate the time difference between the absolute timestamp of the data packet in the time window and the reference time or the timestamp of the previous data packet to obtain the time dimension encoded value. The spatial dimension encoding module is used to perform spatial dimension encoding. If it is the first inserted data packet, its absolute offset is recorded as the base offset. Otherwise, the offset difference between the absolute offset of the current data packet and the absolute offset of the previous data packet is calculated to obtain the spatial dimension encoded value. The variable-length integer encoding compression module is used to perform variable-length integer encoding compression, and to perform variable-length integer encoding on the time dimension encoding value and the spatial dimension encoding value respectively to obtain compressed differential data; The storage block management module is used to perform storage block management, sequentially write the generated compressed differential data into pre-allocated memory blocks, and allocate new memory blocks when the memory blocks are full; The batch disk write module is used to perform batch disk writes. When the memory occupied reaches a set threshold, it writes the compressed differential data in multiple memory blocks to the disk in batches to form a segmented data file.

[0052] To further illustrate the technical effects of the spatiotemporal multi-coding joint compression scheme of the present invention, the following explanation is based on specific application examples and test data.

[0053] In a typical scenario of massive network traffic storage, suppose the system needs to continuously store n network data packets. If a traditional fixed-length storage method is used, the absolute timestamp of each data packet must occupy a fixed 16 bytes, and the absolute offset must occupy a fixed 4 bytes. The storage space requirement for its original metadata is... byte.

[0054] After employing the method provided in this embodiment of the invention, through spatiotemporal two-dimensional differential transformation and variable-length integer encoding, the compressed storage space requirement becomes... .

[0055] Because the time difference and offset difference between adjacent data packets in continuous network traffic are extremely small, the encoded single-packet two-dimensional metadata typically requires only 1 to 2 bytes. This means that in most conventional traffic scenarios, it is feasible to achieve... byte.

[0056] The specific space-saving effect is reflected in: In terms of time dimension, the average storage overhead of a single packet of timestamp data has been reduced from the traditional 16 bytes to 1-4 bytes, with a single-item space saving rate of 75%-94%. For example, a timestamp with an absolute value of up to 1,000,000 microseconds may only need to occupy a tiny byte corresponding to about 15 microseconds after being differentiated, and the compression rate of local data can even reach 99.99%.

[0057] In terms of spatial dimension, the average storage overhead per packet of offset data is reduced from the traditional 4 bytes to 1-4 bytes, with a single-item space saving rate of 0%-75%. For example, when the physical file offset reaches 1,048,576 bytes, its differential increment is usually only the size of a single data packet (such as 1,024 bytes), and its local redundancy elimination rate can reach 99.9%.

[0058] By combining the above-mentioned dual-dimensional encoding mechanism, this invention ensures 100% lossless restoration of the original time and physical location data and supports fast random access, while controlling the overall data compression rate between 10% and 30%. This saves the system approximately 70% to 90% of persistent storage space, thereby completely eliminating the space performance bottleneck of traditional solutions when facing massive network traffic.

[0059] The electronic device provided in this embodiment of the invention, exemplarily, can be a network device, or a chip (system) or other component or assembly that can be disposed in a network device. The electronic device may include a processor. Optionally, the electronic device may also include a memory and / or a transceiver. The processor is coupled to the memory and transceiver, for example, by means of a communication bus connection.

[0060] The following is a detailed introduction to the various components of the electronic device: In this context, the processor is the control center of the electronic device. It can be a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0061] Alternatively, the processor can perform various functions of the electronic device, such as the methods described above, by running or executing software programs stored in memory and by calling data stored in memory.

[0062] In a specific implementation, as one example, the processor may include one or more CPUs, such as CPU0 and CPU1.

[0063] In a specific implementation, as one example, the electronic device may also include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0064] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0065] Optionally, the memory can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory can be integrated with the processor or exist independently and coupled to the processor through the interface circuit of the electronic device; the embodiments of the present invention do not specifically limit this.

[0066] A transceiver is used for communication with other electronic devices. For example, if the electronic device is a terminal, the transceiver can be used to communicate with a network device or with another terminal device. Similarly, if the electronic device is a network device, the transceiver can be used to communicate with a terminal or with another network device.

[0067] Optionally, the transceiver may include a receiver and a transmitter. The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0068] Optionally, the transceiver can be integrated with the processor or exist independently and coupled to the processor through the interface circuit of the electronic device. This embodiment of the invention does not specifically limit this.

[0069] It is understood that the structure of the electronic device in this embodiment does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components, or combine certain components, or have different component arrangements.

[0070] Furthermore, the technical effects of the electronic device can be referred to the technical effects of the method described in the above method embodiments, and will not be repeated here.

[0071] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0072] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0073] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0074] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0075] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0076] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0078] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0079] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0082] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A spatiotemporal multi-coding joint compression and storage method, characterized in that, Applied to electronic devices, the method includes: Obtain the network data packets to be stored, and extract the absolute timestamp of the network data packets and their absolute byte offset in the storage file; Perform time dimension encoding, divide the consecutive absolute timestamps into corresponding time windows, record the base time of the time window, and calculate the time difference between the absolute timestamp of the data packet in the time window and the base time or the timestamp of the previous data packet to obtain the time dimension encoded value. Perform spatial dimension encoding. If it is the first inserted data packet, record its absolute offset as the base offset. Otherwise, calculate the offset difference between the absolute offset of the current data packet and the absolute offset of the previous data packet to obtain the spatial dimension encoded value. Variable-length integer encoding compression is performed on the time dimension encoding value and the spatial dimension encoding value respectively to obtain compressed differential data; Execute storage block management, sequentially write the generated compressed differential data into pre-allocated memory blocks, and allocate new memory blocks when the memory blocks are full; Perform batch disk writes. When the memory usage reaches a set threshold, write the compressed differential data from multiple memory blocks to the disk in batches to form a segmented data file. Also includes: The calculation of the time difference and offset difference is further optimized by incorporating a dynamic prediction and error correction mechanism, specifically including: The average bandwidth is calculated based on the historical data packet size and absolute timestamp within the time window and used as a parameter for the prediction model. Based on the prediction model parameters and the size of the previous data packet, the time interval and offset increment of the current data packet are jointly predicted to obtain the prediction increment data. Calculate the difference between the actual time interval and actual offset increment of the current data packet and the predicted increment data to obtain joint error data, and use the joint error data as the input object for the final variable-length integer encoding; The calculation of the actual time interval of the current data packet and the difference between the actual offset increment and the predicted increment data yields joint error data, including: The time error component is obtained by calculating the difference between the actual time interval and the predicted time interval, and the offset error component is obtained by calculating the difference between the actual offset increment and the predicted offset increment. After scaling the time error component using a preset scaling factor, it is added to the offset error component to obtain the single-dimensional joint error data.

2. The spatiotemporal multi-coding joint compression and storage method according to claim 1, characterized in that, The data generated by the execution of time-dimension encoding and spatial-dimension encoding is managed using a data structure of baseline values ​​and difference values; The first inserted data packet records the reference time and the reference offset, and sets a first packet flag; The subsequently inserted data packet records the time difference and the offset difference between the previous data packet and the previous data packet.

3. The spatiotemporal multi-coding joint compression and storage method according to claim 1, characterized in that, The step of writing data from multiple memory blocks to disk in batches to form a data file includes: The data file adopts a segmented organizational structure, including a file header, data segments, and block storage areas; The file header records the global identifier and start time; the data segment records the segment size, number of packets, reference time, and reference offset. The block storage area is used to store the time dimension encoded value and the spatial dimension encoded value after variable-length integer encoding.

4. The spatiotemporal multi-coding joint compression and storage method according to claim 1, characterized in that, When dividing consecutive absolute timestamps into corresponding time windows, an adaptive window mechanism is adopted, specifically including: The moving average interval data is obtained by calculating the time interval between adjacent data packets. A dynamic threshold is generated based on the moving average interval data and a preset multiple. When the time interval between the current data packet and the previous data packet exceeds the dynamic threshold, or when the number of data packets in the current time window reaches the packet number threshold, the current time window is closed and a new time window is opened to adaptively adjust the window size.

5. The spatiotemporal multi-coding joint compression and storage method according to claim 1, characterized in that, The formula for converting the absolute timestamp to a microsecond value is as follows: ； in and These are the second and microsecond portions of the timestamp for the i-th data packet, respectively.

6. A spatiotemporal multi-encoding joint compression storage device, characterized in that, To implement the method of claim 1, the method comprises: The data acquisition module is used to acquire network data packets to be stored, and extract the absolute timestamp of the network data packets and their absolute byte offset in the storage file; The time dimension encoding module is used to perform time dimension encoding, divide the consecutive absolute timestamps into corresponding time windows, record the reference time of the time window, and calculate the time difference between the absolute timestamp of the data packet in the time window and the reference time or the timestamp of the previous data packet to obtain the time dimension encoded value. The spatial dimension encoding module is used to perform spatial dimension encoding. If it is the first inserted data packet, its absolute offset is recorded as the base offset. Otherwise, the offset difference between the absolute offset of the current data packet and the absolute offset of the previous data packet is calculated to obtain the spatial dimension encoded value. The variable-length integer encoding compression module is used to perform variable-length integer encoding compression, and to perform variable-length integer encoding on the time dimension encoding value and the spatial dimension encoding value respectively to obtain compressed differential data; The storage block management module is used to perform storage block management, sequentially write the generated compressed differential data into pre-allocated memory blocks, and allocate new memory blocks when the memory blocks are full; The batch disk write module is used to perform batch disk writes. When the memory occupied reaches a set threshold, it writes the compressed differential data in multiple memory blocks to the disk in batches to form a segmented data file.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the spatiotemporal multi-coding joint compression storage method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to execute the spatiotemporal multi-coding joint compression storage method as described in any one of claims 1 to 5.

Images