A lightweight network packet storage method with low storage overhead

Abstract

The application provides a low-storage-overhead lightweight network data packet storage method, and belongs to the technical field of computer data processing, and aims to solve the problems of large data storage and slow search in traditional network storage, greatly saves storage space under the premise of not losing precision, and realizes fast search in massive data. In the application, a dictionary coding processing mechanism is introduced, long and high-frequency repeated network address information in the original message is stripped and mapped into equal-length short identifiers, static text redundancy is fundamentally eliminated, high-consumption string mode scanning is converted into integer value comparison friendly to the underlying architecture, the read-write throughput of the underlying storage medium is optimized to reduce fragmentation, adaptive flow classification evaluation is implemented based on the packet quantity threshold, slicing and blocking are adopted for elephant flow, and independent index management is adopted to guarantee the ordered loading of long connections.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a lightweight network data packet storage method with low storage overhead. Background Technology

[0002] With the evolution of network communication technology and the expansion of network scale, the data traffic between network nodes has experienced explosive growth. In application scenarios such as network security auditing, network performance monitoring, and abnormal traffic tracing, persistent storage and rapid retrieval of raw network data packets are indispensable basic supporting technologies.

[0003] Traditional network packet storage schemes typically use direct linear writing to record raw packets. This coarse storage mode retains a large amount of redundant information in the packet header, especially long text metadata such as the frequently appearing source network address, destination network address, and media access control address. This excessively consumes physical storage resources, and the actual physical network traffic exhibits extreme imbalance, with a few large long connections containing a large number of packets coexisting with a large number of short, instantaneous connections containing only a few packets. Traditional technologies employ a uniform storage and indexing strategy for network flows of all sizes. This not only leads to severe disk fragmentation in the underlying storage system due to frequent random writes from massive fragmented streams, slowing down the overall system throughput, but also causes the retrieval index library to expand rapidly. In the retrieval and backtracking of historical packets, existing technologies often rely on linear traversal of the entire file or long string pattern matching, which not only incurs high overhead for reading input and output but also wastes processor computing power. There is room for improvement. Summary of the Invention

[0004] This invention provides a lightweight network data packet storage method with low storage overhead, which solves the problems of data taking up too much space and slow searching in traditional network storage. Without losing accuracy, it significantly saves storage space and enables fast searching in massive amounts of data.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a lightweight network packet storage method with low storage overhead is provided, comprising the following steps: Parse the original network data packet file to obtain the original traffic packets, and extract the multi-dimensional network feature information of the original traffic packets. The multi-dimensional network feature information includes network address information, timestamp sequence and physical offset location. The network address information is mapped to short identifiers and then dictionary-encoded to generate dictionary-based data. Identify network sessions in the original traffic packets, and use a packet number threshold for determining the size of the network session to perform adaptive flow classification evaluation on the network sessions. Network sessions with a packet number threshold greater than or equal to the packet number threshold are classified as elephant flows, and network sessions with a packet number threshold less than the packet number threshold are classified as small flows. Based on the results of the stream classification evaluation, block storage operation is performed on the elephant stream and a corresponding block index structure is established. Continuous write storage operation is performed on the small stream, and the data block files after split storage are merged and output. Write the timestamp sequence and the physical offset position into an offset file to generate the offset file that maps to the data block file; A hierarchical retrieval master index file is constructed based on the dictionary-based data and the block index structure.

[0006] Optionally, the step of mapping the network address information to short identifiers for dictionary encoding to generate dictionary-based data includes: Extract the source network protocol address, destination network protocol address, and media access control address from the network address information; Establish a mapping and conversion relationship, and use numerical numbers to replace the source network protocol address, the destination network protocol address, and the media access control address respectively to form a dictionary object; The dictionary-based objects are organized and written into the dictionary storage area to generate the dictionary-based data.

[0007] Optionally, before the step of performing adaptive flow classification evaluation of the network session using a packet count threshold for determining the network session size, the method further includes: Obtain the five-tuple communication attributes corresponding to the network session; Based on the five-tuple communication attributes, the network session is split into a first unidirectional data stream from client to server and a second unidirectional data stream from server to client. Independent flow statistics context environments are established for the first unidirectional data stream and the second unidirectional data stream, and flow packet count information is recorded independently in the flow statistics context environment; The process of using the packet count information to trigger the step of adaptive flow classification evaluation of the network session using a packet count threshold used to determine the size of the network session.

[0008] Optionally, the step of performing block storage operation on the elephant stream and establishing a corresponding block index structure includes: The elephant stream is sliced ​​according to the size parameter used to limit the number of messages contained in a single block, generating multiple data block sequences of equal magnitude. Calculate the starting offset pointer and time span range for each data block sequence to generate the block index structure; The block index structure is associated with the corresponding data block sequence and stored in the physical medium.

[0009] Optionally, the step of constructing a hierarchical retrieval master index file based on the dictionary-based data and the block index structure includes: Construct a file header area, and record basic positioning information containing a global time span within the file header area; Construct a basic information region containing offset addressing rules, and establish an access link to the offset structure of the dictionary area through the basic information region; Under the dictionary offset structure, a first dictionary sub-region containing the dictionary data is encapsulated, and a second index sub-region containing session index data and destination network protocol address index data is constructed, which is then encapsulated as the main retrieval index file.

[0010] Optionally, after the step of encapsulating the entire process into the main index file, an external retrieval step based on the main index file is further included, the external retrieval step including: Receive external query commands that carry time range conditions and target address attributes; Read the file header area of ​​the main index file and compare the time range condition with the global time span range; If the comparison confirms the existence of overlapping intervals and the verification is successful, the basic information area and the first dictionary sub-area of ​​the main index file are loaded, and the target address attribute is translated into the corresponding target short identifier. Based on the target short identifier, an inverted index match is performed in the second index sub-area to output the target session identifier.

[0011] Optionally, the external retrieval step, before loading the basic information area and the first dictionary sub-area in the main retrieval index file, further includes: A probabilistic data filter is activated, and multiple sets of hash mapping functions are used to perform bit array position projection calculations on the target address attributes. Verify that all projection positions output by the projection calculation are in the active calibration state in the bit array; If a projection location in an inactive calibration state is detected, it is directly determined that the network packet associated with the target address attribute is missing, and the execution link of the current external query instruction is terminated.

[0012] Optionally, after the step of performing inverted index matching in the second index sub-region based on the target short identifier and outputting the target session identifier, the method further includes: Parse the target session identifier to query the block index structure and locate the reading reference coordinates of the target data packet corresponding to the target session identifier; Based on the read reference coordinates, the offset file is triggered to extract the timestamp and physical offset position corresponding to the target data packet; Based on the physical offset position, the original continuous message payload data segments are restored in the data block file.

[0013] Optionally, the step of performing continuous write storage operations on the small streams and merging and outputting the split storage data block files includes: Detect network packets belonging to the small flow; Based on the arrival time sequence of the network data packets at the physical port, the network data packets are directly processed into payload serialization to generate a continuous payload data stream; The continuous payload data stream is sequentially appended and output to the end region of the data block file.

[0014] Secondly, a lightweight network packet storage and retrieval system with low storage overhead is provided, including: The data parsing module is configured to parse the original network data packet file to obtain the original traffic packets and extract the multi-dimensional network feature information of the original traffic packets. The multi-dimensional network feature information includes network address information, timestamp sequence and physical offset position. The dictionary encoding module is configured to communicate with the data parsing module to map the network address information into short identifiers for dictionary encoding processing, thereby generating dictionary-based data. The flow classification module is configured to communicate with the dictionary encoding module, identify network sessions in the original traffic packets, and perform adaptive flow classification evaluation on the network sessions using a packet number threshold for determining the size of the network sessions. Network sessions with a packet number threshold greater than or equal to the packet number threshold are classified as elephant flows, and network sessions with a packet number threshold less than the packet number threshold are classified as small flows. The compression writing module is configured to communicate with the stream classification module, perform block storage operation on the elephant stream and establish a corresponding block index structure according to the stream classification evaluation result, perform continuous write storage operation on the small stream, and merge and output the data block file after stream split storage; The compression writing module is further configured to write the timestamp sequence and the physical offset position to generate an offset file that maps to the data block file, and to construct a hierarchical retrieval master index file based on the dictionary data and the block index structure.

[0015] Thirdly, an electronic device is provided, comprising: a processor and a memory; the memory is used to store a computer program, which, when executed by the processor, causes the electronic device to perform the lightweight network packet storage method with low storage overhead described in the first aspect.

[0016] In one possible design, the electronic device described in the third aspect may further include a transceiver. This transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the electronic device described in the third aspect and other electronic devices.

[0017] In the embodiments of the present invention, the electronic device described in the third aspect may be a terminal, or a chip (system) or other component or assembly disposed in the terminal, or a system containing the terminal.

[0018] Fourthly, a computer-readable storage medium is provided, comprising: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer causes the computer to perform the lightweight network data packet storage method with low storage overhead described in the first aspect.

[0019] In summary, the above methods and systems have the following technical effects: 1. In this invention, by introducing a dictionary encoding processing mechanism, the redundant and frequently reused network address information in the original message is stripped and mapped to short identifiers of equal length, which fundamentally eliminates static text redundancy. At the same time, the high-consumption string pattern scanning is converted into integer numerical comparison that is friendly to the underlying architecture.

[0020] 2. In this invention, fragmentation is reduced by optimizing the read and write throughput of the underlying storage medium, adaptive flow classification evaluation is implemented based on the packet number threshold, slice and block and independent index management are adopted for large flow to ensure the orderly loading of long connection, and a physical write strategy of continuously appending to the end according to the arrival time sequence is adopted for massive small flow, thus eliminating the random seek penalty and metadata expansion caused by discrete short connection.

[0021] 3. In this invention, a fast time window initial screening is achieved by constructing a hierarchical master index file with a global time span header, and a probabilistic data filter is creatively placed in front. The missing status of the target address is determined instantly by hash projection, thereby directly blocking external query commands that fail and unloading the unnecessary scanning pressure of the underlying disk. Attached Figure Description

[0022] Figure 1 A flowchart illustrating a lightweight network packet storage method with low storage overhead provided in an embodiment of the present invention; Figure 2 A block diagram of a lightweight network packet storage system with low storage overhead provided in an embodiment of the present invention; Figure 3 A schematic diagram of the storage process of a lightweight network packet storage system with low storage overhead provided in an embodiment of the present invention; Figure 4 A schematic diagram of the retrieval process for a lightweight network data packet storage method with low storage overhead provided in an embodiment of the present invention; Figure 5 A schematic diagram of the index file structure for a lightweight network packet storage method with low storage overhead provided in an embodiment of the present invention; Figure 6 A schematic diagram of the data transmission direction for a lightweight network packet storage method with low storage overhead provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a session scheme based on existing technology. Figure 8 This is a schematic diagram of a session scheme for a lightweight network packet storage method with low storage overhead provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0024] In this embodiment of the invention, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information is called the information to be instructed. In specific implementation, there are many ways to instruct the information to be instructed, such as, but not limited to, directly instructing the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly instruct the information to be instructed by instructing other information, where there is a correlation between the other information and the information to be instructed. It can also instruct only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing instruction overhead to some extent. Simultaneously, common parts of various pieces of information can be identified and uniformly indicated to reduce the instruction overhead caused by individually indicating the same information.

[0025] Furthermore, the specific indication method can also be any existing indication method, such as, but not limited to, the above-mentioned indication methods and their various combinations. Specific details of various indication methods can be found in existing technologies, and will not be elaborated upon here. As described above, for example, when multiple pieces of information of the same type need to be indicated, the indication methods for different pieces of information may differ. In specific implementation, the required indication method can be selected according to specific needs. This embodiment of the invention does not limit the selected indication method; therefore, the indication methods involved in this embodiment of the invention should be understood to cover various methods that enable the party to be indicated to obtain the information to be indicated.

[0026] It should be understood that the information to be indicated can be sent as a whole or divided into multiple sub-information messages sent separately, and the sending period and / or timing of these sub-information messages can be the same or different. The specific sending method is not limited in this embodiment of the invention. The sending period and / or timing of these sub-information messages can be predefined, for example, according to a protocol, or configured by the sending device by sending configuration information to the receiving device.

[0027] "Predefined" or "pre-configured" can be achieved by pre-saving corresponding codes, tables, or other means that can be used to indicate relevant information in the device. This embodiment of the invention does not limit the specific implementation method. "Saving" can refer to saving in one or more memories. These memories can be separate installations or integrated into the encoder, decoder, processor, or electronic device. Alternatively, some memories can be separately installed, while others are integrated into the decoder, processor, or electronic device. The type of memory can be any form of storage medium, and this embodiment of the invention does not limit this.

[0028] In the embodiments of this invention, "protocol" may refer to a protocol family in the field of communication, a standard protocol with a similar protocol family frame structure, or a related protocol applied to a future lightweight network data packet storage method system with low storage overhead. The embodiments of this invention do not specifically limit this.

[0029] In this embodiment of the invention, descriptions such as "when," "under the circumstances," "if," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not limited to a specific time. They do not require the device to make a judgment action during implementation, nor do they imply any other limitations.

[0030] In the description of the embodiments of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in the embodiments of the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of the embodiments of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or order of execution, and that "first," "second," etc., are not necessarily different. Furthermore, in the embodiments of this invention, words such as "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0031] The network architecture and business scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0032] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention. The lightweight network packet storage method with low storage overhead includes the following steps: Parse the original network packet file to obtain the original traffic packets, and extract the multi-dimensional network feature information of the original traffic packets. The multi-dimensional network feature information includes network address information, timestamp sequence and physical offset location. Network address information is mapped to short identifiers and then dictionary-encoded to generate dictionary-based data. Identify network sessions in the raw traffic packets, and use the packet number threshold used to determine the size of the network session to perform adaptive flow classification evaluation of the network session. Network sessions with a packet number greater than or equal to the packet number threshold are classified as elephant flows, and network sessions with a packet number less than the packet number threshold are classified as small flows. Based on the results of the stream classification evaluation, block storage operations are performed on the large stream and a corresponding block index structure is established, while continuous write storage operations are performed on the small stream, and the data block files after split storage are merged and output. Write the timestamp sequence and physical offset position to the offset file to generate an offset file that maps to the data block file; The hierarchical retrieval master index file is constructed based on dictionary-based data and block index structure.

[0033] Specifically, in one embodiment, lightweight processing of network packets is achieved based on feature extraction and adaptive traffic splitting mechanisms. First, the original network packets are parsed to extract multi-dimensional features such as network address timestamps and physical offsets. Long network addresses are mapped to short identifiers for dictionary encoding. Then, network sessions in the original packets are identified, and flow classification is performed using a packet count threshold to dynamically divide traffic into large and small flows. Differentiated storage strategies are adopted for different flow sizes: large flows are stored in blocks and a block index is built; small flows are continuously written. Finally, the split-stored data block files are merged and output. The timestamps and physical offsets are written to independent offset files, and a hierarchical retrieval master index file is constructed by combining dictionary data and the block index structure. This approach reduces network packet storage overhead and significantly improves the efficiency of subsequent feature retrieval and packet reconstruction. By replacing long addresses with short identifiers through dictionary encoding, it eliminates address data redundancy that frequently occurs in massive packets. The adaptive flow classification and differentiated storage mechanism based on packet quantity thresholds accurately adapts to the uneven distribution of network traffic. This avoids disk fragmentation caused by frequent random writes of small flows and ensures the orderly management of large flows through block storage, thereby optimizing the overall storage space utilization. The independently generated offset file and the hierarchical retrieval master index file effectively decouple metadata from the underlying physical payload, enabling external query commands to quickly locate the target session through a lightweight index link. This greatly reduces the input / output overhead of system reading and accelerates the accurate tracing process of network packets.

[0034] Furthermore, the steps of mapping network address information to short identifiers and performing dictionary encoding to generate dictionary-based data include: Extract the source network protocol address, destination network protocol address, and media access control address from the network address information; Establish a mapping and conversion relationship, and use numerical numbers to replace the source network protocol address, destination network protocol address, and media access control address respectively to form a dictionary object; Organize dictionary-based objects and write them to the dictionary storage area to generate dictionary-based data.

[0035] By extracting the source network protocol address, destination network protocol address, and media access control address, a multidimensional network address mapping relationship is established. These high-dimensional features are then replaced with numerical identifiers to form dictionary-based objects stored in the dictionary storage area. This metadata dictionaryging process can be expressed by a compression mapping formula: in This represents the total storage overhead after compression, where n is the total number of unique network address strings extracted from the original traffic. The frequency of a specific network address appearing in a message. A fixed length is allocated for short identifiers with numerical designations. To maintain the static space usage of the independent dictionary storage area.

[0036] In the underlying processing path, all repetitive network feature information in long string format is stripped and converted into globally unique short integer values ​​of fixed length. This reduces physical storage redundancy at the metadata level and eliminates the high computational cost of string matching in subsequent retrieval stages.

[0037] Decapping replaces the long string addresses that appear repeatedly in massive packets with numerical numbers of equal length, which can compress the header size of the original traffic packets, freeing up valuable storage space. It also enables external retrieval to directly perform continuous integer value comparisons in memory through short identifiers, completely avoiding the inefficient long text pattern scanning mechanism that is prone to cache misses.

[0038] Furthermore, before the step of performing adaptive flow classification assessment of a network session using a packet count threshold used to determine the size of the network session, the following steps are also included: Retrieve the five-tuple communication attributes corresponding to the network session; Based on the five-tuple communication properties, the network session is split into a first unidirectional data stream from the client to the server and a second unidirectional data stream from the server to the client. Independent flow statistics contexts are established for the first unidirectional data stream and the second unidirectional data stream, and flow packet count information is recorded independently in the flow statistics contexts. The process uses flow packet count information to trigger an adaptive flow classification assessment of a network session using a packet count threshold used to determine the size of the network session.

[0039] The decapping mechanism finely decomposes the network session entity identified based on the five-tuple communication attributes into two independent unidirectional data streams: client to server and server to client. It also creates independent flow statistics context environments for these two directions to independently record flow packet count information. The five-tuple information includes source IP address, destination IP address, source port number, destination port number, and transport layer protocol.

[0040] The determination logic of this bidirectional splitting and statistical triggering mechanism can be defined by the direction packet counting formula: in This represents the cumulative number of data packets on a specific one-way communication link dir, where m is the total number of packets captured in that direction. .

[0041] By independently monitoring the count values ​​in both directions, when the count result in any one-way context reaches the preset packet count threshold, an adaptive flow classification evaluation action is immediately triggered without waiting for the overall convergence of bidirectional communication. It overcomes the risk of misjudging the flow scale in asymmetric network communication scenarios, improves the sensitivity of capturing large network flows and the rigor of state tracking. To avoid the problem that if only the sum of the two-way flow of the session is used as the sole statistical metric due to the asymmetry in the order of the request and response messages, it is easy for important large-scale one-way data transmission to be masked by the average of a large number of small two-way flows. By isolating and tracking one-way packet counts in an independent context, it can more accurately identify hidden large flows with one-sided high-frequency concurrency characteristics. At the same time, it effectively avoids memory conflicts caused by directional state confusion during multi-threaded packet processing.

[0042] Furthermore, the steps of performing block storage operations on the elephant stream and establishing the corresponding block index structure include: The elephant stream is sliced ​​according to the size parameter used to limit the number of messages contained in a single block, generating multiple data block sequences of equal size. Calculate the starting offset pointer and time span range for each data block sequence to generate a block index structure; The block index structure is associated with the corresponding data block sequence and stored in the physical medium.

[0043] For massive session sets identified as "elephant streams," forced segmentation is performed based on preset size parameters, generating multiple data block sequences of equal magnitude. For each slice, a starting offset pointer and time span are independently calculated to construct a mapped block index structure. This segmentation encapsulation logic can be abstracted as a sequence generation formula. in This represents the total number of data block sequences ultimately generated by splitting the elephant stream. The floor operator ensures that even if the last slice is not full, it is allocated as a separate data block. This represents the total number of messages accumulated by the elephant stream over its lifecycle. The system's preset size limit parameter is the maximum number of messages that can be included in a single data block.

[0044] By introducing this threshold intervention, the infinitely extending long-connection, high-volume traffic is proactively truncated into multiple storage units with clear boundaries, standard capacity, and independent physical association attributes. This reduces the stringent dependence of ultra-large network flows on the underlying contiguous physical storage space and significantly lowers the risk of system memory overflow during data backtracking. During periods of sustained activity, network traffic generates extremely large continuous packets. Forcing these packets as a single entity to be written to the physical medium would not only easily lead to disk contiguous space allocation failures but would also instantly exhaust the server's available memory during subsequent single read operations. Slicing these packets into equal-order sequences and combining them with time-span block indexes allows the retrieval system to only page and load data blocks overlapping with the query time interval into memory during point-to-point verification. This mechanism fundamentally endows the system with the resilience to handle and parse extreme traffic surges. Furthermore, the steps for constructing a hierarchical retrieval master index file based on dictionary-based data and block index structure include: Construct a file header area, and record basic positioning information containing a global time span within the file header area; Construct a basic information area containing offset addressing rules, and establish access links to the offset structure of the dictionary area through the basic information area; The first dictionary sub-region containing dictionary-based data is encapsulated under the dictionary offset structure, and a second index sub-region containing session index data and destination network protocol address index data is constructed. The whole is encapsulated into a retrieval main index file.

[0045] Please see Figure 5 A tree-structured master index file, containing a file header area, a basic information area, and multiple underlying data sub-areas, is constructed from top to bottom using logical space isolation. The physical offset jump of the index addressing path can be expressed by the addressing rule formula: in This indicates the actual physical starting address of the target's first dictionary sub-region or second index sub-region in memory or on disk. This is the absolute reference starting point obtained from the basic positioning information recorded in the file header area. This represents the structured relative offset constant for a specific type of data sub-region parsed from the basic information region, thereby extracting the global time span to the top layer, while encapsulating the index data containing the source address and destination network protocol address at the bottom layer. By establishing a high-concurrency query entry structure, the scanning path for invalid files and disk seek latency are shortened. The file header containing the global time span is placed directly at the beginning of the file, allowing external retrieval commands to filter out file entities with mismatched times through tiny byte reads in the extremely short instant of reaching the physical medium, without the need to expand a huge main index. The layered design of isolating and decoupling the basic offset rules and business index data ensures that even if the size of the underlying session index library expands dramatically due to a sudden increase in traffic, the stability of the core jump structure of the main index file will not be compromised in the slightest, thus guaranteeing long-term retrieval vitality in a full data archiving environment.

[0046] Furthermore, after the overall process is encapsulated into the step of retrieving the main index file, it also includes external retrieval steps based on the main index file. These external retrieval steps include: Receive external query commands that carry time range conditions and target address attributes; Read the file header area of ​​the main index file and compare the time range conditions with the global time span range; If the comparison confirms the existence of overlapping intervals and the verification is successful, the basic information area and the first dictionary sub-area in the main index file are loaded, and the target address attribute is translated into the corresponding target short identifier. Based on the target short identifier, inverted index matching is performed in the second index sub-area to output the target session identifier.

[0047] An external instruction execution chain based on multi-dimensional conditional dimensionality reduction and stacking is implemented. After verifying the global time overlap interval, the external target address attributes are translated into short identifiers by loading a dictionary sub-region, and finally, inverted tree matching is completed within the index sub-region. The mathematical model of this retrieval verification logic can be expressed as the result set formula: in The final set of target session identifiers output by the system. and These represent the time range conditions of the external query command and the global time span in the main index header, respectively. The short identifier translated from the first dictionary sub-region for the target address attribute. The inverted index matching space residing in the second index sub-region is executed by strictly adhering to the serial constraint of first comparing time, then translating identifiers, and finally performing inverted matching; Through a progressive filtering and load reduction mechanism, the system achieves extremely low-latency target communication session location amidst massive and chaotic network data packets. By utilizing the initial overlap comparison of the outer time dimension, it shields the blind unpacking behavior of a large number of irrelevant time period data blocks, quickly translates the lengthy and complex string addresses of external input into fixed-length short integer identifiers agreed upon by the system, and reduces the string pattern matching step, which consumes the most processor cycles, into a continuous integer search operation that is extremely user-friendly to the underlying instruction set, thereby reducing the computational power consumption of query instructions.

[0048] Furthermore, the external retrieval step, before loading the basic information area and the first dictionary sub-area in the main retrieval index file, also includes: Start a probabilistic data filter and use multiple sets of hash mapping functions to perform bit array position projection calculations on the target address attributes; Verify that all projection positions output by the projection calculation are in the active calibration state in the bit array; If a projection location in an inactive calibration state is detected, it is directly determined that the network packet associated with the target address attribute is missing, and the execution link of the current external query command is terminated.

[0049] Before the core information area of ​​the main index file is loaded in large quantities, a data filter based on a probabilistic bit array structure is introduced as a pre-filter. Multiple hash mapping functions are used to perform hash operations on the target address attributes to pre-determine the data missing status. The core Boolean decision equation of this pre-detection mechanism is: in This represents the final output state determining whether an associated network packet may exist, where k represents the number of sets of independent hash mapping functions executed concurrently. is the hash projection coordinate value of the target address attribute by the i-th independent hash function, and B is a continuous bit array resident in the system cache. If any bit projection in the inactive calibration state is detected in the operation of this equation, the entire Boolean product will immediately output a false value and forcibly interrupt all subsequent loading and matching operations of the current query link.

[0050] By trading extremely low cache usage costs for the ability to instantly intercept invalid query commands, the read pressure is effectively reduced. In real traffic security investigation scenarios, most externally specified risk target addresses do not actually exist in the traffic slices currently stored in the system. If the missed queries are not blocked and the inverted index is directly accessed, a huge amount of disk bandwidth will be wasted.

[0051] The upfront probabilistic hash projection verification can directly block the verification of target addresses that are in a missing state from the outside, eliminating the need for subsequent loading of large dictionary sub-regions and preserving ample throughput margin.

[0052] Furthermore, after performing inverted index matching based on the target short identifier in the second index sub-region and outputting the target session identifier, the process also includes: Parse the target session identifier to query the block index structure and locate the reading reference coordinates of the target data packet corresponding to the target session identifier; Based on the read baseline coordinates trigger offset file, extract the timestamp and physical offset position corresponding to the target data packet; The original continuous message payload data segments are restored in the data block file based on the physical offset position.

[0053] The reference coordinates are extracted and read by reverse querying the block index structure using the target session identifier output by inverted matching, and the physical offset position is obtained by combining it with an independent offset file. Finally, the original message payload is extracted and reconstructed at a fixed point in the data block file. This precise physical delimitation and payload extraction process is controlled by the absolute vernier calculation formula: in This represents the absolute physical reference cursor used by the system pointer to read the raw, continuous message payload from the data block file. These are the reference coordinates for reading the j-th slice data block under the target session identifier, located backwards from the block index structure. It is based on local timestamps Trigger and extract additional physical offset positions from the offset file.

[0054] The accumulated physical cursor directly drills down into the underlying media to extract the required binary payload, achieving zero-redundancy and precise jump extraction from the macro session index to the micro underlying payload bytes. It abandons the coarse-grained restoration mode of full file linear traversal and full parsing. The lightweight storage strategy has highly condensed network traffic. Relying solely on block indexes can only locate the macro slice boundaries. By introducing an independent and parallel offset file system, it can accurately jump within data blocks, avoiding the dumping of a large number of bypass irrelevant payloads into the server memory when restoring specific target data packets. This ensures that the restored continuous packet payload data segments can perfectly maintain their original network topology arrival order, meeting the stringent evidence requirements of security audits for high-fidelity traffic playback.

[0055] Furthermore, the steps of performing continuous write storage operations on the small streams and merging the output data block files after split storage include: Detect network packets belonging to small flows; Based on the arrival time sequence of network data packets at the physical port, the network data packets are directly processed into payload serialization to generate a continuous payload data stream; The continuous payload data stream is appended sequentially to the end area of ​​the data block file.

[0056] For low-load, small-flow network packets detected during the flow classification and evaluation phase, the complex indexing and block-splitting logic is completely avoided. Instead, the packets are linearly serialized according to their actual arrival time sequence at the physical network port, and the continuous payload data stream is directly appended to the end of the file.

[0057] The continuously appending output logic follows a time series merging formula: Where ⊕ is the ordered byte stream concatenation operator, which forces that the subsequent data stream must be tightly attached to the end of the previous data stream, and does not allow any position to be interchanged, ensuring the unidirectional extension of physical writes. This represents the final physical state of the end of the data block file after merging and appending the output; This is the static data block area at the end of the current data block file where the history has been fixed. Then at the point in time The captured payload of the m-th message belonging to the small stream, where every message in the sequence satisfies the time monotonically increasing constraint. .

[0058] By maximizing the optimal sequential write characteristics of the underlying physical disk media, the impact of file fragmentation caused by massive discrete short connections is eliminated at the source. By integrating these fragmented network data packets into a continuous payload stream and appending them uniformly to the output, the underlying driver can complete persistent disk writing at the maximum peak throughput, pushing the overall system's throughput capacity limit for high-concurrency connections.

[0059] In another embodiment, a lightweight network packet storage and retrieval system with low storage overhead is also disclosed, comprising: The data parsing module is configured to parse the original network data packet file to obtain the original traffic packets and extract the multi-dimensional network feature information of the original traffic packets. The multi-dimensional network feature information includes network address information, timestamp sequence and physical offset location. The dictionary encoding module is configured to communicate with the data parsing module, map network address information into short identifiers for dictionary encoding, and generate dictionary-based data. The flow classification module is configured to communicate with the dictionary encoding module to identify network sessions in the original traffic packets. It uses a packet number threshold to determine the size of the network session to perform adaptive flow classification evaluation of the network session. Network sessions with a packet number threshold greater than or equal to the packet number threshold are classified as elephant flows, and network sessions with a packet number threshold less than the packet number threshold are classified as small flows. The compression writing module is configured to communicate with the stream classification module. Based on the stream classification evaluation results, it performs block storage operations on large streams and establishes corresponding block index structures, performs continuous write storage operations on small streams, and merges and outputs the data block files after stream splitting and storage. The compression writing module is also configured to write timestamp sequences and physical offset positions to generate offset files that map to data block files, and to build a hierarchical retrieval master index file based on dictionary-based data and block index structure.

[0060] The electronic device provided in this embodiment of the invention, exemplarily, can be a network device, or a chip (system) or other component or assembly that can be disposed in a network device. The electronic device may include a processor. Optionally, the electronic device may also include a memory and / or a transceiver. The processor is coupled to the memory and transceiver, for example, by means of a communication bus connection.

[0061] In the several embodiments provided in this invention, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0062] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0063] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0064] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A lightweight network data packet storage method with low storage overhead, characterized in that, Includes the following steps: Parse the original network data packet file to obtain the original traffic packets, and extract the multi-dimensional network feature information of the original traffic packets. The multi-dimensional network feature information includes network address information, timestamp sequence and physical offset location. The network address information is mapped to short identifiers and then dictionary-encoded to generate dictionary-based data. Identify network sessions in the original traffic packets, and use a packet number threshold for determining the size of the network session to perform adaptive flow classification evaluation on the network sessions. Network sessions with a packet number threshold greater than or equal to the packet number threshold are classified as elephant flows, and network sessions with a packet number threshold less than the packet number threshold are classified as small flows. Based on the results of the stream classification evaluation, block storage operation is performed on the elephant stream and a corresponding block index structure is established. Continuous write storage operation is performed on the small stream, and the data block files after split storage are merged and output. Write the timestamp sequence and the physical offset position into an offset file to generate the offset file that maps to the data block file; A hierarchical retrieval master index file is constructed based on the dictionary-based data and the block index structure.

2. The lightweight network packet storage method with low storage overhead according to claim 1, characterized in that, The step of mapping the network address information to short identifiers and performing dictionary encoding to generate dictionary-based data includes: Extract the source network protocol address, destination network protocol address, and media access control address from the network address information; Establish a mapping and conversion relationship, and use numerical numbers to replace the source network protocol address, the destination network protocol address, and the media access control address respectively to form a dictionary object; The dictionary-based objects are organized and written into the dictionary storage area to generate the dictionary-based data.

3. The lightweight network packet storage method with low storage overhead according to claim 1, characterized in that, Before the step of performing adaptive flow classification evaluation of the network session using a packet count threshold for determining the network session size, the method further includes: Obtain the five-tuple communication attributes corresponding to the network session; Based on the five-tuple communication attributes, the network session is split into a first unidirectional data stream from client to server and a second unidirectional data stream from server to client. Independent flow statistics context environments are established for the first unidirectional data stream and the second unidirectional data stream, and flow packet count information is recorded independently in the flow statistics context environment; The process of using the packet count information to trigger the step of adaptive flow classification evaluation of the network session using a packet count threshold used to determine the size of the network session.

4. The lightweight network packet storage method with low storage overhead according to claim 1, characterized in that, The steps of performing block storage operations on the elephant stream and establishing corresponding block index structures include: The elephant stream is sliced ​​according to the size parameter used to limit the number of messages contained in a single block, generating multiple data block sequences of equal magnitude. Calculate the starting offset pointer and time span range for each data block sequence to generate the block index structure; The block index structure is associated with the corresponding data block sequence and stored in the physical medium.

5. The lightweight network packet storage method with low storage overhead according to claim 2, characterized in that, The step of constructing a hierarchical retrieval master index file based on the dictionary-based data and the block index structure includes: Construct a file header area, and record basic positioning information containing a global time span within the file header area; Construct a basic information region containing offset addressing rules, and establish an access link to the offset structure of the dictionary area through the basic information region; Under the dictionary offset structure, a first dictionary sub-region containing the dictionary data is encapsulated, and a second index sub-region containing session index data and destination network protocol address index data is constructed, which is then encapsulated as the main retrieval index file.

6. The lightweight network packet storage method with low storage overhead according to claim 5, characterized in that, Following the step of encapsulating the entire process into the main index file, the system further includes an external retrieval step based on the main index file. This external retrieval step includes: Receive external query commands that carry time range conditions and target address attributes; Read the file header area of ​​the main index file and compare the time range condition with the global time span range; If the comparison confirms the existence of overlapping intervals and the verification is successful, the basic information area and the first dictionary sub-area of ​​the main index file are loaded, and the target address attribute is translated into the corresponding target short identifier. Based on the target short identifier, an inverted index match is performed in the second index sub-area to output the target session identifier.

7. The lightweight network packet storage method with low storage overhead according to claim 6, characterized in that, The external retrieval step, before loading the basic information area and the first dictionary sub-area in the main retrieval index file, further includes: A probabilistic data filter is activated, and multiple sets of hash mapping functions are used to perform bit array position projection calculations on the target address attributes. Verify that all projection positions output by the projection calculation are in the active calibration state in the bit array; If a projection location in an inactive calibration state is detected, it is directly determined that the network packet associated with the target address attribute is missing, and the execution link of the current external query instruction is terminated.

8. The lightweight network packet storage method with low storage overhead according to claim 6, characterized in that, After the step of performing inverted index matching in the second index sub-region based on the target short identifier and outputting the target session identifier, the method further includes: Parse the target session identifier to query the block index structure and locate the reading reference coordinates of the target data packet corresponding to the target session identifier; Based on the read reference coordinates, the offset file is triggered to extract the timestamp and physical offset position corresponding to the target data packet; Based on the physical offset position, the original continuous message payload data segments are restored in the data block file.

9. The lightweight network packet storage method with low storage overhead according to claim 1, characterized in that, The step of performing continuous write storage operations on the small streams and merging and outputting the split storage data block files includes: Detect network packets belonging to the small flow; Based on the arrival time sequence of the network data packets at the physical port, the network data packets are directly processed into payload serialization to generate a continuous payload data stream; The continuous payload data stream is sequentially appended and output to the end region of the data block file.

10. A lightweight network packet storage and retrieval system with low storage overhead, comprising the lightweight network packet storage method with low storage overhead according to any one of claims 1-9, characterized in that, include: The data parsing module is configured to parse the original network data packet file to obtain the original traffic packets and extract the multi-dimensional network feature information of the original traffic packets. The multi-dimensional network feature information includes network address information, timestamp sequence and physical offset position. The dictionary encoding module is configured to communicate with the data parsing module to map the network address information into short identifiers for dictionary encoding processing, thereby generating dictionary-based data. The flow classification module is configured to communicate with the dictionary encoding module, identify network sessions in the original traffic packets, and perform adaptive flow classification evaluation on the network sessions using a packet number threshold for determining the size of the network sessions. Network sessions with a packet number threshold greater than or equal to the packet number threshold are classified as elephant flows, and network sessions with a packet number threshold less than the packet number threshold are classified as small flows. The compression writing module is configured to communicate with the stream classification module, perform block storage operation on the elephant stream and establish a corresponding block index structure according to the stream classification evaluation result, perform continuous write storage operation on the small stream, and merge and output the data block file after stream split storage; The compression writing module is further configured to write the timestamp sequence and the physical offset position to generate an offset file that maps to the data block file, and to construct a hierarchical retrieval master index file based on the dictionary data and the block index structure.

Images