A network data packet storage and query method and related device

Abstract

The present application relates to a kind of network data packet storage and query method and related equipment.The method includes: obtaining the load data of network data packet, load data is divided into multiple data segments;Based on the value range of fixed-length byte length, construct the multi-level index structure containing directory level index and file level index;Load data is stored in data file, and multi-level index structure is stored in index database;Receive query request, determine target data file range according to query time period, according to fixed-length byte length, the target query data is divided into multiple query substrings;Utilize multiple query substrings in target data file range to query directory level index, filter out candidate file set;Utilize multiple query substrings to query file level index corresponding to candidate file set, determine target file;Search in target file, output query result.The method solves the problems of large storage overhead, low query efficiency and poor query accuracy in related art.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and related equipment for storing and retrieving network data packets. Background Technology

[0002] In scenarios such as network security management, network forensics, and fault backtracking analysis, the full storage of raw data packets on network links and the support for post-event query retrieval are critical requirements. Traditional data packet storage and retrieval systems typically use indexes based on timestamps, five-tuples (source / destination IP address, source / destination port, protocol), or application-layer session metadata (such as URIs (Uniform Resource Identifiers) and User Agents in the HTTP (Hypertext Transfer Protocol) protocol). Users can quickly locate data packets within a specific time period or specific network session based on these indexes.

[0003] However, when it is necessary to retrieve the original content of the data packet payload (i.e. the effective payload transmitted in the data packet), the solutions in the relevant technologies have the following significant defects: (1) The payload content in a large number of original data packets is compressed, which makes it impossible for conventional keyword matching or pattern search to hit the target, thus leading to the failure of forensic analysis; (2) Since attackers widely use encoding or obfuscation techniques to evade detection, traditional signature-based rules or real-time query systems have difficulty identifying these transformed attack traffic, and are also difficult to hit the target, making it difficult to trace the source of zero-day attacks or complex attacks; (3) Since the amount of network data packets is extremely large, if such a large amount of data is to be fully stored and an index supporting content retrieval is to be built, traditional methods (such as full-text word segmentation index, inverted index) will cause index explosion - the index volume may even exceed the original data, and a large number of random I / O operations during the query will seriously squeeze the bandwidth of the data packet writing process, resulting in overall system packet loss, and ultimately failing to achieve lossless storage of the full amount of data packets.

[0004] There is currently no effective solution to the problems of high storage overhead, low query efficiency, and poor query accuracy in related technologies. Summary of the Invention

[0005] The present invention provides a method and related device for storing and querying network data packets, which at least partially solves the problems of high storage overhead, low query efficiency, and poor query accuracy in related technologies.

[0006] To address the aforementioned problems, one aspect of this invention provides a method for storing and retrieving network data packets, comprising: Capture network packet streams on the network link and obtain the payload data of each network packet in the network packet stream; Based on a preset fixed-length byte, the payload data is divided into multiple continuous data segments, each corresponding to a specific value within the fixed-length byte. Based on the value range corresponding to the fixed-length byte, a multi-level index structure including directory-level and file-level indexes is constructed, and the data segments are used to populate the corresponding index entries in the multi-level index structure. The payload data is stored in a data file, organized by time or network stream, and stored in a corresponding directory or subdirectory. The multi-level index structure is stored in an index database. The directory-level index records the existence distribution of each value corresponding to the value range in the directory-level storage space, and the file-level index records the first matching position information of each value corresponding to the value range within the corresponding data file. Receive a query request containing target query data and a query time period, determine the target data file range based on the query time period, and divide the target query data into multiple consecutive query substrings based on the fixed-length byte length; The directory-level index is queried within the target data file using multiple query substrings to obtain the existence distribution information corresponding to each query substring. Logical intersection is performed on the existence distribution information of all query substrings to filter out a candidate file set that contains all query substrings. The file-level index corresponding to each candidate file in the candidate file set is queried using multiple query substrings to obtain the first matching offset of each query substring in each candidate file. For any candidate file, based on the offset step size between adjacent query substrings when the target query data is segmented, it is verified whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size. If it does, the corresponding candidate file is confirmed as the target file. A local search is performed in the target file based on the first matching offset, and the query result containing the target query data is output.

[0007] In some embodiments, after the step of receiving a query request containing target query data and a query time period, the method further includes: If the query request also carries 5-tuple information and / or application layer metadata, then based on the 5-tuple information and / or application layer metadata, mismatched data files are filtered from the target data file range to achieve filtering updates of the target data file range.

[0008] In some embodiments, the value range includes fixed-length character combinations or fixed-length binary data blocks, wherein the fixed-length byte length is a preset positive integer; Each index entry in the directory-level index includes: a value corresponding to the value range and its existence distribution information in each data file or subdirectory under the corresponding directory; wherein, the existence distribution information adopts a bitmap data structure, and each bit in the bitmap corresponds to the sequence number of a data file or subdirectory, which is used to identify whether the value exists in the corresponding data file or subdirectory; Each index entry in the file-level index includes: a value corresponding to the value range and its position association information within the corresponding data file; wherein, the position association information includes the first matching offset when the value first appears in the data file; the position association information also includes the matching offset when the value appears a second and subsequent times in the data file.

[0009] In some embodiments, the step of dividing the target query data into multiple consecutive query substrings includes: dividing the target query data into multiple consecutive query substrings with a length equal to the fixed-length byte length by using a sliding window method, and the starting positions of adjacent query substrings differ by a preset offset step. The step of verifying whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size includes: for adjacent query substrings obtained after segmenting the target query data, calculating the difference between the first matching offset of the next query substring and the first matching offset of the previous query substring; if the difference between all adjacent query substrings is less than or equal to the offset step size, then the verification is deemed successful and conforms to the continuity feature.

[0010] In some embodiments, the step of obtaining the payload data of each network packet in the network packet stream further includes: Perform a preprocessing and restoration operation on the original payload content of each network data packet in the network data packet stream to obtain the payload data; The preprocessing and restoration operation includes at least one of the following steps: performing decompression processing on the original payload content; performing multi-layer nested decoding processing on the original payload content; and performing stream reassembly operation on network data packets belonging to the same session according to the transport layer protocol.

[0011] In some of these embodiments, it also includes: The link layer information, network layer information, and transport layer information of the network data packets are stored separately from the payload data; Among them, network packets belonging to the same network flow share a five-tuple information and link layer address information, while each network packet independently stores its sequence number, acknowledgment sequence number, payload length information and transport layer flag bit.

[0012] In some embodiments, the step of performing a logical intersection operation on the existence distribution information of all query substrings to filter out a candidate file set that simultaneously contains all query substrings includes: Perform a bitwise AND operation on the bitmap data structure corresponding to each query substring; Based on the bits with a value of 1 in the calculation result, the file number of the corresponding candidate file is determined, thus forming the candidate file set.

[0013] To address the aforementioned problems, one aspect of this invention provides a network data packet storage and retrieval apparatus, comprising: The acquisition module is used to capture network data packet streams on the network link and acquire the payload data of each network data packet in the network data packet stream; A storage module is used to divide the payload data into multiple continuous data segments based on a preset fixed-length byte length, each data segment corresponding to a specific value within the fixed-length byte length; based on the value range corresponding to the fixed-length byte length, a multi-level index structure including directory-level indexes and file-level indexes is constructed, and the data segments are used to populate the corresponding index entries in the multi-level index structure; the payload data is stored in a data file, the data file is organized by time or network stream, and stored in the corresponding directory or subdirectory; the multi-level index structure is stored in an index database; wherein, the directory-level index is used to record the existence distribution of each value corresponding to the value range in the directory-level storage space, and the file-level index is used to record the first matching position information of each value corresponding to the value range in the corresponding data file; The receiving module is used to receive a query request containing target query data and a query time period, determine the target data file range according to the query time period, and divide the target query data into multiple consecutive query substrings according to the fixed-length byte length. The filtering module is used to query the directory-level index within the target data file range using multiple query substrings, obtain the existence distribution information corresponding to each query substring, perform a logical intersection operation on the existence distribution information of all query substrings, and filter out a candidate file set that contains all query substrings. The query module is used to query the file-level index corresponding to each candidate file in the candidate file set using multiple query substrings, and obtain the first matching offset of each query substring in each candidate file; for any candidate file, based on the offset step size between adjacent query substrings when the target query data is segmented, it verifies whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size. If it does, the corresponding candidate file is confirmed as the target file; based on the first matching offset, a local search is performed in the target file, and a query result containing the target query data is output.

[0014] To address the aforementioned problems, one aspect of this invention provides an electronic device, including: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform any of the aforementioned methods for storing and retrieving network data packets.

[0015] To address the aforementioned problems, one aspect of this invention provides a non-transitory machine-readable medium storing computer instructions, the computer instructions being used to cause a computer to execute any of the aforementioned methods for storing and retrieving network data packets.

[0016] The beneficial effects of this invention are as follows: By capturing network data packet streams on a network link, the payload data of each network data packet in the network data packet stream is obtained; based on a preset fixed-length byte length, the payload data is divided into multiple continuous data segments, each data segment corresponding to a specific value under the fixed-length byte length; based on the value range corresponding to the fixed-length byte length, a multi-level index structure including directory-level index and file-level index is constructed, and the corresponding index entries in the multi-level index structure are filled using data segments; the payload data is stored in a data file, the data file is organized by time or network stream, and stored in the corresponding directory or subdirectory; the multi-level index structure is stored in an index database; wherein, the directory-level index is used to record the existence distribution information of each value corresponding to the value range in the directory-level storage space, and the file-level index is used to record the first matching position information of each value corresponding to the value range in the corresponding data file; a query request containing target query data and query time period is received, the target data file range is determined according to the query time period, and the fixed-length byte length is determined according to the fixed-length byte length. The target query data is divided into multiple consecutive query substrings based on byte length. These substrings are then used to query the directory-level index within the target data file to obtain the existence distribution information for each substring. A logical intersection operation is performed on the existence distribution information of all substrings to filter out a candidate file set that simultaneously contains all query substrings. The file-level index of each candidate file in the candidate file set is then used to obtain the first matching offset of each query substring within each candidate file. For any given candidate file, the first matching offset of the multiple query substrings is verified based on the offset step size between adjacent query substrings during the segmentation of the target query data. If the difference between the offsets of the first matching matches conforms to the continuity characteristic corresponding to the offset step size, the corresponding candidate file is confirmed as the target file. This technique involves performing a local search within the target file based on the first matching offset and outputting query results containing the target query data. It overcomes the problems of high storage overhead, low query efficiency, and poor query accuracy in related technologies. It captures network data packet streams to obtain the payload data of each network data packet. By constructing a multi-level index structure containing directory-level and file-level indexes based on a preset long byte length, it provides a space-controllable filtering and positioning basis for the search. By segmenting the target query data into the same fixed-length byte segments, it maintains consistency between the query granularity and the index granularity. By utilizing the bitmap structure of the directory-level index to perform logical intersection operations on the existence distribution of all query substrings, it achieves rapid filtering of the vast majority of irrelevant files. By using the file-level index to obtain the first matching offset and verify the continuity characteristic of adjacent substring offsets, it completes the accurate positioning and local search of the target file. Under the premise of ensuring controllable index storage space, it achieves efficient and accurate retrieval of the payload content of massive compressed and encoded network data packets, significantly improving query response speed and hit accuracy.

[0017] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the main flow of a network data packet storage and retrieval method according to one embodiment of the present invention; Figure 2 This is a schematic diagram of the main modules of a network data packet storage and retrieval device according to one embodiment of the present invention; Figure 3 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0020] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0021] When it is necessary to query and retrieve the original content of the data packet payload (i.e., the effective payload transmitted in the data packet), the network data packet storage and retrieval solutions provided by related technologies have at least the following significant drawbacks: First, the payload content in a large number of raw data packets is compressed. For example, the HTTP protocol often uses compression algorithms to compress the response body to save bandwidth. If only the compressed raw data packets are stored, when a user needs to search for a specific attack payload or sensitive information segment, the stored content has been scrambled by the compression algorithm, making conventional keyword matching or pattern search unable to find the target, leading to the failure of forensic analysis. Second, attackers widely use encoding or obfuscation techniques to evade detection. For example, they encode attack payloads using URIEncode, Base64, or Unicode, and even employ multi-level nested encoding (such as URIEncode nested with Unicode). Traditional signature-based rules or real-time detection systems struggle to identify these modified attack traffic. Even if string matching is used for retrieval afterward, the payload content has been encoded into a non-original readable form, making it difficult to find the source of zero-day attacks or complex attacks. Third, the volume of network data packets is extremely large. Taking a network with a 10Gbps outbound bandwidth as an example, the volume of raw data packets generated per second can reach 1GB. If such massive amounts of data are to be fully stored and an index supporting content retrieval is to be built, traditional methods (such as full-text word segmentation index and inverted index) will result in index explosion—the index volume may even exceed the original data, and the large number of random I / O operations during queries will severely crowd out the bandwidth of the data packet writing process, leading to overall system packet loss, and ultimately failing to achieve lossless storage of the entire data packet.

[0022] In summary, the existing technology lacks a solution that can efficiently retrieve compressed and encoded payload content in the context of massive network raw data packets, with controllable indexing overhead and fast query response.

[0023] To address the aforementioned problems, embodiments of the present invention provide a method for storing and retrieving network data packets, such as... Figure 1 As shown, the methods for storing and retrieving network data packets mainly include: Step S101: Capture the network data packet stream on the network link and obtain the payload data of each network data packet in the network data packet stream; Step S102: Based on a preset fixed-length byte, the payload data is divided into multiple continuous data segments, each corresponding to a specific value within the fixed-length byte. Based on the value range corresponding to the fixed-length byte, a multi-level index structure containing directory-level and file-level indexes is constructed, and the corresponding index entries in the multi-level index structure are filled using the data segments. The payload data is stored in a data file, which is organized by time or network stream and stored in the corresponding directory or subdirectory. The multi-level index structure is stored in the index database. The directory-level index is used to record the existence distribution of each value corresponding to the value range in the directory-level storage space, and the file-level index is used to record the first matching position information of each value corresponding to the value range in the corresponding data file. Step S103: Receive a query request containing target query data and query time period, determine the target data file range based on the query time period, and divide the target query data into multiple consecutive query substrings based on the fixed-length byte length. Step S104: Use multiple query substrings to query the directory-level index within the target data file range, obtain the existence distribution information corresponding to each query substring, perform a logical intersection operation on the existence distribution information of all query substrings, and filter out the candidate file set that contains all query substrings at the same time. Step S105: Utilize multiple query substrings to query the file-level index corresponding to each candidate file in the candidate file set, and obtain the first matching offset of each query substring within each candidate file; for any candidate file, based on the offset step size between adjacent query substrings when splitting the target query data, verify whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size. If it does, confirm the corresponding candidate file as the target file; perform a local search in the target file based on the first matching offset, and output the query results containing the target query data.

[0024] Based on the above settings, the payload data of each network packet is obtained by capturing network packet streams; a multi-level index structure containing directory-level and file-level indexes is constructed based on a preset long byte length, providing a spatially controllable filtering and positioning foundation for searching; the query granularity is kept consistent with the index granularity by dividing the target query data into segments of the same fixed byte length; the logical intersection operation of the existence distribution of all query substrings is performed using the bitmap structure of the directory-level index, enabling fast filtering of the vast majority of irrelevant files; and the offset of the first match is obtained by using the file-level index, and the offsets of adjacent substrings are verified. The continuous characteristics of the data enable precise location and local search of the target file. The progressive operations of "acquiring load data, fixed-length byte mapping, two-level query filtering, and storage location verification" ensure that each query operation is based on the previous stage of significantly narrowing the scope, ultimately obtaining the most accurate query results with minimal I / O overhead. In this way, while ensuring controllable index storage space, it overcomes the shortcomings of large storage overhead, low query efficiency, and poor accuracy in related technologies, and achieves efficient and accurate retrieval of massive compressed and encoded network data packet load content, significantly improving query response speed and hit accuracy.

[0025] According to an embodiment of the present invention, based on the above step S101, the load data can be obtained by directly using the original load content, or by obtaining the restored load data after performing preprocessing and restoration. In a specific implementation of the present invention, network packets from a specified network port (multiple packet capture ports can be set) can be captured, and the packets can be restored to a certain extent according to the current time and directory settings, as well as the restoration length settings (which can be set to not restore or restore a maximum length). Specifically, by performing preprocessing and restoration operations on the load content of the network packets, the load content in the captured original network packets, which may be in a compressed, encoded, or fragmented state, is uniformly restored into a continuous, unprocessed original byte sequence (converting the original load content into a standard format that can be directly processed by subsequent indexing and retrieval), thereby avoiding retrieval omissions due to possible compression, encoding, or fragmentation distortions in the original load.

[0026] In some embodiments, based on step S102 above, the fixed-length byte limits the total number of possible values, thus the number of index entries is fixed, the index space can be pre-allocated and does not grow with the amount of data; the directory-level index independently stores the existence distribution information of each value in the directory-level storage space, and the file-level index independently stores the first matching position information of each value in the file. The separation of the two allows for quick filtering by existence distribution and then location by the first matching position information during querying, without traversing all the data. Based on the above settings, a multi-level index structure covering all load content is established with a fixed index space, so that the existence information of each possible value at the directory level and its first matching position in the data file are pre-recorded, providing a structured filtering and location basis for subsequent queries.

[0027] In some examples, based on step S103 above, during the query phase, the range of target data files to be queried is first determined according to the query time period indicated by the query request. This determination process utilizes the time-organized directory structure of the data files, which can quickly exclude files outside the time period, thereby accelerating query efficiency. Secondly, the target query data is segmented using the same fixed-length bytes as in the storage phase, ensuring that the length of each query substring is equal to the length of the value recorded in the index. This allows each query substring to find a corresponding entry in both the directory-level and file-level indexes. In other words, the granularity of the target query data segmentation is consistent with the granularity of the index in the storage phase, enabling direct querying using the already constructed index.

[0028] In some examples, based on step S104 above, the directory-level index records the existence distribution information of each value in the directory-level storage space. By querying the existence distribution information corresponding to each query substring, it is possible to obtain which files contain the query substring. Performing a logical intersection operation on the existence distribution information of all query substrings yields a set of files that simultaneously contain all query substrings. This operation does not require accessing the data files themselves and can be completed solely based on the index information. This allows the search scope to be quickly narrowed from all data files to a set of candidate files that only contain all query substrings, significantly reducing the number of files that need further examination and helping to improve query efficiency.

[0029] In some instances, based on step S105 above, the file-level index provides the first matching offset of each query substring within the candidate file. By comparing the difference between the first matching offsets of adjacent query substrings with the offset step size during segmentation, it can be determined whether these query substrings appear consecutively in the file: only consecutive occurrences indicate that the file truly contains the complete target query content. Furthermore, after confirming the target file, a local search is performed directly to the approximate area of ​​the query content based on the known first matching offset, without needing to perform a full file scan. It is understandable that by further identifying the target file that truly contains continuous and complete query content within the candidate file set, and locating the position of the query content for a local search, both the accuracy and efficiency of the query results can be guaranteed.

[0030] According to a specific embodiment of the present invention, the step of performing a local search in the target file based on the first matching offset and outputting the query result containing the target query data may include: (1) Determining the search starting point: taking the first matching offset (denoted as first_offset) of the first query substring in the sequence of query substrings that have passed the continuity verification. Since the continuity verification has ensured that all query substrings appear consecutively in the target file, first_offset is the starting byte position of the target query data in the target file. (2) Determining the search range: in order to avoid scanning the entire target file (which may be several GB), the search is limited to a small local window. The starting point of the window is first_offset, and the ending position of the window may be: first_offset + len (target query data) + preset expansion margin (considering the possibility of overlapping matches or file alignment factors); or, based on the multiple matching offsets recorded in the file-level index (i.e., the second matching offset, the third matching offset, etc.), it is possible to directly jump to the subsequent matching position for searching. (3) Perform exact matching: Within a limited local area, use efficient string search algorithms (such as Boyer-Moore, KMP) or binary mode search to find all occurrences of the target query data. Since the search range is usually only a few times the length of the target query data (e.g., a few KB), the search speed is extremely fast, and the consumption of I / O and CPU is negligible. (4) Output query results: Organize the information of each found match (such as file path, offset, matched content fragments, etc.) and output it to the user. Furthermore, if the user requests to replay the data packet, the complete original data packet is extracted from the data file according to the offset and data packet length.

[0031] The essence of the aforementioned local search is to take the first matching offset of the first query substring that has been verified for continuity as the starting position of the target query data in the target file, and then perform precise string or binary pattern matching only within a small window near that starting position, thereby obtaining the complete query results at minimal cost.

[0032] In some embodiments, after receiving the query request containing the target query data and the query time period, the method further includes: if the query request also carries 5-tuple information and / or application layer metadata, filtering out mismatched data files from the target data file range based on the 5-tuple information and / or application layer metadata, so as to achieve filtering and updating of the target data file range.

[0033] Based on the above settings, a pre-filtering step is provided before executing hierarchical queries. Specifically, time-period filtering utilizes the directory structure of data files organized by time to quickly extract a subset of files within a time window from the global dataset; the 5-tuple / metadata filtering, based on this time subset, further eliminates files that do not belong to the target session or application using structured identifiers from network protocols. Without increasing additional I / O overhead, the two combined achieve two rapid exclusions of irrelevant data files, minimizing the range of files that ultimately enter the directory-level index query. This multi-level filtering strategy, based on information from different dimensions, effectively avoids dealing with a large number of obviously irrelevant files in the subsequent directory-level bitmap intersection operation and file-level offset verification stages, thus significantly improving the execution efficiency of the entire query process and creating better input conditions for subsequent more refined index queries and local searches.

[0034] Specifically, according to embodiments of the present invention, network data packets have already established corresponding indexes based on 5-tuples and application layer metadata during the storage phase. When a query request simultaneously contains a time period, 5-tuples / metadata, and the target query data, the system can prioritize using the low-cost metadata index for range reduction according to the principle of "coarse to fine," and then assign the highly narrowed range to the relatively more expensive content index for processing. Time period filtering utilizes the physical directory structure of the file system, while 5-tuple / metadata filtering utilizes structured indexes in memory or on disk. Both are lightweight operations that do not involve scanning the load content or randomly reading data files. Therefore, based on the above settings, the overall query accuracy and speed can be effectively improved without introducing additional performance bottlenecks.

[0035] In some embodiments, the aforementioned value range includes fixed-length character combinations or fixed-length binary data blocks, where the fixed-length byte length is a preset positive integer; each index entry in the directory-level index includes: a value corresponding to the value range and its existence distribution information in each data file or subdirectory under the corresponding directory; wherein, the existence distribution information adopts a bitmap data structure, where each bit in the bitmap corresponds to the sequence number of a data file or subdirectory, used to identify whether the value exists in the corresponding data file or subdirectory; each index entry in the file-level index includes: a value corresponding to the value range and its position association information within the corresponding data file; wherein, the position association information includes the first matching offset when the value first appears in the data file.

[0036] Based on the above settings, the fixed-length byte ensures the determinism of the total number of index entries, making storage overhead controllable. The directory-level index uses a bitmap structure, compressing existence information into bits to support high-speed set operations. The file-level index only stores the first matching offset, recording location information at minimal cost. Accordingly, the index space is determined by the value range, global filtering capabilities are provided through the directory-level bitmap, and local location capabilities are provided through the file-level offset. This constructs an index system that is both storage-efficient and supports fast queries, allowing subsequent queries to first quickly filter candidate files through the directory-level bitmap and then accurately locate the position using the file-level offset. The entire process does not require reading or scanning the data files themselves. It is understandable that the above settings, with minimal fixed storage space, achieve rapid determination of the existence of each possible value across all data files and direct location of its first occurrence, providing a data structure foundation for efficient two-level filtering queries.

[0037] Specifically, according to an embodiment of the present invention, the fixed-length byte length L determines the total number of all possible values ​​(e.g., there are 256^L character combinations, each byte corresponds to 256 possibilities (0~255), and binary data blocks have 2^(8L) possibilities). This total number is a fixed value that will not change with the increase of the actual load data volume, thereby fundamentally limiting the type and quantity of all possible values ​​covered by the index (i.e., controlling the upper limit of index storage), making the index space pre-calculated and fixed, thereby significantly reducing storage overhead.

[0038] In some embodiments, for directory-level indexes, the bitmap data structure compresses the existence distribution information of each data file or subdirectory into a single bit, and the existence distribution information of all data files or subdirectories is organized into a compact bit sequence. By accessing the bit corresponding to the file or subdirectory number in this bit sequence, it can be immediately determined whether the value exists in the corresponding data file or subdirectory. When performing bitwise logical operations on bitmaps corresponding to multiple values, only a few machine words need to be processed in parallel. Therefore, querying the existence distribution information of a value can be completed with a single bitmap access without traversing the file list, and the existence information of multiple values ​​can be quickly merged through bitwise operations. For file-level indexes, the first matching offset is an integer that records the byte distance of the value from the beginning of the file when it first appears. This offset can serve as a direct pointer, guiding subsequent searches to start directly from the vicinity of this position, avoiding a linear scan of the entire file. Therefore, without scanning the file content, a location query for a value can directly obtain its precise location in the file, effectively improving query efficiency and accuracy.

[0039] According to a specific embodiment of the present invention, the payload data can be mapped and calculated based on a fixed-length ASCII setting (i.e., the aforementioned preset long byte length, generally 3 characters). If a certain combination exists in the currently saved data file (i.e., the data segment obtained by splitting corresponds to a specific value under the fixed-length byte length), then its first offset within the data file is recorded. The offset is generally 5 bytes. Therefore, the file-level index is the ASCII combination (a value corresponding to the value range) plus the first offset within the data file, totaling 8 bytes. Thus, for this method, the space occupied by the index entry corresponding to the file-level index is 256^3 * 8 = 134,217,728 bytes (i.e., 128M). Compared to the original data packet which is often several gigabytes in size, this part of the data storage is not very large. The expression for the file-level index (FileIndex) is as follows: FileIndex ={<partial_pattern, first_match_offset> ,……} In the above formula, FileIndex is defined as a set, where each element is a tuple. The first component, partial_pattern, is the partial matching pattern (i.e., a value corresponding to a range of values), and the second component, first_match_offset, is the offset of the first match. If a second match offset, second_match_offset, or a third match offset is needed, the index size can be increased (this needs to be determined based on the actual situation). The corresponding expression is as follows: FileIndex ={<partial_pattern, first_match_offset,second_match_offset,...> ,……} According to another specific embodiment of the present invention, for a directory-level index, its structure is similar to that of a file-level index. The first n bytes are still combinations (i.e., a value corresponding to a range of values), while the last 5 bytes form a bitmap, indicating which data files or subdirectories contain this ASCII character combination. That is, 5 bytes can identify up to 40 data files or subdirectories. The original data files storing the payload content can only appear in the leaf nodes of the index tree, and cannot appear in the non-terminal nodes of the index tree. In this way, the storage status of the original data packets over a relatively long period of time can be expressed through this multi-level organization. The directory-level index (DirIndex) can be expressed as follows: DirIndex ={<partial_pattern, dir_or_file_bitmap> ,……} Similar to file indexes, directory indexes are also a set of tuples. The first component of this tuple, partial_pattern, is a partial matching pattern (that is, a value corresponding to a range of values), and the second component, dir_or_file_bitmap, is the bitmap index of each data file or subdirectory in the directory corresponding to the matching pattern (the index in the bitmap is used to identify whether the value exists in the corresponding data file or subdirectory, and thus also represents the existence distribution information).

[0040] In some embodiments, the aforementioned location association information further includes: the matching offset when the value appears a second or more times in the data file.

[0041] Based on the above settings, a scheme is provided to store multiple matching offsets corresponding to a single value in a file-level index. By pre-storing multiple matching offsets, the repetitive scanning work that may be required during a query is completed in advance during the index building stage. This achieves instant access to multiple occurrences during the query stage with minimal storage increments (each additional offset requires only a few bytes), thereby expanding the index's applicability and query flexibility without significantly increasing index storage overhead. Specifically, by enhancing query support for scenarios where the same value appears multiple times (by pre-storing multiple matching offsets), a direct data foundation is provided for complex queries requiring multiple position references (such as verifying whether continuous query substrings appear in different regions, counting the frequency of a segment, or avoiding missed detections due to the initial matching position not satisfying continuity but subsequent matching positions satisfying continuity), thus expanding the applicable scenarios of the method.

[0042] According to an embodiment of the present invention, each index entry of the file-level index is expanded into a variable-length sequence in terms of storage structure. Originally, only one initial matching offset was stored; now, multiple matching offsets can be stored as needed, each corresponding to the starting position of a single occurrence. When building the index, after segmenting the load data and identifying the specific value of each data segment, the system not only records the file offset when that value first appears, but also continues to monitor subsequent data content: whenever that value appears again, its new matching offset is appended to the index entry, thus completely recording all occurrences of that value in the file. In subsequent queries, when it is necessary to obtain all occurrences of that value (such as the second, third, or even more occurrences), the multiple matching offsets stored in the index entry can be read directly.

[0043] According to a specific embodiment of the present invention, when the same fixed-length value appears multiple times in a single data file, the query process can directly obtain all occurrence positions without rescanning the file to find other occurrence positions after the first match during retrieval. This not only improves query efficiency but also further expands the applicable scenarios of the method.

[0044] In some embodiments, the step of dividing the target query data into multiple consecutive query substrings includes: dividing the target query data into multiple consecutive query substrings of a length equal to a fixed-length byte by using a sliding window, and the starting positions of adjacent query substrings differ by a preset offset step.

[0045] Based on the above settings, the query substring sequence generated by the sliding window segmentation method has dual determinism: "fixed length and fixed interval." The length determinism ensures the feasibility of index queries, while the interval determinism provides an expected value for offset difference verification. Accordingly, the above embodiment establishes a precise correspondence between query substrings and index entries, and quantifies the positional interval that adjacent query substrings should have under ideal continuity, simplifying the computational complexity of the overall query process and providing directly usable mathematical judgment conditions for subsequent continuity verification based on the first matching offset difference.

[0046] According to an embodiment of the present invention, in the above-described sliding window segmentation mechanism, the system first obtains the total length of the target query data, and then, starting from the first byte of the target query data, extracts the first query substring according to a preset length of bytes. After extraction, the starting position is moved backward by a preset offset step (e.g., 1 byte), and the second query substring is extracted. This process is repeated until the starting position plus the fixed-length byte exceeds the end of the target query data. The resulting query substring sequence has the following properties: the length of each substring is equal to the fixed-length byte, the contents of adjacent substrings overlap significantly (when the offset step is less than the fixed-length byte), and the difference between the starting positions of adjacent substrings (i.e., the window sliding step) is strictly equal to the preset offset step. This difference is a known, fixed value that can be directly used as the expected value or threshold for comparing offset differences in subsequent steps. Since the length of each query substring is the same as the fixed-length byte used in the storage stage index, each query substring can find corresponding entries in both the directory-level index and the file-level index, and there is no situation where a query cannot be performed due to length mismatch. Meanwhile, the starting positions of adjacent query substrings differ by a preset offset step (usually 1 byte). This fixed difference constitutes the quantitative benchmark for subsequent verification of continuity, making it possible to determine whether multiple query substrings appear in the data file in a continuous and adjacent manner.

[0047] According to a specific embodiment of the present invention, for directory-level indexing, the present invention uses a method similar to "pillar set" to further narrow down the query scope. For example, if the original data packet content specified by the user contains a string of length m (supporting not only text but also binary search), it is split into m+n-1 query substrings of length n using a sliding window method (here, n is a fixed-length ASCII string, generally 3, n≤m; 1 refers to the window sliding step size, i.e., the aforementioned offset step size).

[0048] In some embodiments, the step of performing a logical intersection operation on the existence distribution information of all query substrings to filter out a candidate file set that simultaneously contains all query substrings includes: performing a bitwise logical AND operation on the bitmap data structure corresponding to each query substring; determining the file number of the corresponding candidate file based on the bit with a value of 1 in the operation result, thereby forming a candidate file set.

[0049] Based on the above settings, the inherent characteristic of directory-level indexes storing existence information in bitmap form is fully utilized. Logical intersection operations are mapped from general set traversal comparison operations to bitwise AND operations directly supported by hardware. This simplifies the filtering task, which might have required multiple iterative scans of the file list, into a single in-memory bitmap batch operation. This setup effectively avoids high I / O overhead or CPU-intensive computation in the early stages of the query, allowing the system to quickly eliminate a large number of files that are unlikely to contain the complete target query content, drastically narrowing the search scope. Simultaneously, since the operation results directly provide candidate file indices in bitmap form, no additional format conversion or parsing is required, and the system can seamlessly use these indices for subsequent file-level index queries. Therefore, while ensuring filtering accuracy, multi-condition existence queries are transformed into a set of simple bit-level parallel operations, achieving precise filtering of the candidate file set with extremely low computational resource overhead, providing a minimized input range for subsequent, more refined query steps.

[0050] According to a specific embodiment of the present invention, when searching a directory-level index using the above-described query substring, the bitmap sequence number corresponding to the candidate file is... The expression is as follows:

[0051] Where M represents the sum of all possible values ​​within the range corresponding to the fixed-length byte. This represents the i-th tuple in the set of directory-level indices; the function π is the projection operation of the directory-level indices, with subscripts 1 and 2 indicating different dimensions of the projection. i) represents the first component in the corresponding binary tuple. This represents the second component in the corresponding binary tuple. This represents finding the intersection of a finite number of elements (M) to determine the candidate first component. (Inverse function) The candidate tuple to which the first candidate component belongs is determined by inverse operation.

[0052] That is, the m+n-1 query substrings obtained by splitting the original query data packet content are used as the target search partial index. The directory-level index is searched, and the first component of the query is logically intersected to obtain the candidate first component. Then, the inverse operation is performed to obtain the complete candidate pair, and finally, the candidate second component in the candidate pair is obtained. The second component is the bitmap sequence number in each data file or subdirectory under the directory corresponding to the matching pattern (the sequence number in the bitmap is used to identify whether the value exists in the corresponding data file or subdirectory). Therefore, the corresponding file sequence number can be obtained to get the candidate file set.

[0053] Specifically, according to embodiments of the present invention, the above embodiments utilize the compact representation and parallel operation characteristics inherent in the bitmap data structure itself. Specifically, the existence distribution information corresponding to each query substring in the directory-level index is stored as a bitmap, where the length of the bitmap is equal to the total number of data files in the directory, each bit's index corresponds to a data file, a bit value of "1" indicates that the query substring exists in the file, and a bit value of "0" indicates that it does not exist. When it is necessary to perform a logical intersection operation on the existence distribution information of multiple query substrings (i.e., filter out files that simultaneously contain all query substrings), the system treats the bitmap corresponding to each query substring as a binary vector and performs a logical AND operation on the corresponding bits of each vector. Since computer processors can process multiple bit operations in parallel at machine word lengths (e.g., 64 bits), the entire bitmap bitwise AND operation only requires approximately (total number of files / machine word length) instructions, and the time complexity is linearly related to the total number of files but has an extremely low constant factor. After the operation is complete, the result is still a bitmap, where a bit is 1 if and only if all bitmaps corresponding to the query substrings are 1 at that bit (i.e., the file contains every query substring). The system then iterates through this bitmap, extracting the file indices corresponding to all bits with a value of 1. The data files pointed to by these indices are the candidate file set containing all query substrings. The entire process involves only reading bitmap data from memory and performing bitwise logical operations. It eliminates the need to check each file individually for each query substring, and avoids reading or scanning any data file content. This allows for the rapid filtering of the candidate file set, significantly improving the efficiency of the initial query stage.

[0054] In some embodiments, the step of verifying whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size includes: for adjacent query substrings obtained after segmenting the target query data, calculating the difference between the first matching offset of the next query substring and the first matching offset of the previous query substring; if the difference between all adjacent query substrings is less than or equal to the offset step size, the verification is deemed successful and conforms to the continuity feature.

[0055] Based on the above settings, a specific implementation method is provided to verify continuity by calculating the difference in the first matching offsets of adjacent query substrings and comparing it with the offset step size. This implementation method fully utilizes the first matching offsets pre-stored during index construction and the offset step size determined during querying—two known values. This verification process does not rely on the data file content itself, nor does it involve complex string matching or regular expression operations; it can be completed simply through arithmetic comparison. This lightweight verification method based on positional relationships transforms the continuity judgment, which might otherwise require traversing the entire candidate file, into a fast operation that only requires reading a few integer values ​​and performing subtraction and comparison, greatly improving the overall efficiency of retrieval. Simultaneously, this verification standard ensures that truly continuous sequences pass (because the difference in the offsets of adjacent substrings in a continuous sequence will not exceed the offset step size) while eliminating the vast majority of false positive candidates, providing a highly accurate starting range for subsequent local searches, thereby ultimately achieving fast and accurate query result output. It is understandable that the above-described specific implementation method completes a high-precision secondary screening at the candidate file level with extremely low computational overhead, so that only files that pass the continuity verification enter the final local search stage, thereby greatly reducing unnecessary search operations while ensuring no missed detections.

[0056] According to a specific embodiment of the present invention, after determining the candidate file set by performing a directory-level index query, a file-level index search can be performed. The specific steps are similar to those of searching a directory-level index, which also involves searching the query substring that matches the relevant search results. The formalization is as follows:

[0057] Where M represents the sum of all possible values ​​within the range corresponding to the fixed-length byte. This represents the i-th tuple in the file-level index set; the function π is the projection operation of the file-level index, with subscripts 1 and 2 indicating different dimensions of the projection. i) represents the first component in the corresponding binary tuple. This represents the second component in the corresponding binary tuple. This represents finding the intersection of a finite number of elements (M) to determine the candidate first component. (Inverse function) The candidate tuple to which the first candidate component belongs is determined by inverse operation.

[0058] In other words, after obtaining the candidate file set containing these query substrings, we can further calculate whether these candidate files actually contain at least one instance of the desired search string (i.e., the target query substring) based on the file-level index. The logic is that if a file contains all m+n-1 query substrings of the desired search string, similar to the directory-level index query step, the function π is the projection operation of the file-level index, with subscripts 1 and 2 indicating different dimensions of the projection, while the function θ is to obtain the matching partial index in the original session, and its inverse function... This involves retrieving the relevant original session record set based on a partial index.

[0059] The first matching offsets of the above query substrings satisfy the following formal representation: first_match_offset j - first_match_offset i ≤ 1 (for all ji=1); where ji=1 means that the j-th query substring and the i-th query substring are adjacent query substrings; first_match_offset j and first_match_offset i FIRST_MATCH_OFFSET represents the first match offset for the j-th and i-th query substrings, respectively.

[0060] If the above conditions are met (the difference in the first matching offset between adjacent query substrings is less than or equal to 1, that is, less than or equal to the offset step size), it means that relevant content can be retrieved in this target file. When the specific original data packet storage file is located, the search tool can be used to further search in the original data packet storage file (starting from the first offset). The system can be designed so that the size of each file is not too large, for example, each file does not exceed 10GB, in order to facilitate the rapid retrieval of relevant content.

[0061] Specifically, the above implementation utilizes the physical location pattern of the target query data when it is stored continuously in the original payload. Specifically, when the target query data exists as a continuous whole in the data file, its adjacent query substrings after segmentation must be arranged adjacently. Assuming the offset of the first byte of the target query data in the file is P, the preset fixed-length byte length is L, and the offset step size is S (usually S = 1 byte), then the ideal starting offset of the i-th query substring should be P + (i-1) × S. Therefore, the difference between the first matching offset of the subsequent query substring and the first matching offset of the preceding query substring should be exactly equal to the offset step size S. However, considering the potential interference factors in the data file (such as multiple occurrences of the same value, file alignment, etc.), a certain range of deviation is allowed in practice. The judgment condition is relaxed to "less than or equal to the offset step size" rather than strictly equal to the offset step size. This can tolerate reasonable deviations caused by the matching algorithm's initial matching offset not being the precise starting point in the continuous sequence (for example, the initial matching offset of the previous query substring is exactly at an isolated occurrence position before the continuous sequence), while effectively identifying cases where the distance between query substrings is too large. When the difference between the initial matching offsets of any adjacent query substrings is greater than the offset step size, it indicates that the two query substrings are far apart in the data file and cannot belong to the same continuous query data sequence. Therefore, by calculating the difference and comparing it with the offset step, it is possible to accurately determine whether all substrings are arranged in a way that is close to each other (continuous or quasi-continuous), thereby determining whether the candidate file truly contains the complete target query data. Since the whole process only involves integer subtraction and comparison operations, there is no need to scan the content of the candidate file or perform string matching. False positive files that exist in the candidate file but are not continuous in position can be quickly eliminated, thereby further narrowing the candidate range before entering the local search. At the same time, it ensures that the verified candidate file does indeed contain continuous query content, with extremely low computational cost.

[0062] In some embodiments, the step of obtaining the payload data of each network packet in the network packet stream further includes: performing a preprocessing and restoration operation on the original payload content of each network packet in the network packet stream to obtain payload data; wherein the preprocessing and restoration operation includes at least one of the following steps: performing decompression processing on the original payload content; performing multi-layer nested decoding processing on the original payload content; and performing stream reassembly operation on network packets belonging to the same session according to the transport layer protocol.

[0063] Based on the above settings, potential distortion factors in network data packet payloads are eliminated from multiple dimensions. Payload content in the original data packets, which becomes difficult to directly retrieve due to compression, encoding, and fragmented transmission, is uniformly restored into a continuous, original byte sequence that can be directly used for text or pattern matching. This provides high-quality foundational data for subsequent index construction and content retrieval. It should be noted that this preprocessing restoration operation is optional; users can choose whether to perform it and which specific operations to perform based on their actual needs.

[0064] The preprocessing and restoration operation provided in this embodiment of the invention can perform decompression (mainly for the HTTP protocol) and decoding according to a user-specified protocol. Specifically, the decoding process refers to decoding the encoded content (mainly for the HTTP protocol) in the original network data, and supports multi-level nested encoding to obtain the actual potential attack payload. According to a specific embodiment of the invention, the corresponding nested encoding generally has a depth of 2, such as supporting URIENCODE nested UNICODE encoding, etc.

[0065] Specifically, according to embodiments of the present invention, network data packets undergo various morphological changes during transmission. These changes are often superimposed, meaning a single network data packet may simultaneously involve compression, encoding, and fragmentation. By providing three complementary restoration methods (decompression, decoding, and stream reassembly), the system can selectively execute one or more of these operations based on the characteristics of the actual data packet. Decompression addresses the data volume compression problem, multi-layer nested decoding addresses the character encoding obfuscation problem, and stream reassembly addresses the data cross-packet dispersion problem. Each of these three methods independently eliminates the corresponding impact of deformation factors in different dimensions. When multiple deformations exist simultaneously, the system can combine and execute multiple operations to restore each layer of deformation sequentially, ultimately obtaining the completely original payload content. It is understood that this layered, optional preprocessing restoration mechanism ensures that regardless of the processing (compression, encoding, fragmentation, and their combinations) the original data packet has undergone, it can be restored to a standardized form suitable for subsequent fixed-length indexing and content retrieval, thereby maximizing retrieval coverage and hit rate and avoiding missed detections due to data deformation.

[0066] In some embodiments, data compression commonly used in network transmission (such as gzip compression in the HTTP protocol) alters the byte sequence of the original data, replacing it with a compressed bitstream. The compressed data lacks readable original text or binary format, and direct storage and indexing of it will result in users being unable to find results when searching using the original keywords. Decompression processing reverses the compression process according to the corresponding compression algorithm (such as inflate decompression), restoring the payload content to the original byte sequence before compression. This eliminates the morphological changes caused by compression to the data content, allowing subsequent fixed-length segmentation and index construction to be based on real and valid data.

[0067] In some examples, attackers or certain application scenarios often encode the payload multiple times (e.g., first Unicode encoding, then URI encoding the encoded string), forming a nested encoding structure. Each layer of encoding maps the original characters to a specific encoded representation. Multi-layered nested decoding proceeds in reverse order of encoding: first decoding the outermost layer (e.g., URI Encode decoding), then decoding the inner layers (e.g., Unicode decoding), until all encoding layers are completely decrypted, restoring the original character sequence. Through this process, attack payloads or sensitive information that were originally masked by multiple layers of encoding are revealed, ensuring that subsequent indexing and retrieval do not miss critical content due to encoding distortion.

[0068] In some cases, transport layer protocols (such as TCP) typically split a complete application layer data block into multiple packets for transmission to avoid network congestion or to adapt to maximum transmission unit (MTU) limitations. These packets may travel through different network paths, arrive at the receiving end out of order, or be interleaved with packets from other sessions. Stream reassembly (R&D) rearranges all packets belonging to the same session and data stream in the correct order based on the transport layer protocol's sequence number field (such as TCP's seq number), extracts the payload portion of each packet, and concatenates them into a continuous and complete payload data block. After R&D, the application layer payload data, originally scattered across multiple packets, is merged into a complete and continuous byte sequence, avoiding retrieval breaks caused by data crossing packets, and enabling indexing and querying of long payload content to cover all data.

[0069] In some embodiments, the method further includes: storing the link layer information, network layer information, and transport layer information of network data packets separately from the payload data; wherein network data packets belonging to the same network flow share a single copy of the five-tuple information and link layer address information, and each network data packet independently stores its sequence number, acknowledgment sequence number, payload length information, and transport layer flag.

[0070] Based on the above settings, while ensuring that the original header information of each data packet in the network flow can be fully recovered to support subsequent playback, the total space required to store the header information is effectively reduced by eliminating storage redundancy between multiple data packets in the same network flow, thereby improving the utilization efficiency of storage resources.

[0071] Specifically, according to embodiments of the present invention, the essential differences between the "static common part" and the "dynamic private part" in the header information of network data packets are identified. The link-layer address and the 5-tuple are flow identification attributes that remain almost unchanged throughout the lifecycle of the network flow and are suitable for shared storage; while the sequence number, acknowledgment sequence number, payload length, and flag bits are individual attributes of each data packet and must be stored separately. Separating the two avoids the space waste caused by repeatedly storing large amounts of identical information while preserving an accurate record of the independent characteristics of each data packet. This design achieves a balance between disk space and playback requirements—compared to storing the complete header of each data packet (approximately 60 bytes / packet), this scheme only stores the dynamic part (approximately 20 bytes / packet) plus one copy of the static part (approximately 40 bytes / flow); for a flow containing thousands of data packets, the storage overhead of the static part is amortized across each packet and is negligible, thus making the header storage cost of each packet approach the size of the dynamic information, achieving near-optimal storage efficiency while retaining all the key information required for playback. This satisfies the functional requirement of data packet playback for post-event retrospective analysis, and also achieves significant economic benefits in engineering practices with limited storage space.

[0072] According to a specific embodiment of the present invention, in a network flow (a series of data packets identified by the same 5-tuple), the link layer information (MAC address) and the network layer and transport layer 5-tuple information of each data packet are usually identical across different data packets, and may only change in a few cases (such as routing changes). Therefore, extracting these unchanging public information separately and storing them only once throughout the entire lifecycle of the flow can avoid repeatedly saving the same content for each data packet. For each data packet, the truly dynamically changing information mainly includes: TCP or UDP sequence number (used to identify the order of data packets in the flow), acknowledgment sequence number (used to acknowledge receipt), payload length (the amount of application data carried by each data packet), and transport layer flags (such as TCP SYN, ACK, FIN, RST, and other control flags). These dynamic information are different for each data packet and need to be stored independently. By separating the static public information from the dynamic private information and using a sharing mechanism to manage the static information, the overall storage overhead is significantly reduced. Through this separate storage method, storage space is saved for each packet, resulting in an overall saving of storage capacity. Meanwhile, since the sequence number, acknowledgment sequence number, and payload length of each data packet are stored independently, which are sufficient to uniquely determine the position, size, and direction of the data packet in the flow, if it is necessary to replay the network data packets later (e.g., to replay the attack process for forensic analysis), the complete original data packets can be reconstructed based on the shared static information and the dynamic information of each packet, including recalculating the checksum, thus ensuring the feasibility of replay.

[0073] According to another specific embodiment of the present invention, a simplified approach can be adopted in the storage of link layer information, network layer information, and transport layer information in network data packets. For example, for data link layer information, only the source / destination MAC address and upper-layer protocol content need to be saved once. For network layer information, only the source / destination address needs to be saved once, while information such as the length of each transmission packet and whether it is fragmented is not maintained. For transport layer information, only necessary information needs to be retained. In particular, for the TCP protocol, only the source / destination port needs to be retained once; for the UDP protocol, only the source / destination port and the UDP packet payload length for each transmission need to be retained once. If the user is not sensitive to the original information other than the application layer, it can be omitted, as they only have an impact when replaying data packets. For other transport protocols, only the sequence number of each transmitted data packet and the ACK sequence number (absolute or relative sequence numbers can be used), the payload length, and the TCP flag bit (one byte) need to be retained. Taking TCP as an example, the common information (flow ID, MAC, IP, port) of all packets in a TCP network flow (denoted as TCP_Flow) and the dynamic information of each packet (sequence number, acknowledgment number, payload length, flags) are formally represented as follows: TCP_Flow = {<flow_id> ,<dst_mac, src_mac, protocol,[ vlan_id]> ,<src_ip, dst_ip> ,<sport, dst_port> , {<seq, ack_seq, payload_length, tcp_flag>}} Wherein, flow_id (Flow Identifier): a unique identifier for the network data packet flow; In Layer 2 information: dst_mac (Destination Media Access Control): Destination MAC address; src_mac (Source Media Access Control): Source MAC address; protocol: Network layer protocol type; vlan_id (Virtual Local Area Network Identifier): VLAN identifier (square brackets indicate that it is an optional field); In the three-layer information: src_ip (Source Internet Protocol address): source IP address; dst_ip (Destination Internet Protocol address): destination IP address; In the four-layer information: sport (Source Port): source port number; dst_port (Destination Port): destination port number; Dynamic fields include: seq (Sequence Number); ack_seq (Acknowledgment Sequence Number); payload_length: payload length (in bytes); tcp_flag: TCP flags.

[0074] During storage, packets belonging to the same network flow share a single Layer 2 / Layer 3 / Layer 4 common information. Each packet only stores its dynamic fields (seq, ack_seq, payload_length, tcp_flag), thus saving storage space. However, the dynamic fields (sequence number, ACK sequence number, and payload length) may change after the payload is decompressed or deencoded, requiring recalculation of the checksum during playback.

[0075] According to a specific embodiment of the present invention, if it is necessary to replay the stored network data packets corresponding to the query results, after selecting the interface for replaying the data packets, the network flow to which the network data packets belong and the relevant data packet headers can be used to reconstruct and replay these data according to their protocol nature. Since decompression and decoding operations are performed when storing these original data packets, the content of the original data packet headers may not be consistent, especially for TCP protocols, which contain application layer payload data packets. Therefore, they can be constructed and replayed according to certain principles. During the replay process, the maximum transmission unit (MTU) of each network traffic packet needs to be constrained.

[0076] The network data packet storage and query method provided in this embodiment of the invention captures the network data packet stream on the network link to obtain the payload data of each network data packet in the network data packet stream; based on a preset fixed-length byte length, the payload data is divided into multiple continuous data segments, each data segment corresponding to a specific value under the fixed-length byte length; based on the value range corresponding to the fixed-length byte length, a multi-level index structure including directory-level index and file-level index is constructed, and the corresponding index entries in the multi-level index structure are filled using data segments; the payload data is stored in a data file, the data file is organized by time or network stream, and stored in the corresponding directory or subdirectory; the multi-level index structure is stored in an index database; wherein, the directory-level index is used to record the existence distribution information of each value corresponding to the value range in the directory-level storage space, and the file-level index is used to record the first matching position information of each value corresponding to the value range in the corresponding data file; a query request containing target query data and query time period is received, the target data file range is determined according to the query time period, and the target query data is divided into multiple continuous query substrings according to the fixed-length byte length; the multiple query substrings are used to query the directory within the target data file range. This method employs a multi-level index to obtain the existence distribution information corresponding to each query substring. A logical intersection operation is performed on the existence distribution information of all query substrings to filter out a candidate file set that simultaneously contains all query substrings. The file-level indexes corresponding to each candidate file in the candidate file set are then used to obtain the first matching offset of each query substring within each candidate file. For any candidate file, based on the offset step size between adjacent query substrings when splitting the target query data, the difference between the first matching offsets of multiple query substrings is verified to ensure it conforms to the continuity characteristic corresponding to the offset step size. If it does, the corresponding candidate file is confirmed as the target file. A local search is then performed within the target file based on the first matching offset, outputting the query results containing the target query data. This technique achieves the technical effect of reducing storage overhead and significantly improving query response speed and hit accuracy.

[0077] Based on the network data packet storage and retrieval method provided in the embodiments of the present invention, the embodiments of the present invention also provide a network data packet storage and retrieval device. Figure 2 As shown, the network data packet storage and retrieval device 200 includes: The acquisition module 201 is used to capture the network data packet stream on the network link and obtain the payload data of each network data packet in the network data packet stream; Storage module 202 is used to divide the payload data into multiple continuous data segments based on a preset fixed-length byte length, with each data segment corresponding to a specific value within the fixed-length byte length; based on the value range corresponding to the fixed-length byte length, a multi-level index structure including directory-level index and file-level index is constructed, and the corresponding index entries in the multi-level index structure are populated using data segments; the payload data is stored in data files, which are organized by time or network stream and stored in corresponding directories or subdirectories; the multi-level index structure is stored in the index database; wherein, the directory-level index is used to record the existence distribution of each value corresponding to the value range in the directory-level storage space, and the file-level index is used to record the first matching position information of each value corresponding to the value range in the corresponding data file; The receiving module 203 is used to receive a query request containing target query data and a query time period, determine the target data file range according to the query time period, and divide the target query data into multiple consecutive query substrings according to the fixed-length byte length. The filtering module 204 is used to query the directory-level index within the target data file range using multiple query substrings, obtain the existence distribution information corresponding to each query substring, perform a logical intersection operation on the existence distribution information of all query substrings, and filter out a candidate file set that contains all query substrings. The query module 205 is used to query the file-level index corresponding to each candidate file in the candidate file set using multiple query substrings, and obtain the first matching offset of each query substring in each candidate file; for any candidate file, based on the offset step size between adjacent query substrings when splitting the target query data, it verifies whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size. If it does, the corresponding candidate file is confirmed as the target file; based on the first matching offset, a local search is performed in the target file, and the query results containing the target query data are output.

[0078] Based on the above settings, the payload data of each network packet is obtained by capturing network packet streams; a multi-level index structure containing directory-level and file-level indexes is constructed based on a preset long byte length, providing a spatially controllable filtering and positioning foundation for searching; the query granularity is kept consistent with the index granularity by dividing the target query data into segments of the same fixed byte length; the logical intersection operation of the existence distribution of all query substrings is performed using the bitmap structure of the directory-level index, enabling fast filtering of the vast majority of irrelevant files; and the offset of the first match is obtained by using the file-level index, and the offsets of adjacent substrings are verified. The continuous characteristics of the data enable precise location and local search of the target file. The progressive operations of "acquiring load data, fixed-length byte mapping, two-level query filtering, and storage location verification" ensure that each query operation is based on the previous stage of significantly narrowing the scope, ultimately obtaining the most accurate query results with minimal I / O overhead. In this way, while ensuring controllable index storage space, it overcomes the shortcomings of large storage overhead, low query efficiency, and poor accuracy in related technologies, and achieves efficient and accurate retrieval of massive compressed and encoded network data packet load content, significantly improving query response speed and hit accuracy.

[0079] Meanwhile, the aforementioned network data packet storage and retrieval device 200 is configured to execute any of the aforementioned network data packet storage and retrieval methods. Therefore, the relevant modules in the aforementioned network data packet storage and retrieval device are also used to execute the corresponding operations in any of the aforementioned network data packet storage and retrieval methods. Accordingly, it also possesses all the beneficial effects of any of the aforementioned network data packet storage and retrieval methods, which will not be elaborated further in this application.

[0080] It should be noted that the specific modules in the aforementioned network data packet storage and retrieval device are defined primarily based on the corresponding operations performed, and are not intended to limit the specific modules.

[0081] This invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this invention.

[0082] This invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the methods of embodiments of this invention. The computer program product should be understood as a software product that primarily implements the methods of this invention through a computer program.

[0083] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of this invention.

[0084] refer to Figure 3 The present invention will now be described in the form of a structural block diagram of an electronic device that can serve as an embodiment of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0085] like Figure 3 As shown, the electronic device includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0086] Multiple components in the electronic device are connected to I / O interface 305, including: input unit 306, output unit 307, storage unit 308, and communication unit 309. Input unit 306 can be any type of device capable of inputting information into the electronic device. Input unit 306 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 307 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 308 may include, but is not limited to, disks and optical discs. Communication unit 309 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0087] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 302 and / or communication unit 309. In some embodiments, the computing unit 301 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0088] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0089] In the context of embodiments of the present invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0090] It should be noted that the term "comprising" and its variations used in the embodiments of the present invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that, unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0091] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data that have been permitted by the user or have been fully agreed upon by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to agree or refuse.

[0092] The steps described in the method embodiments provided by this invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of this invention is not limited in this respect.

[0093] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0094] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for storing and retrieving network data packets, characterized in that, include: Capture network packet streams on the network link and obtain the payload data of each network packet in the network packet stream; Based on a preset fixed-length byte, the payload data is divided into multiple continuous data segments, each data segment corresponding to a specific value under the fixed-length byte; based on the value range corresponding to the fixed-length byte, a multi-level index structure including directory-level index and file-level index is constructed, and the data segments are used to populate the corresponding index entries in the multi-level index structure; The load data is stored in a data file, which is organized by time or network stream and stored in the corresponding directory or subdirectory. The multi-level index structure is stored in the index database. The directory-level index is used to record the existence distribution information of each value corresponding to the value range in the directory-level storage space, and the file-level index is used to record the first matching position information of each value corresponding to the value range in the corresponding data file. Receive a query request containing target query data and a query time period, determine the target data file range based on the query time period, and divide the target query data into multiple consecutive query substrings based on the fixed-length byte length; The directory-level index is queried within the target data file using multiple query substrings to obtain the existence distribution information corresponding to each query substring. Logical intersection is performed on the existence distribution information of all query substrings to filter out a candidate file set that contains all query substrings. The file-level index corresponding to each candidate file in the candidate file set is queried using multiple query substrings to obtain the first matching offset of each query substring in each candidate file. For any candidate file, based on the offset step size between adjacent query substrings when the target query data is segmented, it is verified whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size. If it does, the corresponding candidate file is confirmed as the target file. A local search is performed in the target file based on the first matching offset, and the query result containing the target query data is output.

2. The method according to claim 1, characterized in that, Following the step of receiving a query request containing the target query data and the query time period, the method further includes: If the query request also carries 5-tuple information and / or application layer metadata, then based on the 5-tuple information and / or application layer metadata, mismatched data files are filtered from the target data file range to achieve filtering updates of the target data file range.

3. The method according to claim 1, characterized in that, The value range includes fixed-length character combinations or fixed-length binary data blocks, and the length of the fixed-length byte is a preset positive integer. Each index entry in the directory-level index includes: a value corresponding to the value range and its existence distribution information in each data file or subdirectory under the corresponding directory; wherein, the existence distribution information adopts a bitmap data structure, and each bit in the bitmap corresponds to the sequence number of a data file or subdirectory, which is used to identify whether the value exists in the corresponding data file or subdirectory; Each index entry in the file-level index includes: a value corresponding to the value range and its position association information within the corresponding data file; wherein, the position association information includes the first matching offset when the value first appears in the data file; the position association information also includes the matching offset when the value appears a second and subsequent times in the data file.

4. The method according to claim 1, characterized in that, The step of dividing the target query data into multiple consecutive query substrings includes: dividing the target query data into multiple consecutive query substrings with a length equal to the fixed-length byte length by using a sliding window method, and the starting positions of adjacent query substrings differ by a preset offset step. The step of verifying whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size includes: for adjacent query substrings obtained after segmenting the target query data, calculating the difference between the first matching offset of the next query substring and the first matching offset of the previous query substring; if the difference between all adjacent query substrings is less than or equal to the offset step size, then the verification is deemed successful and conforms to the continuity feature.

5. The method according to claim 1, characterized in that, The step of obtaining the payload data of each network packet in the network packet stream further includes: Perform a preprocessing and restoration operation on the original payload content of each network data packet in the network data packet stream to obtain the payload data; The preprocessing and restoration operation includes at least one of the following steps: performing decompression processing on the original payload content; performing multi-layer nested decoding processing on the original payload content; and performing stream reassembly operation on network data packets belonging to the same session according to the transport layer protocol.

6. The method according to claim 1, characterized in that, Also includes: The link layer information, network layer information, and transport layer information of the network data packets are stored separately from the payload data; Among them, network packets belonging to the same network flow share a five-tuple information and link layer address information, while each network packet independently stores its sequence number, acknowledgment sequence number, payload length information and transport layer flag bit.

7. The method according to claim 3, characterized in that, The step of performing a logical intersection operation on the existence distribution information of all query substrings to filter out a candidate file set that simultaneously contains all query substrings includes: Perform a bitwise AND operation on the bitmap data structure corresponding to each query substring; Based on the bits with a value of 1 in the calculation result, the file number of the corresponding candidate file is determined, thus forming the candidate file set.

8. A device for storing and retrieving network data packets, characterized in that, include: The acquisition module is used to capture network data packet streams on the network link and acquire the payload data of each network data packet in the network data packet stream; The storage module is used to divide the payload data into multiple continuous data segments based on a preset fixed-length byte length, each data segment corresponding to a specific value under the fixed-length byte length; based on the value range corresponding to the fixed-length byte length, a multi-level index structure including directory-level index and file-level index is constructed, and the data segments are used to fill the corresponding index entries in the multi-level index structure; The load data is stored in a data file, which is organized by time or network stream and stored in the corresponding directory or subdirectory. The multi-level index structure is stored in the index database. The directory-level index is used to record the existence distribution of each value corresponding to the value range in the directory-level storage space, and the file-level index is used to record the first matching position information of each value corresponding to the value range in the corresponding data file. The receiving module is used to receive a query request containing target query data and a query time period, determine the target data file range according to the query time period, and divide the target query data into multiple consecutive query substrings according to the fixed-length byte length. The filtering module is used to query the directory-level index within the target data file range using multiple query substrings, obtain the existence distribution information corresponding to each query substring, perform a logical intersection operation on the existence distribution information of all query substrings, and filter out a candidate file set that contains all query substrings. The query module is used to query the file-level index corresponding to each candidate file in the candidate file set using multiple query substrings, and obtain the first matching offset of each query substring in each candidate file; for any candidate file, based on the offset step size between adjacent query substrings when the target query data is segmented, it verifies whether the difference between the first matching offsets of multiple query substrings conforms to the continuity feature corresponding to the offset step size. If it does, the corresponding candidate file is confirmed as the target file; based on the first matching offset, a local search is performed in the target file, and a query result containing the target query data is output.

9. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images