Network traffic multi-dimensional retrieval method, system, computer device and storage medium

Abstract

The application relates to a network traffic multidimensional retrieval method, system, computer device and storage medium. The method comprises the following steps: assigning a continuously increasing internal row ID to a network traffic record, performing fixed-length byte splitting on each dimensional feature according to attribute cardinality difference, converting into a fixed-format binary byte sequence, building a hybrid index according to a single attribute, layer-by-layer routing based on a prefix dictionary tree of the shortest disjoint prefix, splitting a node when a prefix conflict occurs, binding an adaptive compression bitmap of a leaf node to store a row ID, splitting and serializing the constructed index topology structure and bitmap data, storing the data in files and arranging bitmap storage addresses according to hardware byte alignment rules, converting a request into the same specification code during retrieval, loading the index through memory mapping, traversing to obtain corresponding bitmap pointers and performing logical operation, and restoring and outputting original traffic data. The method can realize high-speed index construction and low-delay multidimensional retrieval in a massive network traffic environment.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, system, computer device, and storage medium for multidimensional retrieval of network traffic. Background Technology

[0002] With the rapid development of internet technology and the deepening of global digitalization, the network has become an indispensable infrastructure for the operation of modern society. Massive amounts of data flow through global networks every moment, carrying all kinds of information, from daily communications and business transactions to critical infrastructure control. These network traffic records are crucial for forensics, anomaly investigations, and troubleshooting. Administrators typically archive these massive records for retrospective analysis. To investigate complex cyberattacks, they urgently need the ability to retrieve multi-attribute network traffic to quickly locate specific traffic, such as simultaneously matching source IP addresses, destination ports, behavioral patterns, and statistical characteristics. However, traditional analysis tools like Wireshark and Tcpdump do not inherently possess efficient multi-dimensional traffic indexing capabilities. Without a dedicated index, retrieving large-scale traffic records becomes a time-consuming linear sequential scan, completely failing to meet the requirements of modern network security.

[0003] Current research on web traffic retrieval primarily focuses on improving index structures, such as optimizing K-ary tree indexes and bitmap indexes. However, existing technologies have significant and insurmountable limitations in addressing the critical need for accelerated multi-attribute retrieval: (1) Performance degradation of index building and retrieval queries caused by the growth of data scale: In terms of index building, traditional tree-based indexes (such as B+ trees or certain balanced K-ary trees) need to frequently perform node splitting and rebalancing operations during large-scale data writing to maintain the stability of their structure. This results in huge computational overhead, making it difficult to meet the real-time access requirements of line-speed throughput in a network environment, and easily causing data writing to be slow or even lost.

[0004] In terms of retrieval and query, its complexity is usually related to the total amount of data. When processing tens of millions or even hundreds of millions of streaming records, frequent memory jumps and node comparisons lead to a significant increase in retrieval latency, making it difficult to maintain a stable millisecond-level response.

[0005] (2) Computational bottleneck of multi-attribute joint retrieval: Existing research largely focuses on building independent index structures for single attribute dimensions, meaning an index structure is built for each attribute dimension. Under this architecture, handling multi-dimensional joint queries must follow the inherent paradigm of first retrieving and then merging; that is, the system needs to retrieve a list of candidate IDs that meet a single condition from each index tree, and then obtain the intersection result through global logical operations. In large-scale network traffic scenarios, a single attribute often corresponds to tens of thousands of data records, resulting in an extremely large intermediate result set. As the query dimensions increase, performing intersection operations on these long lists not only consumes huge amounts of memory bandwidth and CPU cycles, but also causes serious I / O waste due to reading a large amount of invalid data.

[0006] (3) Performance challenges of index persistence architecture and memory loading: High-speed network traffic indexing systems require a balance between line-speed writes and millisecond-level queries, placing extremely high demands on the underlying storage structure. Traditional index structures perform reasonably well in memory, but often face efficiency challenges when persisting to disk. On the one hand, the large number of pointer references in memory leads to a significant increase in the serialized storage space, and the lack of an effective sparse data compression mechanism further exacerbates space waste. On the other hand, logically adjacent nodes in the index are often discretely distributed on the physical disk, resulting in a large number of random I / O accesses and cache misses during queries. Due to the lack of a disk storage layout aligned with memory, the system has to rely on expensive deserialization operations when loading the index, which greatly limits the system's real-time responsiveness.

[0007] In summary, in the complex environment of massive traffic data, traditional traffic indexing methods are limited by frequent structural adjustments and random writes during the construction phase, making it difficult to meet the line-speed real-time processing requirements of high-speed networks. Furthermore, during the retrieval phase, the redundant overhead of multi-dimensional logical operations and deserialization bottlenecks make it difficult to achieve the goal of millisecond-level response. Therefore, in scenarios involving multi-attribute joint retrieval of massive network traffic data, how to achieve low-latency multi-dimensional retrieval of large-scale traffic, and improve the proactive defense capabilities of network security and the efficiency of post-event forensic tracing, has become an urgent technical problem to be solved. Summary of the Invention

[0008] Therefore, it is necessary to provide a method, system, computer device, and storage medium for multidimensional network traffic retrieval to address the aforementioned technical problems.

[0009] A method for multidimensional retrieval of network traffic, the method comprising: Assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence. Hybrid indexes are built for each single attribute. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, it is split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the bitmap cardinality. The index topology and bitmap data of each batch are split and serialized, and stored in different files. The bitmap storage addresses are arranged according to the hardware byte alignment rules. It receives search requests and converts the search criteria in the search requests into binary byte sequences of the same specification, and loads the index topology structure and bitmap data through memory mapping. Based on the transformed search criteria, traverse the hybrid index tree corresponding to each dimension attribute, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node that the search criteria hit, and use the memory pointer to read the corresponding adaptive compressed bitmap data. Based on the logical operators in the search criteria, the corresponding logical operations are performed on each read adaptive compressed bitmap to generate a result bitmap. The result bitmap is parsed to obtain an internal row ID set. The original network traffic data is restored and output by addressing the internal row IDs according to the pre-established forward mapping table.

[0010] A multidimensional network traffic retrieval system, the system comprising: The traffic preprocessing module is used to assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence. The hybrid index building module is used to build hybrid indexes based on a single attribute. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, it is split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the corresponding adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the cardinality of the bitmap. The index storage module is used to split and serialize the index topology structure and bitmap data of each batch of construction, and store them in different files, arranging the bitmap storage addresses according to hardware byte alignment rules; The retrieval loading module is used to receive retrieval requests and convert the retrieval conditions in the retrieval requests into binary byte sequences of the same specification, and load the index topology structure and bitmap data through memory mapping. The bitmap operation module is used to traverse the hybrid index tree corresponding to each dimension attribute according to the transformed search conditions, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node hit by the search conditions, and use the memory pointer to read the corresponding adaptive compressed bitmap data. The result restoration module is used to perform corresponding logical operations on each read adaptive compressed bitmap according to the logical operators in the search conditions, generate a result bitmap, parse the result bitmap to obtain an internal row ID set, restore and output the original network traffic data by addressing through the internal row IDs according to the pre-established positive mapping table.

[0011] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence. Hybrid indexes are built for each single attribute. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, it is split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the bitmap cardinality. The index topology and bitmap data of each batch are split and serialized, and stored in different files. The bitmap storage addresses are arranged according to the hardware byte alignment rules. It receives search requests and converts the search criteria in the search requests into binary byte sequences of the same specification, and loads the index topology structure and bitmap data through memory mapping. Based on the transformed search criteria, traverse the hybrid index tree corresponding to each dimension attribute, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node that the search criteria hit, and use the memory pointer to read the corresponding adaptive compressed bitmap data. Based on the logical operators in the search criteria, the corresponding logical operations are performed on each read adaptive compressed bitmap to generate a result bitmap. The result bitmap is parsed to obtain an internal row ID set. The original network traffic data is restored and output by addressing the internal row IDs according to the pre-established forward mapping table.

[0012] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence. Hybrid indexes are built for each single attribute. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, it is split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the bitmap cardinality. The index topology and bitmap data of each batch are split and serialized, and stored in different files. The bitmap storage addresses are arranged according to the hardware byte alignment rules. It receives search requests and converts the search criteria in the search requests into binary byte sequences of the same specification, and loads the index topology structure and bitmap data through memory mapping. Based on the transformed search criteria, traverse the hybrid index tree corresponding to each dimension attribute, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node that the search criteria hit, and use the memory pointer to read the corresponding adaptive compressed bitmap data. Based on the logical operators in the search criteria, the corresponding logical operations are performed on each read adaptive compressed bitmap to generate a result bitmap. The result bitmap is parsed to obtain an internal row ID set. The original network traffic data is restored and output by addressing the internal row IDs according to the pre-established forward mapping table.

[0013] The aforementioned multidimensional network traffic retrieval method, system, computer equipment, and storage medium, by configuring continuous internal row IDs and a unified attribute binary encoding format for each traffic item, and designing a hybrid index architecture based on a prefix trie with leaf binding adaptive compressed bitmaps, can eliminate the frequent node splitting and structural rebalancing operations of traditional tree indexes, significantly improving the index write speed in massive network traffic scenarios. By splitting and serializing the index topology and bitmap data and storing them according to hardware alignment rules, the space redundancy caused by overall serialization can be avoided, compressing the overall disk usage of the index. By loading the index through a memory mapping mechanism and only obtaining the bitmap pointer, the deserialization operation required for traditional index loading can be eliminated, shortening the index cold start time. Relying on bitmap set operations combined with the blacklist unified difference processing method, it can replace the retrieval mode of traversing the candidate list and merging each item, reducing CPU overhead and invalid I / O reads in the multidimensional query process. The embodiments of the present invention can simultaneously achieve high-speed index construction and low-latency multidimensional retrieval in massive network traffic environments. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a multidimensional network traffic retrieval method in one embodiment; Figure 2 This is a schematic diagram of the hybrid index structure in one embodiment, wherein, Figure 2 (a) is a schematic diagram of the prefix trie index structure. Figure 2 (b) is a schematic diagram of the adaptive compressed bitmap structure. Figure 2 (c) is a schematic diagram of the overall structure of the hybrid index; Figure 3 This is a flowchart illustrating a single-pass accumulation algorithm for a single-attribute range query in one embodiment. Figure 4 This is a schematic diagram of a double intersection algorithm in a multi-attribute combined query in one embodiment; Figure 5 This is a schematic diagram of the storage format of the hybrid index structure on disk in one embodiment; Figure 6 This is a flowchart illustrating a multi-dimensional network traffic retrieval method in a specific embodiment; Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0016] In one embodiment, such as Figure 1 As shown, a multidimensional network traffic retrieval method is provided, including the following steps: Step 102: Assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence.

[0017] The continuously increasing internal row ID is a unique traffic identifier assigned sequentially from the initial value to each processing batch. Attribute cardinality refers to the total number of selectable values ​​for a corresponding field attribute; a wide range of values ​​indicates a high cardinality attribute, while a narrow range indicates a low cardinality attribute. Fixed-length byte splitting involves defining a specific fixed byte length based on the value range of each attribute dimension. The binary byte sequence is the standardized byte data stream formed after splitting. Specifically, based on the preset numerical range of each attribute dimension, standardized byte splitting and transformation are performed. For high cardinality attributes (such as IPv4 addresses), they are converted into a 4-byte array in big-endian order. For low cardinality attributes (such as protocol numbers), the number of bytes split (such as 1 byte) is customized according to their value space, thus transforming discrete feature values ​​into a fixed-length binary byte sequence. This process, through vectorized alignment of features, provides a standardized input path for constructing the prefix routing tree in subsequent steps.

[0018] It is understandable that standardized ID allocation and attribute encoding can organize heterogeneous traffic raw data, unify the input format of the entire data, lay the data foundation for the subsequent standardization of index structure, and facilitate the batch writing of bitmaps by leveraging continuous ID attributes.

[0019] Step 104: Construct hybrid indexes based on single attributes. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, the nodes are split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the bitmap cardinality.

[0020] The prefix trie based on the shortest disjoint prefix uses the shortest unique byte of a feature as the route termination criterion, simplifying the tree's hierarchy. Prefix collisions occur when two traffic streams have identical prefix bytes but different complete attribute content; in such cases, they are split and new internal nodes and leaf nodes are generated. An adaptive compressed bitmap is a bitmap carrier that can switch its internal storage format according to changes in storage entries. The bitmap cardinality represents the total number of internal row IDs stored in a single bitmap. Adaptive storage format switching means automatically changing the underlying storage structure of the bitmap based on the cardinality value.

[0021] Step 106: Split and serialize the index topology and bitmap data of each batch, and store them in different files, arranging the bitmap storage addresses according to hardware byte alignment rules.

[0022] Index topology refers to the tree-like skeleton information of the prefix trie, such as node hierarchy and branch relationships. Split serialization involves encoding the index skeleton information and bitmap entity data separately and storing them in independent files. Hardware byte alignment rules constrain the starting storage location of the bitmap data block according to the CPU hardware addressing standard.

[0023] It is understandable that separating topology and data storage can eliminate the space redundancy caused by invalid pointers during serialization, and the byte-aligned arrangement can adapt to the hardware read and write logic, reduce the disk space occupied by the index, and facilitate direct data access through memory mapping in the future.

[0024] Step 108: Receive the retrieval request and convert the retrieval conditions in the retrieval request into a binary byte sequence of the same specification, and load the index topology structure and bitmap data through memory mapping.

[0025] Search condition vectorization reuses the byte encoding rules used during database creation to transform query content. Memory mapping is an access mechanism where the operating system directly maps disk files to the process's virtual address space. A bitmap memory pointer is the starting address identifier of the corresponding bitmap data block within the mapped space.

[0026] It is understandable that using the same encoding rules for querying and database creation can ensure the accuracy of tree route matching, memory mapping eliminates the need for data copying between user space and kernel space, which can shorten the time spent loading indexes during cold starts, and obtaining only pointers without pre-reading the complete bitmap can reduce unnecessary disk data reads.

[0027] Step 110: Traverse the hybrid index tree corresponding to each dimension attribute according to the transformed search conditions, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node that the search conditions match, and use the memory pointer to read the corresponding adaptive compressed bitmap data.

[0028] Negation conditions are the filtering rules in a query command that require the removal of corresponding traffic. The blacklist set is used to summarize the compressed bitmaps corresponding to all exclusion conditions. Bitmap logical operations include AND and OR set bitwise operations. The unified difference operation refers to the operation method that uses the result of the positive operation to remove data corresponding to the blacklist.

[0029] It is understandable that centralizing all exclusion conditions and uniformly performing subtraction operations can avoid ambiguity in Boolean operation priority. Selecting dedicated optimization algorithms based on query type can reduce the amount of data involved in the calculation and reduce the computational power consumption of multidimensional composite query processes.

[0030] Step 112: Based on the logical operators in the search conditions, perform corresponding logical operations on each read adaptive compressed bitmap to generate a result bitmap. Parse the result bitmap to obtain an internal row ID set. Based on the pre-established forward mapping table, address the internal row IDs to restore and output the original network traffic data.

[0031] The resulting bitmap is a compressed bitmap data containing the compliant row IDs after all logical operations are completed. Fixed-point addressing uses the internal row ID as the array offset to directly locate the storage position. A forward mapping table pre-stores the correspondence between the internal row IDs and all fields of the original traffic.

[0032] It is understandable that fixed-point addressing based on ID offset does not require traversing and searching the entire table, which can speed up the location of original traffic information and facilitate the rapid completion of query data restoration and result encapsulation and output.

[0033] In the aforementioned multidimensional network traffic retrieval method, by configuring continuous internal row IDs and uniform attribute binary encoding formats for each traffic item, and designing a hybrid index architecture based on a prefix trie with the shortest disjoint prefix and leaf binding adaptive compressed bitmaps, the frequent node splitting and structural rebalancing operations of traditional tree indexes can be eliminated, significantly improving the index write speed in massive network traffic scenarios. By splitting and serializing the index topology and bitmap data and storing them according to hardware alignment rules, the space redundancy caused by overall serialization can be avoided, compressing the overall disk usage of the index. By loading the index through a memory mapping mechanism and only obtaining the bitmap pointer, the deserialization operation required for traditional index loading can be eliminated, shortening the index cold start time. Relying on bitmap set operations combined with the blacklist unified difference processing method, the retrieval mode of traversing the candidate list and merging each item can be replaced, reducing CPU overhead and invalid I / O reads in the multidimensional query process. In this embodiment of the invention, high-speed index construction and low-latency multidimensional retrieval can be achieved simultaneously in massive network traffic environments.

[0034] In one embodiment, the step of assigning continuously increasing internal row IDs to the currently collected network traffic records further includes: counting the number of network traffic records for which internal row IDs have been assigned since the previous batch was written to disk; when the number reaches a preset threshold, triggering the persistent storage operation of the currently accumulated current batch index and resetting the internal row ID counter; during the accumulation process, recording the minimum and maximum timestamps of all network traffic records in the current batch in real time, and encoding the minimum and maximum timestamps as time-series metadata into the filename of the current batch index file when persistent storage is triggered, so as to filter batches whose time range does not overlap with the retrieval request during the retrieval stage.

[0035] In this embodiment, to match the 32-bit Roaring Bitmap compressed bitmap structure, after receiving the original traffic metadata, the system assigns a 32-bit internal row ID (InternalRowID) starting from 0 and incrementing sequentially to each flow record. This internal row ID, along with the original flow ID, multi-dimensional attribute feature values, and the physical offset of the flow record in the underlying original storage (PCAP file or log file), is synchronously recorded in the row ID forward mapping table. Because the internal row IDs are continuously incremented, the forward mapping table is physically represented as a fixed-length linear array with the internal row ID as the index. In subsequent table lookup stages, the corresponding record is read directly using the internal row ID as the fixed-length array index, achieving O(1) complexity addressing. Utilizing the continuously incrementing characteristic of the internal row IDs, subsequent compressed bitmap containers can be constructed through sequential appending, significantly improving the index construction speed. At this point, the internal row ID serves as the unique identifier for the current batch of flows. While allocating internal row IDs, the system performs vectorization and byte splitting of flow features: based on the preset numerical range of each dimension attribute feature (e.g., IP address in 4 bytes, protocol number in 1 byte), the system performs standardized byte splitting and conversion on each attribute feature value to generate a fixed-length binary byte sequence.

[0036] During the extraction of stream features, the system simultaneously performs time-series boundary detection: by comparing and recording the extreme start times of all stream records within the current processing batch in real time, the minimum timestamp (min_t) and maximum timestamp (max_t) of the batch are obtained. These two extreme values ​​are encapsulated as time-series metadata of the batch's index file, used in subsequent retrieval stages to quickly filter and skip irrelevant data blocks based on time range determination. The minimum and maximum timestamps are encoded into the row map table filename in a fixed format, such as batch_index_start_timestamp_end_timestamp_rowmap.bin. The retrieval engine can quickly determine whether the time range of the batch overlaps with the query request by scanning the filename, avoiding loading invalid batch indexes. The system continuously monitors the scale of records processed in the current batch (i.e., the number of records assigned internal row IDs since the previous batch was written to disk). When the number reaches a preset flow threshold (e.g., 1 million), the persistent storage operation for the current batch is triggered: first, the persistent storage of the current row ID mapping table is triggered, and a global dictionary maintaining the original traffic file paths for the current batch is exported for subsequent reverse parsing. Then, the internal row ID counter is reset to zero, and the index persistence and disk write operations are started simultaneously, thereby initiating the construction of the forward mapping table for the next batch. Through the above batch-order processing, it is ensured that large-scale traffic data can be managed according to a defined data block size, avoiding memory overflow and index bloat.

[0037] In one embodiment, the step of routing the internal nodes of the prefix trie layer by layer according to the feature bytes includes: traversing the prefix trie layer by layer from the high byte to the low byte according to the binary byte sequence of the attribute feature to be inserted; when the current byte path uniquely distinguishes the attribute feature to be inserted from other attribute features already stored in the trie, stopping the traversal of the tree structure downwards; generating a leaf node at the current position; and storing the complete feature value of the attribute feature to be inserted in the leaf node.

[0038] In this embodiment, when data traverses downwards along the index tree and encounters an empty branch for the first time, the creation of subsequent internal nodes is immediately terminated, and a leaf node is directly generated at the current position. To handle potential prefix conflicts, the complete feature value of the attribute is recorded in the leaf node, and a RoaringBitmap (RB) bitmap object is initialized simultaneously. When a leaf node is encountered during traversal, the complete attribute value stored in the leaf node is used for comparison. If the attribute values ​​are the same, the internal row ID (or offset, etc., a unique identifier) ​​corresponding to the network traffic record to be inserted is directly appended to the RB. If the attribute values ​​are different, a collision occurs. In this case, a new internal node needs to be created at the current depth as a prefix for differentiation. The original leaf node is appended to the newly created internal node according to the byte value, and the new leaf node is used as the writing target for the currently inserted data.

[0039] RB is a high-efficiency compressed bitmap structure that divides a 32-bit integer space into a high 16-bit container key (i.e., container number) and a low 16-bit offset (within the container). Each high 16 bits corresponds to a container. When elements are sparse, an ordered array (ArrayContainer) is used to save space; when elements are dense, a fixed-size bitmap (Bitmap Container) is used to improve bitwise operation speed. It balances high compression ratio with fast intersection, union, and difference operations, making it suitable for large-scale data set computation scenarios. In this embodiment, the routing layer uses a Trie-Index structure, combined with the Roaring Bitmap storage layer to form a hybrid index structure. A schematic diagram of the hybrid index structure is shown below. Figure 2 As shown, where, Figure 2 (a) is a schematic diagram of the prefix trie index structure. Figure 2 (b) is a schematic diagram of the adaptive compressed bitmap structure. Figure 2 (c) is a schematic diagram of the overall structure of the hybrid index.

[0040] When a routing path reaches a leaf node of the prefix trie, the system retrieves the corresponding compressed bitmap (RB) container. The internal row ID corresponding to the current network traffic record is then appended to the Roaring Bitmap. During the bitmap appending process, the cardinality stored within the current leaf node is monitored in real time. Based on a preset threshold (e.g., 4096), the physical format of the underlying container is dynamically adjusted: when the number of flows is sparse, an array container is used to save space; when the data becomes dense and exceeds the threshold, it is automatically converted to a bitmap container to improve computational performance. This adaptive storage strategy achieves an optimal balance between index compression ratio and computational efficiency. After this step, a hybrid index tree combining the prefix tree and compressed bitmap is constructed for each dimension.

[0041] In one embodiment, the step of appending the internal row ID of the corresponding traffic to the corresponding adaptive compression bitmap includes: calling the bitmap append interface through a batch write context to map consecutive internal row IDs to the corresponding positions in the adaptive compression bitmap.

[0042] In this embodiment, based on the monotonically increasing characteristic of the internal row IDs, the continuous row IDs are mapped to the position information in the bitmap by calling the bitmap append interface through the BulkContext. This avoids frequent memory reallocation of the bitmap container and achieves O(1) complexity writing of index data.

[0043] In one embodiment, the step of splitting and serializing the index topology structure and bitmap data constructed in each batch specifically involves: scanning the prefix trie in memory using a depth-first traversal algorithm to serialize the index topology structure into a first file; the first file stores the routing information of internal nodes, the attribute values ​​of leaf nodes, and the starting offset and byte length of the corresponding adaptive compressed bitmap; and concatenating the original binary data of all adaptive compressed bitmaps consecutively into a second file, with the starting address of each compressed bitmap data block aligned to the byte boundary aligned with the hardware instruction.

[0044] Specifically, the index topology (internal routing nodes, path information) is separated from the underlying bitmap data: the tree skeleton is transformed into a compact linear address space layout and serialized into a first file (e.g., the main index file `index_out`). The first file stores the routing information of internal nodes, the attribute values ​​of leaf nodes, and the starting offset and byte length of the corresponding adaptive compressed bitmap in the second file; it no longer stores business row ID data. Simultaneously, the adaptive compressed bitmap containers scattered throughout memory are aggregated into contiguous binary data blocks and serialized into a second file (e.g., the data file `data_out`), with the starting address of each compressed bitmap data block aligned to the byte boundary aligned to the hardware instruction. The index storage format is as follows: Figure 5 As shown in the figure, the separate storage structure of the index_out topology file and the data_out bitmap file is illustrated: the leaf nodes of the prefix trie store the complete attribute values ​​of the corresponding attributes, as well as the offset and length of the corresponding bitmap data blocks, which are used to locate the Roaring Bitmap data stored contiguously in data_out; the bitmap data is stored sequentially in units of containers, and the separation and arrangement of the topology and data are completed by depth-first traversal serialization.

[0045] Before each bitmap data block is written, the system performs low-level physical cursor probing: obtaining the physical address of the current file write cursor. If the current write address does not meet the addressing requirements of the underlying SIMD hardware instructions, the system injects a very small zero-value padding block to force the physical start address of each bitmap data block to strictly adhere to 32-byte boundary alignment. This physical arrangement paves the way for subsequent retrieval processes to directly utilize the operating system's low-level mapping (mmap) to trigger high-speed hardware-level computation. This separate storage mechanism establishes the decoupling of indexing logic from physical data, providing a foundation for subsequent on-demand loading.

[0046] After the index tree traversal is complete, the system merges the first and second files of the current batch into a unified metadata index file (e.g., index.bin) and writes it to disk. Simultaneously, it exports a global dictionary maintaining the original traffic file paths for use in reverse lookup throughout the global lifecycle. Furthermore, the minimum and maximum timestamps of the batch recorded during the accumulation process are encoded in the row map table filename (e.g., batch_0_mint_maxt_rowmap.bin), allowing the subsequent query engine to obtain the time span of each batch simply by scanning the directory upon startup. After all processing is complete, the system safely releases all memory resources occupied by the current batch in the prefix trie and bitmap container.

[0047] In one embodiment, for the current batch index that has not been persisted, the retrieval operation is performed directly in the hybrid index structure in memory.

[0048] In one embodiment, the step of receiving a retrieval request and converting the retrieval conditions in the retrieval request into a binary byte sequence of the same specification includes: using the fixed-length byte splitting rules and encoding format of the traffic preprocessing stage to parse and encode the retrieval conditions, and generating a binary byte sequence consistent with the attribute feature specification.

[0049] In this embodiment, the user-input search criteria (such as a specific IP address, port, or protocol type) are parsed and converted into a fixed-length byte sequence consistent with the index building phase. For example, the input query IP string is parsed into a 4-byte big-endian array, which serves as the standard key for drilling down the path in the prefix trie.

[0050] In one embodiment, the steps of loading the index topology and bitmap data through memory mapping and traversing the prefix trie to obtain the memory pointers of the adaptive compressed bitmaps corresponding to each search condition include: loading the index topology file and bitmap data file through the memory mapping interface, traversing the prefix trie to the target leaf node, and obtaining the memory pointer of the corresponding bitmap data block according to the offset and length information stored in the leaf node.

[0051] In this embodiment, upon receiving a retrieval request, instead of using traditional file I / O to read the index data into the user-space buffer, the binary index file on disk is directly mapped to the process's virtual address space via a memory mapping interface (such as the mmap system call). Since the index file has undergone strict physical layout alignment during disk writing (e.g., bitmap data blocks are aligned to 32-byte boundaries), the mapped virtual address can be directly accessed as a memory pointer without additional address translation or data copying. This process eliminates redundant data copying between kernel and user modes, achieving zero-load startup of the index data and reducing memory pressure during system cold starts.

[0052] To address multi-dimensional query requirements, search criteria are converted into the same binary byte sequence as during the index building phase, and layer-by-layer routing matching is performed within each index tree, starting from the root node. Offsets stored in the routing nodes are used to quickly locate the next-level node address until the target leaf node is reached. Once the path addressing reaches the leaf node, the system does not immediately read the actual bitmap container data. Instead, it directly obtains the physical memory start pointer of the adaptively compressed bitmap container stored within the mapped region, which is pointed to by the leaf node. This pointer directly points to the aligned bitmap data block, providing a direct memory operation handle for subsequent bitmap acceleration calculations, thus avoiding the overhead of traditional index page swapping in and out. Because the bitmap data block is forcibly aligned to the byte boundaries required by hardware instructions when written to disk, the virtual address returned after mapping can be directly used as the source operand for SIMD instructions (such as AVX2), avoiding additional memory copying or address adjustments, achieving zero-overhead data access.

[0053] In one embodiment, the step of performing logical operations on the adaptive compressed bitmap according to the logical operators in the search conditions includes: when the search condition is a single attribute range query, obtaining the adaptive compressed bitmap corresponding to all leaf nodes within the single attribute range; merging the adaptive compressed bitmap by traversing the full mapping form once, and adaptively switching the storage format according to the cardinality of the merged bitmap to generate a single-dimensional result bitmap.

[0054] In this embodiment, when the retrieval condition is an exact query, the single adaptive compressed bitmap obtained after route matching is directly used as the query result for the current attribute dimension. When the retrieval condition is a single attribute range query, the adaptive compressed bitmaps bound to all leaf nodes corresponding to the query interval are aggregated, and multi-bitmap merging is performed using a single-pass accumulation algorithm. Since the data mapped to different leaf nodes of the prefix tree are logically strictly disjoint, a "single-pass accumulation" algorithm optimized for disjoint sets is adopted. See the detailed process below. Figure 3 The steps are as follows: 1) Full mapping and memory pre-allocation: During the merge preparation phase, the system directly allocates a 65536×8 byte pointer array in physical memory as a full mapping table, simultaneously allocates a 65536×4 byte integer array as a radix counting table, and initializes min_key and max_key to minimize the subsequent scan boundaries.

[0055] 2) Container Classification and High-Speed ​​Writing: During a single traversal of the input bitmap, if the corresponding mapping table `fast_map[key]` is empty, an 8KB standard bitmap container is lazily allocated, and the input radix is ​​accumulated in the radix counting mapping table `card_map`. For the diverse underlying container physical formats, the system implements customized high-speed writing: if the input is an array, it uses 4-way loop unrolling bit operations for writing; if the input is a bitmap, it directly calls AVX2 (SIMD) instructions to perform 256-bit aligned memory block merging operations.

[0056] 3) Adaptive Degradation and Result Reorganization: After the full traversal is completed, the system linearly scans the entire mapping table from min_key to max_key. Based on the precise cardinality of the card_map record, if the cardinality of the merged container is strictly less than 4096, the system will dynamically degrade it back to a high-compression array container; otherwise, it will maintain the bitmap form and eventually mount it to the result bitmap object.

[0057] In one embodiment, the step of performing logical operations on the adaptive compressed bitmap according to the logical operator in the search condition further includes: when the search condition is a multi-attribute joint query and the logical operator is AND, sorting all the adaptive compressed bitmaps participating in the operation in ascending order of the number of internal containers, and selecting the adaptive compressed bitmap with the fewest containers as the operation axis; traversing the containers of the operation axis, performing a jump search on other adaptive compressed bitmaps, and performing pairwise intersection operations only on containers where all the adaptive compressed bitmaps participating in the operation exist, sorting them in ascending order of container cardinality.

[0058] In this embodiment, if the query condition is a multi-dimensional joint query, mainly targeting the intersection of multiple attributes in traffic retrieval, then a "double intersection" optimization operation for multi-dimensional combined queries is adopted. This algorithm mainly performs intersection calculations after obtaining the corresponding bitmaps for multiple attributes. See the detailed process below. Figure 4 The example covers three search dimensions: SIP (Source IP address), Plength (Packet Length), and SPORT (Source Port). The steps are as follows: 1) Key-level pivot skip search: First, a global forward sort is performed based on the key_size (number of container keys) of each parameter bitmap, forcibly selecting the bitmap with the fewest keys as the pivot for retrieval. When traversing the pivot key, an exponential gallop search (jump search) is performed on the high-order key arrays of all other bitmaps. Once a bitmap is found to be missing a key, a short-circuit skip mechanism is immediately triggered, blocking the loading and computation of invalid underlying containers.

[0059] 2) Container-level radix-sorted intersection: The system extracts the corresponding container and sorts it a second time by its cardinality only when a key exists in all participating bitmaps. In the subsequent pairwise intersection operations, this strategy makes deep use of the short-circuiting effect of intersection operations: intersection of small cardinality containers is very likely to produce smaller results or even an empty set, thus terminating the loop as early as possible and greatly avoiding the computational waste of subsequent large containers participating in bitwise operations.

[0060] In one embodiment, the step of performing logical operations on the adaptive compressed bitmap according to the logical operators in the search conditions further includes: when the search conditions contain a negative clause, aggregating the adaptive compressed bitmap corresponding to the negative clause into a blacklisted unit map, performing logical operations on the adaptive compressed bitmap corresponding to the positive clause to obtain a positive result bitmap, and then performing a difference operation between the positive result bitmap and the blacklisted unit map to generate a result bitmap.

[0061] In this embodiment, a global blacklist architecture is implemented to address the complex combined query rules in real-world network security situational awareness. In a parenthesis-free query terminal, when a user inputs a "NOT" operation, their true business intent is often to globally remove a certain type of traffic from the final result, rather than creating priority ambiguity with local OR conditions. Therefore, the engine scans all query conditions during execution, extracts all candidate bitmaps with the NOT operator, and stores them in a "global blacklist pool." After the remaining AND and OR conditions are calculated, the engine uses the underlying C language interface to aggregate all bitmaps in the blacklist pool into a large "blacklist unit graph" and performs a single bitmap difference operation (ANDNOT) on the positive result.

[0062] For the remaining positive conditions (all AND or OR) after removing NOT, the engine will not calculate them one by one. Instead, it will dynamically split the execution path into three main branches through a lightweight O(N) probe: 1) Full AND: Utilizing a "double intersection" optimization algorithm; 2) Full OR: Utilizing the library's native algorithm; 3) Mixed logic (including AND and OR): AND is performed first, followed by OR, for pairwise calculations.

[0063] The final result is a row ID bitmap. The intermediate containers obtained from the above operations and their corresponding key values ​​are encapsulated into the final bitmap object. This bitmap only maintains a compressed internal set of row IDs in memory and does not trigger actual traffic data access.

[0064] In one embodiment, the step of restoring and outputting the original network traffic data by addressing the internal row IDs according to the pre-established forward mapping table includes: parsing the result bitmap and extracting the set of internal row IDs stored therein; using each internal row ID in the set of internal row IDs as an offset address, performing direct addressing in the forward mapping table to obtain the corresponding original flow ID, multi-dimensional attribute feature value, and the physical location of the record in the original storage; and summarizing and encapsulating the obtained original flow ID, multi-dimensional attribute feature value, and physical location according to a preset output format and outputting them to the client.

[0065] In this implementation, the final bitmap obtained through logical operations is parsed, and the 32-bit internal row ID set stored therein is extracted. The forward mapping table is the row mapping table that is constructed synchronously when allocating internal row IDs and persisted to disk in batches. Each internal row ID is used as a physical offset address, and O(1) complexity direct addressing is performed in the forward mapping table to obtain the corresponding original stream ID, the complete network seven-tuple feature value, and the physical location of the record in the original storage. Finally, the above information is summarized and encapsulated according to the output format specified by the user (such as JSON, CSV, or structured list) and delivered to the client.

[0066] After completing the multi-attribute bitmap acceleration operation, the final result Roaring Bitmap is decompressed to obtain the corresponding stream ID (or unique identifier) ​​result.

[0067] In one specific embodiment, such as Figure 6The diagram illustrates a process flow for a multi-dimensional network traffic retrieval method. The overall process is divided into six sequentially connected functional modules, following a business timeline of index construction, index persistence, and online retrieval computation. Data enters the data extraction stage of module one via a streaming message queue. This stage first completes the access of raw traffic data and constructs a positive mapping table linking row IDs with raw flow identifiers, attributes, and physical offsets. Then, feature processing generates multi-dimensional feature structures. The structured data flows down to the index construction stage of module two. The index is split into a Trie_Index routing layer and a RoaringBitmap storage layer. The routing layer generates multi-branch internal nodes based on big-endian prefix routing. The storage layer appends data to the leaf nodes bound to the RB structure. After index construction is complete, the process proceeds to module three for index persistence. Serialization is performed using depth-first search (DFS), and frozen serialization generates an index_out file storing topology information and a data_out file storing bitmap data. These two types of files are merged and packaged into a .bin file. The index file is stored on disk; subsequent retrieval requests enter the retrieval query stage of module four, which loads the index_out and data_out data split from bin through mmap memory mapping, distinguishing between two branches: precise retrieval and range retrieval. Precise retrieval locates the offset parameters of the target leaf node through prefix routing, while range retrieval relies on a single traversal accumulation algorithm to match the parameters of all leaf nodes corresponding to the interval. Both types of queries use memory mapping to obtain the RB bitmap results of each dimension in memory. All bitmap data is sent to the computation acceleration module of module five. Single-attribute data completes range merging, and multi-attribute data uses a two-layer intersection acceleration algorithm to complete multi-dimensional bitmap logical operations and output the final RB result. Finally, the row ID array obtained by RB parsing is sent to the result mapping of module six. Relying on the pre-stored forward mapping table, the original flow ID, full attributes, and physical offset positions in the original traffic capture file (pcap) are restored in reverse, completing the end-to-end traffic retrieval and data restoration.

[0068] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0069] In one embodiment, a multidimensional network traffic retrieval system is provided, comprising: The traffic preprocessing module is used to assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence. The hybrid index building module is used to build hybrid indexes based on a single attribute. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, it is split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the corresponding adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the cardinality of the bitmap. The index storage module is used to split and serialize the index topology structure and bitmap data of each batch of construction, and store them in different files, arranging the bitmap storage addresses according to hardware byte alignment rules; The retrieval loading module is used to receive retrieval requests and convert the retrieval conditions in the retrieval requests into binary byte sequences of the same specification, and load the index topology structure and bitmap data through memory mapping. The bitmap operation module is used to traverse the hybrid index tree corresponding to each dimension attribute according to the transformed search conditions, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node hit by the search conditions, and use the memory pointer to read the corresponding adaptive compressed bitmap data. The result restoration module is used to perform corresponding logical operations on each read adaptive compressed bitmap according to the logical operators in the search conditions, generate a result bitmap, parse the result bitmap to obtain an internal row ID set, restore and output the original network traffic data by addressing through the internal row IDs according to the pre-established positive mapping table.

[0070] For specific limitations regarding the multidimensional network traffic retrieval system, please refer to the limitations of the multidimensional network traffic retrieval method mentioned above, which will not be repeated here. Each module in the aforementioned multidimensional network traffic retrieval system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0071] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, display screen, and input system connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-dimensional network traffic retrieval method. The display screen can be an LCD screen or an e-ink screen. The input system can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0072] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0073] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0074] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0075] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for multidimensional retrieval of network traffic, characterized in that, The method includes: Assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence. Hybrid indexes are built for each single attribute. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, it is split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the bitmap cardinality. The index topology and bitmap data of each batch are split and serialized, and stored in different files. The bitmap storage addresses are arranged according to the hardware byte alignment rules. It receives search requests and converts the search criteria in the search requests into binary byte sequences of the same specification, and loads the index topology structure and bitmap data through memory mapping. Based on the transformed search criteria, traverse the hybrid index tree corresponding to each dimension attribute, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node that the search criteria hit, and use the memory pointer to read the corresponding adaptive compressed bitmap data. Based on the logical operators in the search criteria, the corresponding logical operations are performed on each read adaptive compressed bitmap to generate a result bitmap. The result bitmap is parsed to obtain an internal row ID set. The original network traffic data is restored and output by addressing the internal row IDs according to the pre-established forward mapping table.

2. The method according to claim 1, characterized in that, The step of assigning continuously incrementing internal row IDs to the currently collected network traffic records further includes: The system counts the number of network traffic records with assigned internal row IDs since the last batch was written to disk. When the number reaches a preset threshold, it triggers the persistent storage operation of the currently accumulated index of the current batch and resets the internal row ID counter. During the accumulation process, the minimum and maximum timestamps of all network traffic records in the current batch are recorded in real time. When persistent storage is triggered, the minimum and maximum timestamps are encoded as time-series metadata into the filename of the current batch index file to filter batches whose time ranges do not overlap with the retrieval request during the retrieval phase.

3. The method according to claim 1, characterized in that, The steps of routing the internal nodes of the prefix trie layer by layer according to the feature bytes include: According to the binary byte sequence of the attribute feature to be inserted, the prefix trie is traversed layer by layer from the high byte to the low byte. When the current byte path uniquely distinguishes the attribute feature to be inserted from other attribute features already stored in the trie, the traversal of the tree structure is stopped, a leaf node is generated at the current position, and the complete feature value of the attribute feature to be inserted is stored in the leaf node.

4. The method according to claim 1, characterized in that, The step of appending the internal row ID of the corresponding traffic to the corresponding adaptive compression bitmap includes: By calling the bitmap append interface in the batch write context, consecutive internal row IDs are mapped to the corresponding positions in the adaptive compressed bitmap.

5. The method according to claim 1, characterized in that, The specific steps for splitting and serializing the index topology structure and bitmap data constructed in each batch are as follows: The prefix trie in memory is scanned by a depth-first traversal algorithm, and the index topology structure is serialized into a first file; the first file stores the routing information of the internal nodes, the attribute values ​​of the leaf nodes, and the starting offset and byte length of the corresponding adaptive compressed bitmap; The raw binary data of all adaptive compressed bitmaps are concatenated into a second file, with the starting address of each compressed bitmap data block aligned to the byte boundary aligned with the hardware instruction.

6. The method according to claim 1, characterized in that, The step of performing logical operations on the adaptive compressed bitmap based on the logical operators in the search criteria includes: When the search condition is a single attribute range query, obtain the adaptive compressed bitmap corresponding to all leaf nodes within the single attribute range; The adaptive compressed bitmap is merged by traversing the full mapping form one by one, and the storage format is adaptively switched according to the cardinality of the merged bitmap to generate a single-dimensional result bitmap.

7. The method according to claim 1, characterized in that, The step of performing logical operations on the adaptive compressed bitmap based on the logical operators in the search conditions further includes: When the search condition is a multi-attribute joint query and the logical operator is AND, all adaptive compressed bitmaps participating in the operation are sorted in ascending order of the number of internal containers, and the adaptive compressed bitmap with the fewest number of containers is selected as the main axis of the operation. Traverse the containers of the main axis of operation, perform a jump search on other adaptive compressed bitmaps, and only perform pairwise intersection operations on containers that exist in all adaptive compressed bitmaps involved in the operation, sorted by container cardinality in ascending order.

8. A multi-dimensional network traffic retrieval system, characterized in that, The system includes: The traffic preprocessing module is used to assign continuously increasing internal row IDs to the currently collected network traffic records, extract the attribute features of each dimension and split them into fixed-length bytes according to the difference in attribute cardinality, and convert each dimension attribute into a fixed-format binary byte sequence. The hybrid index building module is used to build hybrid indexes based on a single attribute. The hybrid index includes a prefix trie based on the shortest disjoint prefix. The internal nodes of the prefix trie are routed layer by layer according to the feature bytes. When a feature prefix conflict occurs, it is split to generate new internal nodes and leaf nodes. Each leaf node is bound to an adaptive compressed bitmap for storing the set of internal row IDs. The internal row IDs of the corresponding traffic are appended to the corresponding adaptive compressed bitmap, and the internal storage format of the bitmap is adaptively switched according to the cardinality of the bitmap. The index storage module is used to split and serialize the index topology structure and bitmap data of each batch of construction, and store them in different files, arranging the bitmap storage addresses according to hardware byte alignment rules; The retrieval loading module is used to receive retrieval requests and convert the retrieval conditions in the retrieval requests into binary byte sequences of the same specification, and load the index topology structure and bitmap data through memory mapping. The bitmap operation module is used to traverse the hybrid index tree corresponding to each dimension attribute according to the transformed search conditions, obtain the memory pointer of the adaptive compressed bitmap pointed to by the leaf node hit by the search conditions, and use the memory pointer to read the corresponding adaptive compressed bitmap data. The result restoration module is used to perform corresponding logical operations on each read adaptive compressed bitmap according to the logical operators in the search conditions, generate a result bitmap, parse the result bitmap to obtain an internal row ID set, restore and output the original network traffic data by addressing through the internal row IDs according to the pre-established positive mapping table.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images