Network traffic load search method, device and equipment

Abstract

In order to overcome the accuracy and storage efficiency problems of existing load search methods, the present invention proposes a network traffic load search method, device and equipment, including using traffic layering and traffic load blocking to generate a bitmap index structure: dividing the flow to be processed into levels according to size; configuring index parameters for each level differently; using index parameters to block the traffic load; filling the mapping records of the flow table and building a load fragment addressing matrix; compressing and storing on disk to generate a bitmap index structure; performing load query based on the generated bitmap index structure: linearly loading the bitmap index structure of each level and reading it into memory; reading the index parameters configured for the current level; noise reduction search within the level; precise search within the level; using a parallel reduction algorithm to merge the precise result sets obtained at each level and then feed back to the user. Through the above method, the present invention achieves a significant improvement in load search accuracy and storage efficiency, and has broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of network traffic processing, and in particular to a network traffic load search method, device and equipment. Background Art

[0002] With the rapid development of internet technology and the deepening global digitalization, networks have become an indispensable infrastructure for the functioning of modern society. Vast amounts of data flow constantly through global networks, carrying a wide range of information, from daily communications and commercial transactions to critical infrastructure control. The packet payload within these network flows, known as traffic payload, serves as the core carrier of information transmission and contains extremely rich and sensitive content. It is crucial for network behavior analysis, security incident detection, and tracing. Accurately identifying and locating specific segments within traffic payloads (such as unsafe code signatures, sensitive information leaks, attack command signatures, or specific communication protocol identifiers) is crucial for enterprises to build real-time intrusion detection systems (IDS), defend against advanced persistent threats (APTs), prevent data leaks, and conduct security situational awareness. Furthermore, in the field of network forensics, quickly and accurately retrieving payload segments related to specific events from massive amounts of historical traffic and precisely locating the network flows to which they belong has broad application value in information technology tracing.

[0003] Currently, research on network traffic content analysis focuses primarily on deep packet inspection (DPI) techniques based on regular expression rules and payload attribution techniques using Bloom filters. However, existing technologies have significant and difficult-to-overcome limitations in addressing the critical need for efficient and accurate fragment-level search and location of payload content:

[0004] (1) Rule dependence causes lack of flexibility and efficiency bottlenecks:

[0005] Although regular expression-based DPI technology is mature and has rich expressive capabilities, it has two fundamental problems:

[0006] Poor flexibility: Once a new, undefined payload segment needs to be searched, the rules must be re-established and the entire historical or real-time traffic data set must be rescanned, making dynamic, on-demand, instant search impossible.

[0007] Efficiency bottleneck: In a large-scale traffic environment, repeatedly executing full rule matching results in huge computational overhead, making it difficult to meet the needs of real-time or near-real-time fragment search.

[0008] (2) Storage-granularity dilemma and limited search scope:

[0009] Bloom filter-based payload attribution supports existence queries by storing fixed-length sub-segments (also known as seeds) of the payload. However, its design has an inherent trade-off between storage overhead and search capability:

[0010] Small seed size: Choosing a shorter seed length to support finer-grained fragment searches requires a very large amount of storage space to build the filter (the ratio of raw data size to storage overhead is approximately 3:1) to maintain an acceptably low false positive rate. This is prohibitively expensive when processing massive network traffic.

[0011] Large seed size: If a longer seed size is chosen to control storage overhead, the minimum length of payload segments that can be effectively searched will also increase significantly. This means that a large number of key segments shorter than the seed length cannot be effectively searched, and the searched segments must strictly consist of these long seeds, which greatly limits the scope and flexibility of the searchable segments.

[0012] (3) Weak positioning capabilities, resulting in higher computing costs and lower accuracy:

[0013] When answering which flows a payload fragment is related to, Bloom filters require a high traversal cost and have a high false positive rate, failing to meet the requirement of "precise positioning". This is also a key obstacle to the use of existing technical methods for deep network forensics.

[0014] In summary, due to the diverse types of payloads and their uncertain lengths and meanings, using traditional search methods, such as string matching, when faced with millions or even billions of payload data, would be very costly. For searches on specific payload fragments, existing technologies can only provide approximate results and cannot pinpoint specific packets. Therefore, research on efficient and accurate fragment-level search and positioning of network traffic payloads is dedicated to overcoming the above limitations, deeply mining the internal information of the payload, and achieving precise positioning from "whether it exists" to "where it is located." This will greatly enhance the efficiency and accuracy of proactive network defense capabilities and post-investigation and evidence collection, and has significant potential for technological breakthroughs, broad commercial application prospects, and far-reaching social security value. Summary of the Invention

[0015] In order to overcome the above-mentioned problems of existing load search methods, the present invention provides a network traffic load search method, device and equipment, which is based on a load fragment addressing bitmap indexing mechanism with hierarchical flow size, and realizes efficient search of traffic load in the network through a differentiated indexing strategy.

[0016] To achieve the above objectives, the present invention provides a network traffic load search method, comprising:

[0017] By using traffic stratification and traffic load segmentation, a bitmap index structure is generated, including:

[0018] Based on the Pareto characteristics of traffic distribution and the characteristics of the bitmap index structure, the traffic to be processed is stratified by size to obtain multiple traffic levels. A differentiated indexing strategy is designed between the levels, and index parameters are configured for each traffic level.

[0019] Use index parameters to chunk traffic load;

[0020] Fill in the mapping records of the flow table and build the payload fragment addressing matrix to obtain the bitmap index structure; compress the mapping records of the flow table and the bitmap index structure and write them to disk to generate the bitmap index table;

[0021] Based on the generated bitmap index structure, the traffic load query and search are performed on the query payload fragment by linearly traversing the bitmap index structure at each level, including:

[0022] Linearly load the bitmap index structure of each level and read it into memory;

[0023] Read the index parameters of the current layer configuration;

[0024] Perform intra-level noise reduction search on query payload segments;

[0025] Perform precise searches within a hierarchy to obtain the exact result set for payload fragment queries within the same hierarchy.

[0026] Using a parallel reduction algorithm, the precise result sets obtained at each level are aggregated across levels to obtain the final query and search results for the payload segment to be queried and feed them back to the user.

[0027] In addition, the present invention also provides a network traffic load search device, which includes:

[0028] The first module is used to generate a bitmap index structure by using traffic layering and traffic load segmentation, including:

[0029] The first submodule is used to stratify the to-be-processed flows by size based on the Pareto characteristics of traffic distribution and the characteristics of the bitmap index structure, obtaining multiple traffic levels. It also designs a differentiated indexing strategy between levels and configures index parameters for each traffic level.

[0030] The second submodule is used to divide the traffic load into blocks using index parameters;

[0031] The third submodule is used to fill the mapping records of the flow table, build the payload segment addressing matrix, and obtain the bitmap index structure; compress the mapping records of the flow table and the bitmap index structure to disk, and generate the bitmap index table;

[0032] The second module is used to perform traffic load query and search for the query payload fragment by linearly traversing the bitmap index structure at each level based on the generated bitmap index structure, including:

[0033] The fourth submodule is used to linearly load the bitmap index structure of each level and read it into memory;

[0034] The fifth submodule is used to read the index parameters of the current level configuration;

[0035] The sixth submodule is used to perform intra-level noise reduction search on the query payload fragment;

[0036] The seventh submodule is used to perform precise search within the hierarchy and obtain the precise result set of the payload fragment query within the same hierarchy;

[0037] The eighth submodule is used to use a parallel reduction algorithm to aggregate the precise result sets obtained at each level across levels, obtain the final query and search results of the payload segment to be queried, and feed them back to the user.

[0038] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned network traffic load search method when executing the computer program.

[0039] Compared with the prior art, the present invention achieves the following beneficial technical effects:

[0040] (1) Storage performance optimization: By utilizing the Pareto characteristics of traffic distribution and adopting a hierarchical configuration and compression processing strategy, we can significantly optimize the storage space occupied by the bitmap index structure while effectively ensuring search performance;

[0041] (2) Flexible query methods: The constructed bitmap index structure can perform instant queries on any payload fragment larger than 8 bytes, without the need to reconstruct the index for different query contents;

[0042] (3) Significantly improved noise reduction accuracy: Compared with the load attribution system based on Bloom filters, the noise reduction search link of the present invention can limit the target traffic to a smaller and more accurate set range, significantly reducing the amount of candidate data;

[0043] (4) Retrieval efficiency is greatly improved: A two-stage mechanism of intra-level “noise reduction search” + “precision search” is adopted: first, efficient intra-level noise reduction search is used to quickly eliminate most of the irrelevant data interference and narrow the search scope; then, intra-level precise search is performed on small-scale target sets, which greatly improves the overall retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0045] Figure 1 This is a flowchart of the steps of the network traffic load search method in the first embodiment of the present invention;

[0046] Figure 2 This is a flow example operation diagram of constructing a bitmap index structure in the second embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] In the first embodiment, as Figure 1 As shown, the present invention proposes a network traffic load search method, which mainly includes the following steps:

[0049] Using traffic layering and traffic load segmentation, a bitmap index structure is generated, including the following steps:

[0050] (1) Divide the traffic levels, design differentiated indexing strategies between levels, and configure relevant index parameters for each traffic level.

[0051] Based on the traffic distribution characteristics of data streams (flows) and the bitmap index structure features, the flows to be processed are divided into traffic layers according to their sizes to obtain multiple traffic levels.

[0052] Traffic distribution has a Pareto characteristic: the number of small flows is high but the total space is small, while the opposite is true for large flows; in addition, the bitmap index implements a noise reduction search based on filtering. Based on the above rules, the bitmap index structure should have the following characteristics: for small flows, a stronger filtering effect needs to be achieved, and the storage ratio of its bitmap index structure can be relatively large; for large flows with the same storage scale, since the total number of flows has decreased, the effect of filtering large flows can be weaker than that of small flows, but the storage ratio of its bitmap index structure needs to be further balanced. In summary, based on the balance between filtering effect and index storage space occupancy, the flows to be processed are stratified by size: the first stratum is set according to different size ranges. Tier-level traffic levels , ; is the total number of traffic tiers.

[0053] Secondly, for different traffic levels, we designed a differentiated indexing strategy between these levels to achieve size-driven dynamic precision control of the bitmap index structure between flows at different levels. The basic principles of this differentiated indexing strategy are: low-level levels use full-precision indexing with uninterrupted sampling, while other levels use interval byte sampling and compressed indexing; the length of the block data selected at the low level is smaller than the length of the block data selected at the high level.

[0054] The storage overhead ratio is defined as the ratio of the traffic size to be indexed (raw data size) to the bitmap size (index size). A larger storage overhead ratio indicates a smaller proportion of the space overhead required by the bitmap structure. For a given level of bitmap index structure, a larger block length will require a larger bitmap index structure to index the same number of streams, resulting in an increased storage overhead ratio.

[0055] For each level of stream, the index parameters corresponding to the differentiated configuration include: block length (block data length, in bits) and step value (in bytes). The step value refers to the byte sampling step of the stream.

[0056] (2) Use index parameters to block the traffic load and divide each flow at each level into multiple blocks.

[0057] Each flow contains a variable number of packets. Traffic payload segmentation essentially involves segmenting the payload data of each packet within the flow.

[0058] Find the traffic level corresponding to the flow and obtain the corresponding configuration index parameters: block length and step value.

[0059] The chunking process mainly uses two index parameters: chunk length and step value. Take the stream with sequence number 357 as an example. Figure 2 As shown, stream 357 has a block length of 16 bits (2 bytes) and a step value of 2 bytes, belonging to the second level of the aforementioned hierarchy. During processing, each message segment begins with the first byte of the payload and extracts bits corresponding to the block length as a block. Bytes corresponding to the step value are then skipped, marking the start of the next block.

[0060] The above process is repeated until the payload data of all packets contained in the flow are divided into blocks.

[0061] If the data at the end of the current payload is less than the length of a block, it will be filled with the predetermined content and then treated as a block. After all the messages in the flow are processed in sequence, the flow payload block is completed.

[0062] (3) Filling the hierarchical bitmap index structure, including filling the mapping records of the flow table and constructing the payload fragment addressing matrix to obtain the bitmap index structure; compressing the mapping records of the flow table and the bitmap index structure, and storing them on the SSD hard disk, and generating a bitmap index table.

[0063] The flow table (MapTable) is mainly used to record the mapping relationship between the flow identifiers processed in the current bitmap index structure and their sequence number values. It is essentially an array structure.

[0064] Assume that the level of the current stream F to be processed is , the corresponding stream number in the current bitmap index structure is (For example, the previous flow 357, the flow number is 357), then use the index parameter configuration to obtain the corresponding unique flow identifier, represented by Flow Identifier; Block ( is the chunk number of the stream; It's a flow The total number of blocks) is mapped to Val( ), and then fill in the mapping records of the flow table, and build the payload fragment addressing matrix to obtain the bitmap index structure.

[0065] Specifically, block Mapped to The process is to directly divide the blocks The corresponding binary numerical content is converted into the column value of the bitmap matrix, and the stream sequence number is converted into the row value of the bitmap matrix. The overall process is similar to addressing, as shown in the pseudo code of Table 1.

[0066] Table 1: Pseudo code for filling the flow table mapping record and constructing the payload fragment addressing matrix to obtain the bitmap index structure

[0067]

[0068] Among them, step 1 defines the array structure of the flow table "MapTable" to store the flow sequence number The flow identifier corresponding to the flow; Step 2 defines a set of bitmaps "Bitmapk" of a two-dimensional matrix structure, and the row value of the bitmap matrix is ​​the flow sequence number , the column value is the serial number Chunking of streams The corresponding binary value conversion (similar to fragment position addressing) is obtained, The serial number is In step 3, the above operation is performed on the next stream sequence number until all streams with the same sequence number are traversed.

[0069] Then, the user can pre-set Tier-level configuration storage thresholds , when the memory consumption of the mapping records of the level flow table and its bitmap index structure reaches When the system stores the data on the corresponding SSD (solid-state) hard disk (the same level shares the same SSD hard disk). The data stored on the disk must meet the following requirements:

[0070] Continuous storage: Data at the same level is stored continuously.

[0071] Boundary alignment: All data blocks are aligned on 4KB boundaries to optimize storage performance.

[0072] When storing the mapping records of the flow table, a bitmap index table is generated in real time.

[0073] Bitmap index compression: For bitmap index tables, the Roaring Bitmap sparse compression algorithm is used to effectively reduce storage overhead.

[0074] The Roar bitmap sparse compression algorithm is an efficient sparse bitmap compression algorithm designed specifically for processing large-scale integer sets. Its core advantage is that it dynamically selects the optimal bitmap container to store integer set data, achieving an excellent balance between storage space and computing performance. It is widely used in databases, search engines, big data analysis and other fields.

[0075] After the disk is written, the system will save the The level creates an empty traffic mapping record and bitmap index structure to store the index data of new flows that subsequently enter the level.

[0076] The above steps (1) to (3) belong to the index construction phase of generating a bitmap index structure by using traffic stratification and load data blocking.

[0077] Once the bitmap index structure at all levels is constructed, the system can perform payload queries. Next, we enter the traffic payload query processing phase. Based on the generated bitmap index structure, we perform traffic payload queries by linearly traversing the bitmap index structures at each level. During this process, the user needs to provide a payload segment of at least 8 bytes in length. After the query is completed, the system will return to the user the set of all flow identifiers containing the payload segment. This includes:

[0078] (4) By reading the configuration of index parameters, the bitmap index structure of each level is linearly loaded and read into memory.

[0079] The system starts from level 1 and loads the bitmap index structure linearly in level order.

[0080] Read When indexing at the level:

[0081] First, parallel optimization is performed: based on the available system memory and the size of the current level bitmap index structure, multi-threaded loading is dynamically enabled to improve efficiency;

[0082] Next, compression and restoration are performed: Once the bitmap index structure is loaded into memory, the Roar bitmap decompression algorithm is immediately applied to restore it to a usable in-memory bitmap index. The Roar bitmap algorithm combines intelligent bitmap container selection with on-demand access to avoid the overhead of traditional "full decompression" of compressed bitmaps. The Roar bitmap decompression algorithm essentially operates on efficient local operations on the bitmap container, as shown in the pseudocode in Table 2.

[0083] Table 2: Pseudo code for linearly loading the bitmap index structure at each level and reading it into memory

[0084]

[0085] Step 1 defines the stream processing framework. Steps 2 and 3 define the fixed-size stream table data structure buffer[] and the bitmap indexing mechanism. The row index is the stream sequence number, and the column index is the binary value of the payload segment. Step 4 is a read loop. Steps 5 and 6 read 4KB aligned blocks (partial reads and errors must be handled). Step 7 dynamically generates a two-dimensional bitmap matrix, chunk_bitmap, mapping the 4KB raw data blocks to bitmap segments. Steps 8-10 check the generated chunk_bitmap. Step 11 performs a bitwise OR operation to aggregate the bitmaps. Step 12 releases the associated memory. Steps 14-16 process the hierarchical bitmap index structure, sequentially loading each level of the bitmap index structure.

[0086] (5) By traversing the bitmap indexes at the same level, read the index parameters configured at the current level.

[0087] Read the block length and step value from the index parameters of the current level and process the payload segments to be searched into blocks.

[0088] (6) Block query is performed on the payload fragments to be queried, and noise reduction search is performed within the traffic layer.

[0089] Based on the payload fragment to be queried, each bitmap index structure at the same level is traversed in turn, and a noise reduction search is performed within the traffic set corresponding to each bitmap index structure at the same level.

[0090] Specifically, assuming the current level is Hierarchy , there are bitmap index structure, corresponding to Traffic set, currently processing the Bitmap index structure . Divide the payload fragment into seed sequences based on the block length and step value , for seeds ( ) Query the stream containing the seed in the current bitmap index table and record it in the collection The intersection of the results of various seeds is the final result of the current bitmap index table query, recorded in the set middle:

[0091] ;

[0092] For the rest of the current level The same query is executed on the bitmap index structure. The initial results after index filtering are , , ,…, .make For the current All the results (stream numbers) obtained by index query at the level are:

[0093] .

[0094] The noise reduction search process within the above hierarchy can limit the target traffic to a smaller and more accurate set range, significantly reducing the amount of candidate data.

[0095] (7) Perform precise search within the same level and accurately match the query results to obtain the precise result set of the payload fragment query within the same level.

[0096] Perform a secondary precise search on the results of the noise reduction search, including the above The payload fragment to be queried is converted into a deterministic finite automaton (DFA), and then a secondary filtering search is performed using deep packet inspection (DPI) technology to obtain an accurate result set. .

[0097] (8) Perform cross-level aggregation on the exact result set after exact matching.

[0098] Using a parallel reduction algorithm, the precise result sets obtained at each level are aggregated across levels, merged to obtain the final search results, and fed back to the user. The specific process is shown in the pseudo code in Table 3.

[0099] Table 3: Pseudo code for using parallel reduction algorithm to perform cross-level aggregation to obtain the final search results

[0100]

[0101] Among them, step 1 defines a reduction operation called "Merge" for std::set <flowid>Container type, all elements of the thread-local collection omp_in are merged into the target collection omp_out through the insert method. This design avoids the lock contention problem caused by directly operating shared containers; step 2 declares the global result container global_results as the reduction target; step 3 instructions start a parallel loop, where the key parameter merge:global_results explicitly specifies the use of the previously defined custom reduction method; steps 4 and 5, inside the parallel region, loop through K data partitions (from C[1] to C[K]), and each thread merges the currently processed subset C[k] into the thread-local copy through the merge method. Finally, the local copies of all threads will be automatically merged into the global_results global collection according to the declared Merge reduction rule.

[0102] In the second embodiment of the present invention, when dividing the traffic levels in step (1), the flows to be processed are specifically divided into the following traffic levels according to their sizes:

[0103] Level 1: L1, size range 0-4KB;

[0104] Level 2: L2, size range 4KB to 16KB;

[0105] Level 3: L3, size range 16KB to 64KB;

[0106] Level 4: L4, size range 64KB to 256KB;

[0107] Level 5: L5, size range 256KB~1MB;

[0108] Level 6: L6, size range 1MB to 4MB;

[0109] Level 7: L7, size range 4MB to 16MB;

[0110] Level 8: L8, size range 16MB to 64MB;

[0111] Level 9: L9, size range 64MB and above.

[0112] That is, the total number of traffic levels .

[0113] Furthermore, in step (2), this embodiment configures index parameters according to a differentiated indexing strategy that uses full-precision indexing with no interval sampling at the first level and interval byte sampling with compressed indexing at other levels. Specifically, the index parameters are configured as follows: for the first level, the block length is set to 15 bits, the step value is 1 byte, and the corresponding storage overhead ratio is 1:2; for the second level, the block length is set to 16 bits, the step value is 2 bytes, and the corresponding storage overhead ratio is 1:1; for the third level, the block length is set to 17 bits, the step value is 2 bytes, and the corresponding storage overhead ratio is 2:1; for the fourth level, the block length is set to 19 bits, the step value is 2 bytes, and the corresponding storage overhead ratio is 2:1; for the fifth level, the block length is set to 20 bits, the step value is 2 bytes, and the corresponding storage overhead ratio is 4:1; for the sixth level, the block length is set to 21 bits, the step value is 2 bytes, and the corresponding storage overhead ratio is 8:1; for the seventh level, the block length is set to 23 bits, the step value is 2 bytes, and the corresponding storage overhead ratio is 2:1. The corresponding storage overhead ratio is 8:1. For the 8th level, the block length is set to 25 bits and the step value is 2 bytes, which corresponds to a storage overhead ratio of 8:1. For the 9th level, the block length is set to 27 bits and the step value is 2 bytes, which corresponds to a storage overhead ratio of 8:1. The pseudo code for the above process is shown in Table 4.

[0114] Table 4: Pseudocode example of configuring index parameters using differentiated indexing strategies between tiers

[0115]

[0116] Specifically, in step 1, the "define" function is used to define an array "STORAGE_RATIO" to store the core performance metric: storage overhead ratio. "Flow_Data_Size" represents the size of the traffic to be indexed, and "BitMap_size" represents the bitmap size. In step 2, a data structure "layer_config" is defined to store index parameter configurations for different layers. "Chunk_size" represents the chunk size, and "step" represents the step size.

[0117] The inter-layer differentiated indexing strategy adopted in the above-mentioned process of configuring index parameters sets parameters at layers 1 and 2 to make the storage overhead ratio greater than 1, trading space for search efficiency, and ensuring the storage accuracy of low-level small-size data; at layers 3, 4, and 5, the storage overhead ratio is maintained at 1 / 2 to 1 / 4 through parameter settings, so that the middle layer maintains a balance between storage accuracy and efficiency; at layers 6, 7, 8, and 9, the storage overhead ratio is maintained at 1 / 8 through parameter settings, sacrificing search results in exchange for storage space savings, ensuring that the processing efficiency of high-level large-size data is prioritized and avoiding index expansion.

[0118] In summary, the network traffic payload search methods disclosed in the first and second embodiments utilize traffic stratification and traffic payload segmentation to construct a bitmap index structure. Specifically, they leverage the Pareto characteristic of traffic distribution and employ a hierarchical configuration and compression strategy. While effectively ensuring search performance, they significantly optimize the storage space usage of the bitmap index structure, achieving optimized data storage performance for network traffic payloads. The constructed bitmap index structure enables instant queries for any payload fragments larger than 8 bytes during subsequent searches, eliminating the need to restructure the index for different query content and accommodating flexible query design. Furthermore, compared to Bloom filter-based payload attribution systems, the noise reduction search phase of the present invention can limit target traffic to a smaller, more precise set, significantly reducing the amount of candidate data and significantly improving noise reduction accuracy. A two-stage mechanism of intra-level "noise reduction search" and "precision search" is employed: First, an efficient intra-level noise reduction search is used to quickly eliminate most irrelevant data interference, narrowing the search scope; then, an intra-level precision search is performed on the small target set, significantly improving overall retrieval and query efficiency.

[0119] In a third embodiment, the present invention provides a network traffic load search device, the device comprising:

[0120] The first module is used to generate a bitmap index structure by using traffic layering and traffic load segmentation, including:

[0121] The first submodule is used to stratify the to-be-processed flows by size based on the Pareto characteristics of traffic distribution and the characteristics of the bitmap index structure, obtaining multiple traffic levels. It also designs a differentiated indexing strategy between levels and configures index parameters for each traffic level.

[0122] The second submodule is used to divide the traffic load into blocks using index parameters;

[0123] The third submodule is used to fill the mapping records of the flow table, build the payload segment addressing matrix, and obtain the bitmap index structure; compress the mapping records of the flow table and the bitmap index structure to disk, and generate the bitmap index table;

[0124] The second module is used to perform traffic load query and search for the query payload fragment by linearly traversing the bitmap index structure at each level based on the generated bitmap index structure, including:

[0125] The fourth submodule is used to linearly load the bitmap index structure of each level and read it into memory;

[0126] The fifth submodule is used to read the index parameters of the current level configuration;

[0127] The sixth submodule is used to perform intra-level noise reduction search on the query payload fragment;

[0128] The seventh submodule is used to perform precise search within the hierarchy and obtain the precise result set of the payload fragment query within the same hierarchy;

[0129] The eighth submodule is used to use a parallel reduction algorithm to aggregate the precise result sets obtained at each level across levels, obtain the final query and search results of the payload segment to be queried, and feed them back to the user.

[0130] On the other hand, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the network traffic load search method provided in any of the above embodiments are implemented. The computer device may be a server. The computer device comprises a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with an external terminal via a network connection.

[0131] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the network traffic load search method provided in any of the above embodiments are implemented.

[0132] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0133] Matters not covered by the present invention are known technologies.

[0134] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are all within the scope of protection of the present application.

[0136] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.< / flowid>

Claims

1. A network traffic load search method, characterized in that: include: By using traffic stratification and traffic load segmentation, a bitmap index structure is generated, including: Based on the Pareto characteristics of traffic distribution and the characteristics of the bitmap index structure, the traffic to be processed is stratified by size to obtain multiple traffic levels. A differentiated indexing strategy is designed between the levels, and index parameters are configured for each traffic level. Use index parameters to chunk traffic load; Fill in the mapping records of the flow table and build the payload fragment addressing matrix to obtain the bitmap index structure; compress the mapping records of the flow table and the bitmap index structure and write them to disk to generate the bitmap index table; Based on the generated bitmap index structure, the traffic load query and search are performed on the query payload fragment by linearly traversing the bitmap index structure at each level, including: Linearly load the bitmap index structure of each level and read it into memory; Read the index parameters of the current layer configuration; Perform intra-level noise reduction search on query payload segments; Perform precise searches within a hierarchy to obtain the exact result set for payload fragment queries within the same hierarchy. Using a parallel reduction algorithm, the precise result sets obtained at each level are aggregated across levels to obtain the final query and search results for the payload segment to be queried and feed them back to the user.

2. The network traffic load search method according to claim 1, characterized in that: The Pareto characteristics of the traffic distribution include: a high proportion of small flows but a small total space, and a low proportion of large flows but a large total space; the characteristics of the bitmap index structure include: for small flows, a strong filtering effect needs to be provided, and the corresponding bitmap index structure storage ratio increases; for large flows with the same storage scale as small flows, the filtering effect is weaker than that of small flows due to the decrease in the total number of flows, but the bitmap index structure storage ratio needs to be balanced; According to the Pareto characteristics of traffic distribution and the characteristics of bitmap index structure, the flow to be processed is divided into traffic layers according to size to obtain multiple traffic levels, including: based on the balance between filtering effect and storage ratio of bitmap index structure, the first layer is set according to different size ranges. Tier Traffic Tier , ; is the total number of traffic levels; Design differentiated indexing strategies between tiers and configure indexing parameters for each traffic tier, including: For different traffic levels, we design differentiated indexing strategies between levels: low-level levels use full-precision indexing with non-interval sampling, while other levels use interval byte sampling and compressed indexing; the length of the block data selected by the low-level level is smaller than that of the block data selected by the high-level level; The storage overhead ratio of the bitmap index is defined as the ratio of the traffic size to be indexed to the size of the bitmap index structure; the traffic size to be indexed is the original data size, and the bitmap index structure size is the index size; Index parameters are configured for each traffic level. The index parameters include block length and step value. The block length describes the length of the block data in bits. The step value refers to the byte sampling step of the flow in bytes.

3. The network traffic load search method according to claim 2, characterized in that: Use index parameters to segment traffic payloads, including: Find the traffic level corresponding to the flow and obtain the corresponding configuration index parameters: block length and step value; For each message contained in the stream, starting from the first byte of the payload part, the bit data corresponding to the block length is intercepted as a block; Skip the bytes of the step value as the starting position of the next block, and intercept the bit data corresponding to the block length as the next block; Repeat the above process until the payload data of all packets contained in the flow are divided into blocks; If the data length at the end of the payload of the current message is less than the length of a block, it will be filled with the predetermined content as the last block of the message.

4. The network traffic load search method according to claim 3, characterized in that: The mapping record of the filling flow table is constructed, and a payload segment addressing matrix is ​​constructed to obtain a bitmap index structure; Compress the mapping records and bitmap index structure of the flow table and write them to disk, and generate a bitmap index table, including: Get the corresponding traffic level number of the current processing flow , and the stream sequence number corresponding to the current processing stream in the bitmap index structure ; Use the The index parameter of the traffic level configuration of the level obtains the corresponding unique flow identifier, which is represented by FlowIdentifier; will flow Block Mapping to payload fragment addressing matrix , ; It's a flow The total number of blocks; the mapping process includes: Convert the stream sequence number into the row value of the bitmap matrix; Directly divide the blocks The corresponding binary numerical content is converted into column values ​​of the bitmap matrix; Pre-order Tier-level traffic tier configuration storage threshold ; The Roar bitmap sparse compression algorithm is used to compress the mapping records and bitmap index structure of the flow table; When the storage consumption in memory reaches When The mapping records and bitmap index structures of the flow tables in the hierarchy are written to the corresponding SSD hard disks. This process satisfies the following requirements: The same level shares the same SSD hard drive; Data at the same level are stored continuously; To optimize storage performance, all data blocks are aligned on 4KB boundaries; After the disk storage is completed, the system stores the At the level, a new empty flow table mapping record and bitmap index structure are created to store the subsequent entry into the Index data of the new stream of the level; When storing the mapping records of the flow table, a bitmap index table is generated in real time.

5. The network traffic load search method according to claim 4, characterized in that: The linear loading of the bitmap index structure at each level and reading it into memory includes: The system starts from level 1 and loads the bitmap index structure linearly in level order; Load and read When using a hierarchical bitmap index structure, the following operating rules are implemented: First, perform parallel optimization: dynamically enable multi-threaded loading based on the system's available memory and the size of the current hierarchical bitmap index structure; Then perform compression and restoration: After the bitmap index structure is loaded into memory, the roar bitmap decompression algorithm is immediately applied to restore the loaded bitmap index structure to an operational memory bitmap index.

6. The network traffic load search method according to claim 5, characterized in that: The intra-level noise reduction search for the query payload segment includes: Set up the first The traffic level of the level is bitmap index structure, corresponding to Traffic set, currently processing the A bitmap index structure, ; Based on the The block length and step value configured by the traffic level of the level will The payload segments of the bitmap index structure are divided into seed sequences ; Targeting seeds , , query the stream containing the seed in the current bitmap index table and record it in the result set middle; Take the intersection of all seed result sets to get the final result of the current bitmap index table query and record it in the set middle: ； For the current The same query is performed on all bitmap index structures within the hierarchy. After filtering by index, the query results of noise reduction search within the hierarchy are initially obtained for each bitmap index structure: , , ,…, ; Use each of the above bitmap index structures to reduce the noise of the search query results within the hierarchy and obtain the first Query results for noise reduction search within the traffic layer: 。 7. The network traffic load search method according to claim 6, characterized in that: The precise search within the hierarchy is performed to obtain a precise result set for the payload fragment query within the same hierarchy, including: Use the The query results of the denoising search within the hierarchical traffic level, convert the payload fragment to be queried into a deterministic finite automaton; Use deep packet inspection technology to perform secondary filtering search and obtain the Accurate result set within the traffic level of the level .

8. The network traffic load search method according to claim 7, characterized in that: According to the Pareto characteristics of the traffic distribution and the bitmap index structure characteristics, the flows to be processed are stratified by size: Level 1: L1, size range 0-4KB; Level 2: L2, size range 4KB to 16KB; Level 3: L3, size range 16KB to 64KB; Level 4: L4, size range 64KB to 256KB; Level 5: L5, size range 256KB~1MB; Level 6: L6, size range 1MB to 4MB; Level 7: L7, size range 4MB to 16MB; Level 8: L8, size range 16MB to 64MB; Level 9: L9, size range 64MB and above; Furthermore, configure index parameters according to the differentiated indexing strategy: For level 1, set the block length to 15 bits, the step value to 1 byte, and the storage overhead ratio to 1:2; For the second level, set the block length to 16 bits, the step value to 2 bytes, and the storage overhead ratio to 1:1; For level 3, set the block length to 17 bits, the step value to 2 bytes, and the storage overhead ratio to 2:1; For level 4, set the block length to 19 bits, the step value to 2 bytes, and the storage overhead ratio to 2:1; For level 5, set the block length to 20 bits, the step value to 2 bytes, and the storage overhead ratio to 4:1; For level 6, set the block length to 21 bits, the step value to 2 bytes, and the storage overhead ratio to 8:1; For level 7, set the block length to 23 bits, the step value to 2 bytes, and the storage overhead ratio to 8:1; For level 8, set the block length to 25 bits, the step value to 2 bytes, and the storage overhead ratio to 8:1; For level 9, the block length is set to 27 bits, the step value is set to 2 bytes, and the storage overhead ratio is set to 8:

1.

9. A network traffic load search device, characterized in that: include: The first module is used to generate a bitmap index structure by using traffic layering and traffic load segmentation, including: The first submodule is used to stratify the to-be-processed flows by size based on the Pareto characteristics of traffic distribution and the characteristics of the bitmap index structure, obtaining multiple traffic levels. It also designs a differentiated indexing strategy between levels and configures index parameters for each traffic level. The second submodule is used to divide the traffic load into blocks using index parameters; The third submodule is used to fill the mapping records of the flow table, build the payload segment addressing matrix, and obtain the bitmap index structure; compress the mapping records of the flow table and the bitmap index structure to disk, and generate the bitmap index table; The second module is used to perform traffic load query and search for the query payload fragment by linearly traversing the bitmap index structure at each level based on the generated bitmap index structure, including: The fourth submodule is used to linearly load the bitmap index structure of each level and read it into memory; The fifth submodule is used to read the index parameters of the current level configuration; The sixth submodule is used to perform intra-level noise reduction search on the query payload fragment; The seventh submodule is used to perform precise search within the hierarchy and obtain the precise result set of the payload fragment query within the same hierarchy; The eighth submodule is used to use a parallel reduction algorithm to aggregate the precise result sets obtained at each level across levels, obtain the final query and search results of the payload segment to be queried, and feed them back to the user.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images