Data packet storage query method and device based on network segment index in network environment

Abstract

The present invention discloses a data packet storage query method and device based on network segment index in a network environment, belonging to the technical field of data packet detection. The data packet storage query method comprises: creating an IP data packet index based on multi-threaded parallelism; merging IP data packet indexes belonging to the same time and the same network segment to form a network segment index; serializing the network segment indexes at the same time to form a network segment index file block; writing the network segment index file block to a disk, and recording the offset of the network segment index file block in a memory. When performing an IP data packet query, the present invention only needs one query to locate an index file block at a time point, thereby improving the speed of querying IP data packets and saving a large amount of CPU and IO resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of data packet detection, and in particular relates to a data packet storage query method and device based on network segment index in a network environment. Background Art

[0002] In the field of network traffic analysis, network data packets are stored and retrieved from different dimensions, such as network segments, IP sessions, TCP sessions, etc. In order to quickly retrieve network data packets, an index is usually created for each IP data packet.

[0003] The existing technical solution is as follows: create an index for each IP according to the time point, record the offset of the IP and the data packet of the IP in the data packet storage file, and store the index in the index file through a certain strategy. When querying the network segment data packet, query the index file one by one for all IPs belonging to the query network segment to obtain the data offset of the data packet in the data packet file, and then read the original data packet of the IP in the data packet file according to the data offset.

[0004] The existing technical solution is slow in searching and querying network segment data packets and seriously wastes CPU and IO resources: for network segment search, all IPs belonging to the network segment need to be indexed and queried. The index file block data read is large and has high discreteness, resulting in large consumption of IO resources. At the same time, different IPs may locate the same index file block, and the same index file block may be read multiple times, which will also seriously waste IO resources. At the same time, a large number of IP query index operations will also cause the CPU load to be too high. In addition, after querying a single IP data packet index and obtaining the data packet file offset, the original data packet file is read. Different IPs may also locate the same data packet original file data block, which causes the IP data packet original file block to be read multiple times, with a lot of repeated IO and extremely low IO utilization. At the same time, indexing and locating all IPs in the same network segment itself also consumes a lot of CPU and IO resources, which will cause the query efficiency of the entire network segment data packet to be very slow. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method and device for storing and querying data packets based on network segment indexes in a network environment.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] The first aspect of the present invention discloses a data packet storage query method based on a network segment index in a network environment, comprising:

[0008] Create IP packet index based on multi-threaded parallelism;

[0009] Merge the IP data packet indexes belonging to the same time and the same network segment to form a network segment index;

[0010] Serialize the network segment indexes at the same time to form a network segment index file block;

[0011] The network segment index file block is written to the disk, and the offset of the network segment index file block is recorded in the memory.

[0012] Furthermore, the IP data packet index includes an IP address of the IP data packet, a timestamp of the IP data packet, an offset of the IP data packet in the data packet storage file, and a length of the IP data packet in the data packet storage file.

[0013] Furthermore, the network segment index includes the network segment number, the timestamp corresponding to the network segment number, the number of minimum index units corresponding to each timestamp and the minimum index unit corresponding to each timestamp, and the minimum index unit includes the offset of the IP data packet in the data packet storage file and the length of the IP data packet in the data packet storage file.

[0014] Furthermore, the network segment index file block includes the number of bytes of the network segment index file block, the number of network segment indexes in the network segment index file block, the number of each network segment index in the network segment index file block, the relative offset of each network segment index in the network segment index file block in the network segment index file block, and the content of each network segment index in the network segment index file block.

[0015] Furthermore, the data packet storage query method also includes:

[0016] Get the network segment and time range to be queried;

[0017] Determine whether there is network segment index information within the time range in the memory, and if there is network segment index information within the time range in the memory, read the network segment index file block of the network segment within the time range;

[0018] The offset of the IP data packet in the data packet storage file is obtained based on the network segment index file block, and the IP data packet is read and returned according to the offset of the IP data packet in the data packet storage file.

[0019] Furthermore, if the network segment index information of the time range does not exist in the memory, empty data is returned.

[0020] Further, when reading the network segment index file blocks of the network segment within the time range, a network segment index file block of a preset size is read each time.

[0021] The second aspect of the present invention discloses a data packet storage query device based on a network segment index in a network environment, characterized in that it includes:

[0022] Index creation module, used to create IP data packet index based on multi-threaded parallelism;

[0023] The network segment index creation module is used to merge the IP data packet indexes belonging to the same network segment at the same time to form a network segment index;

[0024] A file block creation module is used to serialize the network segment indexes at the same time to form a network segment index file block;

[0025] The file block storage module is used to store the network segment index file block on the disk and record the offset of the network segment index file block in the memory.

[0026] Furthermore, the data packet storage query device also includes:

[0027] The query condition acquisition module is used to obtain the network segment and time range to be queried;

[0028] A file block reading module, used for judging whether there is network segment index information within the time range in the memory, and if there is network segment index information within the time range in the memory, reading the network segment index file block of the network segment within the time range;

[0029] The data packet query module is used to obtain the offset of the IP data packet in the data packet storage file based on the network segment index file block, and read and return the IP data packet according to the offset of the IP data packet in the data packet storage file.

[0030] The beneficial effects of the present invention are as follows: when performing an IP data packet query, the present invention only needs one query to locate an index file block at a time point, thereby improving the speed of querying IP data packets and saving a large amount of CPU and IO resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a flow chart of a data packet storage query method in the present invention;

[0032] Figure 2 A schematic diagram for creating an IP packet index based on multi-threading;

[0033] Figure 3 A schematic diagram for performing index merging of IP data packets;

[0034] Figure 4 A schematic diagram of a structure of a network segment index file block;

[0035] Figure 5 A schematic diagram of the layout of the network segment index file block in the memory;

[0036] Figure 6It is another flow chart of the data packet storage query method in the present invention;

[0037] Figure 7 It is a query logic diagram based on network segments in the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0039] See also Figures 1 to 7 The present invention provides a method and device for storing and querying data packets based on network segment index in a network environment:

[0040] The first aspect of this embodiment discloses a data packet storage query method based on a network segment index in a network environment. Figure 1 As shown, the data packet storage query method includes steps S100 to S400.

[0041] Step S100: Creating an IP data packet index based on multi-threaded parallelism.

[0042] Specifically, a multi-threaded approach is used to create IP data packet indexes in parallel.

[0043] In order to speed up the query of data packets, the present embodiment uses a multi-threaded method to record the index of the storage location of each IP data packet. Using multi-threads to process network traffic can give full play to the advantages of multi-core and improve processing efficiency.

[0044] The IP data packet index includes the IP address of the IP data packet, the timestamp of the IP data packet, the offset of the IP data packet in the data packet storage file, and the length of the IP data packet in the data packet storage file.

[0045] Figure 2 The index of IP packets created by thread 1 (Thread1) and thread 2 (Thread2) at time T (such as time T1, time T2, and time T3), where time T represents a time range, which can be 1 second, 10 seconds, 1 minute, etc. The first column represents the IP address of the IP packet (such as the source address or destination address of the IP packet), the second column represents the timestamp of the IP packet, the third column represents the offset of the IP packet in the packet storage file, and the fourth column represents the length of the IP packet in the packet storage file.

[0046] For example, Figure 2Taking the IP data packet index at time T1 in thread 1 as an example, IP1, timestamp1, offset1 and datalen1 in the first row respectively represent the IP address of the IP data packet, the timestamp of the IP data packet corresponding to IP1, the offset of the IP data packet corresponding to IP1 in the data packet storage file, and the length of the IP data packet corresponding to IP1 in the data packet storage file. IP2, timestamp2, offset2 and datalen2 in the second row respectively represent the IP address of the IP data packet, the timestamp of the IP data packet corresponding to IP2, the offset of the IP data packet corresponding to IP2 in the data packet storage file, and the length of the IP data packet corresponding to IP2 in the data packet storage file. IP3, timestamp3, offset3 and datalen3 in the third row respectively represent the IP address of the IP data packet, the timestamp of the IP data packet corresponding to IP3, the offset of the IP data packet corresponding to IP3 in the data packet storage file, and the length of the IP data packet corresponding to IP3 in the data packet storage file. IP1, timestamp3, offset4 and datalen4 in the third row respectively represent the IP address of the IP data packet, the timestamp of the IP data packet corresponding to IP4, the offset of the IP data packet corresponding to IP4 in the data packet storage file, and the length of the IP data packet corresponding to IP4 in the data packet storage file.

[0047] Step S200: merge the IP data packet indexes belonging to the same time and the same network segment to form a network segment index.

[0048] Specifically, for the IP data packet indexes created by each thread, the IP data packet indexes belonging to the same time and the same network segment are merged to form a network segment index.

[0049] In this embodiment, the IP data packet indexes at different times in each thread are merged.

[0050] The network segment index includes the network segment number, the timestamp corresponding to the network segment number, the number of minimum index units corresponding to each timestamp and the minimum index unit corresponding to each timestamp. The minimum index unit includes the offset of the IP data packet in the data packet storage file and the length of the IP data packet in the data packet storage file.

[0051] like Figure 3As shown, first, the IP packet index of each thread is aligned according to time T. In the figure, IP1, IP4, and IP7 belong to network segment 1 (netsegment1); IP2, IP5, and IP6 belong to network segment 2 (netsegment2); and IP3, IP6, IP9, and IP10 belong to network segment 3 (netsegment3). For the same time T1 in different threads, a multi-way merging strategy is adopted to merge different IP packet indexes belonging to the same network segment to form a network segment index.

[0052] The network segment index organization format of a time point of each network segment is arranged in ascending order of timestamps. For example, because timestamp2 > timestamp1, the IP data packet index corresponding to timestamp2 is arranged behind the IP data packet index corresponding to timestamp1.

[0053] by Figure 3 Taking the organizational format of the network segment netsegment1 at timestamp1 as an example, IP1, IP4, and IP7 belong to network segment netsegment1. The offset of the IP data packet of IP1 at timestamp1 is offset1, and the length of the IP data packet is datalen1; the offset of the IP data packet of IP4 at timestamp1 is offset5, and the length of the IP data packet is datalen5; the offset of the IP data packet of IP7 at timestamp1 is offset9, and the length of the IP data packet is datalen9. Arrange the offsets and lengths of the IP data packets corresponding to IP1, IP4, and IP7, and take (offset, datalen) as a minimum index unit. Record the number of minimum index units in the "count" field of the third column of the network segment index one (index-1). It is easy to get that "count" here is equal to 3.

[0054] Step S300: Serialize the network segment indexes at the same time to form a network segment index file block.

[0055] The network segment index file block includes the number of bytes of the network segment index file block, the number of network segment indexes in the network segment index file block, the number of each network segment index in the network segment index file block, the relative offset of each network segment index in the network segment index file block in the network segment index file block, and the content of each network segment index in the network segment index file block (such as the timestamp of the IP data packet, the offset of the IP data packet in the data packet storage file, and the length of the IP data packet in the data packet storage file, etc.).

[0056] like Figure 4As shown, multiple network segment indexes are formed at time T1. When the space occupied by the network segment index in the memory is greater than a preset value (such as 1M), the network segment index in the memory is serialized to form a network segment index file block.

[0057] Figure 4 In the network segment index file block shown, "totalbytes" represents the number of bytes of the network segment index file block, "net-gement-num" represents the number of network segment indexes in the network segment index file block, "net-segment-1" represents the network segment index net-segment-1, and "fileoffset-1" represents the relative offset of the network segment index net-segment-1 in the network segment index file block.

[0058] Step S400: Write the network segment index file block to the disk, and record the offset of the network segment index file block in the memory.

[0059] Figure 5 Shows the layout of the network segment index file blocks in memory.

[0060] In some embodiments, Figure 6 As shown, the data packet storage query method also includes steps S500 to S700.

[0061] Step S500: Obtain the network segment and time range to be queried.

[0062] Step S600: Determine whether there is network segment index information within the time range in the memory. If there is network segment index information within the time range in the memory, read the network segment index file block within the time range.

[0063] For example, the network segment index file block of the network segment in the time range is read based on the offset of the network segment index file block.

[0064] In some embodiments, if the network segment index information of the time range does not exist in the memory, empty data is returned.

[0065] Step S700. Obtain the offset of the IP data packet in the data packet storage file based on the network segment index file block, and read and return the IP data packet according to the offset of the IP data packet in the data packet storage file.

[0066] In some embodiments, when reading the network segment index file blocks of the network segment in the time range, network segment index file blocks of a preset size are read each time.

[0067] In the method of this embodiment, for the index file block information obtained by the query, the file offset of the IP data packet is aggregated according to a preset size, and the IP data packet file is read. For the IP data packet file block within the preset size, it is read only once, which greatly saves the IO resources consumed by reading the original IP data packet file, and greatly improves the speed of querying the network segment data packet.

[0068] For example, Figure 7 As shown, for all IP data packets in netsegment1 at time T1-T2, the index file offsets indexfile-offset-1 and indexfile-offset-2 of netsegment1 at time T1 and T2 will be queried. The index file is read according to the index file offsets indexfile-offset-1 and indexfile-offset-2 to obtain the indexes (offset and length) of all IP data packets in the data packet storage file. All IP data packet indexes in netsegment1 are sorted in ascending order according to offset. Then, starting from the smallest offset offset1, the data blocks are read in order of 4M data blocks (or data blocks of other preset sizes), and the hit IP data packets are parsed according to offset and datalen.

[0069] In order to improve IO efficiency, 4M data packet file blocks will be read at one time. Figure 7 In the example, the two offsets offset1 and offset2 of netsegment1 are in a 4M data packet file block, and the data of the IP data packet is parsed out according to offset and datalen. Similarly, the four offsets offset3, offset4, offset5 and offset6 of netsegment1 are in a 4M data packet file block, so four IP data packets can be parsed out at one time according to the datalen of each IP data packet.

[0070] Compared with the existing technical solution of querying index file blocks for all IPs in a network segment, the method of this embodiment merges different network segments in memory at each time point, aggregates the IP data packet indexes belonging to the same network segment, and writes the index files to the disk at one time point. When performing a network segment query, for data at a time point, only one query needs to be executed to obtain the index file block information, thereby saving several orders of magnitude of query operations and saving a large amount of CPU, memory and IO resources.

[0071] A second aspect of this embodiment discloses a data packet storage query device based on a network segment index in a network environment, including an index creation module, a network segment index creation module, a file block creation module and a file block storage module.

[0072] The index creation module is used to create IP data packet indexes based on multi-threaded parallelism.

[0073] The network segment index creation module is used to merge the IP data packet indexes belonging to the same network segment at the same time to form a network segment index.

[0074] The file block creation module is used to serialize the network segment indexes at the same time to form a network segment index file block.

[0075] The file block storage module is used to store the network segment index file block on the disk and record the offset of the network segment index file block in the memory.

[0076] The data packet storage query device also includes a query condition acquisition module, a file block reading module and a data packet query module.

[0077] The query condition acquisition module is used to obtain the network segment and time range to be queried.

[0078] The file block reading module is used to determine whether there is network segment index information within the time range in the memory. If there is network segment index information within the time range in the memory, the network segment index file block of the network segment within the time range is read.

[0079] The data packet query module is used to obtain the offset of the IP data packet in the data packet storage file based on the network segment index file block, and read and return the IP data packet according to the offset of the IP data packet in the data packet storage file.

[0080] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system or device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0081] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A data packet storage query method based on network segment index in a network environment, characterized in that: include: Create IP packet index based on multi-threaded parallelism; Merge the IP data packet indexes belonging to the same time and the same network segment to form a network segment index; Serialize the network segment indexes at the same time to form a network segment index file block; The network segment index file block is written to a disk, and the offset of the network segment index file block is recorded in a memory; The network segment index includes a network segment number, a timestamp corresponding to the network segment number, the number of minimum index units corresponding to each timestamp, and a minimum index unit corresponding to each timestamp, wherein the minimum index unit includes an offset of an IP data packet in a data packet storage file and a length of an IP data packet in a data packet storage file; The network segment index file block includes the number of bytes of the network segment index file block, the number of network segment indexes in the network segment index file block, the number of each network segment index in the network segment index file block, the relative offset of each network segment index in the network segment index file block in the network segment index file block and the content of each network segment index in the network segment index file block.

2. The data packet storage query method based on network segment index in a network environment according to claim 1, characterized in that: The IP data packet index includes the IP address of the IP data packet, the timestamp of the IP data packet, the offset of the IP data packet in the data packet storage file, and the length of the IP data packet in the data packet storage file.

3. The data packet storage query method based on network segment index in a network environment according to claim 1, characterized in that: The data packet storage query method also includes: Get the network segment and time range to be queried; Determine whether there is network segment index information within the time range in the memory, and if there is network segment index information within the time range in the memory, read the network segment index file block of the network segment within the time range; The offset of the IP data packet in the data packet storage file is obtained based on the network segment index file block, and the IP data packet is read and returned according to the offset of the IP data packet in the data packet storage file.

4. The method for storing and querying data packets based on network segment index in a network environment according to claim 3, characterized in that: If the network segment index information of the time range does not exist in the memory, empty data is returned.

5. The method for storing and querying data packets based on network segment index in a network environment according to claim 3, characterized in that: When reading the network segment index file blocks of the network segment within the time range, a network segment index file block of a preset size is read each time.

6. A data packet storage query device based on network segment index in a network environment, characterized in that: include: Index creation module, used to create IP data packet index based on multi-threaded parallelism; A network segment index creation module is used to merge the IP data packet indexes belonging to the same network segment at the same time to form a network segment index; A file block creation module is used to serialize the network segment indexes at the same time to form a network segment index file block; A file block storage module, used for storing the network segment index file block on the disk and recording the offset of the network segment index file block in the memory; The network segment index includes a network segment number, a timestamp corresponding to the network segment number, the number of minimum index units corresponding to each timestamp, and a minimum index unit corresponding to each timestamp, wherein the minimum index unit includes an offset of an IP data packet in a data packet storage file and a length of an IP data packet in a data packet storage file; The network segment index file block includes the number of bytes of the network segment index file block, the number of network segment indexes in the network segment index file block, the number of each network segment index in the network segment index file block, the relative offset of each network segment index in the network segment index file block in the network segment index file block and the content of each network segment index in the network segment index file block.

7. The data packet storage query device based on network segment index in a network environment according to claim 6, characterized in that: The data packet storage query device also includes: The query condition acquisition module is used to obtain the network segment and time range to be queried; A file block reading module, used for judging whether there is network segment index information within the time range in the memory, and if there is network segment index information within the time range in the memory, reading the network segment index file block of the network segment within the time range; The data packet query module is used to obtain the offset of the IP data packet in the data packet storage file based on the network segment index file block, and read and return the IP data packet according to the offset of the IP data packet in the data packet storage file.

Images