Method, device, equipment and medium for quickly querying network session data packets

Abstract

The present invention discloses a method, apparatus, device, and medium for quickly querying network session data packets. The method includes the following steps: storing a data stream in a data stream table; caching the data stream in a data packet index table; performing a primary feature index in the data stream table based on preset search conditions to generate a primary feature index result; further performing a secondary feature index in the data packet index table based on the primary feature index result to generate a secondary feature index result; and writing the secondary feature index result into a result file as a query result. The method of the present invention utilizes multi-level feature index storage for data, fully integrating data packets, rapidly locating data packets based on feature queries, and extracting tens, hundreds, or even thousands of data packets with only four or fewer file data segment jumps, effectively improving query speed and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for quickly querying network session data packets. Background Art

[0002] With the continuous development of the information age, network traffic analysis is of great significance for the optimization of network resources and network security. However, due to the increasing network data traffic, the storage scale of data packets is also increasing. Retrieving network session data packets from massive data packet files is a difficult task. In the existing technology, data packets are usually stored in chronological order, resulting in the cross-saving of data packets of concurrent data connections. When performing queries based on the above situation, there are usually many problems. For example, the search requires frequent jumps in the file offset to read discontinuous data segments, the large search volume cannot accurately retrieve the required files, and the search consumes a lot of resources and time. The low efficiency of retrieving the original data packets of network sessions leads to a poor user experience.

[0003] Therefore, there is still a need in the prior art for improving the method for quickly querying network session data packets. Summary of the Invention

[0004] In view of this, the purpose of the embodiments of the present invention is to propose a method, device, equipment and medium for quickly querying network session data packets. The method of the present invention establishes a multi-level index feature storage, fully integrates the data packets, and effectively improves the query efficiency.

[0005] Based on the above purpose, an embodiment of the present invention provides a method for quickly querying network session data packets, including the following steps:

[0006] Store the data stream in the data stream table;

[0007] Cache the data stream into a data packet index table;

[0008] Performing a first-level feature index in the data stream table based on a preset search condition to generate a first-level feature index result;

[0009] Further performing secondary feature indexing in the data packet index table based on the primary feature indexing result to generate a secondary feature indexing result;

[0010] The secondary feature index result is written into a result file as a query result.

[0011] In some implementations, the data flow table includes: data flow ID, start time, end time, source IP, source port, destination IP, destination port, IP protocol type, and total number of bytes.

[0012] In some implementations, the data packet index table includes: a data stream ID, a data packet file name, a storage start position, and a storage end position.

[0013] In some implementations, caching the data stream into a data packet index table includes:

[0014] The short connection data streams in the data stream are merged, and the merged short connection data streams are cached in a data packet index table.

[0015] In some implementations, caching the data stream into a data packet index table includes:

[0016] Segmenting the long connection data stream in the data stream, and caching the segmented long connection data streams into a data packet index table respectively;

[0017] When the number of cache segments exceeds a cache segment number threshold, the portion of the long connection data stream that exceeds the cache segment number threshold is cached into an independent data packet.

[0018] In some implementations, the method further includes: the search condition is quintuple data, and data query is performed based on the quintuple data.

[0019] In some embodiments, further performing a secondary feature index in the data packet index table based on the primary feature index result to generate a secondary feature index result includes:

[0020] Based on the primary feature indexing result, a secondary feature index is further performed in the data packet index table to obtain multiple index items, and the index items are merged to generate a secondary feature indexing result.

[0021] In some embodiments, comprising:

[0022] a first index building module configured to store the data stream in a data stream table;

[0023] a second index building module, the second index building module being configured to cache the data stream into a data packet index table;

[0024] a first indexing module, configured to perform a first-level feature index in the data stream table based on a preset search condition to generate a first-level feature index result;

[0025] A second indexing module is configured to further perform a secondary feature index in the data packet index table based on the primary feature indexing result to generate a secondary feature indexing result.

[0026] A result output module is configured to write the secondary feature index result to a result file as a query result. Another aspect of an embodiment of the present invention further provides a computer device comprising: at least one processor; and a memory storing computer instructions executable on the processor, the instructions implementing the steps of the above method when executed by the processor.

[0027] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores a computer program that implements the above method steps when executed by a processor.

[0028] The present invention has at least the following beneficial technical effects:

[0029] The present invention provides a method for quickly querying network session data packets. The method of the present invention establishes primary index storage and secondary index storage for data streams, fully merges data streams, and integrates data packets. During the query process, the primary and secondary indexes are used to quickly locate the position of the data packets, and tens of thousands, hundreds of thousands, or even more data packets can be extracted by only jumping through no more than four file data segments. In addition, the entire segment of data is read during the query process rather than reading scattered data packets, thereby avoiding the verification of filtering conditions such as the IP address and port of each data packet, and effectively improving the query speed and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of an embodiment of a method for quickly querying network session data packets provided by the present invention;

[0032] Figure 2 A schematic diagram of an embodiment of a data flow table provided by the present invention;

[0033] Figure 3 A schematic diagram of an embodiment of a data packet index table provided by the present invention;

[0034] Figure 4 A schematic diagram of an embodiment of a data packet storage structure provided by the present invention;

[0035] Figure 5 A schematic diagram of an embodiment of a feature index query provided by the present invention;

[0036] Figure 6A schematic diagram of an embodiment of a network session data packet fast query device provided by the present invention;

[0037] Figure 7 A schematic diagram of an embodiment of a computer device for executing a method for quickly querying network session data packets provided by the present invention;

[0038] Figure 8 A schematic diagram of an embodiment of a computer-readable storage medium for implementing a method for rapidly querying network session data packets provided by the present invention. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0040] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. Subsequent embodiments will not explain this one by one.

[0041] Based on the above objectives, a first aspect of an embodiment of the present invention provides an embodiment of a method for quickly querying network session data packets. Figure 1 FIG. 1 is a schematic diagram of an embodiment of a method for quickly querying network session data packets provided by the present invention. Figure 1 As shown, the method for quickly querying network session data packets according to an embodiment of the present invention includes the following steps:

[0042] S1. Store the data stream in the data stream table;

[0043] S2. caching the data stream into a data packet index table;

[0044] S3. Performing a first-level feature index in the data stream table based on a preset search condition to generate a first-level feature index result;

[0045] S4. Further perform secondary feature indexing in the data packet index table based on the primary feature indexing result to generate a secondary feature indexing result;

[0046] S5. Write the secondary feature index result into a result file as the query result.

[0047] In some embodiments of the present invention, the purpose of S1 storing the data stream in the data stream table is to establish a first-level feature index, wherein the data stream table is as follows: Figure 2As shown in FIG, the data flow table includes information such as data flow ID (Flow ID), start time (StartTime), end time (End Time), source IP (Src IP), source port (Src Port), destination IP (Dst IP), destination port (Dst Port), IP protocol type (IP Protocol), and total number of bytes (Total Bytes). The data flow is stored in the data flow table based on the above information to establish a primary feature index.

[0048] like Figure 3 The diagram shows an embodiment of a packet index table provided by the present invention, namely, the packet index table in S2. This packet index table includes a data flow ID (Flow ID), a packet file name (File Name), a storage start offset (Start Offset), and a storage end offset (End Offset). This table records the file name and location within the file where the packets for each data flow are stored. The packet file is used to store raw packet information. To ensure that the packets of a data flow are stored as continuously as possible, each data flow writes the packet file all at once after caching a certain amount of packets.

[0049] Furthermore, the present invention proposes short connection data packets and long connection data packets, wherein data packets whose cache segments in the data packet file are within the cache segment number threshold are marked as short connection data packets, while data packets whose cache segment number exceeds the threshold are marked as long connection data packets, and the data packets exceeding the threshold are cached into independent data packets. In some embodiments of the present invention, the cache segment number threshold is set to 3. It should be understood by those skilled in the art that the threshold can be flexibly set according to actual application scenarios, such as Figure 3 and Figure 4 As shown, Flow ID = 3 is stored in the data packet file 1644977640-103.pcap. The number of cached segments in this data packet file exceeds 3. During the fourth cache, the storage start offset (StartOffset) is set to 0 and the storage end offset (EndOffset) is set to -1, indicating that the data packet is infinite. The portion of the data packet exceeding the threshold is stored in a separate data packet file. Because long connection data packets often have long time spans and a large total number of bytes, storing the portion exceeding the cache threshold separately helps improve retrieval efficiency.

[0050] The method of the present invention establishes a primary feature index storage and a secondary feature index storage for the data packets of the network session data stream. During the storage process, the data packets of the same data stream are saved as continuously as possible, and short connection data packets and long connection data packets are distinguished. The short connection data packets are merged and saved, and the long connection data packets are saved independently, so as to speed up the indexing speed during query.

[0051] like Figure 5 A schematic diagram of an embodiment of a feature index query provided by the present invention is provided. First, the search conditions are set. In some embodiments of the present invention, the search conditions are five-tuple data of the network session connection, namely, the source IP address, source port, destination port, destination IP address and transport layer protocol. For example, by entering the source IP address as 32.76.1.5, the source port as 43994, the destination port as 6, the destination IP address as 22.66.1.112 and the transport layer protocol as 80, the data query can be started.

[0052] Further, S3 is executed to perform a first-level feature index in the data stream table based on the search conditions to generate a first-level feature index result, such as Figure 5 As shown, based on the search conditions in the data flow table corresponding to the information in the table, the first-level feature index result is quickly located and generated as Flow ID=3.

[0053] Furthermore, S4 is executed to further perform secondary feature indexing in the data packet index table based on the primary feature indexing result to generate a secondary feature indexing result, such as Figure 5 As shown, based on the primary feature index result Flow ID = 3, a secondary feature index is performed in the data packet index table. The index obtains that Flow ID = 3 has three segments cached in the data packet file 1644977640-103.pcap, and further indexes the remaining data packets of Flow ID = 3 cached in the data packet file 1644977640-flow-3.pcap. When the index obtains multiple index items, the index items are merged to generate a secondary feature index result, that is, the results of all data packets of Flow ID = 3 are merged. The method of the present invention marks Flow ID = 3 as a long connection data packet during the storage phase, and saves the portion exceeding the cache segment number threshold in a separate file. This allows tens of thousands, hundreds of thousands, or even more data packets to be extracted with only no more than four file data segment jumps.

[0054] Furthermore, S5 is executed to write the secondary feature index result into the result file as the query result, and the four data packets of the secondary feature index result Flow ID=3 are merged and written into the Result-pkts.pcap result file, and the query result is output.

[0055] The present invention stores data packets through secondary indexes and a method of merging data streams. During the data packet query stage, the secondary index is used to quickly locate the position of the data packet data segment, and then continuous data packets are read in the data segments that have been merged according to the session connection, thereby greatly improving the query efficiency.

[0056] It should be pointed out in particular that the various steps in the various embodiments of the above-mentioned network session data packet rapid query method can be cross-linked, replaced, added, and deleted with each other. Therefore, these reasonable permutations, combinations, and transformations of the network session data packet rapid query method should also fall within the scope of protection of the present invention, and the scope of protection of the present invention should not be limited to the embodiments.

[0057] Based on the above purpose, a second aspect of an embodiment of the present invention provides a device for quickly querying network session data packets. Figure 6 The diagram shows an embodiment of a network session data packet fast query device provided by the present invention. Figure 6 As shown, the network session data packet fast query device of the embodiment of the present invention includes the following modules: a first index construction module 011, which is configured to store the data stream in the data stream table; a second index construction module 012, which is configured to cache the data stream in the data packet index table; a first index module 013, which is configured to perform a first-level feature index in the data stream table based on a preset search condition to generate a first-level feature index result; a second index module 014, which is configured to further perform a second-level feature index in the data packet index table based on the first-level feature index result to generate a second-level feature index result. A result output module 015, which is configured to write the second-level feature index result into a result file as a query result.

[0058] Based on the above objectives, a third aspect of an embodiment of the present invention provides a computer device. Figure 7 FIG. 1 is a schematic diagram of an embodiment of a computer device for executing a method for quickly querying network session data packets provided by the present invention. Figure 7 As shown, the computer device of the embodiment of the present invention includes the following devices: at least one processor 021; and a memory 022, the memory 022 stores computer instructions 023 that can be run on the processor, and the instructions implement the steps of the above method when executed by the processor.

[0059] The present invention also provides a computer-readable storage medium. Figure 8 FIG. 1 is a schematic diagram of an embodiment of a computer-readable storage medium for implementing a method for quickly querying network session data packets provided by the present invention. Figure 8 As shown, the computer-readable storage medium 031 stores a computer program 032 that performs the above method when executed by a processor.

[0060] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program for the method for rapidly querying network session data packets can be stored in a computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. The storage medium for the program can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM). The above-described computer program embodiments can achieve the same or similar effects as any of the corresponding aforementioned method embodiments.

[0061] In addition, the method disclosed in the embodiment of the present invention can also be implemented as a computer program executed by a processor, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the above functions defined in the method disclosed in the embodiment of the present invention are performed.

[0062] In addition, the above method steps and system units can also be implemented using a controller and a computer-readable storage medium for storing a computer program that enables the controller to implement the above steps or unit functions.

[0063] It will also be appreciated by those skilled in the art that the various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given of the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0064] In one or more exemplary designs, the function can be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Computer-readable media include computer storage media and communication media, and the communication media include any media that helps to transmit a computer program from one location to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. As an example and not limitation, the computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of an instruction or data structure and can be accessed by a general or special-purpose computer or a general or special-purpose processor. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if a coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DOL) or wireless technologies such as infrared, radio and microwaves are used to send software from a website, server or other remote source, the above-mentioned coaxial cable, optical fiber cable, twisted pair, DOL or wireless technologies such as infrared, radio and microwaves are all included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0065] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless expressly limited to the singular.

[0066] It should be understood that, as used herein, the singular forms "a" and "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" is intended to include any and all possible combinations of one or more of the associated listed items.

[0067] The serial numbers of the embodiments disclosed in the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0068] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program instructing the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0069] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples. Within the spirit of the embodiments of the present invention, the technical features of the above embodiments or different embodiments may be combined, and there are many other variations of different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention.

Claims

1. A method for quickly querying network session data packets, characterized in that: The following steps are involved: Store the data stream in the data stream table; Cache the data stream into a data packet index table; Performing a first-level feature index in the data stream table based on a preset search condition to generate a first-level feature index result, which includes: locating and generating a first-level feature index result in the data stream table corresponding to the information in the table based on the search condition; Further performing secondary feature indexing in the data packet index table based on the primary feature index result to generate a secondary feature index result, which includes: further performing secondary feature indexing in the data packet index table based on the primary feature index result to obtain multiple index items, and merging the index items to generate a secondary feature index result; Writing the secondary feature index result into a result file as a query result; Caching the data stream into the data packet index table includes: marking the data packets whose cache segment number is within the cache segment number threshold as short connection data packets, and merging and saving them; marking the data packets that exceed the cache segment number threshold as long connection data packets, and caching the data packets that exceed the cache segment number threshold into independent data packets.

2. The method for quickly querying network session data packets according to claim 1, characterized in that: The data flow table includes: data flow ID, start time, end time, source IP, source port, destination IP, destination port, IP protocol type and total number of bytes.

3. The method for quickly querying network session data packets according to claim 1, characterized in that: The data packet index table includes: data stream ID, data packet file name, storage start position and storage end position.

4. The method for quickly querying network session data packets according to claim 1, characterized in that: Caching the data stream into the data packet index table includes: The short connection data streams in the data stream are merged, and the merged short connection data streams are cached in the data packet index table.

5. The method for quickly querying network session data packets according to claim 1, characterized in that: Caching the data stream into the data packet index table includes: Segmenting the long connection data stream in the data stream, and caching the segmented long connection data streams into a data packet index table respectively; When the number of cache segments exceeds a cache segment number threshold, the portion of the long connection data stream that exceeds the cache segment number threshold is cached into an independent data packet.

6. The method for quickly querying network session data packets according to claim 1, characterized in that: Also includes: The search condition is quintuple data, and data query is performed based on the quintuple data.

7. A network session data packet fast query device, characterized in that: include: a first index building module configured to store the data stream in a data stream table; a second index building module, the second index building module being configured to cache the data stream into a data packet index table; a first indexing module, configured to perform a first-level feature index in the data stream table based on a preset search condition to generate a first-level feature index result; a second indexing module configured to further perform a secondary feature index in the data packet index table based on the primary feature indexing result to generate a secondary feature indexing result; A result output module configured to write the secondary feature index result into a result file as a query result; The second index construction module is used to: mark the data packets whose cache segment number is within the cache segment number threshold as short connection data packets, merge and save them; mark the data packets exceeding the cache segment number threshold as long connection data packets, and cache the data packets exceeding the cache segment number threshold into independent data packets; The second indexing module is further configured to: further perform secondary feature indexing in the data packet index table based on the primary feature indexing result to obtain multiple index items, and merge the index items to generate a secondary feature indexing result.

8. A computer device, characterized in that: include: at least one processor; as well as A memory storing computer instructions executable on the processor, wherein the instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images