Network traffic indexing method, apparatus, and medium based on columnar store database
By separating the primary index table and multiple secondary index tables in a columnar storage database, and selecting the appropriate index table for retrieval based on query conditions, the problem of low performance in network traffic data packet retrieval is solved, achieving efficient data packet retrieval and storage optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DBAPPSECURITY CO LTD
- Filing Date
- 2023-08-28
- Publication Date
- 2026-05-19
AI Technical Summary
In existing technologies, network traffic packet retrieval performance is low, especially in large-scale network traffic analysis. Existing indexing methods increase memory pressure and disk usage, and affect retrieval performance.
A network traffic indexing method based on columnar storage database is adopted, which splits session metadata into session fragment information and stores it in a main index table and multiple secondary index tables. The index IDs are linked, and the appropriate secondary index table is selected for retrieval based on the user's query conditions, thereby optimizing retrieval performance.
It improves data packet retrieval performance, reduces memory pressure and disk usage, and optimizes retrieval efficiency in various scenarios.
Smart Images
Figure CN117112565B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information security technology, and in particular to a method, apparatus and medium for indexing network traffic based on a columnar storage database. Background Technology
[0002] With the rapid development of information technology, especially internet technology, the number of network applications is also growing rapidly. Due to analysis speed limitations, large amounts of network traffic may not be processed in real time and need to be saved as data packets for later analysis. Searching sequentially through massive data packet records would be very inefficient; therefore, indexing can enable fast retrieval.
[0003] In the existing technology, HashMap can be used to implement indexing, but it has the following drawbacks: 1) In order to ensure that the packets in the session are concentrated together, session reorganization needs to be performed in memory, which increases memory pressure; 2) Implementing indexing through HashMap greatly increases the disk usage of the index file. Although the disk usage is alleviated by compressing the index, the compression reduces the retrieval performance.
[0004] There is currently no effective solution to the problem of low data packet retrieval performance in related technologies. Summary of the Invention
[0005] Therefore, it is necessary to provide a network traffic indexing method, apparatus, and medium based on columnar storage database to address the aforementioned technical problems.
[0006] In a first aspect, embodiments of this application provide a network traffic indexing method based on a columnar storage database, the method comprising:
[0007] Capture network traffic packets from the network adapter in real time and obtain the corresponding session metadata;
[0008] The session metadata is split into several session fragment information, and the several session fragment information are stored in an index table in a columnar storage database; wherein, the index table includes a main index table and multiple secondary index tables, the main index table includes a list of data packet locations, and the main index table and each of the secondary index tables are associated through an index ID;
[0009] Based on the query conditions provided by the user, the appropriate secondary index table is selected for retrieval to obtain the index ID corresponding to the query conditions;
[0010] Use the index ID to query the main index table to obtain the corresponding list of data packet locations;
[0011] Based on the data packet location list, the corresponding network traffic data packets are downloaded from the disk data packet file.
[0012] In one embodiment, the secondary index table includes a first secondary index table, a second secondary index table, and a third secondary index table; the step of selecting the appropriate secondary index table for retrieval based on the query conditions provided by the user includes:
[0013] Determine whether the query conditions provided by the user include IP address-related fields, which include the destination IP address and the source IP address;
[0014] If the query conditions provided by the user include the destination IP address, then the first secondary index table is selected for retrieval;
[0015] If the query conditions provided by the user include the source IP address but do not include the destination IP address, then the second-level index table is selected for retrieval;
[0016] Otherwise, determine whether the query conditions provided by the user include fields other than the time range field; if so, select the third and second-level index tables for retrieval, wherein the time range field includes the time of the first packet of the session fragment and the time of the last packet of the session fragment.
[0017] In one embodiment, the method further includes:
[0018] The retrieval priority is sorted for multiple fields in each of the secondary index tables;
[0019] If the query conditions provided by the user include multiple query fields, the retrieval will be performed in the appropriate secondary index table according to the priority order.
[0020] In one embodiment, the method further includes:
[0021] If the query conditions provided by the user only include the fields of the time range, then the main index table is selected for retrieval to obtain the corresponding list of data packet locations;
[0022] Based on the data packet location list, the corresponding network traffic data packets are downloaded from the disk data packet file.
[0023] In one embodiment, downloading the corresponding network traffic data packet from the disk data packet file based on the data packet location list includes:
[0024] Based on the data packet location list, obtain the corresponding data packet file ID and file offset;
[0025] Download the corresponding network traffic data from the disk data packet file based on the data packet file ID and the file offset.
[0026] In one embodiment, the method further includes:
[0027] The index ID is used to query the main index table to obtain the corresponding display field;
[0028] The display fields are presented to the user as search results.
[0029] In one embodiment, the real-time capture of network traffic packets from the network adapter further includes:
[0030] Write the network traffic data packets into the buffer;
[0031] When the buffer is full or the preset time window ends, the network traffic data packets are written to the disk data packet file in chronological order.
[0032] Secondly, embodiments of this application also provide a network traffic indexing device based on a columnar storage database, the device comprising:
[0033] The first acquisition module is used to capture network traffic data packets from the network adapter in real time and obtain the corresponding session metadata.
[0034] The storage module is used to split the session metadata into several session fragment information and store several session fragment information in an index table of a columnar storage database; wherein, the index table includes a main index table and multiple secondary index tables, the main index table includes a list of data packet locations, and the main index table and each of the secondary index tables are associated through an index ID;
[0035] The retrieval module is used to select the appropriate secondary index table for retrieval based on the query conditions provided by the user, and obtain the index ID corresponding to the query conditions.
[0036] The second acquisition module is used to query the main index table using the index ID to obtain the corresponding data packet location list;
[0037] The download module is used to download the corresponding network traffic data packets from the disk data packet file based on the data packet location list.
[0038] Thirdly, embodiments of this application also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method described in the first aspect above.
[0039] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect above.
[0040] The aforementioned network traffic indexing method, apparatus, and medium based on a columnar storage database capture network traffic data packets from a network adapter in real time to obtain corresponding session metadata. The session metadata is then split into several session fragments, which are stored in an index table within the columnar storage database. Each index table includes a primary index table and multiple secondary index tables. The primary index table includes a list of data packet locations, and the primary and secondary index tables are associated via index IDs. Based on user-provided query conditions, a suitable secondary index table is selected for retrieval to obtain the index ID corresponding to the query conditions. The index ID is then used to query the primary index table to obtain the corresponding list of data packet locations. Based on the list of data packet locations, the corresponding network traffic data packets are downloaded from a disk data packet file. This application, based on a columnar storage database, divides the index table into a primary index table and multiple secondary index tables, and uses a suitable secondary index table according to different retrieval conditions, thus optimizing retrieval performance under various scenarios and improving data packet retrieval performance.
[0041] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0043] Figure 1 This is a schematic diagram illustrating the application environment of a network traffic indexing method based on a columnar storage database in one embodiment.
[0044] Figure 2 This is a flowchart illustrating a network traffic indexing method based on a columnar storage database in one embodiment;
[0045] Figure 3 This is a structural diagram of the primary index table and the secondary index table in one embodiment;
[0046] Figure 4 This is a schematic diagram of the data packet storage and index creation process in one embodiment;
[0047] Figure 5This is a schematic diagram of a network traffic indexing method based on a columnar storage database in a preferred embodiment;
[0048] Figure 6 This is a structural block diagram of a network traffic indexing device based on a columnar storage database in one embodiment;
[0049] Figure 7 This is a schematic diagram of the computer device structure in one embodiment. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.
[0051] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.
[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0053] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.
[0054] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the network traffic indexing method based on columnar storage database in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.
[0055] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the network traffic indexing method based on a columnar storage database in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0056] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a radio frequency (RF) module used for wireless communication with the Internet.
[0057] This application provides a network traffic indexing method based on a columnar storage database, such as... Figure 2 As shown, the method includes the following steps:
[0058] Step S201: Capture network traffic data packets from the network adapter in real time and obtain the corresponding session metadata.
[0059] In a network, a session is a temporary, interactive exchange of information between two or more communication devices, or between a computer and a user. Network traffic packets are the basic unit of network traffic; the transmission of multiple packets forms traffic. Packet files are the international standard PCAP files, which can be viewed by mainstream packet viewing tools (such as Wireshark).
[0060] For each captured data packet, its information is extracted and attributed to session metadata. Session metadata contains session attribute information and identifies a specific session through a unique session ID.
[0061] Step S202: The session metadata is split into several session fragment information, and the several session fragment information are stored in an index table in a columnar storage database; wherein, the index table includes a main index table and multiple secondary index tables, the main index table includes a list of data packet locations, and the main index table and each of the secondary index tables are associated through an index ID.
[0062] Specifically, each session metadata can be broken down into several session fragments. When a specified number of data packets are reached or a specified time is reached, the session fragment information will be stored in an index table in the columnar database, and a new session fragment will be created for that session. A columnar database is a database design pattern that stores data by columns rather than rows. In a columnar database, each column of each table is stored separately, rather than storing the entire row together.
[0063] To improve retrieval performance, this implementation divides the index table into a main index table and multiple secondary index tables. The main index table stores columns including: searchable fields, display fields, and a list of data packet locations. Specifically, the searchable fields include: source IP address, destination IP address, source port, destination port, transport layer protocol, application layer protocol, time of the first packet of a session fragment, time of the last packet of a session fragment, index ID, and user-defined fields. Multiple secondary index tables store several searchable fields, and the main index table and each secondary index table are linked by an index ID.
[0064] Step S203: Based on the query conditions provided by the user, select the appropriate secondary index table for retrieval to obtain the index ID corresponding to the query conditions.
[0065] In this embodiment, based on the query conditions provided by the user and according to the preset matching rules, the appropriate secondary index table is selected for retrieval to obtain the index ID corresponding to the query conditions.
[0066] Step S204: Use the index ID to query the main index table to obtain the corresponding data packet location list. The data packet location list describes the location information of the data packet to be queried in the disk file.
[0067] Step S205: Based on the data packet location list, download the corresponding network traffic data packets from the disk data packet file.
[0068] In this embodiment, based on a columnar storage database, the index table is divided into a primary index table and multiple secondary index tables. Depending on the different search conditions, an appropriate secondary index table is used, thereby optimizing the search performance under various scenarios and improving the data packet retrieval performance.
[0069] In one embodiment, such as Figure 3 As shown, the secondary index tables in this embodiment include three tables: a first secondary index table (secondary index table 1), a second secondary index table (secondary index table 2), and a third secondary index table (secondary index table 3). The first secondary index table contains all searchable fields, specifically including: destination IP address, source IP address, source port, destination port, transport layer protocol, application layer protocol, time of the first packet of a session fragment, time of the last packet of a session fragment, and index ID. The second secondary index table contains all searchable fields except the destination IP address. The third secondary index table contains all searchable fields except the destination IP address and source IP address. The step of selecting the appropriate secondary index table for retrieval based on the user-provided query conditions includes the following steps:
[0070] Step 1: If the query conditions provided by the user include the destination IP address, then the first secondary index table is selected for retrieval.
[0071] Step 2: If the query conditions provided by the user include the source IP address but do not include the destination IP address, then the second secondary index table is selected for retrieval.
[0072] Step 3: If the query conditions provided by the user do not include either the source IP address or the destination IP address, but include fields other than the time range field, then the third-level index table is selected for retrieval. In this embodiment, the time range field includes the time of the first packet of the session fragment and the time of the last packet of the session fragment. This can be understood as follows: if the query conditions provided by the user only include the time of the first packet of the session fragment and / or the time of the last packet of the session fragment, then the third-level index table will not be selected for retrieval.
[0073] In one embodiment, multiple fields in each of the secondary index tables are sorted according to search priority. Specifically, the destination IP address, source IP address, time of the first packet of the session fragment, and time of the last packet of the session fragment in the first secondary index table are sorted from high to low priority; the source IP address, time of the first packet of the session fragment, and time of the last packet of the session fragment in the second secondary index table are sorted from high to low priority; and the transport layer protocol, application layer protocol, destination port, source port, time of the first packet of the session fragment, and time of the last packet of the session fragment in the third secondary index table are sorted from high to low priority.
[0074] In this embodiment, if the query conditions provided by the user include multiple query fields, the search is performed in the adapted secondary index table according to the priority order. For example, if the query conditions provided by the user include a destination IP address and a source IP address, the adapted secondary index table is confirmed to be the first secondary index table. Next, the destination IP address is used to search the first secondary index table to obtain a first search result matching the destination IP address. Finally, the source IP address is used to further search the first search result to obtain a second search result matching the source IP address, which is the final search result.
[0075] In columnar storage databases, sorting has a far greater impact on retrieval performance than the database's built-in indexes. Each table can allow multiple sort keys, and the higher the priority of the sort key, the better the retrieval performance. In the same table, it's impossible to have all searchable fields with high sort priority. Therefore, this embodiment uses multiple secondary index tables to split the fields that need to be sorted. The search priority is then ranked for multiple searchable fields in each secondary index table. If the user-provided query conditions include multiple query fields, the retrieval is performed in the appropriate secondary index table according to the priority ranking, thus improving retrieval efficiency.
[0076] In one embodiment, if the query conditions provided by the user only include the time range field, the main index table is directly selected for retrieval to obtain the corresponding data packet location list; and based on the data packet location list, the corresponding network traffic data packets are downloaded from the disk data packet file.
[0077] In this embodiment, when the user-provided query conditions only include the time of the first packet and / or the time of the last packet of the session segment, the secondary index table is not used for retrieval; instead, the query is performed directly in the primary index table. Because the index ID is time-related and can be located down to the second level, the time range can be queried using the index ID.
[0078] In one embodiment, downloading the corresponding network traffic data packet from the disk data packet file based on the data packet location list includes the following steps:
[0079] Step 1: Based on the data packet location list, obtain the corresponding data packet file ID and file offset. For example, a file offset of N for the target content means that the distance between the first byte of the target content and the start position of the file is N bytes, where a byte is the basic unit for reading and writing files.
[0080] Step 2: Download the corresponding network traffic data from the disk data packet file based on the data packet file ID and the file offset.
[0081] In one embodiment, when the index ID is used to query the main index table, the obtained display fields are presented to the user as search results.
[0082] In this embodiment, several index IDs that meet the requirements are obtained by querying the adapted secondary index table. These index IDs can be used to query all fields in the main index table. At this time, the fields are displayed to the user in units of session fragments.
[0083] In one embodiment, after capturing network traffic packets from the network adapter in real time, the method further includes: writing the network traffic packets to a buffer; and writing the network traffic packets to a disk data packet file based on time sequence when the buffer is full or when a preset time window ends.
[0084] Specifically, such as Figure 4 As shown, after capturing network traffic packets from the network adapter in real time, the process is divided into two steps. The first step is to store the captured packets on disk, specifically including the following steps: S21, placing the captured packets into a user buffer; S22, determining if the buffer is full or if a preset time window has ended; S23, writing the packets to disk and releasing the buffer. In S23, the buffer contents are written to disk using direct I / O to avoid a large amount of disk I / O filling the operating system buffer. Then, the buffer is released. This application does not require grouping packets from the same session together, nor does it require changing the order of the packets: when using a single thread, packet storage is entirely based on time order; this solution supports multi-threading, and when using multi-threading, the packets stored within each thread are based on time order. The second step is to store the acquired session fragment information in an index table in a columnar storage database, specifically including the following steps: S31, extracting metadata; S32, classifying the metadata into session metadata; S33, determining if a specified number of packets has been reached or if a timeout has occurred; S34, when the specified number of packets has been reached or a timeout has occurred, storing the session fragment information in the database and creating a new session fragment information; S35, determining if the session has ended; S36, releasing the session when the session ends.
[0085] The present embodiment will now be described and illustrated through preferred embodiments.
[0086] Figure 5 This is a preferred flowchart of the network traffic indexing method based on a columnar storage database in this embodiment. The network traffic indexing method based on a columnar storage database includes the following steps:
[0087] Step S41: Determine whether the query conditions include the destination IP address. If yes, proceed to step S42; otherwise, proceed to step S43.
[0088] Step S42: Select secondary index table 1 for retrieval, obtain the index ID corresponding to the query conditions, and use the corresponding index ID to query the main index table.
[0089] Step S43: Determine whether the query conditions include the source IP address. If yes, proceed to step S44; otherwise, proceed to step S45.
[0090] Step S44: Select secondary index table 2 for retrieval, obtain the index ID corresponding to the query conditions, and use the corresponding index ID to query the main index table.
[0091] Step S45: Determine whether the query conditions include search terms other than time, that is, whether the query conditions include search terms other than the time of the first packet of the session fragment and the time of the last packet of the session fragment. If yes, proceed to step S46; otherwise, proceed to step S47.
[0092] Step S46: Select secondary index table 3 for retrieval, obtain the index ID corresponding to the query conditions, and use the corresponding index ID to query the main index table.
[0093] Step S47: Return session fragment information that matches the query conditions, and obtain the data packet file ID and file offset.
[0094] In step S48, the user can select the session fragment information that meets the conditions to obtain the corresponding data packet file ID and file offset.
[0095] Step S49: Download from the data packet file according to the data packet file ID (number) and file offset.
[0096] Step S51: Generate data packet.
[0097] Secondly, embodiments of this application also provide a network traffic indexing device based on a columnar storage database, such as... Figure 6 As shown, the device includes:
[0098] The first acquisition module 510 is used to capture network traffic data packets from the network adapter in real time and obtain the corresponding session metadata.
[0099] Storage module 520 is used to split the session metadata into several session fragment information and store several session fragment information in an index table of a columnar storage database; wherein, the index table includes a main index table and multiple secondary index tables, the main index table includes a list of data packet locations, and the main index table and each of the secondary index tables are associated through an index ID.
[0100] The retrieval module 530 is used to select the appropriate secondary index table for retrieval based on the query conditions provided by the user, and obtain the index ID corresponding to the query conditions.
[0101] The second acquisition module 540 is used to query the main index table using the index ID to obtain the corresponding data packet location list.
[0102] Download module 550 is used to download the corresponding network traffic data packets from the disk data packet file based on the data packet location list.
[0103] In one embodiment, the retrieval module 530 is further configured to: determine whether the query conditions provided by the user include IP address-related fields, wherein the IP address-related fields include the destination IP address and the source IP address; if the query conditions provided by the user include the destination IP address, then the first secondary index table is selected for retrieval; if the query conditions provided by the user include the source IP address but do not include the destination IP address, then the second secondary index table is selected for retrieval; otherwise, determine whether the query conditions provided by the user include fields other than the time range field; if so, then the third secondary index table is selected for retrieval, wherein the time range field includes the time of the first packet of the session fragment and the time of the last packet of the session fragment.
[0104] In one embodiment, the retrieval module 530 is further configured to: sort the multiple fields in each of the secondary index tables according to their retrieval priorities; if the query conditions provided by the user include multiple query fields, then the retrieval is performed in the appropriate secondary index table according to the priority sorting.
[0105] In one embodiment, the retrieval module 530 is further configured to: if the query conditions provided by the user only include the field of the time range, then select the main index table for retrieval to obtain the corresponding data packet location list.
[0106] In one embodiment, the download module 550 is further configured to: obtain the corresponding data packet file ID and file offset based on the data packet location list; and download the corresponding network traffic data from the disk data packet file according to the data packet file ID and the file offset.
[0107] In one embodiment, the retrieval module 530 is further configured to: query the main index table using the index ID to obtain the corresponding display field; and display the display field as the retrieval result to the user.
[0108] In one embodiment, the apparatus further includes a storage model 560, which is configured to: write the network traffic data packets into a buffer; and write the network traffic data packets into a disk data packet file based on time sequence when the buffer is full or when a preset time window ends.
[0109] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.
[0110] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a network traffic indexing method based on a columnar storage database. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0111] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0112] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above embodiments of the network traffic indexing method based on a columnar storage database.
[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0115] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A network traffic indexing method based on a columnar storage database, characterized in that, The method includes: Capture network traffic packets from the network adapter in real time and obtain the corresponding session metadata; The session metadata is split into several session fragment information, and the several session fragment information are stored in an index table in a columnar storage database; wherein, the index table includes a main index table and multiple secondary index tables, the main index table includes a list of data packet locations, and the main index table and each of the secondary index tables are associated through an index ID; wherein, the secondary index tables include a first secondary index table, a second secondary index table, and a third secondary index table; Based on the query conditions provided by the user, the appropriate secondary index table is selected for retrieval to obtain the index ID corresponding to the query conditions; this includes: determining whether the query conditions provided by the user contain IP address-related fields, wherein the IP address-related fields include the destination IP address and the source IP address; if the query conditions provided by the user contain the destination IP address, the first secondary index table is selected for retrieval; if the query conditions provided by the user contain the source IP address but do not contain the destination IP address, the second secondary index table is selected for retrieval; otherwise, determining whether the query conditions provided by the user contain fields other than the time range field; if so, the third secondary index table is selected for retrieval, wherein the time range field includes the time of the first packet of the session fragment and the time of the last packet of the session fragment; Use the index ID to query the main index table to obtain the corresponding list of data packet locations; Based on the data packet location list, the corresponding network traffic data packets are downloaded from the disk data packet file.
2. The method according to claim 1, characterized in that, The method further includes: The retrieval priority is sorted for multiple fields in each of the secondary index tables; If the query conditions provided by the user include multiple query fields, the retrieval will be performed in the appropriate secondary index table according to the priority order.
3. The method according to claim 1, characterized in that, The method further includes: If the query conditions provided by the user only include the fields of the time range, then the main index table is selected for retrieval to obtain the corresponding list of data packet locations; Based on the data packet location list, the corresponding network traffic data packets are downloaded from the disk data packet file.
4. The method according to claim 1, characterized in that, The step of downloading the corresponding network traffic data packets from the disk data packet file based on the data packet location list includes: Based on the data packet location list, obtain the corresponding data packet file ID and file offset; Download the corresponding network traffic data from the disk data packet file based on the data packet file ID and the file offset.
5. The method according to claim 1, characterized in that, The method further includes: The index ID is used to query the main index table to obtain the corresponding display field; The display fields are presented to the user as search results.
6. The method according to claim 1, characterized in that, The real-time capture of network traffic packets from the network adapter also includes: Write the network traffic data packets into the buffer; When the buffer is full or the preset time window ends, the network traffic data packets are written to the disk data packet file in chronological order.
7. A network traffic indexing device based on a columnar storage database, characterized in that, The device includes: The first acquisition module is used to capture network traffic data packets from the network adapter in real time and obtain the corresponding session metadata. A storage module is used to split the session metadata into several session fragment information and store the several session fragment information in an index table of a columnar storage database; wherein, the index table includes a main index table and multiple secondary index tables, the main index table includes a list of data packet locations, and the main index table and each of the secondary index tables are associated through an index ID; wherein, the secondary index tables include a first secondary index table, a second secondary index table, and a third secondary index table; The retrieval module is used to select an appropriate secondary index table for retrieval based on the query conditions provided by the user, and obtain the index ID corresponding to the query conditions. This includes: determining whether the query conditions provided by the user contain IP address-related fields, wherein the IP address-related fields include the destination IP address and the source IP address; if the query conditions provided by the user contain the destination IP address, then the first secondary index table is selected for retrieval; if the query conditions provided by the user contain the source IP address but do not contain the destination IP address, then the second secondary index table is selected for retrieval; otherwise, determining whether the query conditions provided by the user contain fields other than the time range field; if so, then the third secondary index table is selected for retrieval, wherein the time range field includes the time of the first packet of the session fragment and the time of the last packet of the session fragment. The second acquisition module is used to query the main index table using the index ID to obtain the corresponding data packet location list; The download module is used to download the corresponding network traffic data packets from the disk data packet file based on the data packet location list.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.